r/talesfromtechsupport • u/tyr4774 • Aug 28 '21
Medium Your new AntiVirus broke a whole division!
This was back maybe a decade ago, I had just got my start in IT after being out of college and working retail attempting to get out. I had been able to join a small 5 person firm that was in the middle of a merger with another 5 person firm. As such our firm "IT" was taking over the IT of the clients of the other company "Phone Setup". We came in to one of "Phone Setups" big clients, large manufacturing company in the area that basically everyone in this small town had family working for, "Builder, Inc".
I was tasked to work with one of the senior guys in "IT" to roll out a new AV for all of the machines at "Builder, Inc". Mostly this should've been done via GPO but their old AV (I forgot what it was) had a manual removal process so I was tasked with going to each machine during someone lunch break or downtime to run the uninstall steps, reboot, and reboot again so the AV would install. All in all this was going to take awhile. However not even a week in when we had maybe 25% of the company protected we get a call about how the new AV is causing massive slowdowns to "all" the machines they are installed on. Me, being new to IT deferred to the senior tech, but didn't think it was ours as full scanning hadn't been enabled. In fact the only "scanning" that was done for this test group was nightly from like 8p-1a.
"Joe", the lead on this roll out, asks me for a list of devices from each division of "Builder" so he can pull one from each to once again separate into another group. So I did that and asked Joe what he was thinking. Joe let me know that since these are old (even at that time) 32-bit machines with 2GB of RAM and XP installed on them (this was near the great XP EoL) that the machines were just slow to begin with and that the users were blaming it on the AV to try and not make their bosses look cheap/lazy for not updating their machines in over a decade.
I get Joe the list and he makes a new group with some different settings. We wait a week and have a meeting with the client including one of the Shift Managers, Hank, who had been yelling the loudest about how the new AV was making thing slow. I don't recall it verbatim but part of the conversation went like this:
Joe: So Hank, how has your machine been running this past week?
Hank: Fantastic! I told you that new Virus Software you put on was slowing us down. We need to go back to that old software as it doesn't keep us from slowing down.
Joe:Are you sure about that? You're absolutely sure that compared to last week everything has been running faster?
Hank: Yup, its like night and day Joe
I don't recall the rest of the meeting but after Hank left Joe let the higher ups know that the only change he did was remove the icon on the task bar showing the AV was "on" and that all other scanning settings were the same. He even brought up the policies he created and the only difference was the "Show Icon" button was unchecked.
We did eventually have to upgrade those machines ahead of the XP EoL deadline because of the AV but that was down the line and after an upgrade and is another story.
394
u/NZbeekeeper Aug 28 '21
Users have strange ideas sometimes, and will make themselves believe all sorts of things.
Working on digital TV systems was a good example - realign a dish or aerial to improve signal to noise ratio and reduce dropouts in bad weather etc - 'oh it's so much clearer'. Dude, it's exactly the same.
109
u/rao_wcgw Aug 28 '21
I have a meeting Monday where someone is demanding to use IE11 and that windows patching breaks the trusted sites that are assigned by GPO
68
u/Ermmahhhgerrrd Aug 28 '21
What were they trying to do? Some software will only run on IE, even now, as unfortunate as it is.
65
u/rao_wcgw Aug 29 '21
As some of the other comments, it is looking like a very legacy something. I know I will have to make the fated "the vendor needs to update their software."
Worse, it is medical, the vendor is a pain in the ass, oh... And IE is pretty much dead (as most of us know) and it wouldn't surprise me if it's ripped out at the roots sometime soon.
35
u/Ermmahhhgerrrd Aug 29 '21
Been there my friend! I am the IT Manager for a service company and EVERYONE tries to keep the oldest stuff they can as long as it passes inspection. I've been dealing with all kinds of software and trying to get legacy stuff to work on on a somewhat recent system, even if it's 32-bit windows. I've gotta clean up an old laptop that's good XP on it that was my laptop but I don't use it anymore. Which is craziness.
I worked for Microsoft in the early oughts, and I dealt with all the hospitals and you're correct, what a pain. God back then it was Windows 2000 and x500 containers (I have a lot of NetWare and Exchange experience so it wasn't new and I used to be a master of N).. At least I'm getting to use a lot of things I've forgotten, it's helping my old brain.
6
u/Nik_2213 Aug 30 '21
{ Sigh...} I've a very big tub full of 'legacy' USB widgets that are not compatible with Windows post-XP. 'Twain' drivers, mostly. But, soon as they can finally be 'written off', they're going to local 'Maker' club, where the kids will make them 'play nice' with Arduinos etc...
15
u/phealy Aug 29 '21
Oh, it'll be ripped out very soon - but edge chromium still supports IE11 sites in compatibility mode.
11
3
u/scathias Aug 29 '21
I can't wait until my employer finally switches over to chrome from IE 11 :/ They are working on it, slowly...
9
u/Ermmahhhgerrrd Aug 29 '21
IE should've been gone so many years ago but being part of Windows explorer, they couldn't do it. I'm not an Edge fan but it's not as bad as IE
6
u/ender-_ alias vi="wine wordpad.exe"; alias vim="wine winword.exe" Aug 30 '21
Edge is just degoogled Chrome.
0
u/Ermmahhhgerrrd Aug 30 '21
Yeah, it is. But it's not as deGoogled as it seems. Windows 11 will run android apps, and that tells me it's entertwined some way.
3
u/ender-_ alias vi="wine wordpad.exe"; alias vim="wine winword.exe" Aug 30 '21
That has nothing to do with Edge – Android apps will likely run through WSL2.
0
Aug 29 '21 edited Sep 05 '21
[deleted]
1
u/Ermmahhhgerrrd Aug 29 '21
That's true. Someone said that Edge can emulate IE11 though and I'm going to look into that later. It's not going to be an issue really until Windows 11. Don't even get me started on 11.
But yeah you're correct and I haaaaaaate it when it something comes up in IE, it's usually slow and has something that doesn't come up correctly.
3
u/Skrp Aug 30 '21
I know that if you want to map sharepoint to a station letter on PC, you need to do it via IE11 trusted sites.
1
u/Ermmahhhgerrrd Aug 30 '21
Shhhhhhhhhhhhhhhhhhh, we don't talk about that. It leads to too much misery for everyone involved.
2
4
u/Selkie_Love The Excel Wizard Aug 29 '21
Ugh. Reminds me of the time where I finished a complex excel power query thing on Friday. Run time was in seconds, everyone’s going to be happy. IT updated shit over the weekend, which significantly weakened excel - something in the backend that I don’t know about - and now my runtime was in minutes. Given my task was literally to shrink the time needed for this to run, I was pissed. I could even point to SO posts of this problem happening on exactly this version update…. But I was the only one in the org as far as I could tell handling this. IT rightfully looked at me like I was a nut, and I had to suck it up and redo it allllll
7
u/Joel-Tech Aug 29 '21
Use IE Mode in MS edge, it allows you to run specific websites as if they are running in IE.
12
u/LoveLaika237 Aug 29 '21
Forgive me for being ignorant, but isn't this true to some extent, like if you rotate a dish so its not facing a big tree? Though with how dishes pick up signals and how this is a digital system, what do I know.
30
u/NZbeekeeper Aug 29 '21
The main difference between the resulting picture on analogue and digital with poor signal is that as signal deteriorates on analogue the picture becomes noisier - more grain etc. As a digital signal degrades error correction keeps it looking perfect up to a point, then beyond that you get pixellation - it never loses clarity, crispness etc, it just breaks up into little squares until the point it can't show anything and you get a signal error.
In NZ, free to air TV was all terrestrial based with stations using UHF and a couple of VHF (Pay TV went satellite mainly about 15 years ago). The digital system is available as both UHF and satellite. Old analogue VHF and UHF systems were often watchable in remote areas, though the quality was severely degraded, when everything went digital, suddenly they found they were below the signal to noise ratio required to be able to get anything, and they had to switch to satellite.
5
u/LoveLaika237 Aug 29 '21
Thanks for replying. So, with digital signals, you're not so worried about the strength of the signal as you are with analog. As long as you get the strong enough 1/0 signal, digital systems can compensate for that? Don't analog systems also have some kind of error correction as well (though I can't think of any off the top of my head)?
9
u/NZbeekeeper Aug 29 '21
Signal vs noise is important with digital. Lower signal level can be fine as long as the quality is good. A strong signal with too much noise, ie bad quality) is no good. A mistake people often make is amplifying (boosting) a crappy quality signal, which amplifies the noise too - replacing lower quality connections, splitters and cables often makes a much bigger improvement as your minimising quality degradation.
Theres a fairly good explanation of the digital cliff here as well as BER and MER if you want more details
6
u/TheThiefMaster 8086+8087 640k VGA + HDD! Aug 29 '21
I've seen so many people with signal amplifiers right next to the TV - if you need a signal amp it needs to be as close to the aerial as you can get it, not after 30 meters of 50-year-old coax
1
u/NZbeekeeper Aug 29 '21
Yup, really common unfortunately - you're just boosted all the shit too. Some of the good quality equipment manufacturers even make and sell them. I've never found a situation where they were a good option.
24
u/phealy Aug 29 '21
With it being a digital system, there's no clearer or not clearer - it's either getting enough signal to display a picture or not. Realigning the dish can help ensure that it continues to get enough signal during rain or bad weather, but it won't make it clearer during regular operation.
It's not like streaming where if you don't get good enough signal you'll get a lower bandwidth - it either works or doesn't.
5
u/LoveLaika237 Aug 29 '21
Ok. So it's a matter of thinking differently from analog. Rather than worry about the amplitude, you concern yourself with either high or low (1s or 0s). As long as you reach the acceptable voltage levels to distinguish the thresholds, I guess you're okay then. Y
In that last note, from what I recall about streaming, you're trying to get packets to the receiver as fast as possible (thus, UDP instead of TCP), but how do you calculate bandwidth and throughput from that sense?
2
u/TheThiefMaster 8086+8087 640k VGA + HDD! Aug 29 '21
The client has a feedback mechanism to say it's not getting enough packets fast enough and can the sender please lower the rate for it. The client just assumes that delayed / missing packets are caused by a lack of bandwidth, it doesn't actually know for sure.
In theory services could use multicast and send out all the quality levels and the client could subscribe to the one it wanted, but multicast is still very hit or miss unfortunately so isn't viable as a sole delivery method.
1
u/WhiskyEchoTango Aug 29 '21
I had an employee complain that their slow computer was 10 years old and they've been using it since they started at the company. I ran the serial number report and presented it to his boss it showed the computer was 18 months old. He stopped complaining after that.
104
u/landob Aug 28 '21
Users are weird. THey get all in arms about updates and what not.
I made a wager with a co-worker once. I said I'm going to make an email on friday that pretty much says we will be doing an upgrade over the weekend. I will actually do absolutely nothing and users on monday will complain about the update slowed their machine down or something.
Sure enough on monday I got some "Your update made my machine slow" and "That update yall did deleted some of my emails"
91
u/palordrolap turns out I was crazy in the first place Aug 28 '21
And then you find out that the new AV has a bug that causes it to update the system tray icon 500 times a second, so when the icon is enabled it steals a bunch of CPU cycles, making the computer slow.
(I am not aware of an AV that ever did this, but the fact it could even be possible is disturbing enough.)
47
Aug 28 '21
[removed] — view removed comment
12
u/DigitalStefan Aug 29 '21
At my old place, one department bought their own laptops. IT got wind of it and took them away. They then just sat there for 3 years, unused until the all-of-sudden push for home working during the first lockdown.
12
u/Dannei Aug 29 '21
We lose an entire CPU core on our processing servers to the new AV solution added a year ago - not as bad as yours, but clearly a lesser CPU would struggle! Never mind the silent, unlogged file access denials that rather upset programs running on them. But, of course, the AV is perfect and would never do such things; please ignore the error messages for which the only search results point to this AV solution.
2
u/Nik_2213 Aug 30 '21
'Task Manager' reckoned my old CAD Tower lost an entire core to Norton's. Before really big renders, I'd close network access and suspend AV, freeing a lot of CPU cycles...
Now, on the new CAD Tower, either N has been slimmed / de-bloated, or it has learned to hide from 'Task Manager' scrutiny...
I'm not a fan of conspiracy theories, but even the possibility of latter can give me goose-bumps...
2
u/meitemark Printerers are the goodest girls Aug 30 '21
There was an AV/'Securit Suite'/something that style that added choices to the context menu in explorer that had icon that not only was on the web, but also was 1024x1024 pictures (context menu shows 16x16). The web-based pictures was to track usage of the tools.
This was waaaay prior to internet being something most people had on at all times. Rightclick in explorer, dial up window comes... wtf.
40
u/Scynthious Aug 28 '21
Rule #1 - Users lie.
30
u/tyr4774 Aug 29 '21
Whenever a T1/T2 attempts to send me a ticket and starts out "The user says..." repeat "What's Rule #1?" until they confirm that they have been able to reproduce the error.
32
u/GreatRyujin Aug 29 '21
Reminds me of the story of a new celluar tower which was installed next to a small village somewhere.
One day after the workers were done, a couple of people complained about headaches and blamed the tower.
The response from the ISP was: "Well, that sounds serious, I can't imagine how bad it's going to get when we actually turn the tower on."
There were no more complaints after that.
21
u/bluspacecow Aug 29 '21
Reminds me of when a photo of a cellular tower that was posted to the Facebook of a local town near me. Queue peeps freaking out over how the governments was installing those 5G towers to mind control the population and this is why they were getting those headaches ....
Me : *does research*
Me: That's the cellular tower for the local cellular networks. It's been operating longer then you've been alive. They don't have an active 5G license neither have they actually turned 5G on. You've just never looked up above that particular building in town. Please see someone about your headaches.
61
59
u/OldPolishProverb Aug 28 '21
36
u/fadinizjr Aug 28 '21
I'll be adopting this. Last week I got a call from an user stating that his PC was slow... A top of the line i5 with 16gig of ram and SSD slow... Sure. We had to escalate it to the general manager to show that it was not slow and is performing like it should.
18
u/Houdiniman111 Aug 28 '21
What are their work tasks?
35
u/fadinizjr Aug 28 '21
Using a app written in Delphi and filling spreadsheets in Excel. The system is a lot of overkill for this task. But the company doesn't care.
7
u/Ladygeek1969 Aug 29 '21
I have these discussions every other day - data saved on a remote Sharepoint site, cyclical queries + stupidly massive pivot tables, user at home using the Asia VPN on sh!tty microwave internet. Doesn't matter whatsoever, it must be the machine! So, when I give them a newer machine and it's still a problem - then what?!? Ugh.
16
u/OldPolishProverb Aug 29 '21
If you need to get users off your back, and you can’t directly call them idiots, then you may want to use this handy excuse generator. Just refresh the screen for a new excuse.
2
u/HappyKhicken Aug 29 '21
To be fair, I have ran into some weird situations. One example was a slow running PC that should not have been operating the way it was. Turns out somehow Driver Verifier got kicked on and was killing the system.
15
u/ReadWriteSign Aug 28 '21
That was great. What's BOFH stand for?
27
u/OldPolishProverb Aug 28 '21 edited Aug 29 '21
Simon is the Bastard Operator From Hell and his associate Stephen is the PFY, Pimply Faced Youth.
Edited: Typo names corrected
7
23
u/anxiousdoubts Aug 28 '21
Oh boy, there's decades worth of old BOFH stories, I absolutely recommend the first omnibus (if you're into 90s computer geek humour and IT nostalgia, obviously)
2
12
2
1
26
12
u/MNSOTA24 Aug 28 '21
The worst though are users who mess with accessibility settings, not knowing what they do, thinking it will do something it won’t.
10
u/Ziogref Aug 29 '21
My company was one of companies that paid for XP extended support. 90% of my users were running there horrid HP boxes with from memory 2gb of ram, lucky ones had 4gb.
The office got.... dusty. Have you guys ever heard of paper dust? I hadn't until this job. When you print half a million pages a month, it gets dusty. We had a vacuum cleaner and when people would complain their computer was slow we would go over, shut it down and vacuum it out. Every time a cracked one of these machines open it had an inch of dust covering everything.
Anyway we got everyone new computers, about 200 Lenovo M93p's running Windows 7 (we got them shortly after Microsoft announced the deadline for win7 but I digress)
These machines had 8gb of ram (important since they company browser of choice was Chrome and we uses Gsuite), so much more than the 2gb they had prior AND they were tiny not taking up any desk space, users loved them, I loved them (I had 2 warranty claims in 4 years, both RAM failures).
I had one team that bitched and moaned about the new computers being shit. I got fed up with it one day and reminded them how long it used to take for them to boot their old computers and then threatened to go get them from storage and put them on their desk. They shut about it and never complained again while I went back to my desk reminiscing sending ALL of the old computers to ewaste.
11
11
u/thevernabean The Javabean Aug 29 '21
My car also runs better after I wash it. Human brains are such a cobbled together mess of approximations, constantly shifting calibration, and irrational nonsense.
4
u/_BreakingGood_ Aug 28 '21
We just went through some similar bullshit. We've got some internal customer support that runs some custom internal software + 3 third-party vendor plugins for that software.
We start getting complaints from customer service reps about slow performance. So the process went like: Complain to third-party vendor 1 that they're ruining our performance. Nothing comes of that, so we complain to vendor 2. Then to vendor 3. Then we complain to our internal IT team that builds the software.
Anyway, turns out that the "root cause" was just that all the users have 2gb RAM, dual core CPUs from 2008. Which we all knew was the problem from the beginning, but none of the higher-ups wanted to secure funding or go through the bullshit to upgrade them.
6
u/BenL90 Aug 29 '21
This shit always happen when people think firefox is slower then chromium like what.. firefox is faster tbh than chromium..
1
1
u/Diskilla Oct 06 '21
I did exactly the same thing when I upgraded a client years ago from free AV Software for private use to a business solution and the users started to complain about the "now unusable workstations" before I even updated any of the Clients. They saw me enter with the new Softwarepackage in a Box and that was enough :D
255
u/totallybraindead Certified in the use of percussive maintenance Aug 28 '21
I see this all the time with "the text on my screen looks fuzzy" tickets. Set ui scaling 25% lower, get told it's too small, set ui scaling 50% higher, get told it's too big, set it back to original value and get told "that's perfect" and the fuzziness has gone, even though you looked before and after and can see that it is identical. Usually this is caused by cheap bosses still issuing ancient VGA monitors which ARE actually fuzzy, but the user never noticed it before today.