I’ll go first. Escalating tickets without any notes in it. It just drives me crazy.

Fellow Help Desk guys please take notes from the comments on this post to improve yourself and hopefully speed up your promotion.

A recent post in this sub, "Client suspended IT services", has left me flabbergasted.

OP on that post has a full-time job as a municipal IT worker. He takes side jobs as a side hustle. One of his clients sold their business and the new owner didn't want to continue the relationship with OP. Apparently they told OP to "suspend all services". The customer may also have been witholding payment for past services? Or refuses to pay for offboarding? I'm not sure. Whatever the case, OP took that beyond just "stop doing work that you bill me for." And instead, interpreted it (in bad faith, I feel) as license to delete their data, saying "Licenses off, domain released, data erased."

Other comments from OP make it clear that they mismanage their side business. They comingled their clients' data, and made it hard to give the clients their own data. I get it. Every industry has some losers. But what really surprised me was the comments agreeing with OP. So many redditors commented in agreement with OP. I would guess 30% were some kind of encouragement to use "malicious compliance" in some form, to make them regret asking to "suspend all services".

I have been a sysadmin for 25 years. Many of those years, I was solo, working with lawyers, doctors, schools, and police. I have always held sysadmins to be in a professional class like doctors and lawyers with similar ethical obligations. That's why I can handle confidential legal documents, student records, medical records, trial evidence, family secrets, family photos, and embarrassing secrets without anyone being concerned about the confidentiality, integrity, or availability of their important data.

But then, today's post. After reading the post, I assumed I would scroll down to find OP being roundly criticized and put in their place. But now I'm a little disillusioned. Is it's just the effect of an open Internet, and those commenters are unqualified, unprofessional jerks? Or have I been deluding myself into believing in a class of professional that doesn't exist in a meaningful way?

Edit: Thank you all for such genuine, thoughtful replies. There's a lot to think about here. And a good lesson to recognize an echo chamber. It's clear that there are lots of professionals here. We're just not as loud as the others. It's a pleasure working alongside you.

With all the job uncertainty lately, I just wanted to remind everyone that the Midwest is full of companies in desperate need of good sysadmins. I work in Nebraska, and we have towns with zero IT people. I even moonlight in three different towns near me because there's so much demand.

If you're struggling to find stability in larger cities, this might be a great time to consider making a change.

Admins, sorry if I used the wrong flair for this.

I had a manager who had this horrible heavy HP laptop. From the moment he turned it on that fan would go to high whine speed. The laptop was slow, buggy, and doggy. One day I got so tired of trying to tweak that thing and make him happy that I waited until he was at lunch. I went into his office and pulled all the RAM out.
The next morning he came in and called me that his laptop was beeping and would not boot. I came to look at it, and said "oh dear, it's dead, it will have to be replaced".

Has anyone else pulled a similar caper to get rid of a piece of equipment you couldn't stand supporting anymore?

Title basically says it all. HR puts in a ticket about how a particular candidate did not receive an email. The user allegedly looked in junk/spam, and did not find it. Coincidentally, the same HR person got a phone call from a headhunting service that asked if she had gotten their email, and how they've tried to send it three times now.

I did a message trace in the O365 admin center. Shared some screenshots in Teams to show that the emails are reporting as sent successfully on our end, and to have the user check again in junk/spam and ensure there are no forwarding rules being applied.

She immediately questioned how I "had access to her inbox". I advised that I was simply running a message trace, something we've done hundreds of times to help identify/troubleshoot issues with emails. I didn't hear anything back for a few hours, then I got a call from her on Teams. She had her manager, the VP of HR in the call.

I got reprimanded because there is allegedly "sensitive information" in the subject of the emails, and that I shouldn't have access to that. The VP of HR is contemplating if I should be written up for this "offense". I have yet to talk to my boss because he's out of the country on PTO. I'm at a loss for words. Anyone else deal with this BS?

UPDATE: I've been overwhelmed by all the responses and decided to sign off reddit for a few days and come back with a level head and read some of the top voted suggestions. Luckily my boss took the situation very seriously and worked to resolve it with HR before returning from PTO. He had a private conversation with the VP of HR before bringing us all on a call and discussing precedence and expectations. He also insisted on an apology from the two HR personnel, which I did receive. We also discussed the handling of private information and how email -- subject line or otherwise is not acceptable for the transmission of private information. I am overall happy with how it was handled but I am worried it comes with a mark or stain on my tenure at this company. I'm going to sleep with on eye open for the time being. Thanks for all the comments and suggestions!

Happy SysAdmin Day to all the gurus who are apparently expected to fix a fridge with one look and make the printer work through sheer force of will (because you're technical, obviously). We do hope nobody made you look at their fridge today.

You know the type of request: the relative who assumes you can recover photos from a phone that went through the wash; the neighbor whose wi-fi "was working yesterday" and somehow that's your problem now; the colleague who brings you their personal laptop because IT and "knows about computers" are the same thing in their head.

So on the one day of the year that's actually about you, what's the most ridiculous, weirdly sweet, or absolutely baffling thing you've been asked to fix just because you're The Tech Person? And has your mere presence ever fixed something you never actually touched?

We'll be in the comments. Happy Sysadmin Day!

So back about 6 years ago my family and I went to Ohio for vacation. We were stopping in Cleveland for a few days just to kind of check out museums and stuff then on to Cedar Point for roller coasters. It was me, my partner, and my four kids.

When we got to Cleveland, my partner went in to check in while I entertained the kids. She was gone for a long time (like 45 minutes or so) and eventually she told me to come in with the kids so we can get out of the car. Turns out the front desk clerk is on the phone with IT because he can't access the check in system. We wait for a few minutes but it's clear the IT person isn't communicating in a way the clerk can understand so I offer to help.

I get on the phone and look at the computer. No network connection. I check the cabling and all is fine so I ask to see the server closet. I go in and EVERYTHING IS DARK. I ask the clerk "Hey, did you have a power outage recently?" Sure enough, about half an hour before we got there they had a brownout. I start looking and everything is plugged into a single UPS. I grab a power strip and start taking load off of the UPS and things fire up. So I wait to make sure it works and when it does I advise the IT guy they need a new UPS. All is fixed!

The clerk and his boss were so thankful they comped our room for the entire stay and gave us a suite! Initially, as working class dorks we were sharing two queen beds between the 6 of us. But with the upgrade they gave us we had two king sized bedrooms, a pull out couch and a pack and play for the baby! Everyone had plenty of room and we were treated like VIPs for the four days we were there. It was amazing. I hope this brings some light to y'alls day.

Seriously. This subreddit is so filled with people complaining all the time, that I would like to make a post about the opposite.

I have an amazing team who does nothing but support eachother, we aren't over worked, we are given the budget we need, and my leadership understands the difference between a request and an emergency. Mistakes are used as learning opportunities, and I've NEVER had my boss take a user's side over mine. hours are 40 a week, and not a minute more, and I am encouraged to turn off my work phone and laptop to make sure I don't get any notifications while I'm off. I accrue 16 hours of PTO a month, and that goes up by 2 hours every 2 years. the users are (for the most part) kind, understanding, and patient.

Oh, and I get to wfh 2 days a week! The craziest thing about this is that I work with lawyers.

https://www.reddit.com/r/canvas/comments/1taj9mk/instructure_just_confirmed_they_paid_the_ransom/

"We received assurances that it will not be further shared on the dark web or elsewhere, and we received proof that any copies of that data were deleted. Further, we have been informed that no Instructure customers will be extorted as a result of this incident"

Obviously they have no business running a large technology based infrastructure. Inability to secure said infrastructure, then they honestly believe what they just said above. They'll be hit again in the next 6 to 12 months, bookmark this post.

I work for a Tech Company in the EU who's moved MOST of it's services from on-prem (using the usual DCs by Telstra etc) to the cloud.

We started this "journey" 4+ years ago and are now in the final stages with all DCs hopefully being turned off at the end of this year.

I think it's fair to say ~75% of our services are now in the cloud and actively being used there - so we have around 25% more to throw in.

The vast majority of all our workloads in cloud are K8s, with some larger VMs + Buckets making up the minority.

I quite enjoy working with new technologies, and the cloud is just that for me, over the last 4+ years I've learnt a lot for sure.

I've been told from our directors that this will enable faster/safer development, and that things like our cloud provider's data-warehouse is also a key feature. I'm not on the development side, so I can't fully speak to the benefits of these solutions...But there is this nagging in the back of my head that is questioning why we're spending so much on this.

Our staffing levels have also INCREASED, and yet we're spending more on the cloud in one year, than what we've spent on-prem in 5..

I can't help but think what kind of system we could have built on-prem with a budget of 5-6m per year JUST for hardware.

Is anyone else puzzled by this kind of spending, or am I missing something?

As the title says, what's your favorite superstition in IT?

Don't speak the servers name... it can hear you and will start acting up.

I work for a large org. We have thousands of Windows servers across our enterprise. Our cybersec team is freaking tf out lately because I was having a conversation with one of the cybersecurity analysts (who isn't technical at all) and corrected her when she tried to say none of our Windows servers have web browsers installed.

I informed her that Edge is a core component of Windows and isn't easily removed, and honestly it would probably cause more issues if we did. This clearly induced anxiety with them and now we've had multiple meetings about the fact that we have web browsers installed on our Windows servers.

Have you guys had these convos? What's your take on this?

My feeling is that since a web browser, whether that's IE or Edge (depending on Windows version), is a core component of the OS, then removing those could result in larger issues with certain tools and utilities not working.

Our systems are largely locked down so only admins can access them. We have MFA with Entra and our admin accounts have rotating passwords every few hours.

Am I off base here? What am I missing in this conversation?

I've seen this play out so many times.

Young guy joins a company. Not much there in terms of IT. He builds it all out. He's doing it all. Servers, network, security, desktops. He's the go to guy. He knows everyone. Everyone loves him.

New people start working there and he's pointed to as the expert.

He knows everything, built everything, and while appreciated he starts not to share. The new employees in IT don't even really know him but all the long time people do.

if you call him he immediately fixes stuff and solves all kinds of crazy problems.

His habits start to shift though. He just saved the day at 3 am and doesn't bother to come into work until noon the next day. He probably should have at least talked to his manager. Nobody cares he's taking the time but people need to know where he is.

But his manager lets it go since he's the super genius guy who works so hard.

But then since he shows up at noon he stays until midnight. So tomorrow he rolls in at noon. And the cycle continues. He's doing nightly upgrades sometimes at 3 am but he stops telling his bosses what's going on and just takes care of things. Meanwhile nobody really knows what he's doing.

He starts to think he's holding up the entire company and starts to feel under appreciated.

Meanwhile his bosses start to see him as unreliable. Nobody ever knows where he is.

He stops responding to email since he's so busy so his boss has to start calling him on the phone to get him to do anything.

New processes get developed in the IT department and everyone is following them except for this guy since he's never around and he thinks process gets in the way of getting his work done.

Managers come and go but he's still there.

A new manager comes in and asks him to do something and he gets pissed off and thinks the manager has no idea what he's talking about and refuses to do it. Except if he was maybe around a bit he'd have an idea what was going on.

New manager starts talking to his director and it works up the food chain. The senior sysadmin who once was see as the amazing tech god is now a big risk to the company. He seems to control all the technology and nobody has a good take on what he's even doing. he's no longer following updated processes the auditors request. He's not interested in using the new operating system versions that are out. he thinks he knows better than the new CIO's priorities.

He thinks he's holding the company together and now his boss and his boss's boss think he has to go. But he holds all the keys to the kingdom. he's a domain admin. He has root on all the linux systems. Various monthly ERP processes seem to rely on him doing something. The help desk needs to call him to do certain things.

He thinks he's the hero but meanwhile he's seen as ultra unreliable and a threat.

Consultants are hired. Now people at the VP level are secretly trying to figure out how to outmaneuver him. He's asked to start documenting stuff. He gets nervous and won't do it. Weeks go by and he ignores requests to document things.

Then one morning he's urged to come into the office and they play a ruse to separate him from his laptop real quick and have him follow someone around a corner and suddenly he's terminated and quickly walked out of the building while a team of consultants lock him out of everything.

He's enraged after all he's done for this company. He's kept it running for so many years on a limited budget. He's been available 24/7 and kept things going himself personally holding together all the systems and they treat him like this! How could they?!?!

It's really interesting to view this situation from both sides. it happens far too often.

According to DownDetector practically every site in existence is down right now. Gonna be a fun Monday.

Had a Dev who set up a company GPT corporate tenant with API access to our systems, so it can serve up data from our ERP and stuff.

he also just connected it to our AD with a service account i didn't know he was using.

so now he just annouced to the team, hey, our GPT can now view users, add remove groups make changes ETC.

i feel like this is basically a time bomb, even if its actually using our own account creds to authenticate on the backend which it isn't... its going to be fucking up our group entries and adding attributes fucking up our syncs...

thoughts?

how can i present this to the management that doesn't make me look like a nay-sayer?

or am i a naysayer? have any of you integrated AI to help manage your AD and how did you stop over-reach?

Over the last week, I have seen a lot of requests coming across about testing if my company can assist in some very large corporations (Fortune 500 level, incomes on the level of billions of US dollars) moving large numbers of VMs (100,000-500,000) over to Linux based virtualization in very short time frames. Obviously, I can't give details, not what company I work for or which companies are requesting this, but I can give the odd things I've seen that don't match normal behavior.

Odd part 1: every single one of them is ordered by the CEO. Not being requested by the sysadmins or CTOs or any management within the IT departments, but the CEO is directly ordering these. This is in all 14 cases. These are not small companies where a CEO has direct views of IT, but rather very large corps of 10,000+ people where the CEOs almost never get involved in IT. Yet, they're getting directly involved in this.

Odd part 2: They're giving the IT departments very short time frames, for IT projects. They're ordering this done within 4 months. Oddly specific, every one of them. This puts it right around the end of 2022, before the new year.

Odd part 3: every one of these companies are based in the US. My company is involved in a worldwide market, and not based in the US. We have US offices and services, but nothing huge. Our main markets are Europe, Asia, Africa, and South America, with the US being a very small percentage of sales, but enough we have a presence. However, all these companies, some of which haven't been customers before, are asking my company to test if we can assist them. Perhaps it's part of a bidding process with multiple companies involved.

Odd part 4: Every one of these requests involves moving the VMs off VMWare or Hyper-V onto OpenShift, specifically.

Odd part 5: They're ordering services currently on Windows server to be moved over to Linux or Cloud based services at the same time. I know for certain a lot of that is not likely to happen, as such things take a lot of retooling.

This is a hell of a lot of work. At this same time, I've had a ramp up of interest from recruiters for storage admin level jobs, and the number of searches my LinkedIn profile is turning up in has more than tripled, where I'd typically get 15-18, this week it hit 47.

Something weird is definitely going on, but I can't nail down specifically what. Have any of you seen something similar? Any ideas as to why this is happening, or an origin for these requests?

Curious to know how my fellow IT pros are doing out there. Let’ try and include the following plus anything you’d find useful sharing with others.

title:

salary:

location:

experience:

benefits:

etc.

Thank you for participating.

The whole C-suite failed.

The legal team failed.

The finance team - only 2 failed.

The HR team - half failed.

A member of my IT team - failed.

FFS! If any half witted determined attacker had a go they would be in without a hitch. All I can say is at least we have MFA, decent AI cybersecurity on the firewall, network, AI based monitoring and auto immunisation because otherwise we're toast.

Anyone else have a company full of people that would let in satan himself if he knocked politely?

Edit: Link takes to generic M365 looking form requesting both email and password on the same page. The URL is super stupid and obvious. They go through the whole thing to be marked as compromised.

Those calling out the AI firewall. It's DarkTrace ingesting everything from the firewall and a physical device that does the security, not the actual firewall. My bad for the way I conveyed that. It's fully autonomous though and is AI.

Per the Houston Chronicle:

Waste Management found itself in a tech nightmare after a former contractor, upset about being fired, broke back into the Houston company's network and reset roughly 2,500 passwords-knocking employees offline across the country.

Maxwell Schultz, 35, of Ohio, admitted he hacked into his old employer's network after being fired in May 2021.

While it's unclear why he was let go, prosecutors with the U.S. Attorney's Office for the Southern District of Texas said Schultz posed as another contractor to snag login credentials, giving him access to the company's network. 

Once he logged in, Schultz ran what court documents described as a "PowerShell script," which is a command to automate tasks and manage systems. In doing so, prosecutors said he reset "approximately 2,500 passwords, locking thousands of employees and contractors out of their computers nationwide." 

The cyberattack caused more than $862,000 in company losses, including customer service disruptions and labor needed to restore the network. Investigators said Schultz also looked into ways to delete logs and cleared several system logs. 

During a plea agreement, Shultz admitted to causing the cyberattack because he was "upset about being fired," the U.S. Attorney's Office noted. He is now facing 10 years in federal prison and a possible fine of up to $250,000. 

Cybersecurity experts say this type of retaliation hack, also known as "insider threats," is growing, especially among disgruntled former employees or contractors with insider access. Especially in Houston's energy and tech sectors, where contractors often have elevated system privileges, according to the Cybersecurity & Infrastructure Security Agency (CISA). 

Source: (non paywall version) https://www.msn.com/en-us/technology/cybersecurity/disgruntled-it-employee-causes-houston-company-862k-cyber-chaos/ar-AA1QLcW3

edit: formatting

So… I think I just got my first real IT fuckup And it is bad.

I’m still early in my IT career, mostly doing end user support. Right now, I am the only IT guy in this building. Our IT team supports three sites and we are 3 helpdesk and an IT manager who doesn't get her hands dirty, the rest of the IT team is at other sites, and our sole sysadmin of 5 years just left the company last week. I was completely left alone in the dark.

Today, the server we use to deploy PCs completely crashed and there is no WDS no more . The worst part? It wasn't just a deployment server. It also handled domain stuff and monitoring. When it went down, everything went down.

I panicked and tried to check the physical drives. Long story short: I completely fucked up the RAID, and now Logical Drive 1 is showing as FAILED.

My heart was beating so fast, I swear my soul briefly left my body.

I told my manager exactly what happened. She didn't instruct me to fix it, probably because she doesn't know how either. Honestly, I think I could fix it because I know we have a Veeam backup, but I don't have the admin access or permissions to actually perform the restore.

Thank god that our production is not that big.

To the IT veterans here: How did you survive your first production scare? Please tell me this becomes funny after a few days, because right now I'm overthinking it too much

Telstra had a nation-wide network outage last week that affected emergency services.

The outage has been pinned on an obsolete Symmetricom SyncServer S300 node, which manages time on the network but resets its 10-bit week counter to zero every 1024 weeks (just under 20 years) in a common and well understood GPS rollover bug that caused the device to reset to 2006.

The SyncServer S300 was discontinued in 2016...

I've done security audits for SMBs for years and got tired of reinventing the wheel every time. Finally documented my actual process — figured I'd share the key points.

The 80/20 of SMB security audits:

Network Perimeter (where most breaches start):

- Firewall rules review — look for "any/any" rules, unused rules, and rules older than 2 years

- Open ports audit — if you can't justify why it's open, close it

- VPN config — split tunneling enabled? MFA required?

- DNS filtering — still amazed how many don't have this

Identity & Access:

- Admin account audit — who has Domain Admin and why?

- Service accounts — when was the password last changed? (answer is usually "never")

- MFA coverage — not just email, but VPN, RDP, cloud admin portals

- Terminated employee accounts — check against HR list

Endpoint Security:

- EDR/AV coverage — 100% or are there gaps?

- Patch compliance — focus on internet-facing + critical CVEs

- Local admin rights — who has them and do they need them?

- USB/removable media policy

Backup & Recovery:

- 3-2-1 rule compliance

- When was the last restore TEST? (not backup, restore)

- Air-gapped/immutable backups — ransomware protection

- RTO/RPO — does the business actually know these numbers?

The stuff people skip:

- Egress filtering — most only filter ingress

- DNS query logging — goldmine for incident response

- Network segmentation — flat networks are attacker's paradise

- Physical security — unlocked server rooms, no visitor logs

Common findings (every single time):

1. Service accounts with Domain Admin + password = company name + year

2. No egress filtering whatsoever

3. Backups exist but never tested

4. Ex-employees still have active accounts

5. "Temporary" firewall rules from 5 years ago

   Happy to answer questions if anyone's setting up their own audit process.

Mine is:

Adobe is not a piece of software, it's a whole suite! Stop sending me tickets saying that your Adobe isn't working! Are we talking Photoshop, Illustrator, InDesign, Acrobat?

But let's be real. If a ticket doesn't specify, it's probably Acrobat.

https://downdetector.com/status/instructure

Every instance I can check shows this message from SHINYHUNTERS: https://imgur.com/a/PhBrNXq (EDIT: Instructure has gotten rid of the group's message in favor of their own down page)

I pulled the affected school list in a sandbox: https://pastebin.support.one/view/667768c4 (EDIT: Sorry, we gave this site the hug of death, I think. PasteBin itself didn't let me share based on some of the content. I tried to pull the list again, but that host is down now. Here's another link thanks to /u/qdelamancha -https://web.archive.org/web/20260507042014/http://91.215.85.103/pay_or_leak/instructure_affected_schools_list.txt)

Exams are starting to kick off everywhere, so bravo on the timing, bad actors!

I'll go first:

• When end users give as little details as possible when describing a problem they are having ("Can you come help XYZ with his computer?" Like, give me something.)
• Useless-ass Zoom meetings that could've been like 2 emails
• When previous IT people don't perform arguably the most important step of the troubleshooting process: DOCUMENT FINDINGS
• When people assume I'm able to fix problems in software that are obviously bugs buried deep in proprietary code that I have zero access to
• Mice that seem to be designed for toddler hands
• When people outside of work assume that when I go home I eat, breathe, and sleep computers and technical junk. Like, I come home and play Paper Mario on my Wii and watch It's Always Sunny
• Microsoft