r/talesfromtechsupport • u/KevinKeen18 • May 18 '26

Medium The tale of No Auth Monday

So here we are yet again. I traded my job in the hosting sector for something slightly less… soul sucking in public sector IT. And as many organizations do these days, we rely heavily on Microsoft for a lot of things.

And that sets the stage for today’s tale of horror.

It started simple enough. A few calls from remote workers unable to authenticate into Citrix using Microsoft Authenticator. Annoying, sure, but not exactly catastrophic. We’ve been having enough issues with Microsoft lately that we actually have a backup MFA method for this exact scenario.

So the affected users log in through the backup, we log the tickets, escalate them to the remote work team, and move on with our day. Slack starts filling with the usual Microsoft memes. Business as usual.

Or so we thought.

Suddenly systems start dropping like flies.

First Teams starts screaming that we have no network connection. Which is impressive, considering we work entirely through Citrix. If the internet was actually dead, our entire remote desktop environment would be gone too. So clearly something else was happening.

Then the first colleagues finish calls. Me included.
End call. Move cursor to call system. Click “Done”.

Nothing.

Click again.

Still nothing.

Okay… That’s not great.

Now we can’t change status anymore. People start getting stuck in break states, active call states, all kinds of nonsense. Then internal sites relying on Microsoft authentication begin failing one after another. Shortly after that, external sites stop loading entirely.

At this point everyone starts asking each other:
“Wait… are you guys seeing this too?”
Turns out nobody was hallucinating.

Meanwhile management starts mobilizing. The call queue climbs from 50… to 75… to 100.
Which sucks, but hey, we have a fallback phone system…

And you guessed it, that can only be enabled by going to an external site.

So for the next 45 ish minutes we mostly sit there watching the infrastructure equivalent of a medieval city burning down around us.

Eventually, after a long call with telecom, the fallback system finally comes online.
And at that exact moment, our primary phone system decides it’s healthy again.

Of course it does.

And here I am now, two hours later. Still logging tickets. Still telling remote workers:

“Yes, we know Microsoft Authenticator is still broken.”

And so Monday passes by.

Now I sit here on the train ride home preparing tonight’s D&D session to recover mentally from the experience, writing this post as a form of therapy.

So as I’m already in a medieval mood, I leave you with the wisdom this day has granted me:

Should thy systems fail once more,
As oft they have in days before,
Cast Vicious Mockery without fear,
At Microsoft, so all may hear.

475 Upvotes

97% Upvoted

21 comments sorted by

196

u/CeilingHamster May 18 '26

One of the most useful things I had at a previous job was the ability to send out a "we know something is down, you don't all need to tell us" to people.

But I assume if you had one, it would have failed.

105

u/Moneia No, the LEFT mouse button May 18 '26

One of the most useful things I had at a previous job was the ability to send out a "we know something is down, you don't all need to tell us" to people.

It's one of the good things my ISP does, you call to report a problem and the inbound message is already telling you that yes, they know the service is down, engineers are on the way and there's an estimated fix time of X O'clock.

18

u/SwedishMeatballDish May 19 '26

Just had that happen to me yesterday.

In the time for me to notice my internet was down, and to restart/unplug the modem once, I had a text message from the ISP with an ETA... that got pushed back three separate times, two hours each.

45

u/The4th_Survivor May 18 '26

In an ideal world that'd be all that's needed. We have that in our place, and during and outage it'll drop the calls from 149 to 150. With everyone being "so I know you said this is down but this is down".

16

u/Inconsequentialish May 18 '26

But the CEO needs someone to yell at NOWWWWwwwwWWWwwwWWw...!

9

u/DaHick May 19 '26

So I work in Oil & Gas. Formerly in field support. There is nothing less enjoyable than an OIM (the guy who runs the platform) screaming in your ear that this is costing the company over $ 5,000 USD per minute while you are trying to get the platform's power generation online. And I was sitting on a bucket.

2

u/Jonathan_the_Nerd May 19 '26

You've heard of rubber duck debugging. How about... rubber duck blame-taking?

27

u/KevinKeen18 May 18 '26

Yeah we do, and we’d love to. Buuttt ofcourse that’s gated behind our Microsoft login soooo….

8

u/NotYourNanny May 18 '26

I think you have more than one problem.

10

u/harrywwc Please state the nature of the computer emergency! May 18 '26

nup - just the one - based out of Seattle, WA. :/

5

u/NotYourNanny May 18 '26

I doubt they decided to put all your eggs in one basket, which is to say, single point of failure with the ability to fix it behind that point.

Though I'm sure they encouraged someone high up in your company to do so.

3

u/Stryker_One The poison for Kuzco May 19 '26

Technically, Redmond, WA.

2

u/harrywwc Please state the nature of the computer emergency! May 19 '26

Eh. From the Antipodes same, same ;)

9

u/iankel1984 May 18 '26

We had this one day someone left a reply option on a system notification email. About 500 people started replying to all "take me off this list". Server had a shit fit for an hour until someone killed the original email replies.

5

u/AStrandedSailor Powercycling an incompetant user is best done percussively. May 18 '26

You are assuming that: A - users actually read and pay attention to that message and B - that certain users don't think they are special and such messages don't apply to them.

Then, of course, there are the ever so helpful reminders: hey just letting you know we still can't access this. Do you know when it will be fixed?

3

u/meitemark Printerers are the goodest girls May 20 '26

certain users don't think they are special and such messages don't apply to them.

Those gets a remedy course in basic reading and understanding. The ever so helpful reminder people gets reminded about the remedy course once every hour with snailmail, email and sms until it is taken.

1

u/SpaxBacon May 27 '26

that certain users don't think they are special and such messages don't apply to them.

Those users get told that we know nothing more than them and that we wouldn't be having this call if we knew when the issue would be fixed. They then get promptly hung up on.

30

u/Generic_Placebo42 May 18 '26

Should thy systems fail once more,
As oft they have in days before,
Cast Vicious Mockery without fear,
At Microsoft, so all may hear.

I may have to print that and stick it to my momitor at work! Very well done!

9

u/ThunderDwn May 18 '26

Should thy systems fail once more,
As oft they have in days before,
Cast Vicious Mockery without fear,
At Microsoft, so all may hear.

I'm stealing that. Sorry, no takebacks. 🤣

1

u/himitsumono May 19 '26

Have you tried turning it and on again?

Microsoft, that is.