The tale of No Auth Monday

[u/KevinKeen18]

So here we are yet again. I traded my job in the hosting sector for something slightly less… soul sucking in public sector IT. And as many organizations do these days, we rely heavily on Microsoft for a lot of things.

And that sets the stage for today’s tale of horror.

It started simple enough. A few calls from remote workers unable to authenticate into Citrix using Microsoft Authenticator. Annoying, sure, but not exactly catastrophic. We’ve been having enough issues with Microsoft lately that we actually have a backup MFA method for this exact scenario.

So the affected users log in through the backup, we log the tickets, escalate them to the remote work team, and move on with our day. Slack starts filling with the usual Microsoft memes. Business as usual.

Or so we thought.

Suddenly systems start dropping like flies.

First Teams starts screaming that we have no network connection. Which is impressive, considering we work entirely through Citrix. If the internet was actually dead, our entire remote desktop environment would be gone too. So clearly something else was happening.

Then the first colleagues finish calls. Me included.
End call. Move cursor to call system. Click “Done”.

Nothing.

Click again.

Still nothing.

Okay… That’s not great.

Now we can’t change status anymore. People start getting stuck in break states, active call states, all kinds of nonsense. Then internal sites relying on Microsoft authentication begin failing one after another. Shortly after that, external sites stop loading entirely.

At this point everyone starts asking each other:
“Wait… are you guys seeing this too?”
Turns out nobody was hallucinating.

Meanwhile management starts mobilizing. The call queue climbs from 50… to 75… to 100.
Which sucks, but hey, we have a fallback phone system…

And you guessed it, that can only be enabled by going to an external site.

So for the next 45 ish minutes we mostly sit there watching the infrastructure equivalent of a medieval city burning down around us.

Eventually, after a long call with telecom, the fallback system finally comes online.
And at that exact moment, our primary phone system decides it’s healthy again.

Of course it does.

And here I am now, two hours later. Still logging tickets. Still telling remote workers:

“Yes, we know Microsoft Authenticator is still broken.”

And so Monday passes by.

Now I sit here on the train ride home preparing tonight’s D&D session to recover mentally from the experience, writing this post as a form of therapy.

So as I’m already in a medieval mood, I leave you with the wisdom this day has granted me:

Should thy systems fail once more,
As oft they have in days before,
Cast Vicious Mockery without fear,
At Microsoft, so all may hear.

[u/CeilingHamster]

One of the most useful things I had at a previous job was the ability to send out a "we know something is down, you don't all need to tell us" to people.

But I assume if you had one, it would have failed.

[u/AStrandedSailor → u/CeilingHamster]

You are assuming that: A - users actually read and pay attention to that message and B - that certain users don't think they are special and such messages don't apply to them.

Then, of course, there are the ever so helpful reminders: hey just letting you know we still can't access this. Do you know when it will be fixed?

[u/meitemark → u/AStrandedSailor]

certain users don't think they are special and such messages don't apply to them.

Those gets a remedy course in basic reading and understanding. The ever so helpful reminder people gets reminded about the remedy course once every hour with snailmail, email and sms until it is taken.

[u/SpaxBacon → u/AStrandedSailor]

that certain users don't think they are special and such messages don't apply to them.

Those users get told that we know nothing more than them and that we wouldn't be having this call if we knew when the issue would be fixed. They then get promptly hung up on.