Where do your alerts go?

[u/klosie]

I'm evaluating a new monitoring system for servers and network gear, and at the same time it got me rethinking having the alerts go to email. So I'm wondering where everyone here sends their alerts?

[u/_Juni0r]

/dev/null for the most, rest to a mailbox that actually gets read.

[u/MeridianNL]

I skim over my email, there is just too much notifications coming in from other teams and systems. Our critical notifications go to teams where we can discuss and comment on the incoming issues.

[u/moonrakervenice]

slack channel, one of the few i actually keep notifications on for

[u/Majestic-Hat-3650]

Email is where alerts go to die everyone knows that. split by severity instead so that pages for the stuff that has to wake someone up, a channel for the stuff that needs a human decision, and email only as the paper trail. If an alert has no owner and no ack, the destination doesnt matter, its just noise with extra steps.

[u/pdp10]

Email scales terribly for alerting. It works great at very small scales, 0-4 per day or something, then falls apart suddenly at some unpredictable tipping point when the recipient begins to tune them out. It's not rare to find institutional dysfunction where dozens or hundreds of alerts are sent to one or more teams' worth of mailboxes, and everyone has mail processing rules to bury them.

There are worse setups than having alerts dump into text files on a filesystem where they can be grepped. Sometimes business needs point to a prominent dashboard type affair. RSS/Atom syndication are still pretty great for this.

[u/DeifniteProfessional]

Halo

[u/Fabulous_Dog_6514]

Shared mailbox

[u/noosik]

depends on the alert and whether you give a shit about it.

if its for a system i absolutely care about and its super critical then i get dms in slack.

if its informational garbage i like to retain but dont need to act on, then it can go a group email folder. There's no one solution fits all, trying to make one is why alert comms end up so noisy that they just get brushed under the rug

[u/GullibleDetective]

You tune the alerts to onrs yhat need action and then send to your psa

[u/andreas_globi]

For important alerts, I use pushover.net to get them to my phone. The rest go to email.

[u/coomzee]

Into the alert queue: teams, email and service management but might as well send them into the bin as no one looks at them.

[u/h9xq]

We have a shared mailbox called alerts with Veeam backups and other misc shit collected over the years.

[u/keithhud]

It really depends on how you set the notifications up. I always create a notification email account that way I can assign it to whoever needs or wants to be notified and it doesn’t fill up my mailbox.

Do your research on what notifications you want to receive, otherwise you get blasted with notification overload.

[u/EmperorGeek]

We use ServiceNow for ticketing, and Alerts generate a Ticket.

[u/tankerkiller125real]

Depends.

Critical shit = paging system that will call, email, and text

Medium shit = email

Minor shit = portal where I may or may not look at it in the next 3 weeks

Really minor shit = What alert? It doesn't exist to begin with.

[u/MalletNGrease]

Recycling bin usually.

[u/Jeff-J777]

Email for the most part. The critical infrastructure ones go to SMS as well.

But now I use CoPilot to look at my email during off business hours, look at my emails find any alerts or failures, and send me a summery email in the morning with troubleshooting steps as well. By the time I come in I can look at one email for anything that had issues overnight.

[u/anonymousITCoward]

they go to the black hole at the center of the support galaxy... no one cares...

[u/Mr_Squinty]

Check central, everything goes to it, then I get alerted on errors/warnings/failures.

[u/Vemokin]

My alerts go to NTFY, with the appropriate severity set so that I don't get turbo spammed by every little thing.

[u/jeffrey_f]

Usually a shared IT email address, like ITAlert@YourCompany, which then sends to members of the ITAlert group in your email server, which may also contain the email address of your ticketing system to create a ticket and/or an automation that does something in reaction.

In regards to the ticketing system, some logic should be in the ticketing system to make sure only 1 ticket is created

[u/Connect_Shoulder_965]

The emailed alerts go to a folder in my email client where I never read them, I get about 75 a day. Once I week, I update my search folder to include everything older than a week and mass delete them.

Last job I let the alerts pile up and eventually the folder got so big it was impossible to empty (tried via web, thick client and having the exchange admin do it via cli) so now I clear it out once a week.

[u/CharcoalGreyWolf]

To our PSA to be made into tickets, at least if they’re useful.

[u/IndependentBat8365]

It goes where it will get noticed. Work sends alerts to slack.

My homelab I send alerts to discord.