Alerts that people act on
An alert is only useful if someone acts on it. Good alerts fire on conditions that matter, persist long enough to be real, go to the team that owns them, and come with a run book. Too many alerts teach people to ignore them, which is how real problems get missed.
What makes a good alert
- It matters to users or deadlines: response time, failed transactions, a stopped channel, a job on the critical path.
- It is real: the condition holds for several samples, not one spike.
- It has an owner: it goes to the team that can fix it, not to everyone.
- It has a response: a run book entry says what to check and what to do.
- It has a severity: warning (look soon) and critical (act now) are handled differently.
Thresholds and baselines
A threshold is a fixed limit, such as 85% page data set use. A baseline compares with normal for that time, such as "twice the usual queue depth for a Monday at 09:00". Thresholds are simple and right for hard limits; baselines suit measurements that vary with the business day.
Alert fatigue
When a console scrolls with hundreds of alerts that need no action, people learn to ignore them, and the one that matters is missed. Review alerts regularly: remove those nobody acts on, merge duplicates, and tune thresholds that fire too often.
From alert to fix
| Step | What you do |
|---|---|
| 1. Confirm | Is it real, and is it still happening? Look at the graph, not only the message. |
| 2. Size it | Who is affected and how badly? This sets the severity. |
| 3. Correlate | What else changed at the same moment: a change, a batch job, another alert? |
| 4. Act by the run book | Take the documented action or involve the owner. Do not improvise on production. |
| 5. Record | Open or update the ticket with times, evidence and actions. |
What is the name for people ignoring alerts because there are too many? (two words)
Show a hint
They are tired of alerts.
Show the solution
Alert fatigue.
Common mistakes
Each alert should need a human. Others belong on a dashboard or in a report.
Wait for the condition to persist, unless the run book says otherwise, so spikes do not cause actions.
An alert that goes to everyone is owned by no one.
What you will see at work
- Teams often review their noisiest alerts monthly and remove or tune them.
- Alerts from mainframe monitors are commonly forwarded to an enterprise event console or ticketing system, so all teams see one queue.
- The Mainframe observability course covers golden signals and sending mainframe data to enterprise platforms.
Key terms
Check your understanding.
Take this lesson's quiz and save your progress. Free.