An alert that fires too often teaches people to ignore alerts. Once that habit forms, the one page that signals a real outage competes with dozens of meaningless ones, and engineers sleep poorly, burn out or quietly mute their phones. Reducing alert fatigue is both a reliability goal and a people goal, and the two are tightly linked.

Recognize the Symptoms

Alert fatigue rarely announces itself. Look for these signs:

• Pages that are acknowledged and closed with no action taken.
• Alerts that auto-resolve before anyone looks.
• Channels full of notifications that everyone has muted.
• Engineers reluctant to join the on-call rotation.
• Incidents discovered by customers while alerts were technically firing.

If you see several, you have a noise problem worth treating as a project, not a nuisance.

The Core Principle: Page on Symptoms, Not Causes

A page should interrupt a human only when something needs urgent human action. A common guideline is to alert on symptoms users would notice, such as elevated error rates or high latency against agreed objectives, rather than on every internal cause such as one node’s CPU spike or a single failed retry.

Cause-based signals still matter, but most belong on dashboards or in lower-urgency notifications. Ask of every proposed page: is it urgent, is it actionable, and does it need a person right now? If any answer is no, it should not wake anyone.

Classify Alerts by Urgency

Separating alerts into tiers prevents everything from competing for attention.

Tier	Meaning	Delivery	Example
Page	User impact now or imminent, needs immediate action	Phone, escalates if unacknowledged	Checkout error rate above objective
Ticket	Needs attention within working hours	Ticket queue or chat	Disk trending toward full in several days
Log only	Useful for investigation	Dashboard or stored event	Single pod restart that recovered

Moving alerts to the right tier is usually the fastest noise reduction available.

Tune the Alerts Themselves

Many noisy alerts are technically correct but badly configured. Techniques that help:

• Add duration conditions. Require a condition to hold for a sensible window so brief blips do not page.
• Use rates and percentages rather than raw counts that scale with traffic.
• Alert on error budget burn against service level objectives, so pages reflect risk to reliability targets rather than arbitrary thresholds.
• Group related alerts so one incident produces one notification, not fifty.
• Suppress downstream alerts when a known upstream dependency is already down.
• Silence maintenance windows deliberately, with an expiry.

Give Every Alert an Owner and a Runbook

An alert without an owner is orphaned, and an alert without a runbook forces responders to invent a response at their worst moment. Each page-worthy alert should link to a short runbook describing what the alert means, how to confirm impact, first mitigation steps and who to escalate to. If the runbook says “do nothing”, delete or downgrade the alert.

Make Alert Review a Habit

Noise returns unless someone holds the line. Hold a short recurring review, weekly or fortnightly, covering:

1. Which alerts fired most, and what was done about each?
2. Which pages required no action? Fix, downgrade or delete them.
3. Which incidents had no alert at all? Add detection.
4. How were the on-call load and sleep disruption distributed across the team?

Track simple counts over time, such as pages per shift and the share of pages that led to action, but use them to guide improvements rather than to judge individuals.

Look After the Humans

Technical tuning is not enough. Sustainable on-call also needs fair rotation schedules, clear escalation paths, time off after disruptive nights and a blameless culture around incidents. Engineers who fear being blamed for a missed page are more likely to over-alert defensively. Staffing and working-time rules vary by organization and jurisdiction, so involve management and HR in setting expectations.

A Realistic Example

Suppose a team’s on-call rotation receives many pages each week, mostly about individual container restarts and brief CPU spikes that recover on their own. In a review, they sort alerts into tiers. Restart and CPU alerts become dashboard signals, with one ticket-level alert if restarts persist. They replace several per-service error alerts with a single alert based on error rate against an agreed objective, with a duration window. Each remaining page links to a runbook. The following month, the rotation sees far fewer pages, nearly all of them actionable, and two previously missed failure modes gain proper detection.

Frequently Asked Questions

How many alerts is too many? There is no universal number. A useful test is whether nearly every page requires action. If many do not, you have too many.

Should we delete alerts that never fire? Review them. Some guard rare but severe failures and are worth keeping, while others are obsolete. Test that they still work.

Can automation replace some pages? Yes. If the response is always the same safe action, such as restarting a stuck worker, automate it and alert only if automation fails.

Who should own alert quality? The team that owns the service owns its alerts, with platform or reliability teams providing shared standards and tooling.

How do we start with limited time? Take last month’s pages, rank them by frequency and fix the top three offenders. Small, visible wins build momentum.

Next Step

Export your last month of pages and classify each as actionable, informational or noise. Downgrade or delete the noise, attach runbooks to what remains, and book a recurring review so the improvements stick.