Add the alert that would have paged sooner
An alert on the signal that moved first, back-tested against the incident window.
The ticket
Add the alert that would have paged before the incident became visible to users.
Acceptance criteria
- The rule fires when replayed against the incident window
- It does not fire against a comparable healthy window
- It links to a runbook that exists
- The routing destination is a person or rotation, not an unwatched channel
What lands as proof
The rule replayed against both the incident window and a healthy one, showing it fires for the first and not the second.
Why teams defer it
- An alert that fires too often gets muted within a week, so tuning it is the actual work.
- Back-testing needs retained metrics at the right resolution, and the retention window may already have passed.
Questions
- What does the agent actually change?
- The ticket is scoped to one outcome: add the alert that would have paged before the incident became visible to users. Work that serves that outcome is in scope, and anything outside it is left for a separate ticket, so the pull request stays reviewable.
- How do I know the work is done?
- The pull request carries the evidence, not only the diff. Here that means rule and runbook in the same pull request, so a reviewer can confirm the result without reproducing the work locally.
- How much oversight does this need?
- The run stops once the plan is written. Somebody reads the approach and approves it before any code exists, which is the cheapest moment to redirect the work.