Routing

Routes decide which channels hear about an incident. Manage them in Account → Alerts → Routing. Owners and admins can change routes.

How routes are tried #

Routes are tried from the top. The first route that matches an incident decides who is told, unless it’s set to Keep checking later routes, in which case later matching routes are used too. If nothing matches, the default route is used.

Use the up and down arrows on a route to change its place in the order.

Creating a route #

Click New route and set:

• Name, for example Payments critical.
• Which incidents:
  • Severity: any severity, warning and critical, or critical only.
  • Environments and Buckets: tick some to match only those. None ticked means all.
• Who is told: one or more steps, each with one or more channels.
• When: quiet hours, digest and reminders (below).

Routes can also match on alert sources and labels through the API. Those conditions show on the route and are kept when you edit it in the app.

Escalation steps #

Step 1 sends at once. Add up to four more steps with Add escalation step; each runs only if the incident is still unacknowledged after the number of minutes you set. For example:

1. At once: #payments-alerts and OPS Jira
2. After 15 minutes if unacknowledged: Payments PagerDuty

Acknowledging the incident cancels the steps that haven’t run yet.

Reminders #

Remind about critical incidents nobody has acknowledged sends step 1’s channels a reminder every 15 minutes, 30 minutes, hour or 4 hours, for as long as a critical incident stays open and unacknowledged. Acknowledging or resolving the incident stops them.

Quiet hours #

Tick Quiet hours, then set From, To and the Time zone. During quiet hours, anything below critical is held and sent as one digest when they end, so a quiet night is one message in the morning. Critical still pages is on by default; turn it off to hold critical too.

PagerDuty and Jira can’t batch, so they get each held notification on its own when quiet hours end.

Digests #

Tick Digest to batch anything below critical into one message instead of sending each. Choose Every hour (on the hour) or Once a day at a time and time zone you pick.

A digest lists each incident once, worst first, with its status when the digest is sent, so an incident that opened and resolved overnight shows as resolved. Info incidents always go to the digest, whatever the route says.

Slack, email and webhooks support digests. PagerDuty and Jira get each notification on its own instead.

The default route #

Every account has a default route: critical incidents email the owners and admins. It’s always tried last and catches anything no other route matched, so critical incidents always reach someone.

You can change who the default route sends to, but you can’t delete or disable it.

Why did (or didn’t) I get notified? #

Open the incident. Its Notifications list shows every channel that was told and why, for example:

• “Route step 1”: a route matched and this was its first step.
• “No route matched; the default route”.
• “Follow-up to a channel already told about this incident”: a resolution, acknowledgement or comment.
• “batched into the digest” or “held for quiet hours, sent as a digest”.

If nobody was notified, either the incident was silenced (its timeline says so), or it was below critical and no route matched it: only critical incidents go to the default route.