Sign in Start free

Alerts

Get told when your API breaks: incidents, who was notified and why, and what counts as broken

This page as Markdown

ApiTraffic watches your traffic and opens an incident when an API gets significantly worse than its own normal. It then tells the right people through your channels (Slack, email, webhooks, PagerDuty, Jira), following your routing.

Alerts are on every plan. Open them from Account → Alerts, or from Incidents in a bucket’s sidebar.

What counts as broken

ApiTraffic compares each endpoint with its own normal, not with a fixed number, so a busy endpoint that always returns some errors doesn’t page you every hour.

A change counts only when it passes three tests at once:

  1. Relative: it’s a multiple of the endpoint’s normal (for example 3× the usual error rate).
  2. Absolute: it’s big in real terms (for example at least 2 percentage points), so 0.01% → 0.03% never pages anyone.
  3. Statistical: it’s very unlikely to be chance for the amount of traffic (99.9% confidence). Two errors in 20 requests don’t count; 200 in 2,000 do.
Signal Relative Absolute Minimum traffic Critical when
Server errors on an endpoint 3× normal +2 percentage points 50 requests +10 points, or 25% or more of requests
Server errors across a bucket 2× normal +1 percentage point 200 requests +5 points, or 25% or more of requests
Client errors (4xx) on an endpoint 3× normal +5 percentage points 50 requests 401/403 on half of all requests
p95 latency on an endpoint 1.5× normal +150 ms 50 requests 3× and +1 second

Server errors are 5xx responses and timeouts. Client errors (4xx) count only when they’re spread across consumers: at least 3 different consumers, with no single one causing more than 80% of them. One client sending bad requests shows as information on the dashboard and never opens an incident.

“Normal”

An endpoint’s normal is the higher of:

  • the previous hour, and
  • the same hour last week, when last week had enough traffic.

So a spike that happens every Monday at 09:00 is that endpoint’s normal on Mondays. While an incident is open, the endpoint keeps being compared with how it was before it broke, so a long outage never becomes the new normal.

A new endpoint with no history needs a clearly high rate (at least 10% server errors, allowing for the amount of traffic) before it counts.

Long enough

The last hour of traffic is checked every 5 minutes. A problem is reported once it holds for two checks in a row, about 10 minutes, or straight away when it’s severe (at least half of 100 or more requests failing). It clears after two clean checks in a row, and the incident resolves 5 minutes after all its alerts have cleared.

Breaking API changes found by drift detection open incidents too.

Incidents

An incident groups related alerts: problems on the same endpoint, after the same deploy, or starting together within a few minutes. The Incidents page lists open incidents first, then by severity and age. Severity is always shown as an icon and a word: Critical, Warning or Info.

Open an incident to see:

  • Alerts: each problem with its numbers, for example server error rate 18.4%.
  • Timeline: everything that happened, with times: opened, alerts added, escalations, comments, notifications that couldn’t be delivered.
  • Notifications: every channel that was told, and why, for example “Route step 1” or “No route matched; the default route”. A notification that failed shows its error and when the next attempt is due.

Actions

Action What it does
Acknowledge Tells everyone someone is on it. Stops escalation steps and reminders.
Resolve Closes the incident. Channels that were told get a resolution.
Reopen Opens a resolved incident again.
Snooze 1 hour / 4 hours / 24 hours Stops notifications about this incident’s alerts for that long. The incident still shows here. See Silences.
Add a comment Adds it to the timeline and sends it to channels already told about the incident.

Anyone on the team can take these actions.

On the dashboard

A bucket’s dashboard shows its open incidents at the top, in the needs attention panel. Each incident is one row that you can open, acknowledge or resolve. The signals an incident already covers aren’t listed again next to it. Signals that haven’t opened an incident are still listed, for example ones below the alerting thresholds.

Acknowledging works both ways:

  • Acknowledging an incident acknowledges the dashboard signals and drift findings in it.
  • Dismissing or snoozing a signal on the dashboard, or acknowledging a drift finding, acknowledges its alert. The incident stops escalating, but it stays open until someone resolves it or the problem clears.

Info never pages

Info-level incidents never send a notification on their own. They’re batched into the digest instead.

When alerts can’t reach anyone

If every channel a route sends to is failing or disabled, ApiTraffic opens a critical incident saying so and sends it to the default route (owners and admins by email). Silences don’t hide it. It clears when a channel recovers.

Owners and admins are also emailed whenever a channel starts failing, is disabled, or recovers. You can’t unsubscribe from these emails.

Next steps