Sign in Start free

If ApiTraffic is unreachable

What the ApiTraffic SDK does to your application when ApiTraffic, or your network path to it, is down: requests, startup, telemetry, redaction and the firewall.

This page as Markdown

Your application keeps serving traffic. On the default configuration, an ApiTraffic outage does not take your application down, block its startup or make its requests wait. That holds for a partial outage (telemetry ingest down) and for a full one (ingest, the configuration API and the realtime connection all down at once).

There is exactly one setting with which an ApiTraffic outage can stop your application serving requests, and it is off unless you turn it on: offlineWafMode: 'closed'. Read that section before you set it.

This page describes what the SDK does in an outage, and the side effects you might notice. It applies to @apitraffic/node and to the per-framework packages (@apitraffic/express, -fastify, -koa, -hapi) alike.

At a glance

Area What happens while ApiTraffic is unreachable
Your requests Served normally. Capture is fire-and-forget: no middleware waits on ApiTraffic before responding.
Startup Not blocked. The SDK loads its configuration in the background. See Startup for the one exception.
Telemetry Retried briefly from a small, bounded in-memory queue, then dropped. Nothing grows without limit.
Redaction Your last-known rules keep applying. With no rules available at all, data is held briefly and then dropped, never sent unredacted.
Firewall, default Running servers keep enforcing their last-known policy. A server that never loaded one allows traffic and logs a warning.
Firewall, offlineWafMode: 'closed' A server that never loaded a policy refuses every request with 503.
Running servers Keep their last-known configuration and reconnect on their own when ApiTraffic returns. No restart.
CLI in your CI definitions check exits 2, which fails the build unless you pass --ignore-unreachable.
Embedded request log Shows an error inside the component. The rest of your page is unaffected.

Telemetry delivery

Every captured request is handed off and your response goes out without waiting for it. When the ingest endpoint cannot be reached, delivery is bounded at every step:

  1. Timeouts and concurrency. Each send times out after 10 seconds. At most 50 payloads are in flight at once, over at most 20 keep-alive sockets.
  2. A small retry queue. Failures that are worth retrying - a timeout, a reset or refused connection, a DNS failure, a 5xx - go into a queue capped at 100 items or 2 MB, whichever comes first. Anything older than 30 seconds is dropped rather than retried. The queue is there to ride out a blip, not an outage.
  3. A circuit breaker. After 5 consecutive failures the SDK stops trying and lets a single probe through per window, backing off from 5 seconds up to 5 minutes.
  4. Cheap rejection. Once the breaker is open and the queue is full, new payloads are dropped before exclusion matching, redaction and serialization run. During a sustained outage the SDK therefore costs each request less than it does normally.
  5. Recovery. The first successful probe closes the breaker. Because each server sends one probe per window, your fleet does not resume at full rate in a single burst.

Telemetry captured during an outage, beyond what the retry queue delivers once ApiTraffic returns, is not recovered. stats() reports how much was dropped and why:

javascript
const apitraffic = require('@apitraffic/node');

console.log(apitraffic.stats());
// { pending, queued, delivered, recovered, dropped, droppedRejected, breakerOpen, ... }

Side effects you might notice

Losing telemetry is the expected cost of an outage. These are the other effects, all bounded:

Effect When How large, and for how long
Memory The ingest endpoint accepts connections but never answers, rather than refusing them Up to 50 in-flight payloads held for up to 10 seconds, plus the 2 MB retry queue. Usually a few MB; in the worst case, with large bodies and many spans, about 100–150 MB. It drains once the breaker opens.
CPU The first seconds of an outage, before the breaker opens The same per-request cost as normal. After that it falls below normal. Every 30 seconds, as stale queue entries expire, a few requests are prepared again before being dropped.
Open sockets Same as the memory case Up to 20 file descriptors.
DNS lookups DNS for ApiTraffic is slow or unresponsive, rather than failing fast The SDK resolves its own hosts on a separate resolver that does not use Node’s thread pool, so your fs, crypto and zlib work is not held up. Where it falls back to the system resolver, its lookups run one at a time, using at most one thread-pool thread.
Log volume debug: true in production One line per failed send. Without debug, the SDK logs one line when it falls back to cached configuration and one when live configuration returns.

Startup

None of the ApiTraffic middleware or plugins wait for the SDK to load its configuration: they start it in the background and your server starts listening immediately, outage or not.

Note: The exception is your own code. If something in your startup path awaits the SDK’s setup() before calling listen(), that wait lasts up to about 15 seconds while ApiTraffic is unreachable: up to 10 seconds for the configuration API, then up to 5 seconds for the CDN mirror, then the on-disk snapshot. Don’t put that await in front of listen(); the middleware does not need it.

Configuration comes from the first source that answers: live, then the disk cache, the CDN mirror and finally the file pinned in your repository. Offline definitions explains that ladder, and how to make sure a brand-new container still has your rules during an outage.

Redaction rules

Your redaction rules are applied in your process, before anything leaves it, and an outage does not change that:

  • A server that has already loaded its rules keeps using them.
  • A server that boots during an outage uses the cached, mirrored or pinned copy.
  • A server with no copy of your rules at all holds captured requests in memory - at most 50 of them, 1 MB in total, for 30 seconds - waiting for the rules to arrive. If they do not arrive in time, those requests are dropped. They are never sent unredacted.

The firewall

A server that has already loaded your firewall policy keeps enforcing it, in the mode you chose, for as long as the outage lasts. So does a server that boots from the cached, mirrored or pinned copy. The only open question is a server that never loaded a policy - booted during an outage, with no snapshot on disk. offlineWafMode decides what happens to its traffic:

Setting A server with no policy
inherit (default) or open Allows every request, unfiltered, and logs a warning once.
closed, or API_TRAFFIC_OFFLINE_WAF_MODE=closed Refuses every request with 503 until a policy loads.

Warning: offlineWafMode: 'closed' makes ApiTraffic’s availability part of yours. It is the only setting with which an ApiTraffic outage takes your application down: every request on an affected server gets a 503 until a policy loads. That is working as designed, for teams who would rather refuse traffic than serve it unfiltered - but choose it deliberately.

  • Commit pinned definitions if you use it. A server that boots from a pinned file has a policy, so closed never applies to it. This is what keeps closed from turning our outage into yours.
  • Removing or disabling the firewall policy for a bucket is not an outage: a server that has loaded settings with no firewall policy allows traffic, whatever closed says. (Earlier @apitraffic/waf releases refused every request in that case; turn closed off before removing a policy if you run one.)
  • API_TRAFFIC_OFFLINE_WAF_MODE overrides the code setting, so during an incident you can set it to open and restart, without a deploy.

How to set it, and how it differs from a policy’s failMode, is on the offline definitions page.

Running servers

A server that was running when the outage began keeps its last-known configuration - redaction rules, exclusions and firewall policy. It keeps retrying the configuration API for the life of the process, with backoff and jitter, and the realtime connection reconnects on its own. When ApiTraffic returns, the server switches back to live configuration without a restart.

Host telemetry (CPU, memory and runtime samples) queues up to 240 samples with backoff and then sheds the oldest. Its footprint is negligible.

Your CI pipeline

npx @apitraffic/cli definitions check exits with code 2 when it cannot reach ApiTraffic. By default that fails your build, which makes our availability a dependency of your deploys. Most teams should pass --ignore-unreachable:

bash
npx @apitraffic/cli definitions check --ignore-unreachable

Real drift still fails the build, and so does a rejected token; only “ApiTraffic could not be reached” becomes a pass. See Choosing how strict to be and the CLI reference.

The embedded request log

The embeddable request log shows an error state inside the component. The rest of your page renders and behaves normally.

  • Offline definitions: Keep your rules and firewall policy available to a server that starts during an outage.

  • CLI: definitions check exit codes and flags.

  • Set up the firewall: Modes, kill switches and failure behavior.

  • Servers: See which servers are running on fallback configuration.