The short answer: every service reads the incoming traceparent header, keeps its trace ID for the duration of the request, writes that ID into every log line, and sends a new traceparent with the same trace ID on every outbound call. With OpenTelemetry’s Node SDK and auto-instrumentations, inbound HTTP, outbound http and fetch calls get this for free. Without OpenTelemetry, it is about 30 lines of code. Both are below.

What is W3C Trace Context? #

W3C Trace Context is a W3C Recommendation (Level 1, last published November 2021) that defines two HTTP headers: traceparent, which carries the trace identity, and tracestate, which carries vendor-specific data. Because it is a standard, a trace can pass through an Express service, a Fastify service, a load balancer and a third-party tool and keep the same ID.

A traceparent looks like this:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

A trace ID or parent ID of all zeros is invalid. Receivers must accept the header name in any case, and tracestate can hold at most 32 list members.

How is traceparent different from X-Request-ID? #

A request ID usually identifies one hop. The trace ID identifies the whole chain. You can keep both, or use the trace ID as your request ID. If you already followed our post on request IDs and AsyncLocalStorage, the structure below will look familiar: the difference is a standard header and a standard format that other tools understand.

Option 1: W3C Trace Context with OpenTelemetry in Node.js #

OpenTelemetry’s default propagators in JavaScript are W3C Trace Context and W3C Baggage. The auto-instrumentations package includes HTTP, Express, undici (which is what Node’s global fetch uses), Postgres and Pino instrumentation, among others.

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

// instrumentation.js  (load before anything else)
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  serviceName: process.env.SERVICE_NAME ?? 'orders-api',
  traceExporter: new OTLPTraceExporter(),   // reads OTEL_EXPORTER_OTLP_ENDPOINT
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

process.on('SIGTERM', () => sdk.shutdown().finally(() => process.exit(0)));

node -r ./instrumentation.js server.js

That is enough for:

• inbound requests: traceparent is extracted, and the request’s span joins the caller’s trace
• outbound http.request and fetch calls: a new traceparent with the same trace ID is injected
• Pino logs: the Pino instrumentation adds trace_id and span_id to log records written inside a traced request

Load order matters. The instrumentation has to load before Express, pg and Pino are required, which is why it goes in -r rather than at the top of server.js. For Fastify, the Fastify team maintains @fastify/otel for route-level spans; inbound propagation already works through the HTTP instrumentation.

Option 2: propagate traceparent by hand #

If you do not want an OpenTelemetry pipeline yet, propagation alone is small. This version accepts version 00 only, which is what every current sender emits:

// trace-context.js
import { AsyncLocalStorage } from 'node:async_hooks';
import { randomBytes } from 'node:crypto';

const als = new AsyncLocalStorage();
const TRACEPARENT = /^00-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$/;
const hex = (bytes) => randomBytes(bytes).toString('hex');

export function parseTraceparent(value) {
  const m = TRACEPARENT.exec((value ?? '').trim().toLowerCase());
  if (!m || /^0+$/.test(m[1]) || /^0+$/.test(m[2])) return null;
  return { traceId: m[1], parentId: m[2], flags: m[3] };
}

export function traceContext({ trustIncoming = true } = {}) {
  return (req, res, next) => {
    const incoming = trustIncoming ? parseTraceparent(req.get('traceparent')) : null;
    const ctx = {
      traceId: incoming?.traceId ?? hex(16),
      spanId: hex(8),
      flags: incoming?.flags ?? '01',
    };
    res.set('x-trace-id', ctx.traceId);   // handy for support tickets
    als.run(ctx, next);
  };
}

export const currentTrace = () => als.getStore();

export function tracedFetch(url, init = {}) {
  const ctx = currentTrace();
  const headers = new Headers(init.headers);
  if (ctx) headers.set('traceparent', `00-${ctx.traceId}-${hex(8)}-${ctx.flags}`);
  return fetch(url, { ...init, headers });
}

Wire it into Express and into your logger:

import express from 'express';
import pino from 'pino';
import { traceContext, currentTrace, tracedFetch } from './trace-context.js';

const log = pino({ mixin: () => ({ trace_id: currentTrace()?.traceId }) });
const app = express();
app.use(traceContext());

app.post('/v1/orders', express.json(), async (req, res) => {
  log.info({ items: req.body.items?.length }, 'creating order');
  const pay = await tracedFetch('http://payments.internal/v1/charges', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ amount: 4200 }),
    signal: AbortSignal.timeout(5000),
  });
  res.status(pay.ok ? 201 : 502).json({ ok: pay.ok });
});

Every log line from that request now carries trace_id, and the payments service sees the same ID in its incoming traceparent. Searching both services’ logs for one ID gives you the whole story. For more on what to record about those outbound calls, see logging outbound HTTP calls.

Where trace context gets lost #

Queues and background jobs #

HTTP headers do not cross a queue. Put the traceparent string into the job payload or message metadata when you enqueue, and parse it in the worker before you do anything else.

Trust boundaries #

The W3C spec warns that blindly continuing any trace with the sampled flag set lets outsiders inflate your tracing costs or pollute your data with colliding IDs. On a public API, either start a fresh trace (traceContext({ trustIncoming: false })) and log the caller’s ID as a separate field, or only trust traceparent from your own front end and internal services.

Baggage and tracestate #

Do not put user emails, tokens or anything personal in baggage or tracestate. They are forwarded to every downstream service, including third parties.

If you only do one thing #

1. Add trace_id to every log line in every service.
2. Accept traceparent on internal calls and send it on every outbound call.
3. Start a new trace at the public edge, or validate before you trust.
4. Carry traceparent through your queue payloads.

A one-person team with two services can use Option 2 this afternoon. A ten-person team with five or more services should adopt OpenTelemetry, so new services and libraries get propagation without anyone remembering to add it.

FAQ #

Does Node’s fetch send traceparent automatically? #

Not by itself. Node’s fetch sends traceparent only when something instruments it, such as OpenTelemetry’s undici instrumentation, or when you set the header yourself.

What is the difference between traceparent and tracestate? #

traceparent carries the IDs and the sampled flag in a fixed format. tracestate is an optional list of vendor-specific key-value pairs that travel alongside it.

Can I use the trace ID as a request ID? #

Yes, as long as you also create a new span ID per hop. Returning it in a response header makes support tickets much easier to trace.

Doing this with ApiTraffic #

If you would rather not stitch logs across services yourself, ApiTraffic records the trace ID on each captured request: from the incoming W3C traceparent header when there is one, otherwise from its first span. Requests from your services that share a trace ID appear together in the request’s Distributed Trace tab, in time order, so you can see which hop failed or was slow. Outbound calls are logged on the request that made them.

The trace ID is read before redaction, so a redaction rule on traceparent does not break grouping.

Start free and see every request your API handles: apitraffic.io

Sources: W3C, “Trace Context” Recommendation (Nov 23, 2021); OpenTelemetry, “Propagation” (JavaScript docs); npm, @opentelemetry/auto-instrumentations-node; Node.js docs, “Asynchronous context tracking”

Imperva’s 2026 Bad Bot Report puts bots at more than 53% of all web traffic, with 40% classified as malicious. It also reports that 27% of attacks now target APIs and identity systems, and that AI-driven bot attacks grew 12.5x. AI tooling makes recon, scraping and credential stuffing cheap to write and cheap to run. Scanners don’t check your headcount before they send requests. A small team just has less time to look at what arrives.

You don’t need a security team to see this traffic. You need the right fields on every request, and a few patterns you know how to spot.

1. What your host logs miss #

The default logs on Render, Railway, Fly or a bare VPS usually give you one line per request: timestamp, method, path, status, maybe latency. That’s enough to see that something got slow. It’s not enough to answer the questions an attack raises:

• Who was calling? Not just the IP, but which user or API key, if any.
• What did they send? The request body is where injection attempts and bad payloads live.
• What else did they do? One request to /v1/users/1042 is normal. Four hundred requests to /v1/users/1 through /v1/users/400, from one caller, in two minutes, is not. You only see that if you can group requests by caller and look at them in order.

Host logs answer the first question partially and the other two not at all.

2. The minimum to capture for every request #

If you log nothing else, log these:

The ASN (the network an IP belongs to) is worth having too, if your edge (Cloudflare, a load balancer) adds it as a header. A sudden wave from a hosting provider your users never come from is a strong signal. You don’t need to resolve it yourself.

3. Five patterns to watch #

All of these are cheap to detect once the fields above are in one searchable place. The log lines below are illustrative, using documentation IP ranges.

Login bursts. Credential stuffing looks like a lot of POST /v1/auth/login with 401s from a small set of IPs, or a slow trickle from many.

203.0.113.24  POST /v1/auth/login  401  38ms
203.0.113.24  POST /v1/auth/login  401  41ms
203.0.113.24  POST /v1/auth/login  401  36ms   (x 412 in 10 min)

Sequential ID walking. One caller requesting /v1/users/1, /2, /3… This is how Broken Object Level Authorization gets found and exploited, and it’s #1 on the OWASP API Security Top 10 (2023). If the responses are 200, you have a data leak, not a probe.

198.51.100.7  GET /v1/users/1041  200
198.51.100.7  GET /v1/users/1042  200
198.51.100.7  GET /v1/users/1043  200

Pagination scraping. GET /v1/products?page=1 through page=900 at a steady rate, often with a generic or missing user agent. It’s not an exploit, but it costs you compute and can lift your whole catalogue. OWASP lists this class of problem as Unrestricted Resource Consumption (API4:2023).

404 sweeps. Requests for paths you never built: /.env, /.git/config, /admin, /wp-login.php. Almost always automated. Mostly harmless on a Node API, until one of them returns 200.

192.0.2.91  GET /.env            404
192.0.2.91  GET /.git/config     404
192.0.2.91  GET /admin           404
192.0.2.91  GET /api/v1/.env     404

New networks. Traffic that suddenly arrives from an ASN or country your users have never come from, often a cloud provider. On its own it proves nothing. Combined with any pattern above, it’s the caller to look at first.

4. When to block and when to watch #

Not everything that looks automated should be blocked. Some of it is your own monitoring, a partner integration, or a search engine.

• Watch first. Log and alert on the patterns above for a week before you block anything. You’ll find the legitimate automation you forgot about.
• Rate-limit sensitive routes. Login, password reset, signup and anything that sends email or SMS. Ten attempts per IP per minute on login is a reasonable start, and it’s the example ApiTraffic’s own firewall policy docs use.
• Block what’s never legitimate. Paths like /.env on an API that has no such file. IPs you’ve confirmed are abusive.
• Allow-list what must never break. Health checks, your own services, known partners.

The goal isn’t zero bot traffic. It’s knowing which bots you have, and making the expensive ones pay a 429.

5. Do it yourself in Express #

Here’s a minimal version: structured logs with the fields above, a stricter rate limit on login, and a cheap 404-sweep block. It assumes express-rate-limit and that your app runs behind one proxy.

const express = require('express');
const rateLimit = require('express-rate-limit');

const app = express();
app.set('trust proxy', 1); // trust exactly one proxy hop, so req.ip is the real client
app.use(express.json({ limit: '100kb' }));

// 1. One structured log line per request
app.use((req, res, next) => {
  const start = process.hrtime.bigint();
  res.on('finish', () => {
    console.log(JSON.stringify({
      ts: new Date().toISOString(),
      method: req.method,
      path: req.originalUrl.split('?')[0],
      route: req.route ? req.baseUrl + req.route.path : null,
      status: res.statusCode,
      latency_ms: Number(process.hrtime.bigint() - start) / 1e6,
      ip: req.ip,
      user_id: req.user?.id ?? null,     // set by your auth middleware
      ua: req.get('user-agent') ?? null,
      req_bytes: Number(req.get('content-length') ?? 0),
      asn: req.get('x-asn') ?? null,     // only if your edge sets and overwrites it
    }));
  });
  next();
});

// 2. Block paths that are never legitimate on this API
const probe = /^\/(\.env|\.git|wp-login\.php|phpmyadmin)/i;
app.use((req, res, next) => (probe.test(req.path) ? res.sendStatus(404) : next()));

// 3. Tighter limit on login
app.use('/v1/auth/login', rateLimit({ windowMs: 60_000, limit: 10 }));

app.use('/v1', apiRouter);
app.listen(3000);

Then point your log drain somewhere searchable and write the queries: 401s on login grouped by IP, distinct IDs per caller on /v1/users/:id, 404s per IP per hour.

Two things to get right:

• trust proxy. If you trust every hop, a caller can write any IP into X-Forwarded-For and your logs and rate limits will believe it. Trust only the proxies you actually run behind.
• The body. This logger leaves bodies out on purpose. If you add them, redact first (see section 7).

This works. The cost is that you now maintain a logger, a log pipeline, the queries and the alerting, and the rate limit counts per process unless you add a shared store.

6. Or do it with ApiTraffic #

ApiTraffic captures every request and response from one middleware, and the same SDK carries a firewall. The quickstart takes about five minutes.

npm install @apitraffic/node

const apitraffic = require('@apitraffic/node');

app.use(express.json());
app.use(apitraffic.express()); // after body parsers, before routes
app.use('/v1', apiRouter);

It reads API_TRAFFIC_TOKEN and API_TRAFFIC_BUCKET from the environment. To tie requests to accounts, tag them in a route or your auth middleware:

apitraffic.tag('userId', String(req.user.id));

The patterns above become searches in the request log:

url_path == "/v1/auth/login" AND status_code == 401
status_code == 404 AND url_path like "/.env*"
tag:userId == "1042" AND status_code >= 400

And the blocking side lives in a firewall policy, pushed to your servers without a deploy:

• Rate limits per IP, header (an API key, say) or path. Over the limit gets a 429 with Retry-After.
• Access lists to allow or deny by IP or CIDR, country, ASN (from a header your edge sets), user agent or path.
• Behavioral protection that scores each caller over a window on exactly these signals: distinct object IDs per endpoint, 401/403 counts, 404 counts and distinct endpoints. Callers who cross the threshold get a temporary deny on every server.
• Monitor mode first, so you can see what a policy would block before it blocks anything.

7. Redact before it leaves your server #

Capturing more detail means capturing more you shouldn’t keep: tokens, passwords, emails, card numbers.

If you roll your own logger, never log raw Authorization or Cookie headers, strip password and token fields from bodies before you write them, and keep body logging off for routes that handle credentials. More on this in redacting sensitive data in logs.

In ApiTraffic, credential headers such as Authorization are always masked. Redaction rules set in the dashboard can run at the API, inside your process, so the value never leaves your server. They’re pushed to connected servers when you save, with no deploy.

FAQ #

What should a small team log on every API request? #

Method, path (raw and the route template), status, latency, IP, a user or API key ID, user agent and request size. Add the ASN if your edge sets it as a header. Log an ID for the caller, never the token.

How do I spot credential stuffing in my logs? #

Look for a lot of POST /v1/auth/login requests returning 401, either from a small set of IPs or as a slow trickle from many. Group 401s on the login route by IP to see it.

Should I block bot traffic as soon as I see it? #

No. Log and alert on the patterns for a week first, because some automation is your own monitoring, a partner integration or a search engine. Then rate-limit sensitive routes, block what is never legitimate and allow-list what must never break.

Is it safe to log request bodies? #

Only after redacting. Never log raw Authorization or Cookie headers, strip password and token fields from bodies before you write them, and keep body logging off for routes that handle credentials.

The short version #

Log method, path, status, latency, IP, user ID, user agent and size on every request. Watch login bursts, ID walking, pagination scraping, 404 sweeps and new networks. Watch before you block. Rate-limit what’s sensitive.

Start free and see the last hour of what’s hitting your API: apitraffic.io

Deploy markers fix that. A deploy marker is a timestamped record of each release (service, environment, commit SHA, who shipped it) drawn on the same charts as your latency and error rates. Add one more thing, the running version on every request log line, and “which release did this” becomes a query: compare p95 per endpoint for the old version and the new one. This post shows how to do both for a Node API in an afternoon, from the build step to a before-and-after comparison script.

What is a deploy marker? #

A deploy marker is an event, not a metric. It needs a handful of fields:

Most monitoring tools have a place for these: Grafana annotations, Sentry releases, Datadog deployment tracking and Honeycomb markers all do the same job with different names. The tool matters less than doing it automatically on every deploy.

How to add deploy markers to a Node API #

Step 1: bake the commit SHA into the build #

Your running process should know which commit it is. Pass it in at build time:

# Dockerfile
FROM node:24-bookworm-slim
ARG GIT_SHA=unknown
ENV GIT_SHA=$GIT_SHA
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
CMD ["node", "server.js"]

docker build --build-arg GIT_SHA=$(git rev-parse --short HEAD) -t orders-api .

Some platforms set it for you. Render exposes RENDER_GIT_COMMIT at build time and runtime. In GitHub Actions, the commit is ${{ github.sha }}.

Step 2: put the version on every log line #

A marker tells you when a release started. During a rolling deploy, old and new instances serve traffic side by side for a while, so the marker alone cannot tell you which version handled a slow request. The version on each log line can:

// logger.js
import pino from 'pino';

export const version = process.env.GIT_SHA ?? process.env.RENDER_GIT_COMMIT?.slice(0, 7) ?? 'dev';

export const log = pino({ base: { service: 'orders-api', version } });
log.info('process started');

// server.js (Express)
import express from 'express';
import { log } from './logger.js';

const app = express();

app.use((req, res, next) => {
  const start = process.hrtime.bigint();
  res.on('finish', () => {
    log.info({
      method: req.method,
      route: req.route?.path ? `${req.baseUrl}${req.route.path}` : 'unmatched',
      status: res.statusCode,
      ms: Number(process.hrtime.bigint() - start) / 1e6,
    }, 'request');
  });
  next();
});

Use the route pattern, not the raw URL, or every ID becomes its own endpoint.

Step 3: send the marker from CI, not from memory #

The most useful markers are the ones nobody has to remember. Add a step at the end of your deploy job. This example writes a Grafana annotation; swap the URL and body for whatever tool you use:

# .github/workflows/deploy.yml (last step of the deploy job)
- name: Record deploy marker
  if: success()
  run: |
    curl -sf -X POST "$GRAFANA_URL/api/annotations" \
      -H "Authorization: Bearer $GRAFANA_TOKEN" \
      -H "Content-Type: application/json" \
      -d "{\"time\": $(date +%s000), \"tags\": [\"deploy\", \"orders-api\", \"production\"],
           \"text\": \"orders-api ${GITHUB_SHA::7} by ${GITHUB_ACTOR}\"}"
  env:
    GRAFANA_URL: ${{ secrets.GRAFANA_URL }}
    GRAFANA_TOKEN: ${{ secrets.GRAFANA_TOKEN }}

Record rollbacks the same way, with a name that says “rollback”. Record feature flag and config changes as markers too: some of the worst regressions ship without a deploy.

Compare p95 before and after a deploy #

With the version on every log line, you can compare releases directly instead of comparing time windows. This script reads NDJSON request logs and prints p95 per endpoint for two versions:

// compare-versions.mjs -- node compare-versions.mjs logs.ndjson a1b2c3d e4f5a6b
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const [file, before, after] = process.argv.slice(2);
const MIN_SAMPLES = 200;
const groups = new Map();   // "GET /v1/search" -> { [version]: number[] }

for await (const line of createInterface({ input: createReadStream(file) })) {
  if (!line.includes('"request"')) continue;
  const r = JSON.parse(line);
  if (r.version !== before && r.version !== after) continue;
  const key = `${r.method} ${r.route}`;
  const g = groups.get(key) ?? { [before]: [], [after]: [] };
  g[r.version].push(r.ms);
  groups.set(key, g);
}

const p95 = (xs) => {
  const s = [...xs].sort((a, b) => a - b);
  return s[Math.ceil(0.95 * s.length) - 1];
};

for (const [key, g] of groups) {
  if (g[before].length < MIN_SAMPLES || g[after].length < MIN_SAMPLES) continue;
  const [a, b] = [p95(g[before]), p95(g[after])];
  const flag = b > a * 1.25 ? '  <-- regression' : '';
  console.log(`${key.padEnd(30)} p95 ${a.toFixed(0)}ms -> ${b.toFixed(0)}ms${flag}`);
}

Why percentiles and not averages: one slow query path can double p95 while the average barely moves. Our post on reading p50, p95 and p99 covers how to read these numbers without fooling yourself.

Why before-and-after comparisons mislead #

• Traffic mix. A deploy at 9am Monday compared with Sunday night is comparing two different workloads. Compare by version during the overlap, or against the same hour a day or a week earlier.
• Warm-up. New processes start with cold caches, connection pools and JIT. Skip the first few minutes after each instance starts.
• Small samples. An endpoint with 40 calls an hour has a noisy p95. The MIN_SAMPLES guard is there for a reason.
• Bundled changes. If one deploy contains a dependency bump, a query change and a new index, the marker tells you which deploy, not which change. Smaller deploys make markers more useful.

If you only do one thing #

1. Bake the commit SHA into the image and log it on every request.
2. Send a marker from CI on every deploy and rollback.
3. Record flag and config changes as markers too.
4. When latency moves, compare p95 per endpoint by version before you debate theories.

A one-person team can get steps 1 and 2 done in an hour and stop guessing. A ten-person team should make the CI step part of the shared deploy workflow and agree on marker names, so markers from different services read the same way. If latency regressions are eating into your reliability targets, connect this to your error budget and to what you page on.

FAQ #

How long should I wait after a deploy before comparing latency? #

Long enough for a few hundred requests per endpoint you care about, after warm-up. For a busy API that is 15 to 30 minutes; for a quiet one, compare against the same window on the previous day.

Should feature flag changes be deploy markers? #

Yes, as their own type. A flag flip can change code paths as much as a deploy, and it is the change people forget to mention.

Do I need a commercial tool for deploy markers? #

No. A log line with the version and a CI step that writes an annotation in the tool you already use is enough to start.

Doing this with ApiTraffic #

If you would rather not build the comparison yourself, record each deploy as an event in ApiTraffic from your pipeline, with the commit in its metadata:

curl -s -X POST "https://api.apitraffic.io/v1/accounts/${APITRAFFIC_ACCOUNT_SID}/events" \
  -H "Authorization: Bearer ${APITRAFFIC_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d "{\"type\": \"deployment\", \"name\": \"orders-api ${GITHUB_SHA::7}\",
       \"metadata\": {\"commitSha\": \"${GITHUB_SHA}\"}}"

Markers appear on the throughput and response time charts. Regressions compares each endpoint’s p95 before and after a deploy event, and the endpoint’s Time Breakdown and Top Operations show whether the new time went to a query, an N+1 pattern or an outbound call.

Start free and see every request your API handles: apitraffic.io

Sources: Render docs, “Default Environment Variables”; GitHub Docs, “Contexts reference” (github.sha); Grafana docs, “Annotations HTTP API”

The short answer: never hold an upload in memory. Stream it to disk or object storage with stream.pipeline, set explicit limits on file size, file count and fields, and for big files skip your API entirely with presigned URLs to S3-compatible storage. And check your multipart parser version, because 2026 has brought a run of denial-of-service fixes in multer, @fastify/multipart and @fastify/busboy.

Why a Node.js large file upload crashes your API #

Three patterns cause most incidents:

• Buffering. Multer’s memoryStorage(), @fastify/multipart’s toBuffer() and hand-written Buffer.concat loops all keep the full file in memory. Ten concurrent 500 MB uploads need 5 GB.
• No limits. Without fileSize, files and fields limits, a single request decides how much work your server does.
• Long requests on a shared process. A slow upload ties up a connection for minutes. Synchronous work on the data, such as hashing a buffer in one go, blocks the event loop for everyone; see event loop lag.

Defaults help only for non-file bodies. Express’s express.json() limits JSON to 100 KB by default and Fastify’s bodyLimit defaults to 1 MiB, but multipart parsing is configured separately, in the multipart library.

Upload strategies compared #

How to stream file uploads in Express with busboy #

Multer is fine with diskStorage and limits, but using busboy directly shows exactly what happens. This Express 5 route streams one file to disk and enforces a size limit:

import express from 'express';
import busboy from 'busboy';
import { createWriteStream } from 'node:fs';
import { unlink } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
import { randomUUID } from 'node:crypto';
import path from 'node:path';

const app = express();
const UPLOAD_DIR = '/var/uploads';
const MAX_BYTES = 500 * 1024 * 1024;

app.post('/v1/uploads', requireAuth, (req, res, next) => {
  const bb = busboy({
    headers: req.headers,
    limits: { fileSize: MAX_BYTES, files: 1, fields: 5, fieldSize: 1024 },
  });
  let saved = null;
  const writes = [];

  bb.on('file', (name, file, info) => {
    // Never trust info.filename for the path
    const dest = path.join(UPLOAD_DIR, randomUUID());
    const write = pipeline(file, createWriteStream(dest))
      .then(() => {
        if (file.truncated) {
          throw Object.assign(new Error('file too large'), { status: 413 });
        }
        saved = { dest, type: info.mimeType };
      })
      .catch(async (err) => {
        await unlink(dest).catch(() => {});
        throw err;
      });
    write.catch(() => {}); // awaited on 'close'; avoids an unhandled rejection
    writes.push(write);
  });

  bb.on('close', async () => {
    try {
      await Promise.all(writes);
      if (!saved) return res.status(400).json({ error: 'no_file' });
      res.status(201).json({ id: path.basename(saved.dest) });
    } catch (err) {
      next(err);
    }
  });
  bb.on('error', next);
  req.on('close', () => { if (!req.complete) bb.destroy(); });
  req.pipe(bb);
});

The details that matter: the file is never in memory as a whole, the filename on disk is random, a truncated file is deleted, and an aborted request tears down the parser. Partial files left behind by aborted uploads are exactly what several 2026 advisories fixed in libraries, so test your own cleanup by killing curl mid-upload.

How to handle file uploads in Fastify with @fastify/multipart #

@fastify/multipart defaults to a 1 MiB fileSize and 1,000 parts. Raise the file limit deliberately, per route if needed:

import Fastify from 'fastify';
import multipart from '@fastify/multipart';
import { createWriteStream } from 'node:fs';
import { unlink } from 'node:fs/promises';
import { pipeline } from 'node:stream/promises';
import { randomUUID } from 'node:crypto';

const app = Fastify({ logger: true });
await app.register(multipart, {
  limits: { fileSize: 500 * 1024 * 1024, files: 1, fields: 5 },
});

app.post('/v1/uploads', async (request, reply) => {
  const data = await request.file();
  if (!data) return reply.code(400).send({ error: 'no_file' });
  const dest = `/var/uploads/${randomUUID()}`;
  try {
    await pipeline(data.file, createWriteStream(dest));
  } catch (err) {
    await unlink(dest).catch(() => {});
    if (data.file.truncated) return reply.code(413).send({ error: 'too_large' });
    throw err;
  }
  return reply.code(201).send({ id: dest.split('/').pop() });
});

Avoid data.toBuffer() for anything large; the plugin’s own README tells you to be careful with it.

Presigned URLs: keep large uploads off your API #

For files in the hundreds of megabytes or more, the best upload is one your API never touches. The API authorizes the upload and returns a short-lived URL, and the client sends the file straight to storage:

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { randomUUID } from 'node:crypto';

const s3 = new S3Client({});

app.post('/v1/uploads/presign', requireAuth, async (req, res) => {
  const key = `uploads/${req.user.accountId}/${randomUUID()}`;
  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({ Bucket: process.env.UPLOAD_BUCKET, Key: key }),
    { expiresIn: 300 },
  );
  res.json({ key, url });
});

A single S3 PUT tops out at 5 GB; beyond that, use S3 multipart upload. Validate the object after upload, not before, and validate any metadata the client sends with a schema, as in validating request bodies with Zod.

Patch your multipart parser: 2026 advisories #

Upload parsers have been a steady source of denial-of-service fixes this year. Check yours with npm ls multer @fastify/multipart @fastify/busboy multiparty busboy.

Multer 2.3.0 also added an opt-in limits.fieldArrayIndexLimit, which the CVE-2026-82333 advisory recommends setting.

Timeouts for long uploads #

Node’s HTTP server has had a requestTimeout of 300,000 ms (five minutes) by default since Node 18. A 2 GB upload on a slow connection can exceed that, and your proxy or platform likely has its own limit too. Raise it only on the instance that serves uploads, or use presigned URLs and avoid the question.

For a one-person team, the afternoon version is: upgrade the parser, set explicit limits, and switch from memory to disk storage. A ten-person team should move large uploads to presigned URLs, add a cleanup job for orphaned temp files and alert on disk usage. If memory still climbs afterwards, heap snapshots will show what is holding it.

If you only do one thing #

Search your code for memoryStorage, toBuffer and Buffer.concat on upload routes, and replace each with a stream to disk or storage.

FAQ #

How do I upload large files in Node.js without running out of memory? #

Stream the request body to disk or object storage with stream.pipeline instead of buffering it, and set a fileSize limit. For very large files, have clients upload directly to storage with presigned URLs.

What is the maximum file upload size in Express? #

Express sets no multipart limit itself. The limit comes from your multipart library’s fileSize option, your proxy and Node’s requestTimeout.

Is multer safe to use in 2026? #

Yes, on a patched version. Upgrade to multer 2.4.0 or later, use diskStorage rather than memoryStorage for large files, and set limits.

Doing this with ApiTraffic #

If you would rather not instrument upload routes yourself:

• ApiTraffic records requests under the route pattern with p50, p95 and p99 over time, so a slow or failing upload endpoint shows up separately from the rest of your API. Search failures with url_path == "/v1/uploads" AND status_code == 413.
• Outbound request logging and APM spans show the time spent sending the file on to storage.
• A WAF rate limit on the upload path, or on an API key header, returns 429 with Retry-After before abusive clients fill your disk. Monitor mode shows what it would block first.

Start free and see every request your API handles: apitraffic.io

Sources: Express, September 2026 Security Releases (Sep 30, 2026); Express, August 2026 Security Releases (Aug 31, 2026); OSV, CVE-2026-18549 @fastify/multipart (Aug 15, 2026); GitLab Advisory Database, CVE-2026-19484 @fastify/busboy (Oct 2, 2026); @fastify/multipart README

In Node, you can collect all of that for every fetch() call without wrapping any code, using diagnostics_channel. This post shows how, what to measure, and what to do with the numbers. If you want one log line per outbound call rather than per-vendor numbers, start with how to log outbound HTTP requests.

Why your own measurements beat the vendor’s status page #

A vendor’s status page reports their aggregate health, usually with a delay. What you need is their health as seen from your servers: your region, your network path, your account’s rate limits, your request shapes. A provider can be fine overall and slow for you because of a regional problem or because you are being throttled.

Your own numbers also answer the internal question: how much of your endpoint’s p95 is actually theirs?

What to measure for each third-party API call #

Group by host and a normalized operation (POST api.vendor.example /v1/charges/:id), not the raw path. Treat 429 separately from other 4xx: it means you are being throttled, and it trends before an outage-shaped problem does.

How to measure fetch latency in Node.js with diagnostics_channel #

Node’s built-in fetch() is powered by undici, which publishes events on diagnostics_channel. The same request object appears on every event for a given request, so a WeakMap can carry timings between them:

// outbound-metrics.js
import diagnostics_channel from 'node:diagnostics_channel';
import client from 'prom-client';

const duration = new client.Histogram({
  name: 'outbound_request_duration_seconds',
  help: 'Outbound HTTP request duration',
  labelNames: ['host', 'op', 'outcome'],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10],
});
const ttfb = new client.Histogram({
  name: 'outbound_request_ttfb_seconds',
  help: 'Outbound time to response headers',
  labelNames: ['host', 'op'],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10],
});

const inflight = new WeakMap();

const normalize = (path) =>
  path.split('?')[0]
    .replace(/\/[0-9a-f]{8}-[0-9a-f-]{27,}/gi, '/:uuid')
    .replace(/\/\d+(?=\/|$)/g, '/:id');

function outcomeOf(status, error) {
  if (error) return error.name === 'TimeoutError' || /TIMEOUT/.test(error.code ?? '') ? 'timeout' : 'network';
  if (status === 429) return '429';
  return `${Math.floor(status / 100)}xx`;
}

diagnostics_channel.subscribe('undici:request:create', ({ request }) => {
  inflight.set(request, {
    t0: performance.now(),
    host: new URL(request.origin).host,
    op: `${request.method} ${normalize(request.path)}`,
  });
});

diagnostics_channel.subscribe('undici:request:headers', ({ request, response }) => {
  const s = inflight.get(request);
  if (!s) return;
  s.status = response.statusCode;
  ttfb.observe({ host: s.host, op: s.op }, (performance.now() - s.t0) / 1000);
});

function finish(request, error) {
  const s = inflight.get(request);
  if (!s) return;
  inflight.delete(request);
  duration.observe(
    { host: s.host, op: s.op, outcome: outcomeOf(s.status, error) },
    (performance.now() - s.t0) / 1000,
  );
}

diagnostics_channel.subscribe('undici:request:trailers', ({ request }) => finish(request));
diagnostics_channel.subscribe('undici:request:error', ({ request, error }) => finish(request, error));

Import this file once at startup, before your app makes outbound calls. request:trailers fires when the response body has been fully received, so “total” includes body time; a body you never read never completes, which is itself worth finding.

Keep label cardinality low. If your normalizer misses an ID format, every customer becomes a new series. Check the op values in a staging run first.

What about axios and node:http clients? #

Libraries built on node:http (axios’s Node adapter, got, many SDKs) do not go through undici. Node publishes its own http.client.* channels for them, and the same WeakMap pattern works with the request object they carry. The subscriber code for those channels is in how to log axios and http.request calls.

Reading the numbers #

• Dependency p95 versus endpoint p95. If checkout’s p95 is 1.8 s and the payment call’s p95 is 1.4 s, there is little to gain by optimizing your own code first.
• TTFB versus total. High TTFB is the vendor thinking. A big gap between TTFB and total is large or slow bodies.
• Error rate per dependency, split by class. Rising 429 means slow down or ask for a higher limit. Rising timeout or network with flat 5xx often points at the network path or connection reuse rather than the vendor’s application.
• Attempts per logical call. If retries average 1.3 attempts, the vendor is failing 30% of first tries even if your success rate looks perfect.

What to do with third-party API latency data #

1. Set timeouts from it. A little above the dependency’s p99, inside your endpoint’s budget. The fetch timeout guide has the code.
2. Add a circuit breaker for dependencies that fail in bursts, so you fail fast instead of piling up waiting requests. opossum is a widely used Node library for this.
3. Cache or move work off the request path. Exchange rates, feature flags and address lookups rarely need a live call per request. Webhook-driven or queued work removes the vendor from user-facing latency entirely.
4. Escalate with evidence. Send the vendor timestamps, the operation, your p95 and error-rate graph, and the request IDs from their response headers. This gets a far better response than “your API feels slow”.
5. Alert on the symptom, not the vendor. Per the alerting guide, page when your users are affected; use per-vendor numbers to diagnose.

A solo developer can start with the code above and a weekly look at per-host p95. A team of ten should put dependency panels next to endpoint latency on the main dashboard and review vendors quarterly with these numbers in hand. When you need the request and response bodies too, see logging outbound HTTP calls.

If you only do one thing #

• Load outbound-metrics.js in every service.
• Chart p95 and error rate per host, with 429 split out.
• Set every outbound timeout from that p99.
• Keep a dated list of incidents per vendor for your next renewal conversation.

FAQ #

How do I measure the latency of fetch requests in Node.js? #

Subscribe to undici’s diagnostics_channel events (undici:request:create, headers, trailers, error), which Node’s built-in fetch() publishes. Record the time between them per request and export it as a histogram.

Does diagnostics_channel work with axios? #

Not through the undici channels, because axios uses node:http in Node. Use Node’s experimental http.client.* channels or an axios interceptor instead.

What is a good timeout for a third-party API? #

Slightly above that dependency’s observed p99 from your own servers, and always inside the deadline your endpoint promises its callers.

Doing this with ApiTraffic #

If you would rather not build and run this instrumentation yourself, ApiTraffic logs outbound requests alongside inbound ones, and its APM records each outbound fetch as a span attributed to the request that started it.

• The request waterfall shows each vendor call on the timeline of the user request that triggered it.
• An endpoint’s Time Breakdown shows how much of its time goes to outbound calls versus queries and your own code.
• span() adds a custom timing around SDK calls you want to see separately.
• Span reports rank outbound time by vendor and failed calls by type.

Start free and see every request your API handles: apitraffic.io

Sources: undici Diagnostics Channel docs; Node.js docs: diagnostics_channel (built-in HTTP channels); prom-client on GitHub; opossum on GitHub

The direct answer: if your API fetches any URL a user can influence (webhooks, link previews, image imports, “import from URL”, OAuth callbacks), allow only http/https, resolve the hostname and refuse private, loopback and link-local addresses at connect time, re-check every redirect, and set a timeout. String checks on the URL are not enough. Below is a tested safeFetch() for Node’s fetch and undici that does all of this.

Where SSRF hides in a Node API #

OWASP lists SSRF as API7 in its 2023 API Security Top 10. In a typical Express or Fastify API it turns up in:

• Webhook endpoints that customers configure.
• Link previews and unfurling, which fetch a page to read its title and image.
• “Import from URL” for avatars, CSVs or documents.
• PDF or screenshot rendering of user-provided HTML or URLs.
• HTTP clients with a baseURL where part of the path comes from the user.

The last one is subtle. In axios before 1.8.2, an absolute URL passed to an instance with a baseURL was sent to that absolute URL rather than appended to the base (CVE-2025-27152, published March 2025). Version 1.8.2 added the allowAbsoluteUrls option to the HTTP adapter, and it still defaults to true; set it to false on instances that should only ever talk to one host.

Why URL string checks fail #

Blocking localhost and 127.0.0.1 in the URL string does not work. The WHATWG URL parser normalizes many spellings of the same address:

http://2130706433/          -> hostname 127.0.0.1
http://0x7f.1/              -> hostname 127.0.0.1
http://127.1/               -> hostname 127.0.0.1
http://017700000001/        -> hostname 127.0.0.1
http://[::ffff:127.0.0.1]/  -> hostname [::ffff:7f00:1]

Hostnames are worse. internal.example.com can resolve to 10.0.0.5, and with DNS rebinding a name can resolve to a public IP when you check it and a private one when you connect. The ip npm package’s isPublic() was itself bypassable with unusual address formats (CVE-2024-29415, May 2024). The reliable place to check is the IP address the socket actually connects to.

How to prevent SSRF in Node.js with undici #

Node’s global fetch is built on undici, and undici’s Agent accepts a custom DNS lookup for its connections. That hook sees the resolved address right before connecting, which closes the rebinding gap.

// safe-fetch.js
import { Agent, fetch } from 'undici';
import { BlockList, isIP } from 'node:net';
import dns from 'node:dns';

const blocked = new BlockList();
for (const [net, prefix] of [
  ['0.0.0.0', 8], ['10.0.0.0', 8], ['100.64.0.0', 10], ['127.0.0.0', 8],
  ['169.254.0.0', 16], ['172.16.0.0', 12], ['192.168.0.0', 16],
]) blocked.addSubnet(net, prefix, 'ipv4');
for (const [net, prefix] of [['::', 128], ['::1', 128], ['fc00::', 7], ['fe80::', 10]])
  blocked.addSubnet(net, prefix, 'ipv6');

function isBlocked(address) {
  const v = isIP(address);
  if (v === 6 && /^::ffff:/i.test(address) && isIP(address.slice(7)) === 4) {
    return blocked.check(address.slice(7), 'ipv4'); // IPv4-mapped IPv6
  }
  return blocked.check(address, v === 6 ? 'ipv6' : 'ipv4');
}

function safeLookup(hostname, options, callback) {
  dns.lookup(hostname, { ...options, all: true }, (err, addresses) => {
    if (err) return callback(err);
    const bad = addresses.find((a) => isBlocked(a.address));
    if (bad) return callback(Object.assign(new Error(`blocked address ${bad.address}`), { code: 'ESSRF' }));
    if (options.all) return callback(null, addresses);
    callback(null, addresses[0].address, addresses[0].family);
  });
}

const agent = new Agent({ connect: { lookup: safeLookup } });

function checkUrl(raw) {
  const url = new URL(raw);
  if (url.protocol !== 'https:' && url.protocol !== 'http:') throw new Error('protocol not allowed');
  if (url.username || url.password) throw new Error('credentials in URL not allowed');
  const host = url.hostname.replace(/^\[|\]$/g, '');
  if (isIP(host) && isBlocked(host)) throw new Error(`blocked address ${host}`); // literal IPs skip DNS
  return url;
}

export async function safeFetch(raw, { maxRedirects = 3, timeoutMs = 5000, ...init } = {}) {
  let url = checkUrl(raw);
  for (let hop = 0; ; hop++) {
    const res = await fetch(url, {
      ...init,
      dispatcher: agent,
      redirect: 'manual',
      signal: AbortSignal.timeout(timeoutMs),
    });
    const location = res.headers.get('location');
    if (res.status < 300 || res.status > 399 || !location) return res;
    if (hop >= maxRedirects) throw new Error('too many redirects');
    await res.body?.cancel();
    url = checkUrl(new URL(location, url)); // re-check every hop
  }
}

Run against a local server, every bypass from the previous section is refused:

http://127.0.0.1:3021/            BLOCKED blocked address 127.0.0.1
http://2130706433:3021/           BLOCKED blocked address 127.0.0.1
http://[::ffff:127.0.0.1]:3021/   BLOCKED blocked address ::ffff:7f00:1
http://localhost:3021/            BLOCKED blocked address 127.0.0.1
file:///etc/passwd                BLOCKED protocol not allowed
http://169.254.169.254/latest/... BLOCKED blocked address 169.254.169.254

Why each piece is there #

1. The literal IP check exists because a URL with an IP address never triggers a DNS lookup, so safeLookup alone would let http://127.0.0.1/ through.
2. redirect: 'manual' stops undici from following a 302 to http://169.254.169.254/ on its own. In Node, a manual redirect returns the real 3xx response with its Location header.
3. The timeout stops a slow or hanging target from tying up your server; see fetch timeouts in Node.js for the details of AbortSignal.timeout().
4. Import fetch and Agent from the same undici package. Passing an npm undici Agent to the global fetch can work, but the bundled and npm versions can drift apart.

Extend the blocklist for your network: add your VPC ranges, and any IPv6 ranges you route internally.

SSRF in Node.js on cloud platforms #

The prize in most SSRF attacks is cloud credentials. On AWS EC2 the instance metadata service lives at 169.254.169.254; on ECS, task credentials come from 169.254.170.2. Both fall inside 169.254.0.0/16, which the code above blocks. Two more layers are worth an afternoon:

• On EC2, require IMDSv2, which needs a PUT request to get a session token first and blocks the simple GET-based SSRF.
• Run URL-fetching work (webhooks, previews) in a separate worker or service with no cloud credentials and outbound-only network access.

On PaaS hosts like Render, Railway, Fly.io or Heroku the metadata risk differs, but internal services on the private network are still reachable. The same rules apply.

SSRF prevention checklist #

• [ ] Every user-influenced URL goes through one safeFetch(); no raw fetch(req.body.url).
• [ ] Only http: and https:; no credentials in URLs.
• [ ] Resolved IPs checked at connect time, including literal IPs and IPv4-mapped IPv6.
• [ ] Redirects followed manually and re-checked, with a hop limit.
• [ ] Timeouts and a maximum response size.
• [ ] axios instances with a baseURL set allowAbsoluteUrls: false.
• [ ] IMDSv2 required on EC2.
• [ ] Outbound requests logged with the destination host.

A solo developer can grep for fetch(, axios and got( calls that take request data, route them through safeFetch(), and be done in an afternoon. A team of ten should also add a lint rule or code-review check that bans fetching user-provided URLs any other way, and move webhook delivery to a worker with its own network policy.

FAQ #

What is SSRF in Node.js? #

Server-side request forgery is when an attacker makes your server send a request to a destination they choose, usually an internal service or cloud metadata endpoint. In Node.js it usually comes from passing user input to fetch, axios or another HTTP client.

Does Node’s fetch protect against SSRF? #

No. Node’s built-in fetch follows redirects and connects to any address, including 127.0.0.1 and 169.254.169.254. You need to add the checks yourself, as above, or use a library that does.

Is blocking private IP ranges enough to stop SSRF? #

It stops the most damaging cases, if you check the resolved IP at connect time and re-check redirects. For high-risk features, also isolate URL fetching in a service with no credentials and restricted network access.

Doing this with ApiTraffic #

If you would rather not stitch together outbound visibility yourself, ApiTraffic logs outbound requests alongside inbound ones, and APM attributes each outbound fetch span to the request that started it. A webhook or preview call to an unexpected host shows up on the request’s waterfall, tied to the user input that caused it (more on logging outbound HTTP requests).

On the inbound side, firewall access lists can deny by IP, CIDR, country, ASN or path, with monitor mode to check a rule before it blocks.

Start free and see every outbound call your API makes: apitraffic.io

Sources: OWASP API Security Top 10 2023: API7 Server Side Request Forgery; undici Agent and connect options; Node.js net.BlockList; NVD: CVE-2025-27152 (axios), March 2025; NVD: CVE-2024-29415 (ip), May 2024.

The short answer on cursor vs offset pagination: offset pagination (LIMIT 50 OFFSET 200000) gets slower the deeper you go because the database still reads every skipped row, and it returns duplicates or skips when rows are inserted mid-scroll. Cursor pagination (also called keyset pagination) asks for “the next 50 after this row”, stays fast at any depth with the right index, and is stable under inserts. Use cursors for anything large, public or growing, and keep offset for small admin tables where jumping to page 7 matters.

Cursor vs offset pagination compared #

Why offset pagination gets slow on large tables #

PostgreSQL’s documentation says it plainly: rows skipped by an OFFSET clause still have to be computed inside the server, so a large offset can be inefficient. OFFSET 200000 means reading and discarding 200,000 rows to return 50. MySQL behaves the same way.

There is also a correctness problem. If a new order is inserted at the top while a client is on page 3, every row shifts down by one, and page 4 repeats the last row of page 3. Deletes cause skipped rows. For a feed, an export or a sync job, that is a bug.

Finally, offset endpoints often return a total count, and SELECT count(*) on a big filtered table is its own slow query on every request.

How to implement cursor pagination in Node.js #

Step 1: pick a stable, unique sort order #

Cursors need a sort key that never ties. created_at alone can tie, so pair it with the primary key: ORDER BY created_at DESC, id DESC.

Step 2: add a matching index #

CREATE INDEX CONCURRENTLY orders_account_created_id
  ON orders (account_id, created_at DESC, id DESC);

Step 3: encode the cursor and query with a row comparison #

This uses Fastify v5 and pg. The cursor is opaque to clients: base64url-encoded JSON of the last row’s sort keys.

import Fastify from 'fastify';
import pg from 'pg';

const app = Fastify({ logger: true });
const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });

const encode = (row) =>
  Buffer.from(JSON.stringify({ t: row.cursor_ts, id: row.id })).toString('base64url');

function decode(cursor) {
  try {
    const { t, id } = JSON.parse(Buffer.from(cursor, 'base64url').toString());
    if (typeof id !== 'string' || Number.isNaN(Date.parse(t))) return null;
    return { t, id };
  } catch {
    return null;
  }
}

app.get('/v1/orders', async (request, reply) => {
  const limit = Math.min(Number(request.query.limit) || 50, 100);
  const after = request.query.after ? decode(request.query.after) : null;
  if (request.query.after && !after) {
    return reply.code(400).send({ error: 'invalid_cursor' });
  }

  const params = [request.user.accountId, limit + 1];
  let where = 'account_id = $1';
  if (after) {
    params.push(after.t, after.id);
    where += ' AND (created_at, id) < ($3::timestamptz, $4)';
  }

  const { rows } = await pool.query(
    `SELECT id, created_at, created_at::text AS cursor_ts, total_cents, status
       FROM orders
      WHERE ${where}
      ORDER BY created_at DESC, id DESC
      LIMIT $2`,
    params,
  );

  const hasMore = rows.length > limit;
  const page = hasMore ? rows.slice(0, limit) : rows;
  return {
    data: page.map(({ cursor_ts, ...order }) => order),
    next_cursor: hasMore ? encode(page.at(-1)) : null,
  };
});

await app.listen({ port: 3000 });

The (created_at, id) < (...) row comparison is PostgreSQL syntax that matches the composite index, so the database seeks straight to the right spot. Fetching limit + 1 rows tells you whether there is a next page without a count query. The account_id condition matters as much as the cursor: a cursor must never let a caller page into another tenant’s rows, the same lesson as BOLA and IDOR.

The cursor uses created_at::text rather than the value pg returns, because pg converts timestamps to JavaScript Date objects, which keep only milliseconds. PostgreSQL stores microseconds, so a cursor built from a Date can skip rows.

Step 4: return the cursor in a predictable place #

Put next_cursor in the body, or use a Link header with rel="next" as described in RFC 8288. Clients loop until next_cursor is null.

When offset pagination is still fine #

• Admin tables with a few thousand rows and a page picker.
• Search results where nobody goes past page 10. Cap the offset so they cannot.
• Reports where the dataset is fixed for the duration of the read.

If you keep offset, cap it: if (offset > 10_000) return reply.code(400).

How pagination design affects scrapers #

Offset endpoints are a gift to scrapers. Pages are independent, so a script can fetch pages 1 through 5,000 in parallel, and the deep ones are your most expensive queries. Cursor endpoints force a scraper to walk the chain one request at a time, which turns a parallel burst into a sequential crawl you can rate limit.

Protections that work with either design:

• Cap limit server-side. A client asking for limit=100000 should get 100.
• Rate limit list endpoints per API key, not only per IP. Rate limiting algorithms compared covers the options.
• Watch for one key walking every page of a list endpoint, or requesting deep offsets in parallel.
• Keep expensive filters behind indexes. List endpoints are often where N+1 queries hide as well.

For a one-person team, the afternoon version is adding cursor pagination to the one or two list endpoints that are actually large, and capping limit and offset everywhere else. A ten-person team should make cursor pagination the default in a shared helper, document the cursor contract, and add a deprecation path for offset parameters on public endpoints.

If you only do one thing #

Cap limit and offset on every list endpoint today. It costs ten minutes and removes the worst-case query from your API.

FAQ #

Is cursor pagination faster than offset pagination? #

At depth, yes. With an index matching the sort order, a cursor query reads only the rows it returns, while OFFSET reads and discards every skipped row. On the first page they perform about the same.

How do I jump to a specific page with cursor pagination? #

You cannot jump directly. If users need that, offer filters such as a date range that narrow the set, or keep offset pagination for small datasets only.

Should pagination cursors be encrypted or signed? #

They should be opaque so clients do not build them by hand. Encoding is enough if the server validates the decoded values and always applies the caller’s tenant filter. Sign them with an HMAC if you do not want clients editing them.

Doing this with ApiTraffic #

If you would rather not build scraper detection and per-key limits yourself:

• ApiTraffic records every request under its route pattern with p50, p95 and p99 per endpoint, so a list endpoint slowing as tables grow shows up before customers report it. The Time Breakdown and request waterfall show whether that time is the query.
• Top Operations ranks queries by calls per request and flags N+1 patterns with the code location that issued them.
• A WAF rate limit keyed on a header such as your API key returns 429 with Retry-After, and behavioral protection scores callers on distinct object IDs per endpoint and distinct endpoints. Monitor mode shows what would be blocked first.

Start free and see every request your API handles: apitraffic.io

Sources: PostgreSQL documentation, LIMIT and OFFSET; PostgreSQL documentation, row constructor comparison; RFC 8288, Web Linking

For a Node API run by a team of 1 to 20, that means four or five alerts, not forty. Here is which ones, how to keep them quiet, and working instrumentation and alert rules.

What should you alert on for an API? #

The Google SRE book’s guidance is to alert on symptoms rather than causes. For an HTTP API the symptom list is short:

Replace the example numbers with values from your own baseline. The structure matters more than the thresholds: ratios instead of counts, a minimum volume, and a duration.

API alerting best practices: what not to page on #

• CPU or memory percentage. High CPU with normal latency is a cost question, not an emergency.
• A single 500. One error is a log line. Many errors are a rate.
• Raw 4xx counts. Clients send bad requests; a 4xx spike is often a scanner or a broken client release. Ticket it, or route it to a security review.
• One process restart. Page on restarts only if they cause errors users see, which the error-rate alert already catches.
• A dependency’s latency alone. If the vendor’s slowness hurts your users, your own latency alert fires.

Causes like these belong on a dashboard you open when a symptom alert fires. Event loop lag is a good example: a vital diagnostic, a poor page.

How to reduce alert noise #

Use rates and minimum volumes #

Five errors out of ten requests at 4am is 50%, and it may be one confused client. Require a minimum request count before an error-rate alert can fire.

Add a duration #

Use a for window (5 to 10 minutes) so a spike during a deploy does not page. Short, real outages are still caught by the external “down” check.

Alert on the routes that matter #

Exclude /health, webhooks you ignore and admin routes from latency alerts, or alert per group of routes. One slow export endpoint should not page as “API slow”.

Consider burn rate once you have an SLO #

If you define an availability target, alerting on how fast you burn the error budget (fast burn pages, slow burn tickets) replaces fixed thresholds well. The SLOs and error budgets post covers how to set one up without much ceremony.

How to instrument a Node API for alerting #

A request histogram labeled by route and status class is enough for every alert above. With prom-client and Express:

import express from 'express';
import client from 'prom-client';

const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status_class'],
  buckets: [0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
});

const app = express();

app.use((req, res, next) => {
  const end = httpDuration.startTimer();
  res.on('finish', () => {
    end({
      method: req.method,
      route: req.route?.path ?? 'unmatched', // the pattern, not the raw URL
      status_class: `${Math.floor(res.statusCode / 100)}xx`,
    });
  });
  next();
});

app.get('/metrics', async (req, res) => {
  res.set('content-type', client.register.contentType);
  res.send(await client.register.metrics());
});

Label by route pattern, never by raw URL, or every user ID becomes a new time series. Note that req.route holds only the last matched path segment in nested routers; prefix it with req.baseUrl if you mount routers.

Then the alert rules, in Prometheus format:

groups:
  - name: api
    rules:
      - alert: ApiHighErrorRate
        expr: |
          sum(rate(http_request_duration_seconds_count{status_class="5xx"}[5m]))
            / sum(rate(http_request_duration_seconds_count[5m])) > 0.02
          and sum(increase(http_request_duration_seconds_count[5m])) > 50
        for: 5m
        labels: { severity: page }
        annotations:
          summary: "5xx rate above 2% for 5m"
          runbook: "https://wiki.example.com/runbooks/api-errors"
      - alert: ApiSlowCheckout
        expr: |
          histogram_quantile(0.95, sum by (le) (
            rate(http_request_duration_seconds_bucket{route=~"/checkout.*"}[10m]))) > 1
        for: 10m
        labels: { severity: page }

The same logic works in Grafana Cloud, Datadog, New Relic or your platform’s built-in alerting; the names change, the structure does not. For the “down” alert, use an external uptime check against a real endpoint, not just /health (see health checks that tell the truth).

Who gets paged on a small team? #

• Solo developer: pages go to your phone, so be strict. Three alerts: error rate, latency on your most important route, and down. Everything else goes to email.
• Team of ~10: a weekly on-call rotation, one channel for pages and another for tickets, and a 15-minute review every week: for each page, was it real, and did someone need to act? Delete or downgrade any alert that fails twice.

Every page should include what fired, the current value, a link to the dashboard and a runbook link with the first three things to check. A runbook can be five lines.

If you only do one thing #

• Delete or downgrade every page that is a cause (CPU, memory, single errors) rather than a symptom.
• Make sure three pages exist: 5xx rate, p95 on key routes, and external “down”.
• Give each a minimum volume and a duration.
• Review pages weekly and delete the ones nobody acted on.

FAQ #

What is the difference between a symptom alert and a cause alert? #

A symptom alert fires on what users experience, such as errors or slow responses. A cause alert fires on something that might lead to that, such as high CPU. Page on symptoms and use causes for diagnosis.

Should I alert on p95 or p99 latency? #

For paging, p95 over several minutes is steadier and harder to trip with a handful of slow requests. Watch p99 on dashboards and for tickets, especially on low-traffic routes.

How many alerts should a small API team have? #

Few enough that every page gets acted on. For a single Node API, three to five paging alerts covering errors, latency, availability and capacity is a reasonable start.

Doing this with ApiTraffic #

Alerts tell you something is wrong; the next ten minutes are spent working out what. If you would rather not build that investigation tooling yourself, ApiTraffic records every request and response, with route patterns and tags.

• Search status_code >= 500 for the alert’s time window to see the failing requests themselves, with bodies and headers (credentials masked by default).
• tag:userId == "1042" AND status_code >= 400 shows whether one customer is behind the spike (tags and dimensions).
• Endpoint details show p50, p95 and p99 per endpoint, and Regressions flags endpoints whose p95 rose after a recorded deploy event.

Start free and see every request your API handles: apitraffic.io

Sources: Google SRE book, “Monitoring Distributed Systems”; Prometheus, “Alerting” best practices; Prometheus alerting rules; prom-client

The direct answer: generate keys with at least 256 bits of randomness, give them a recognizable prefix so leaks are easy to spot, show the key once, store only a SHA-256 hash, look keys up by a separate public ID, attach scopes and an optional expiry, allow two active keys per account so rotation has no downtime, and never log the key itself. Here is how to do each in Node.

What a good API key looks like #

The format we will build:

atk_live_5d0de6ca2ad7_fYF7sqANvWIOd9-JHGS05Snc89N4M13XeBFhunWSSjU_1aqdw5g
└─┬┘ └┬─┘ └────┬────┘ └───────────────────┬───────────────────┘ └──┬──┘
prefix env   key ID          secret (32 random bytes)          checksum

• Prefix (atk_): makes your keys recognizable to people, grep and secret scanners. GitHub moved to prefixed tokens like ghp_ in 2021 for exactly this reason, and Stripe’s sk_live_ / sk_test_ keys are the familiar example.
• Environment (live / test): a test key in production code is a bug you can catch at a glance.
• Key ID: a short, non-secret identifier used to find the row and to show the key in your dashboard.
• Secret: 32 bytes from crypto.randomBytes, base64url encoded. This is the only part that needs protecting.
• Checksum: a CRC32 of the rest, so you can reject typos and truncated keys without a database query. GitHub’s token format includes a checksum for the same reason.

How to generate API keys in Node.js #

// api-keys.js
import { randomBytes, createHash, timingSafeEqual } from 'node:crypto';
import { crc32 } from 'node:zlib'; // Node 20.15+ / 22.2+

const PREFIX = 'atk';

export function generateApiKey(env = 'live') {
  const id = randomBytes(6).toString('hex');
  const secret = randomBytes(32).toString('base64url');
  const body = `${PREFIX}_${env}_${id}_${secret}`;
  const checksum = crc32(body).toString(36).padStart(7, '0');
  return {
    key: `${body}_${checksum}`, // show once, never store
    id,
    hash: createHash('sha256').update(secret).digest('hex'),
  };
}

export function parseApiKey(key) {
  const m = /^atk_(live|test)_([0-9a-f]{12})_([A-Za-z0-9_-]{43})_([0-9a-z]{7})$/.exec(key ?? '');
  if (!m) return null;
  const [, env, id, secret, checksum] = m;
  const expected = crc32(`${PREFIX}_${env}_${id}_${secret}`).toString(36).padStart(7, '0');
  return checksum === expected ? { env, id, secret } : null;
}

export function secretMatches(secret, storedHashHex) {
  const a = createHash('sha256').update(secret).digest();
  const b = Buffer.from(storedHashHex, 'hex');
  return a.length === b.length && timingSafeEqual(a, b);
}

Why hash API keys with SHA-256, not bcrypt #

Passwords need slow hashes (bcrypt, scrypt, Argon2) because people choose guessable passwords. A 256-bit random secret cannot be guessed, so a fast hash such as SHA-256 is enough: a stolen hash is useless without the secret. A slow hash would add tens of milliseconds to every authenticated request for no security gain.

Storing only the hash means a database dump, a backup in the wrong bucket or a support engineer’s SQL query does not expose working keys. It also means you cannot show the key again; users must copy it when it is created.

A minimal table:

CREATE TABLE api_keys (
  id            text PRIMARY KEY,          -- the public key ID
  account_id    bigint NOT NULL REFERENCES accounts(id),
  secret_hash   text NOT NULL,             -- sha256 hex
  env           text NOT NULL,             -- 'live' | 'test'
  scopes        text[] NOT NULL DEFAULT '{}',
  name          text,                      -- "CI deploy key"
  created_at    timestamptz NOT NULL DEFAULT now(),
  expires_at    timestamptz,
  last_used_at  timestamptz,
  revoked_at    timestamptz
);

How to verify API keys in Express #

import { parseApiKey, secretMatches } from './api-keys.js';

export async function apiKeyAuth(req, res, next) {
  const parsed = parseApiKey(req.get('x-api-key'));
  if (!parsed) return res.status(401).json({ error: 'invalid_api_key' });

  const row = await db.apiKeys.findById(parsed.id);
  const valid =
    row &&
    !row.revoked_at &&
    (!row.expires_at || row.expires_at > new Date()) &&
    row.env === parsed.env &&
    secretMatches(parsed.secret, row.secret_hash);

  if (!valid) return res.status(401).json({ error: 'invalid_api_key' });

  req.apiKey = { id: row.id, accountId: row.account_id, scopes: new Set(row.scopes) };
  db.apiKeys.touchLastUsed(row.id).catch(() => {}); // throttle this in production
  next();
}

export const requireScope = (scope) => (req, res, next) =>
  req.apiKey?.scopes.has(scope) ? next() : res.status(403).json({ error: 'insufficient_scope' });

// usage
app.get('/v1/orders', apiKeyAuth, requireScope('orders:read'), listOrders);

Return the same 401 for “malformed”, “not found”, “revoked” and “wrong secret”, so callers cannot probe which keys exist. Use 403 only when the key is valid but lacks the scope.

Scopes: least privilege for API keys #

Most API keys end up in CI systems, Zapier, cron jobs and contractors’ laptops. Scopes limit the damage when one leaks. Start simple: resource:read and resource:write per resource, plus a “full access” option for users who insist. Default new keys to read-only and make write access a deliberate choice.

How to rotate API keys without downtime #

Rotation fails when an account can have only one key. Allow at least two active keys, then:

1. Create the new key and give it to the client.
2. The client deploys it.
3. Watch last_used_at on the old key until it stops changing.
4. Revoke the old key.

Optional expiry (90 days, say) forces this habit for high-risk keys. Showing last_used_at and the key name in your dashboard turns “which of these keys is safe to delete?” into an easy question.

When a key leaks, the order is: revoke immediately, issue a replacement, then check what the leaked key did. That last step needs request logs tied to the key ID, which is a good reason to keep them.

Do not let API keys leak through your own logs #

The most common leak is not GitHub; it is your logs. Mask X-Api-Key and Authorization in request logging (see how to redact sensitive data in logs), log the key ID instead of the key, and never put keys in URLs, where proxies and analytics record them. Pair key auth with rate limiting per API key so a leaked key cannot be used at unlimited speed.

API key security checklist #

• [ ] 32 random bytes per key; prefix, environment and key ID in the format.
• [ ] Shown once; only the SHA-256 hash stored.
• [ ] Constant-time comparison; one generic 401.
• [ ] Scopes, with read-only as the default.
• [ ] Two active keys per account; last_used_at visible.
• [ ] Keys masked in all logs; key ID logged instead.
• [ ] Rate limits per key.

A solo developer can ship the first four in an afternoon, before the first customer gets a key. A team of ten should also write the leak runbook (revoke, replace, review usage) and consider registering the prefix with GitHub’s secret scanning partner program so leaked keys are reported to you.

FAQ #

Should I hash API keys with bcrypt? #

No, not if keys are long random values. bcrypt protects low-entropy passwords from guessing; a 256-bit random secret cannot be guessed, so SHA-256 is enough and keeps authentication fast.

Should API keys expire? #

Make expiry optional and encourage it for high-risk keys. Forced short expiry on every key mostly produces outages when someone forgets to rotate; supporting two active keys and showing last use gets you most of the benefit.

Where should clients send the API key? #

In a header such as Authorization: Bearer ... or X-Api-Key, never in the query string. URLs end up in access logs, browser history and analytics tools.

Doing this with ApiTraffic #

If you would rather not build masking, per-key limits and usage tracking yourself, ApiTraffic covers the parts around your key auth. Authorization, Cookie and X-Api-Key are masked by default in the request log. Firewall rate limits can be keyed on a header, so each API key gets its own limit, with 429 and Retry-After for callers over it.

Tag each request with the account, apitraffic.tag('accountId', String(req.apiKey.accountId)), and the “did anyone use the leaked key?” question becomes a search: tag:accountId == "1042" AND status_code >= 400, or the same tag over the last day.

Start free and see every request each API key makes: apitraffic.io

Sources: GitHub: Behind GitHub’s new authentication token formats (April 2021); Node.js crypto: randomBytes and timingSafeEqual; Node.js zlib.crc32; OWASP API Security Top 10 2023: API2 Broken Authentication; GitHub secret scanning partner program.

The short answer: define a Zod schema per route for the body, query and params; in Express, run safeParse in a small middleware and return a 400 with the issue paths; in Fastify, plug Zod into the schema system with fastify-type-provider-zod so validation, types and response filtering come from one definition. This post uses Zod 4 (stable since July 2025), Express 5 and Fastify 5.

Why Zod validation for request bodies #

Zod gives you three things from one schema: a runtime check, a TypeScript type (z.infer), and normalized data (coerced numbers, defaults, stripped unknown keys). Zod 4 also moved string formats to the top level (z.email(), z.uuid(), z.url()) and unified error customization under a single error parameter, and the maintainers’ benchmarks show object parsing several times faster than Zod 3.

Here are schemas for an order API that the Express and Fastify examples both use:

// schemas.ts
import { z } from 'zod';

export const CreateOrder = z.strictObject({
  customerId: z.uuid(),
  email: z.email(),
  items: z
    .array(z.object({ sku: z.string().min(1).max(64), qty: z.int().min(1).max(100) }))
    .min(1)
    .max(50),
  note: z.string().max(500).optional(),
});
export type CreateOrderInput = z.infer<typeof CreateOrder>;

export const ListOrders = z.object({
  limit: z.coerce.number().int().min(1).max(100).default(20),
  cursor: z.string().max(200).optional(),
});

export const OrderParams = z.object({ id: z.uuid() });

export const OrderOut = z.object({
  id: z.uuid(),
  status: z.enum(['pending', 'paid', 'shipped']),
  total: z.int(),
});

Three choices in there matter for security, not just tidiness:

• z.strictObject rejects unknown keys. A plain z.object strips them silently, which is also safe; what is not safe is spreading the raw req.body into an insert, because then a client can send role: "admin" or accountId and you have a mass-assignment bug.
• Every string and array has a max. Validation is your cheapest defense against a 5 MB note field.
• Query values arrive as strings, so z.coerce.number() turns ?limit=20 into 20.

Zod validation middleware for Express 5 #

Express has no schema system, so you write one small middleware:

// validate.ts
import type { RequestHandler } from 'express';
import type { ZodType } from 'zod';

type Parts = { params?: ZodType; query?: ZodType; body?: ZodType };

export const validate = (schemas: Parts): RequestHandler => (req, res, next) => {
  const input: Record<string, unknown> = {};
  const issues: { in: string; path: string; code: string; message: string }[] = [];

  for (const part of ['params', 'query', 'body'] as const) {
    const schema = schemas[part];
    if (!schema) continue;
    const result = schema.safeParse(req[part]);
    if (result.success) input[part] = result.data;
    else for (const i of result.error.issues) {
      issues.push({ in: part, path: i.path.join('.'), code: i.code, message: i.message });
    }
  }

  if (issues.length) return res.status(400).json({ error: 'invalid request', issues });
  res.locals.input = input;   // req.query is a read-only getter in Express 5
  next();
};

Use it per route:

app.use(express.json({ limit: '100kb' }));

app.post('/v1/orders', authenticate, validate({ body: CreateOrder }), async (req, res) => {
  const body = res.locals.input.body as CreateOrderInput;
  const order = await createOrder(req.user.accountId, body);
  res.status(201).json(OrderOut.parse(order));
});

app.get('/v1/orders', authenticate, validate({ query: ListOrders }), async (req, res) => {
  const { limit, cursor } = res.locals.input.query;
  res.json(await listOrders(req.user.accountId, limit, cursor));
});

The Express 5 gotcha: req.query is read-only #

In Express 4, plenty of validation middleware did req.query = parsed. In Express 5, req.query is a getter, so that assignment does not stick. Put parsed values somewhere else (res.locals above, or a typed req.input you declare yourself). Express 5’s default query parser is also now “simple”, so ?filter[status]=paid no longer becomes a nested object unless you set app.set('query parser', 'extended'). If you are still moving over, our Express 5 migration guide covers the rest.

safeParse or parse? #

Use safeParse in middleware so a bad request is a normal 400, not an exception. Use parse on data you produced, like OrderOut.parse(order) above, where a failure really is a server bug and should be a 500.

Zod validation in Fastify with a type provider #

Fastify validates with JSON Schema and Ajv by default. fastify-type-provider-zod swaps in Zod and gives your handlers typed request.body, request.query and request.params. Versions 5 and 6 of that package need Zod 4; version 7 (June 2026) needs Zod 4.2 or later.

import Fastify from 'fastify';
import {
  serializerCompiler,
  validatorCompiler,
  hasZodFastifySchemaValidationErrors,
  type ZodTypeProvider,
} from 'fastify-type-provider-zod';
import { CreateOrder, OrderOut, OrderParams } from './schemas.js';

const app = Fastify({ logger: true }).withTypeProvider<ZodTypeProvider>();
app.setValidatorCompiler(validatorCompiler);
app.setSerializerCompiler(serializerCompiler);

app.setErrorHandler((err, request, reply) => {
  if (hasZodFastifySchemaValidationErrors(err)) {
    return reply.code(400).send({ error: 'invalid request', issues: err.validation });
  }
  request.log.error({ err }, 'unhandled error');
  return reply.code(500).send({ error: 'internal error' });
});

app.post('/v1/orders', {
  schema: { body: CreateOrder, response: { 201: OrderOut } },
}, async (request, reply) => {
  const order = await createOrder(request.body);   // typed as CreateOrderInput
  return reply.code(201).send(order);
});

app.get('/v1/orders/:id', {
  schema: { params: OrderParams, response: { 200: OrderOut } },
}, async (request) => getOrder(request.params.id));

The response schema is worth the extra lines. If getOrder returns a row with internal_notes or stripe_customer_id, the serializer drops fields that are not in OrderOut. That is a cheap guard against leaking columns, the same class of bug as the object-level access mistakes in our BOLA and IDOR post.

Validation runs before preHandler, so by the time your authorization hook runs, the body is already checked; see Fastify hooks for logging and auth.

Zod or Fastify’s built-in JSON Schema? #

If raw throughput on large bodies is your bottleneck, Fastify’s compiled Ajv validators are hard to beat, and Zod 4 can emit JSON Schema with z.toJSONSchema() if you want to share definitions. For most small teams, one schema language across Express, Fastify and the front end is worth more than the difference.

What to log when validation fails #

Log the issue paths and codes, not the rejected values. A 400 on /v1/login with the full body logged is a password in your logs. A line like { route: "/v1/orders", status: 400, issues: ["items.0.qty"] } is enough to spot a broken client release.

If you only do one thing #

• Add a schema to every route that writes data, using z.strictObject or plain z.object and never spreading raw req.body.
• Put a max on every string and array, and cap body size in the parser.
• Return 400 with { in, path, code, message } issues.
• In Fastify, add a response schema to endpoints that return database rows.

A one-person team should start with the write endpoints and add reads later. A ten-person team should share schemas.ts with the client code so a breaking change fails in CI rather than in production.

FAQ #

How do I validate query parameters with Zod? #

Use z.coerce.number() and z.coerce.boolean() with care, add .default() for optional values, and in Express 5 store the result outside req.query, since it cannot be reassigned.

Should validation errors return 400 or 422? #

Either is defensible; pick one and document it. Many APIs use 400 for malformed input and keep 422 for well-formed requests that break a business rule.

Does Zod strip unknown fields? #

z.object strips unknown keys by default, z.strictObject rejects them, and z.looseObject passes them through. For write endpoints, strict is the most honest with clients.

Doing this with ApiTraffic #

If you would rather not build the logging for this yourself, ApiTraffic captures every request and response under its route pattern, so a client sending bad payloads shows up with one search: url_path == "/v1/orders" AND status_code == 400. Bodies are recorded, credential headers are masked by default, and you can add redaction rules for any other field from the dashboard without a deploy.

Tag requests with the caller (apitraffic.tag('userId', String(req.user.id))) and you can tell one broken integration apart from a general problem.

Start free and see every request your API handles: apitraffic.io

Sources: Zod documentation, “Release notes: Zod 4”; InfoQ, “Zod v4 available” (Aug 23, 2025); fastify-type-provider-zod README; Express, “Migrating to Express 5”

The short answer for API log retention: keep full request logs, including bodies, for 7 to 14 days; keep request metadata without bodies for 30 to 90 days; keep security and audit events for 12 months; and delete everything else on a schedule. Sample fast, successful reads if volume is a problem, but never sample errors. The rest of this post shows how to size that and set it up in an afternoon.

Why API log retention matters for small teams #

Retention is a trade between three things: the questions you need to answer, what storing data costs, and what you are allowed to keep. Keeping everything forever fails the last two. Keeping a week fails the first.

Under GDPR, the storage limitation principle in Article 5(1)(e) says personal data should be kept no longer than necessary for its purpose, and request logs usually contain personal data such as IPs, user IDs and emails in bodies. If you handle card data, PCI DSS requirement 10.5.1 asks for audit log history for at least 12 months, with the most recent three months immediately available for analysis. Most small API teams are not in PCI scope, but it is a sensible reference point for security logs.

How long should you keep API logs? #

Work backwards from the questions logs answer:

Only the first row needs bodies. That single observation is most of the savings.

How to estimate request logging costs #

Do the arithmetic before you pick numbers. Request volume times bytes per entry times days:

// log-size.mjs
const rps = 20;                // average, not peak
const perDay = rps * 86_400;   // 1,728,000 requests
const sizes = { metadata: 1_000, withBodies: 4_000 }; // bytes per entry

for (const [kind, bytes] of Object.entries(sizes)) {
  const gbPerDay = (perDay * bytes) / 1e9;
  console.log(kind, {
    gbPerDay: gbPerDay.toFixed(2),
    gb14d: (gbPerDay * 14).toFixed(0),
    gb90d: (gbPerDay * 90).toFixed(0),
  });
}

At 20 requests per second, metadata is about 1.7 GB a day and full entries about 6.9 GB a day. Ninety days of full entries is over 600 GB. Fourteen days of full entries plus ninety days of metadata is about 250 GB. Measure your own entry sizes rather than trusting these, because a few endpoints with large JSON responses can triple the average. Then apply your provider’s price per GB ingested and stored.

Log sampling: keep errors, sample the boring parts #

Most request volume is fast, successful reads that nobody will ever look at individually. Sampling those cuts storage without losing the requests you will investigate. The rule: decide after the response, so the outcome drives the decision.

import express from 'express';
import pino from 'pino';

const log = pino();
const app = express();
const SAMPLE_RATE = 0.1;
const SLOW_MS = 1000;
const MAX_BODY = 8 * 1024;

app.use((req, res, next) => {
  const start = process.hrtime.bigint();
  res.on('finish', () => {
    const ms = Number(process.hrtime.bigint() - start) / 1e6;
    const route = req.route?.path ?? 'unmatched';
    if (route === '/livez' || route === '/readyz') return;

    const keepFull =
      res.statusCode >= 400 || ms > SLOW_MS || req.method !== 'GET' ||
      Math.random() < SAMPLE_RATE;

    const entry = {
      method: req.method, route, status: res.statusCode,
      ms: Math.round(ms), userId: req.user?.id, sampled: !keepFull,
    };
    if (keepFull && req.body) {
      entry.body = JSON.stringify(req.body).slice(0, MAX_BODY);
    }
    log.info(entry);
  });
  next();
});

Every request still gets a metadata line, so counts and latency stay accurate. Only bodies are sampled. Errors, slow requests and writes are always kept in full. Record sampled: true so nobody mistakes a sampled dataset for a complete one.

How to set up a log retention policy #

Step 1: split logs by type #

Send request metadata, full request entries and security events to separate streams or indexes. Different data, different retention.

Step 2: redact before storage #

Retention is easier to justify when logs hold no secrets. Mask credentials and personal fields at write time; what not to log has the list.

Step 3: automate deletion #

Deletion that depends on someone remembering does not happen. If you archive logs to S3, a lifecycle rule handles it:

{
  "Rules": [
    { "ID": "full-requests", "Filter": { "Prefix": "requests-full/" },
      "Status": "Enabled", "Expiration": { "Days": 14 } },
    { "ID": "metadata", "Filter": { "Prefix": "requests-meta/" },
      "Status": "Enabled", "Expiration": { "Days": 90 } },
    { "ID": "security", "Filter": { "Prefix": "security/" },
      "Status": "Enabled",
      "Transitions": [{ "Days": 90, "StorageClass": "GLACIER_IR" }],
      "Expiration": { "Days": 400 } }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-api-logs --lifecycle-configuration file://lifecycle.json

If logs live in Postgres, partition by day and drop old partitions instead of running large DELETE statements.

Step 4: cut duplicate logging #

Many APIs log the same request three times: the proxy access log, a framework logger and an application line. Keep one request log and make application logs carry the request ID. A clear logging policy with meaningful levels also removes a lot of debug noise from production.

Log retention by team size #

A solo developer mostly needs the last week in detail and a month of history for customer questions. A larger team has more people asking older questions, and more reason to write the policy down.

If you only do one thing #

Write down three retention numbers (full entries, metadata, security events) and set a lifecycle rule that enforces them. A policy that deletes automatically beats a perfect policy on a wiki page.

FAQ #

How long should API logs be retained? #

For most small teams, 7 to 14 days of full request logs, 30 to 90 days of metadata, and 12 months of security events. Adjust for contracts or regulations that apply to you, such as PCI DSS 10.5.1 for card data.

How can I reduce logging costs without losing visibility? #

Keep a metadata line for every request, but store bodies only for errors, slow requests, writes and a sample of successful reads. Drop health check requests and duplicate log lines.

Does GDPR set a maximum log retention period? #

No fixed number. It requires keeping personal data no longer than necessary for the purpose, so document why each retention period exists.

Doing this with ApiTraffic #

If you would rather not build the sampling middleware, the redaction and the storage pipeline yourself, ApiTraffic captures every inbound request and response from one middleware.

• Authorization, Cookie and X-Api-Key are masked by default, and redaction rules for other fields are set in the dashboard and pushed to servers without a deploy, so what you keep is safer to keep.
• Requests are recorded under the matched route pattern, so per-endpoint counts and p50, p95 and p99 stay useful over weeks without storing anything extra yourself.
• Searches like tag:userId == "1042" AND status_code >= 400 answer the six-weeks-ago customer question directly.

Start free and see every request your API handles: apitraffic.io

Sources: GDPR Article 5, principles relating to processing of personal data; Microsoft Learn, PCI DSS requirement 10 (quotes 10.5.1); AWS, S3 Lifecycle configuration

Event loop lag is the delay between when a callback should run and when it actually does. You measure it with perf_hooks.monitorEventLoopDelay(), and you fix it by finding the synchronous code that blocks the loop and moving it off the request path. Here is how to do each, with code.

Why does event loop lag make every endpoint slow? #

Node runs your JavaScript on one thread. While a handler spends 200 ms in JSON.stringify on a huge object, nothing else in that process runs: not other requests, not timers, not health checks. The slow request’s latency is honest. Everybody else’s latency is collateral.

The symptoms are distinctive:

• p99 rises on all endpoints at once, including trivial ones.
• One core is pinned while total machine CPU looks modest (common on multi-core VPS and container hosts).
• Database and outbound spans look normal, but there are unexplained gaps between them.
• Health checks time out under load and the platform restarts a healthy process.

That last one is why liveness checks should be cheap and have generous timeouts.

How to measure Node.js event loop lag #

Node has two built-in tools. monitorEventLoopDelay() samples how late a timer fires and records it in a histogram (in nanoseconds). performance.eventLoopUtilization() reports the fraction of time the loop was busy rather than idle.

// loop-metrics.js
import { monitorEventLoopDelay, performance } from 'node:perf_hooks';

export function startLoopMetrics(log, intervalMs = 10_000) {
  const h = monitorEventLoopDelay({ resolution: 20 });
  h.enable();
  let last = performance.eventLoopUtilization();

  const timer = setInterval(() => {
    const elu = performance.eventLoopUtilization(last);
    last = performance.eventLoopUtilization();
    log.info({
      loop_delay_p50_ms: +(h.percentile(50) / 1e6).toFixed(1),
      loop_delay_p99_ms: +(h.percentile(99) / 1e6).toFixed(1),
      loop_delay_max_ms: +(h.max / 1e6).toFixed(1),
      loop_utilization: +elu.utilization.toFixed(3),
    }, 'event loop');
    h.reset();
  }, intervalMs);
  timer.unref();
}

Call it once at startup with your logger. Log these numbers or export them as gauges; either way, keep the per-interval max, because a single 800 ms block is exactly what this is meant to catch.

How to read the numbers #

• Delay p99 and max tell you how long requests waited for the loop. A max of 300 ms means some request got 300 ms added to it for no reason of its own.
• Utilization near 1.0 for sustained periods means the process is saturated: it is busy all the time, and more traffic will only queue. Low utilization with high max delay means occasional long blocks, which is the “random slowness” case.
• Compare against your own baseline. There is no universal threshold. Record a week of normal values, then alert or investigate when p99 delay moves well above it.

Read the delay alongside request latency percentiles: a p99 latency spike that lines up with a max loop delay spike points at blocking, not at the database.

What blocks the Node.js event loop? #

The usual suspects in API code:

1. Large JSON. JSON.parse and JSON.stringify on multi-megabyte payloads, including logging big objects synchronously.
2. Synchronous crypto. pbkdf2Sync, scryptSync, or a pure-JavaScript bcrypt on a login route.
3. Regular expressions with catastrophic backtracking, especially on user input.
4. Sync I/O on the request path: readFileSync, zlib.gzipSync, template compilation per request.
5. Big in-memory loops: sorting or mapping tens of thousands of rows in JavaScript instead of in the database.
6. Garbage collection. Large heaps mean longer pauses, and GC shows up as loop delay. If the heap grows steadily too, see debugging memory leaks with heap snapshots.

How to find the code that blocks the loop #

Step 1: Profile under realistic load #

node --cpu-prof --cpu-prof-dir=./profiles server.js
# generate load, then stop the process with Ctrl+C

Open the .cpuprofile file in Chrome DevTools (Performance panel) and look for long, flat blocks of your own functions. Run it in staging with replayed or synthetic traffic; profiling production directly is possible but has overhead.

Step 2: Time suspicious sections #

When you have a suspect, measure it directly:

const t0 = performance.now();
const body = JSON.stringify(report);
const ms = performance.now() - t0;
if (ms > 50) req.log.warn({ ms, route: req.routeOptions?.url }, 'slow sync section');

Step 3: Correlate with routes #

Log the loop max next to request logs with timestamps. The route whose slow requests consistently overlap with loop delay spikes is usually the culprit.

How to fix event loop blocking #

• Use async crypto. The callback and promise versions run in libuv’s thread pool:

  js

  import { pbkdf2 } from 'node:crypto';
  import { promisify } from 'node:util';
  const pbkdf2Async = promisify(pbkdf2);
  const hash = await pbkdf2Async(password, salt, 210_000, 32, 'sha256');
  

• Move CPU work to worker threads. For report generation, image work or heavy parsing, use node:worker_threads directly or a pool such as Piscina.

• Shrink the work. Paginate, select fewer columns, cap body sizes, and stream large responses instead of building one giant string.

• Fix regexes. Anchor them, avoid nested quantifiers, and cap input length before matching.

Shedding load when the loop is saturated #

Under a traffic spike, it can be better to return 503 quickly than to make every request slow. Fastify has @fastify/under-pressure:

import underPressure from '@fastify/under-pressure';

await app.register(underPressure, {
  maxEventLoopDelay: 1000,       // ms
  maxEventLoopUtilization: 0.98,
  retryAfter: 10,
});

In Express, you can build the same check on the histogram above, but keep the threshold high: shedding healthy traffic because of one blip causes its own incidents.

A solo developer should add the metrics function and look at the numbers once a week. A team of ten should export them as metrics, put them on the same dashboard as p99 latency, and check them in every performance investigation.

If you only do one thing #

• Add startLoopMetrics() to every service today.
• When p99 latency jumps everywhere at once, check loop max delay before the database.
• Replace any *Sync crypto or file call on a request path.
• Profile one busy endpoint with --cpu-prof this month.

FAQ #

What is a normal event loop delay in Node.js? #

It depends on the timer resolution and your workload, so measure your own baseline. What matters is change: a max delay of hundreds of milliseconds means some requests waited that long for the loop.

Does event loop lag show up as high CPU? #

Usually as one busy core, not high total CPU. On a multi-core host the machine can look mostly idle while the Node process is saturated.

Will adding more instances fix event loop lag? #

It spreads load, so fewer requests queue behind each block, but each blocked request is still slow. Fix the blocking code; scale for throughput.

Doing this with ApiTraffic #

If you would rather not build span tracing to tell blocking from slow dependencies, ApiTraffic records database, cache, queue and outbound spans inside each request via OpenTelemetry.

• An endpoint’s Time Breakdown shows where its time goes. When most of it is not in queries or outbound calls, look at your own code and the loop.
• The request waterfall shows gaps between spans on a slow request; long gaps with no span are where synchronous work ran.
• span() adds a custom timing around a suspect section, such as a big JSON.stringify, so it appears on the waterfall.

Start free and see every request your API handles: apitraffic.io

Sources: Node.js docs: perf_hooks (monitorEventLoopDelay, eventLoopUtilization); Node.js guide: Don’t Block the Event Loop (or the Worker Pool); @fastify/under-pressure on GitHub

The direct answer: if your API uses cookies or other browser credentials, allow only an exact list of origins, never reflect the request’s Origin, never allow null, and never match origins with an unanchored regex. If your API uses bearer tokens only, you usually do not need credentials in CORS at all. Then test it with three curl commands, shown below.

What CORS actually protects (and what it does not) #

Browsers stop a page on https://evil.test from reading responses from https://api.example.com unless the API says otherwise with Access-Control-Allow-Origin. When the response also includes Access-Control-Allow-Credentials: true, the browser will send cookies with the request and let the page read the result.

Three facts about CORS that cause most mistakes:

1. CORS is enforced by the browser, not your server. curl, scripts and other servers ignore it. It is not authentication or authorization.
2. CORS controls reading, not sending. For simple requests, the browser sends the request and your handler runs; it just hides the response from the page. A state-changing POST with a form content type still executes.
3. The dangerous combination is a trusted-looking origin plus credentials. Access-Control-Allow-Origin: * with Access-Control-Allow-Credentials: true is rejected by browsers, so wildcard plus credentials is broken rather than exploitable. Reflecting the origin is what makes it exploitable.

Four CORS misconfigurations, tested #

Here is what the cors middleware for Express returns for four common configurations when probed with different Origin headers (output trimmed to the two CORS headers).

1. Reflecting any origin #

app.use(cors({ origin: true, credentials: true }));

https://evil.test                    ACAO=https://evil.test ACAC=true
https://app.example.com.evil.test    ACAO=https://app.example.com.evil.test ACAC=true
null                                 ACAO=null ACAC=true

origin: true means “echo back whatever Origin was sent”. With credentials, every website on the internet can make authenticated reads as your logged-in users. @fastify/cors with origin: true behaves the same way. This is the one to fix first.

2. An unanchored regex #

app.use(cors({ origin: /https:\/\/app\.example\.com/, credentials: true }));

https://evil.test                    ACAO=-
https://app.example.com.evil.test    ACAO=https://app.example.com.evil.test ACAC=true

The regex matches anywhere in the string, so an attacker registers app.example.com.evil.test. If you need a pattern, anchor it: /^https:\/\/([a-z0-9-]+\.)?example\.com$/. Better still, use a list.

3. Allowing the null origin #

app.use(cors({ origin: ['https://app.example.com', 'null'], credentials: true }));

Developers add 'null' because local files or a sandboxed iframe sent it during testing. Sandboxed iframes and some redirects send Origin: null, and an attacker can create those, so allowing null is close to allowing everyone.

4. Wildcard with credentials #

app.use(cors({ origin: '*', credentials: true }));

This returns ACAO=* with ACAC=true. Browsers refuse it, so your frontend breaks, which is often what leads someone to “fix” it with origin: true.

How to configure CORS safely in Express #

Use an exact allowlist from configuration, and let requests with no Origin (same-origin, server-to-server, curl) through without CORS headers:

import cors from 'cors';

const allowedOrigins = new Set(
  (process.env.CORS_ORIGINS ?? 'https://app.example.com').split(',')
);

app.use(
  cors({
    origin: (origin, callback) => callback(null, !origin || allowedOrigins.has(origin)),
    credentials: true,
    methods: ['GET', 'POST', 'PUT', 'PATCH', 'DELETE'],
    maxAge: 600,
  })
);

The cors package adds Vary: Origin when the allowed origin varies, which stops a CDN or proxy from caching one origin’s CORS headers and serving them to another. If you set CORS headers by hand, add Vary: Origin yourself.

Preview deployments on Render, Vercel or Netlify are where allowlists tend to get loosened. Prefer adding preview origins in the preview environment’s config over a wildcard pattern in production.

How to configure CORS safely in Fastify #

@fastify/cors accepts the same idea:

import Fastify from 'fastify';
import cors from '@fastify/cors';

const app = Fastify();
const allowedOrigins = new Set(process.env.CORS_ORIGINS.split(','));

await app.register(cors, {
  origin: (origin, cb) => cb(null, !origin || allowedOrigins.has(origin)),
  credentials: true,
});

If you add auth in Fastify hooks (see Fastify hooks for logging and auth), remember that preflight OPTIONS requests carry no credentials. Make sure the CORS plugin answers them before your auth hook rejects them with a 401.

How to test your API for CORS misconfiguration #

Step 1: probe with curl #

API=https://api.example.com/v1/me
for origin in https://evil.test https://app.example.com.evil.test null; do
  echo "== $origin"
  curl -s -o /dev/null -D - -H "Origin: $origin" "$API" \
    | grep -i '^access-control-allow-'
done

Any output line containing the probe origin together with access-control-allow-credentials: true is a finding.

Step 2: probe the preflight #

curl -s -o /dev/null -D - -X OPTIONS "$API" \
  -H "Origin: https://evil.test" \
  -H "Access-Control-Request-Method: DELETE" | grep -i '^access-control-'

Step 3: add a regression test #

import { test } from 'node:test';
import assert from 'node:assert/strict';

test('CORS does not trust arbitrary origins', async () => {
  for (const origin of ['https://evil.test', 'https://app.example.com.evil.test', 'null']) {
    const res = await fetch(`${process.env.API_URL}/v1/me`, { headers: { Origin: origin } });
    assert.notEqual(res.headers.get('access-control-allow-origin'), origin);
  }
});

CORS misconfiguration checklist #

• [ ] No origin: true (or hand-written reflection) together with credentials: true.
• [ ] Origins are an exact list from config; any regex is anchored with ^ and $.
• [ ] null is not in the list.
• [ ] Vary: Origin is present when the allowed origin varies.
• [ ] Bearer-token-only APIs do not enable credentials.
• [ ] A test like the one above runs in CI.

A solo developer can do all six in an afternoon. On a team of ten, the useful extra step is making the allowlist a config value owned by one person, so a frontend developer fighting a CORS error opens a PR against config rather than loosening the code. And remember that tight CORS does not replace per-object authorization checks; see BOLA and IDOR.

FAQ #

Is Access-Control-Allow-Origin: * insecure? #

Not on its own. For a public API that does not use cookies, * is fine, and browsers refuse to combine it with credentials. The risk is reflecting specific origins with credentials enabled.

Does CORS protect my API from attackers? #

No. CORS only controls which web pages a browser lets read your responses. Attackers using curl or scripts are unaffected, so authentication, authorization and rate limiting still matter.

Why do I get a CORS error even though my server sends the headers? #

Usually the preflight fails: an auth middleware returns 401 to the OPTIONS request, or the requested method or header is not allowed. Check the OPTIONS response in the browser’s network tab first.

Doing this with ApiTraffic #

If you would rather not guess which origins actually call your API before tightening CORS, ApiTraffic records every request with its headers from one middleware, so you can see the Origin values hitting each endpoint before you change the allowlist. Credential headers like Authorization and Cookie are masked by default.

The firewall’s access lists can also deny by user agent, path, IP or CIDR, and monitor mode shows what a rule would block before it blocks anything.

Start free and see which origins call your API: apitraffic.io

Sources: MDN: Cross-Origin Resource Sharing (CORS); WHATWG Fetch Standard, CORS protocol; PortSwigger Web Security Academy: CORS; cors middleware README; @fastify/cors README.

An idempotency key fixes that. The client generates a unique key (a UUID) once per logical operation and sends it in an Idempotency-Key header on every attempt. The server stores the key with the result of the first attempt; any retry with the same key gets the stored response instead of running the operation again. Below is a working Express implementation on Postgres, the edge cases that bite (concurrent retries, changed payloads, crashes mid-request), and how long to keep keys.

Why POST needs an idempotency key and PUT usually does not #

HTTP defines GET, HEAD, PUT and DELETE as idempotent: sending the same request twice has the same effect as sending it once (RFC 9110, section 9.2.2). POST and PATCH are not. “Create a payment” twice creates two payments.

Clients retry for good reasons: timeouts, dropped connections, a 502 from a load balancer during your deploy. If you have set sensible client timeouts (see timeouts everywhere in Node.js), retries will happen, and the server must be the one that makes them safe.

How the Idempotency-Key header works #

The IETF httpapi working group has a draft standard for this header (revision 07, October 2025). It is not an RFC yet, but its rules match what payment APIs such as Stripe have done for years:

The draft recommends a UUID or similar random value as the key, and says servers should publish how long keys are kept. Stripe’s documentation says keys can be pruned once they are at least 24 hours old and that the stored response is returned even when it was a 500.

Implementing idempotency keys in Express with Postgres #

Step 1: a table with a unique key per caller #

CREATE TABLE idempotency_keys (
  account_id    bigint      NOT NULL,
  key           text        NOT NULL,
  fingerprint   text        NOT NULL,
  status        text        NOT NULL DEFAULT 'running',  -- running | done
  response_code int,
  response_body jsonb,
  created_at    timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (account_id, key)
);
CREATE INDEX ON idempotency_keys (created_at);

Scope keys to the caller. Without account_id in the primary key, two customers who happen to send the same key collide, and one could read the other’s stored response.

Step 2: the middleware #

// idempotency.js
import { createHash } from 'node:crypto';
import { pool } from './db.js';

const KEY_RE = /^[A-Za-z0-9_-]{8,255}$/;
const STALE_MS = 60_000; // a "running" row older than this is treated as abandoned

export function idempotent({ required = true } = {}) {
  return async (req, res, next) => {
    const key = req.get('Idempotency-Key')?.replace(/^"|"$/g, '');
    if (!key) {
      return required ? res.status(400).json({ error: 'Idempotency-Key header required' }) : next();
    }
    if (!KEY_RE.test(key)) return res.status(400).json({ error: 'invalid Idempotency-Key' });

    const accountId = req.user.accountId;
    const fingerprint = createHash('sha256')
      .update(`${req.method} ${req.baseUrl}${req.path}\n${JSON.stringify(req.body ?? null)}`)
      .digest('hex');

    const inserted = await pool.query(
      `INSERT INTO idempotency_keys (account_id, key, fingerprint)
       VALUES ($1, $2, $3) ON CONFLICT DO NOTHING RETURNING key`,
      [accountId, key, fingerprint],
    );

    if (inserted.rowCount === 0) {
      const { rows: [row] } = await pool.query(
        'SELECT * FROM idempotency_keys WHERE account_id = $1 AND key = $2',
        [accountId, key],
      );
      if (row.fingerprint !== fingerprint) {
        return res.status(422).json({ error: 'Idempotency-Key reused with a different request' });
      }
      if (row.status === 'done') {
        res.set('Idempotent-Replayed', 'true');
        return res.status(row.response_code).json(row.response_body);
      }
      if (Date.now() - row.created_at.getTime() < STALE_MS) {
        return res.status(409).json({ error: 'request with this Idempotency-Key is in progress' });
      }
      // Abandoned by a crashed process: take it over.
      await pool.query(
        'UPDATE idempotency_keys SET created_at = now() WHERE account_id = $1 AND key = $2',
        [accountId, key],
      );
    }

    const json = res.json.bind(res);
    res.json = (body) => {
      const done = res.statusCode < 500
        ? pool.query(
            `UPDATE idempotency_keys SET status = 'done', response_code = $3, response_body = $4
             WHERE account_id = $1 AND key = $2`,
            [accountId, key, res.statusCode, body],
          )
        : pool.query('DELETE FROM idempotency_keys WHERE account_id = $1 AND key = $2', [accountId, key]);
      done.catch((err) => req.log?.error({ err }, 'idempotency write failed'));
      return json(body);
    };
    next();
  };
}

Step 3: use it on the routes that move money or create things #

app.post('/v1/payments', authenticate, idempotent(), async (req, res) => {
  const payment = await createPayment(req.user.accountId, req.body);
  res.status(201).json(payment);
});

Express 5 forwards rejected promises from async handlers to your error middleware, so a thrown error here reaches it, and the error handler’s res.json call deletes the key so the client can retry.

The decisions you have to make #

Store 5xx responses or let them retry? #

Stripe stores the first response, including a 500. That is the safest choice when a failure might have happened after the side effect (the charge went through, then the database write failed). The middleware above takes the other path: it deletes the key on a 5xx so the client can retry. That is only safe if your handler is transactional, meaning a 5xx guarantees nothing was committed. If the handler calls an external payment provider, pass the same key through to the provider so their side is idempotent too.

How long to keep keys #

Keep them at least as long as clients retry, plus a margin. 24 hours is a common choice. Delete old rows on a schedule:

DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours';

Document the window in your API docs, as the draft asks.

Redis instead of Postgres #

If you already run Redis, SET key value NX EX 86400 gives you the same “first writer wins” step. Keep Postgres if the operation itself writes to Postgres: storing the key in the same transaction as the payment row removes a whole class of crash-in-between bugs.

The client side #

The key must be created once per user intent, not once per HTTP attempt:

const key = crypto.randomUUID();   // when the user clicks Pay
for (let attempt = 1; attempt <= 3; attempt++) {
  const res = await fetch('https://api.example.com/v1/payments', {
    method: 'POST',
    headers: { 'content-type': 'application/json', 'idempotency-key': key },
    body: JSON.stringify({ amount: 4200, currency: 'usd' }),
    signal: AbortSignal.timeout(10_000),
  }).catch(() => null);
  if (res && res.status !== 409 && res.status < 500) break;
  await new Promise((r) => setTimeout(r, 2 ** attempt * 250));
}

Never use an email address or other personal data as the key.

If you only do one thing #

• Add the idempotency_keys table and middleware to every endpoint that charges, sends or creates something a user would notice twice.
• Scope keys per account, fingerprint the payload, and return 409 and 422 as above.
• Purge keys older than 24 hours.

A one-person team can ship this in an afternoon for the one payment route that matters. A ten-person team should make it a shared middleware and add “is this POST idempotent?” to the review checklist, next to rate limiting.

FAQ #

Is an idempotency key the same as a request ID? #

No. A request ID identifies one HTTP attempt and changes on every retry. An idempotency key identifies the user’s intent and stays the same across retries.

Should the idempotency key go in a header or the body? #

A header. It keeps the business payload clean, works the same on every endpoint, and matches the IETF draft and Stripe’s API.

How do I test idempotency? #

Send the same request twice in parallel and check that only one row was created and the second call got a 409 or the replayed response. Then send the same key with a changed body and expect 422.

Doing this with ApiTraffic #

If you would rather not hunt for duplicate charges in raw logs, ApiTraffic records every request and response, so you can tag each one with its key and see every attempt side by side:

apitraffic.tag('idempotencyKey', key);

Then search for conflicts, url_path == "/v1/payments" AND status_code == 409, or for one customer’s attempts with tag:userId == "1042" AND status_code >= 400. Outbound calls are logged too, so you can confirm your payment provider received the request once.

Start free and see every request your API handles: apitraffic.io

Sources: IETF, “The Idempotency-Key HTTP Header Field”, draft-ietf-httpapi-idempotency-key-header-07 (Oct 15, 2025); RFC 9110, HTTP Semantics, section 9.2.2 (June 2022); Stripe API reference, “Idempotent requests”

The short answer: confirm it is a leak by watching heapUsed low points rise over hours, take three heap snapshots under the same traffic, and compare them in Chrome DevTools. Objects that grow between every pair of snapshots are your leak, and the Retainers panel shows the code holding them. Below is that workflow, with the safety rules for doing it in production.

How to tell if you have a Node.js memory leak #

Not every rising graph is a leak. V8 grows the heap lazily, caches fill up to their limits, and RSS includes native memory the JavaScript heap does not. Log the numbers separately first:

// memory-log.js
const MB = 1024 * 1024;
setInterval(() => {
  const m = process.memoryUsage();
  console.log(JSON.stringify({
    msg: 'memory',
    rssMb: Math.round(m.rss / MB),
    heapUsedMb: Math.round(m.heapUsed / MB),
    heapTotalMb: Math.round(m.heapTotal / MB),
    externalMb: Math.round(m.external / MB),
    arrayBuffersMb: Math.round(m.arrayBuffers / MB),
  }));
}, 30_000).unref();

Check one more thing either way. If the container limit is 512 MB and V8 believes it can grow past that, the kernel kills the process before V8 collects hard. Set --max-old-space-size below the limit to leave room for native memory.

Common causes of memory leaks in Node APIs #

• Module-level maps and arrays used as caches with no size limit or expiry.
• Listeners added per request to a long-lived emitter and never removed. Node prints MaxListenersExceededWarning once an emitter passes 10 listeners for one event. Do not silence it.
• Timers or intervals holding closures that are never cleared.
• Per-request data pushed onto a singleton or the app instance.
• Promises that never settle, often an outbound call with no timeout. See timeouts with fetch, undici and AbortSignal.
• Unbounded queues where the consumer is slower than the producer.

A deliberately leaky Express route to practise on:

import express from 'express';
const app = express();
const seen = new Map(); // grows forever

app.get('/v1/quotes/:id', (req, res) => {
  const key = `${req.params.id}:${req.get('x-request-id') ?? crypto.randomUUID()}`;
  seen.set(key, { at: Date.now(), headers: { ...req.headers } });
  res.json({ id: req.params.id, price: 42 });
});

app.listen(3000);

Every request adds an entry holding a copy of the headers. Run npx autocannon -d 60 http://localhost:3000/v1/quotes/7 and watch heapUsed climb.

How to take a heap snapshot in Node.js #

The Node.js docs are blunt about cost: taking a snapshot stops all other work on the main thread, can take more than a minute on a large heap, and is built in memory, so it can double heap size and crash the process. In production, take snapshots from an instance you can afford to lose, after removing it from load balancer rotation.

From a signal, with no code changes #

node --heapsnapshot-signal=SIGUSR2 server.js
kill -USR2 <pid>
# writes Heap.<date>.<time>.<pid>.<tid>.<seq>.heapsnapshot to the working directory

From code with v8.writeHeapSnapshot() #

import { writeHeapSnapshot } from 'node:v8';

app.post('/internal/heap-snapshot', requireAdmin, (req, res) => {
  const file = writeHeapSnapshot(); // returns the filename
  res.json({ file });
});

Never expose that route publicly. A snapshot contains whatever was in memory, including tokens, request bodies and personal data. Treat the file like a database dump and delete it when you are done.

Automatically, near the heap limit #

--heapsnapshot-near-heap-limit=1 writes a snapshot as the heap approaches its limit, capturing the worst state. It needs headroom to succeed, so pair it with a sensible --max-old-space-size.

Locally, with the inspector #

Start with node --inspect server.js, open chrome://inspect, click “inspect” under your process, and use the Memory tab’s “Take snapshot” button.

How to compare heap snapshots in Chrome DevTools #

One snapshot shows what is in memory, not what is growing. Use three:

1. Start the process and let it warm up.
2. Take snapshot 1.
3. Send steady traffic to the suspect route for a few minutes.
4. Take snapshot 2.
5. Send the same traffic again.
6. Take snapshot 3.

Load them oldest first. Select snapshot 3, switch the view from Summary to Comparison against snapshot 2, and sort by “# Delta” or “Size Delta”. Objects that grew between 1 and 2 and again between 2 and 3 are leak candidates. Objects that grew once are usually caches warming up.

For the leaky route, the comparison shows thousands of new objects and strings. Expand one and read the Retainers panel: the object is held by a Map entry, the Map by seen in the module’s context, the context by the route handler. That points at the line.

Reading tips:

• Shallow size is the object itself. Retained size is everything freed if it went away. Sort by retained size to find the root.
• Skip (compiled code) and (system) entries at first.
• Retainer paths through EventEmitter and _events mean a listener leak. Paths through Timeout mean a timer holding a closure.

Fixing the leak and proving it #

For a cache, add a bound and an expiry. An LRU library or a size check both work:

const MAX = 10_000;
function remember(key, value) {
  if (seen.size >= MAX) seen.delete(seen.keys().next().value);
  seen.set(key, value);
}

Repeat the three-snapshot run. The fix is proven when snapshot 3 shows no consistent growth over snapshot 2, and heapUsed low points stay flat for hours in production.

On a one-person team, do the whole investigation locally with the inspector and a load tool; production snapshots are rarely needed. On a ten-person team, add --heapsnapshot-signal to the start command now, write down who can pull an instance out of rotation, and keep the memory log line in every service.

Memory leak checklist #

If you only do one thing, log heapUsed and rss every 30 seconds so the next leak is diagnosed from a graph instead of a guess. Then:

• --max-old-space-size fits inside the container limit.
• Every module-level cache has a size limit or TTL.
• MaxListenersExceededWarning is treated as a bug.
• Every outbound call has a timeout.
• Snapshot files are deleted after analysis.

FAQ #

How do I find a memory leak in Node.js? #

Confirm heapUsed keeps rising after garbage collection, then take three heap snapshots under the same load and compare them in Chrome DevTools. The Retainers panel shows which code keeps the growing objects alive.

Is it safe to take a heap snapshot in production? #

Only on an instance that can crash without hurting users. The process pauses while the snapshot is written, and memory use can double.

Why is RSS growing when heapUsed is flat? #

The growth is outside the JavaScript heap: buffers, native addons or compression contexts. Check external and arrayBuffers in process.memoryUsage().

Doing this with ApiTraffic #

Heap snapshots show what is held in memory, not which traffic caused it. If you would rather not wire up that correlation yourself:

• ApiTraffic records requests and p50, p95 and p99 per endpoint over time, so latency that rises with memory shows on the endpoint doing the extra work. Read those numbers with latency percentiles in mind.
• Regressions compares p95 before and after a recorded deploy event, narrowing a new leak to the release that introduced it.
• span() and trace() add custom timings around a suspect cache path inside each request.

Start free and see every request your API handles: apitraffic.io

Sources: Node.js, “Using Heap Snapshot”; Node.js CLI options; Node.js v8.writeHeapSnapshot; Chrome DevTools, “Record heap snapshots”

The direct answer: tie each level to an action. error means someone should look at it today, warn means watch the trend, info records events you will want during an investigation, and debug is off in production. Client mistakes (4xx) are never error. Log each failure once, at the boundary where you handle it. The rest of this post is a policy you can paste into your repo, with the code to enforce it.

What each log level means: a policy you can copy #

Pino, Fastify’s logger and most Node loggers use the same six levels, with numeric values in Pino of trace 10, debug 20, info 30, warn 40, error 50 and fatal 60. Syslog (RFC 5424) has eight severities; you do not need that many.

The test for any log line: if this fired 1,000 times tonight, would you want to be woken up? If yes, it is error or fatal. If you would want to see the count on a dashboard, warn. Otherwise info or lower.

Why 4xx responses are not errors #

A 400, 401, 403, 404 or 422 means the client did something wrong and your code did the right thing by refusing. Logging them at error is the most common reason small teams stop trusting their error logs. Scanners, expired sessions and buggy clients produce thousands of them.

Log 4xx at info (or not at all, if your request logger already records the status). Promote a pattern to warn only when the volume itself is the signal, such as a spike in 401s on login. A 429 from your own rate limiter is info; a 429 you received from a third-party API is warn.

How to log each failure once #

The second most common problem is the same failure logged three times: in the repository, in the service and in the error handler. Each line looks like a separate incident. The rule: either handle an error or throw it, never both. Log where you handle.

Carry the intended level on the error itself, so the one place that logs knows what to do:

// errors.js
export class AppError extends Error {
  constructor(message, { status = 500, code = 'internal_error', level, cause } = {}) {
    super(message, { cause });
    this.status = status;
    this.code = code;
    this.level = level ?? (status >= 500 ? 'error' : 'info');
  }
}

// usage in a handler
if (!order) throw new AppError('order not found', { status: 404, code: 'not_found' });

Then a single Express error handler does the logging:

app.use((err, req, res, next) => {
  const status = err.status ?? 500;
  const level = err.level ?? 'error';
  req.log[level]({ err, code: err.code }, err.message);
  res.status(status).json({ error: err.code ?? 'internal_error' });
});

If you also use pino-http (see Pino logging in Node.js), it logs a line for every completed response too. Decide which one owns errors. A simple split: pino-http logs every request at info with its status, and only the error handler logs at error with the stack. Set pino-http’s customLogLevel to return info even for 5xx, so the stack trace is not duplicated.

How to change the log level in production without a deploy #

Running at info in production is the right default. When you are chasing a bug you want debug for ten minutes, not a redeploy. Two cheap options:

Option 1: an environment variable #

const logger = pino({ level: process.env.LOG_LEVEL ?? 'info' });

On Render, Railway, Fly.io or Heroku, changing an environment variable restarts the service. That is fine for a solo developer and loses in-memory state.

Option 2: a signal #

process.on('SIGUSR2', () => {
  logger.level = logger.level === 'debug' ? 'info' : 'debug';
  logger.warn({ newLevel: logger.level }, 'log level changed');
});

kill -USR2 <pid> flips the level with no restart. Use SIGUSR2, not SIGUSR1, which Node reserves for starting the inspector. Log the change at warn, so anyone reading later knows why the volume jumped.

For per-module control, create child loggers and set levels on them: const dbLog = logger.child({ module: 'db' }, { level: 'debug' }).

Node.js log levels and your log bill #

Levels are also a cost control. Most log platforms bill by volume ingested or retained, and debug output left on in production is a classic surprise invoice. Three habits keep it in check:

• Do not log request or response bodies at info. If you need them, log at debug with redaction (see how to redact sensitive data in logs).
• One line per request, not one per middleware.
• Outbound calls: log failures and slow calls, not every success. A dedicated approach to logging outbound HTTP requests covers this in detail.

If you only do one thing: paste this policy into your repo #

## Logging policy
- fatal: process will exit. Pages on-call.
- error: a user-visible failure we did not expect. Reviewed daily.
- warn: unexpected but handled (retries, fallbacks). Watched as a rate.
- info: lifecycle and business events. Default level in production.
- debug/trace: off in production; enable temporarily with SIGUSR2.
- 4xx responses are never error.
- Log an error once, where it is handled. Never log and rethrow.
- Errors are logged as { err }, never err.message.

A solo developer can adopt this in an afternoon: grep for log.error and console.error, demote the 4xx cases and delete the log-and-rethrow lines. A team of ten should add it to the PR checklist so the next new endpoint follows it, and review the error count weekly until it is quiet enough that a new error is news.

FAQ #

What log level should I use in production for Node.js? #

Use info. It records lifecycle and business events without the volume of debug. Enable debug temporarily when investigating, then turn it back off.

Should a 404 be logged as an error? #

No. A 404 means your API correctly refused a request for something that does not exist. Log it at info, and alert only on unusual volume, which often indicates scanning.

What is the difference between warn and error? #

error means a request or job failed in a way the user noticed and nobody expected. warn means something went wrong but was handled, such as a retry that succeeded. Errors deserve individual attention; warnings deserve a trend line.

Doing this with ApiTraffic #

If you would rather not keep per-request log lines at all, ApiTraffic records every inbound request and response (method, path, status, latency, headers, bodies) from one middleware, independent of your log levels. Your application logs can then stay at info or warn for application events, and the request record is still there when you need it.

Finding the failures that matter becomes a search, for example status_code >= 500 for one endpoint, or tag:userId == "1042" AND status_code >= 400 when a specific customer reports a problem.

Start free and keep a full request record without noisy logs: apitraffic.io

Sources: Pino API: logger.level and levels; RFC 5424, The Syslog Protocol, severity levels; Node.js docs: signal events (SIGUSR1 reserved for the debugger).

The short answer: authenticate in onRequest (it runs first and has headers, but no body), authorize in preHandler (body, params and validation are done), and log in onResponse (the status and reply.elapsedTime are final). This post walks through the request lifecycle in Fastify v5, shows working code for each of those three jobs, and lists the mistakes that cause double responses and missing log fields.

The Fastify hooks lifecycle, in order #

Fastify runs these request and reply hooks for every request, in this order:

Two facts from the Fastify docs shape everything else. In onRequest, the body is always undefined because parsing happens later. And onResponse runs after the response has been sent, so you cannot change anything the client sees there.

How to authenticate in the onRequest hook #

Authentication only needs headers, so do it as early as possible. Rejecting a bad token in onRequest means Fastify never parses a body from an anonymous caller, which matters when someone posts 1 MB of JSON at your login-protected endpoints.

// plugins/auth.js
import fp from 'fastify-plugin';
import { verifyToken } from '../lib/tokens.js';

export default fp(async (app) => {
  app.decorateRequest('user', null);

  app.decorate('authenticate', async (request, reply) => {
    const header = request.headers.authorization ?? '';
    const token = header.startsWith('Bearer ') ? header.slice(7) : null;
    const user = token ? await verifyToken(token) : null;

    if (!user) {
      // Send, then return reply so Fastify stops the chain here.
      return reply.code(401).send({ error: 'unauthorized' });
    }
    request.user = user;
  });
});

fastify-plugin matters here. Fastify encapsulates plugins: every hook except onClose, and every decorator, is scoped to the plugin context where it was registered. Wrapping the auth plugin with fp makes app.authenticate visible to the rest of the app, while you still choose where the hook runs.

How to authorize in the preHandler hook #

Authorization usually needs params and sometimes the body: “can this user read order 42”, “can this user move money from this account”. By preHandler, both are parsed and validated. Load the resource once, check ownership, and hand it to the handler:

// routes/orders.js
import { findOrder } from '../lib/orders.js';

export default async function orders(app) {
  app.decorateRequest('order', null);
  app.addHook('onRequest', app.authenticate);   // every route in this plugin

  async function loadOwnOrder(request, reply) {
    const order = await findOrder(request.params.id);
    if (!order || order.accountId !== request.user.accountId) {
      // 404, not 403: do not confirm that someone else's order exists.
      return reply.code(404).send({ error: 'not found' });
    }
    request.order = order;
  }

  app.get('/v1/orders/:id', { preHandler: loadOwnOrder }, async (request) => {
    return request.order;
  });
}

Route-level hooks accept a function or an array, and the docs note that they always run as the last hook in their category. That gives you a predictable order: plugin-wide auth in onRequest, then the route’s own checks in preHandler.

Logging requests in the onResponse hook #

Fastify’s built-in Pino logger already logs “incoming request” and “request completed”. Those lines lack anything app-specific, like the user or the route pattern. Turn the default off and write one line per request yourself:

// app.js
import Fastify from 'fastify';
import auth from './plugins/auth.js';
import orders from './routes/orders.js';

const app = Fastify({
  logger: true,
  disableRequestLogging: true,
  requestIdHeader: 'x-request-id',
});

app.addHook('onRequest', async (request, reply) => {
  reply.header('x-request-id', request.id);
});

app.addHook('onResponse', async (request, reply) => {
  request.log.info({
    method: request.method,
    route: request.routeOptions.url ?? 'unmatched',
    status: reply.statusCode,
    ms: Math.round(reply.elapsedTime),
    userId: request.user?.id,
  }, 'request completed');
});

app.addHook('onError', async (request, reply, error) => {
  request.log.error({ err: error, route: request.routeOptions.url }, 'request failed');
});

await app.register(auth);
await app.register(orders);
await app.listen({ port: 3000, host: '0.0.0.0' });

Three Fastify v5 details are in that snippet. reply.getResponseTime() was removed in v5; use reply.elapsedTime. request.routerPath is deprecated; use request.routeOptions.url, which gives /v1/orders/:id instead of the raw URL, so your logs group by endpoint. And the custom-logger option moved: pass your own Pino instance as loggerInstance, not logger.

For carrying the request ID into code that does not receive request, see request IDs and AsyncLocalStorage. Before you add more fields to that log line, read what not to log.

Common Fastify hook mistakes #

Mixing async and done #

Fastify v5 no longer allows mixing the callback and promise styles in hooks and plugins. Either write async (request, reply) => {} or (request, reply, done) => { ...; done() }. A hook that is async and also calls done can run the chain twice.

Sending without returning #

In an async hook, if you call reply.send() from a callback or timer instead of before the function returns, you must return reply. Otherwise Fastify treats the hook as finished and continues to the handler, and you get “Reply was already sent” errors.

Arrow functions and this #

With function syntax, this is the Fastify instance. With arrow functions it is not. Use the app variable you closed over instead of this.

Logging bodies in onRequest #

The body does not exist yet. If you need it in logs, read it in preHandler, and redact first.

Fastify hooks checklist #

• Authentication: onRequest, through a decorator wrapped in fastify-plugin.
• Authorization and resource loading: route preHandler, returning 404 for other people’s objects.
• One log line per request: onResponse, with routeOptions.url, statusCode, elapsedTime and the user ID.
• Errors: onError for context only; change error responses with setErrorHandler.
• Pick async or callback style per hook, never both.

On a one-person project, put all of this in app.js and one auth plugin. On a ten-person team, keep auth and logging in shared plugins so new route files cannot forget them.

FAQ #

Should I use onRequest or preHandler for auth in Fastify? #

Use onRequest for authentication, since it only needs headers and rejects early. Use preHandler for authorization that needs params, the body or a database lookup.

How do I get the response time in Fastify v5? #

Read reply.elapsedTime in onResponse. It is in milliseconds. reply.getResponseTime() was removed in v5.

Why does my Fastify hook run twice? #

Usually because it mixes styles: it is async and also calls done, or it sends a reply outside the promise chain without returning reply. Pick one style and return reply after sending.

Doing this with ApiTraffic #

If you would rather not maintain the onResponse logging yourself, register apitraffic.fastify() as the first plugin and it captures every request and response, including headers and bodies, under the matched route pattern. Credential headers are masked by default.

Keep your auth hook, and add one line after you identify the user, so requests rejected later by a preHandler still carry it:

apitraffic.tag('userId', String(user.id));

The firewall runs in the same plugin as a preHandler hook, so rate limits and access lists apply before your handler without another hook of your own.

Start free and see every request your API handles: apitraffic.io

Sources: Fastify docs: Hooks; Fastify docs: V5 Migration Guide; Fastify docs: Encapsulation

The short answer: a BOLA vulnerability happens when an endpoint loads an object by the ID the client sent without checking that the caller may access it. The fix is to put the ownership or tenant condition into the database query itself, return 404 for objects the caller cannot see, and add a cross-tenant test for every route that takes an ID. Then watch your logs for callers touching many distinct IDs.

What is a BOLA vulnerability? #

BOLA (broken object level authorization) is the API name for IDOR (insecure direct object reference). OWASP ranks it API1:2023, first in the current API Security Top 10. Authentication is usually fine in these cases. The handler checks “who are you” and forgets “is this yours”.

// Express 5. Authenticated, and still broken.
app.get('/v1/invoices/:id', requireAuth, async (req, res) => {
  const invoice = await db.invoice.findUnique({ where: { id: req.params.id } });
  if (!invoice) return res.status(404).json({ error: 'not_found' });
  res.json(invoice);
});

The same mistake in other shapes:

• The list endpoint filters by tenant, but detail, update and delete do not.
• The ID comes from the body ({ "accountId": 77 } in a PATCH) instead of the path.
• A nested route checks the parent but loads the child by its own ID: /projects/:projectId/files/:fileId.
• A bulk endpoint checks only the first ID in an array.
• A new API version or admin route skips the middleware the others use. Route rewrites during an Express 5 migration are a common moment for this.

Why scanners find IDOR so easily #

BOLA is cheap to test. Sequential IDs make enumeration trivial, success is a 200 where a 404 belongs, and the payoff is real data. Two accounts and a script test every route that takes an ID. UUIDs make guessing harder, but IDs leak through URLs, exports, emails and other responses, so random IDs are a speed bump, not an authorization check.

How to fix broken object level authorization in Node.js #

Step 1: scope the query #

Put the ownership condition in the query, so the object cannot load without it:

app.get('/v1/invoices/:id', requireAuth, async (req, res) => {
  const invoice = await db.invoice.findFirst({
    where: { id: req.params.id, accountId: req.user.accountId },
  });
  if (!invoice) return res.status(404).json({ error: 'not_found' });
  res.json(invoice);
});

Step 2: make the scoped path the only path #

A small repository layer that requires a scope removes the chance to forget:

// invoices.repo.js
export function invoicesFor(scope) {
  if (!scope?.accountId) throw new Error('invoicesFor requires an account scope');
  const base = { accountId: scope.accountId };
  return {
    get: (id) => db.invoice.findFirst({ where: { ...base, id } }),
    list: (opts = {}) => db.invoice.findMany({ where: base, ...opts }),
    update: (id, data) => db.invoice.updateMany({ where: { ...base, id }, data }),
  };
}

updateMany with the scope in where returns a count. Zero means the object does not exist or is not the caller’s, and you return 404 either way, which avoids confirming the ID exists. Use 403 when the caller can see the object but not perform the action, like a viewer trying to delete.

Step 3: in Fastify, load the resource in a preHandler #

import Fastify from 'fastify';
const app = Fastify();
app.decorateRequest('invoice', null);

async function loadInvoice(request, reply) {
  const invoice = await invoicesFor(request.user).get(request.params.id);
  if (!invoice) return reply.code(404).send({ error: 'not_found' });
  request.invoice = invoice;
}

app.get('/v1/invoices/:id', { preHandler: [authenticate, loadInvoice] },
  async (request) => request.invoice);

In NestJS the same idea fits a guard, but the scoped query is what makes it hold.

Step 4: test it like an attacker #

Single-user unit tests rarely catch BOLA. Add a cross-tenant test per route:

import { test } from 'node:test';
import assert from 'node:assert/strict';

test('account B cannot read account A invoice', async () => {
  const a = await createAccountWithInvoice();
  const b = await createAccount();
  const res = await fetch(`${BASE}/v1/invoices/${a.invoiceId}`, {
    headers: { authorization: `Bearer ${b.token}` },
  });
  assert.equal(res.status, 404);
});

Generate the route list from your router so a new route without a cross-tenant test fails CI.

How to detect BOLA attacks in request logs #

Probing has a recognizable shape:

1. One caller, many distinct object IDs on one route pattern.
2. Sequential IDs in order from the same token.
3. A high share of 403 or 404 responses on /:id routes for one caller.
4. Mostly 404s, then a run of 200s. A scanner found a route without a check. This is the one to alert on.
5. IDs requested that never appeared in that caller’s list responses.

Your logs need caller identity and route pattern, not just the raw URL. A quick offline check over exported JSON lines:

// enum-check.mjs: node enum-check.mjs requests.jsonl
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const byCaller = new Map();
const rl = createInterface({ input: createReadStream(process.argv[2]) });

for await (const line of rl) {
  const r = JSON.parse(line); // { userId, route, objectId, status }
  if (!r.objectId) continue;
  const key = `${r.userId} ${r.route}`;
  const s = byCaller.get(key) ?? { ids: new Set(), denied: 0, ok: 0 };
  s.ids.add(r.objectId);
  if (r.status === 403 || r.status === 404) s.denied++;
  else if (r.status < 300) s.ok++;
  byCaller.set(key, s);
}
for (const [key, s] of byCaller) {
  if (s.ids.size > 50) console.log(key, { distinct: s.ids.size, denied: s.denied, ok: s.ok });
}

Tune the threshold. Admin users who touch thousands of records need an exception, which is a good reason to tag callers by role. The same distinct-accounts idea powers credential stuffing detection.

For a one-person team, the afternoon version is scoping every /:id query and running this script weekly. A ten-person team should add the repository layer, generated cross-tenant tests in CI and an alert on pattern 4.

BOLA checklist for every route that takes an ID #

If you only do one thing, grep for findUnique, findById and WHERE id = calls that do not include a tenant or owner condition, and fix those first. Then confirm:

• The ownership condition is in the query, not an if after it.
• Body and query-string IDs get the same check as path IDs.
• Nested resources check the child.
• Bulk endpoints check every ID.
• Invisible objects return 404.
• A cross-tenant test exists.
• Logs record caller, route pattern and object ID.

FAQ #

What is the difference between BOLA and IDOR? #

They describe the same flaw. IDOR is the older web application term, and BOLA is the name OWASP uses in the API Security Top 10, where it is API1:2023.

Do UUIDs prevent IDOR vulnerabilities? #

No. They make IDs harder to guess, but IDs leak through links, logs and other responses. Authorization still has to be checked on every request.

Should an unauthorized object return 403 or 404? #

Return 404 when the caller should not know the object exists, which is the usual case across tenants. Return 403 when they can see it but lack permission for the action.

Doing this with ApiTraffic #

If you would rather not build enumeration detection yourself, ApiTraffic covers the logging and the response.

• Requests are recorded under the matched route pattern (for example /api/v1/invoices/{id}), and apitraffic.tag('userId', String(req.user.id)) ties each one to a user. Tags stick even when a guard rejects the request.
• Review one caller with tag:userId == "1042" AND status_code >= 400.
• Behavioral protection scores each caller on distinct object IDs per endpoint, 401/403 counts, 404 counts and distinct endpoints, and gives callers over the threshold a temporary deny on every server. Monitor mode shows who would be denied first.

Start free and see every request your API handles: apitraffic.io

Sources: OWASP API Security Top 10 2023, API1:2023 Broken Object Level Authorization; OWASP Authorization Cheat Sheet

To find N+1 queries, count database queries per request and flag requests where the same query shape runs many times. You can do that with AsyncLocalStorage and a thin wrapper around your database client in about an hour. To fix them, replace the per-item query with one batched query: a JOIN, a WHERE id = ANY($1), an ORM include, or DataLoader.

What N+1 queries look like in Express #

A typical example with node-postgres:

app.get('/orders', async (req, res) => {
  const { rows: orders } = await db.query(
    'select id, customer_id, total from orders where account_id = $1 order by id desc limit 200',
    [req.user.accountId],
  );
  for (const order of orders) {
    const { rows } = await db.query('select name, email from customers where id = $1', [order.customer_id]);
    order.customer = rows[0];
  }
  res.json(orders);
});

With 10 orders in your seed data, that’s 11 queries at maybe 0.3 ms each against a local database: invisible. With 200 orders and a managed database 1 to 2 ms away, it’s 201 round trips and several hundred milliseconds, all spent waiting on the network.

Why N+1 queries hide until production #

• Seed data is small. N is 5 locally and 500 for your largest account.
• Local databases are close. Network round-trip time dominates each query in production, and it’s near zero on localhost.
• Averages smooth it out. Most accounts are small, so mean latency barely moves. The slowdown shows up in p95 and p99, which is why reading latency percentiles properly matters here.
• ORMs hide the loop. A lazy-loaded relation or a resolver per field issues queries you never wrote.

How to detect N+1 queries per request #

Step 1: track queries in request context #

If you already have request context with AsyncLocalStorage, add a query list to the store. Otherwise this is the minimum:

// query-tracking.js
import { AsyncLocalStorage } from 'node:async_hooks';

export const queryContext = new AsyncLocalStorage();

export function trackQueries(req, res, next) {
  const store = { queries: new Map(), total: 0 };
  res.on('finish', () => report(req, res, store));
  queryContext.run(store, next);
}

function normalize(sql) {
  return sql.replace(/\s+/g, ' ').replace(/'[^']*'/g, '?').replace(/\b\d+\b/g, '?').trim();
}

export function recordQuery(sql) {
  const store = queryContext.getStore();
  if (!store) return;
  const key = normalize(sql);
  store.queries.set(key, (store.queries.get(key) ?? 0) + 1);
  store.total += 1;
}

Step 2: wrap your database client #

Route every query through one function so tracking can’t be skipped:

// db.js
import pg from 'pg';
import { recordQuery } from './query-tracking.js';

const pool = new pg.Pool({ connectionString: process.env.DATABASE_URL });

export const db = {
  query(text, params) {
    recordQuery(text);
    return pool.query(text, params);
  },
};

Step 3: log suspicious requests #

// query-tracking.js (continued)
import { logger } from './logger.js';

const REPEAT_THRESHOLD = 10;
const TOTAL_THRESHOLD = 50;

function report(req, res, store) {
  const repeated = [...store.queries].filter(([, n]) => n >= REPEAT_THRESHOLD);
  if (repeated.length === 0 && store.total < TOTAL_THRESHOLD) return;
  logger.warn({
    route: req.route?.path,
    method: req.method,
    status: res.statusCode,
    totalQueries: store.total,
    repeated: repeated.map(([sql, n]) => ({ sql: sql.slice(0, 200), n })),
  }, 'possible N+1');
}

Register app.use(trackQueries) before your routes. Log the route pattern, not the raw URL, so /orders for every account groups into one line. A day of these warnings gives you a ranked list of endpoints to fix.

Using your ORM’s query log #

If you use an ORM, its query log gives you the same signal:

Hooking those into recordQuery means the same per-request report works regardless of ORM.

Catch N+1 queries in tests #

Production detection finds the ones you already shipped. A test stops new ones:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import request from 'supertest';
import { app } from '../app.js';
import { seedOrders } from './seed.js';
import { lastRequestStats } from '../query-tracking.js';

test('GET /orders does not scale queries with order count', async () => {
  await seedOrders({ count: 50 });
  await request(app).get('/orders').set('authorization', 'Bearer test').expect(200);
  assert.ok(lastRequestStats().total <= 3, `ran ${lastRequestStats().total} queries`);
});

lastRequestStats() is a small test-only export that keeps the most recent store. Seed more rows than a developer would by hand; an N+1 is obvious at 50 and invisible at 3.

How to fix N+1 queries #

Fetch the related rows in one query, then join in memory:

const { rows: orders } = await db.query(
  'select id, customer_id, total from orders where account_id = $1 order by id desc limit 200',
  [req.user.accountId],
);
const ids = [...new Set(orders.map((o) => o.customer_id))];
const { rows: customers } = await db.query(
  'select id, name, email from customers where id = any($1)',
  [ids],
);
const byId = new Map(customers.map((c) => [c.id, c]));
for (const o of orders) o.customer = byId.get(o.customer_id);

Two queries instead of 201, whatever the account size. node-postgres converts the JavaScript array to a Postgres array for any($1). Other options:

• A JOIN when you always need the related data and the result set stays small.
• ORM eager loading, such as Prisma include or Sequelize include, instead of lazy access inside a loop.
• DataLoader for GraphQL resolvers or deeply nested code, where per-item lookups are hard to avoid. It batches all load(id) calls in one tick into a single fetch.

If you only do one thing #

On a 1-person team: add the per-request query counter and read the warnings after a day of traffic. Fix the top endpoint. On a 10-person team: also add the query-count test to the endpoints that list things, so the next N+1 fails in CI instead of in production.

FAQ #

What is an N+1 query? #

One query loads a list of N rows, then N more queries load related data for each row. The total grows with the size of the list, so it gets slower as customers grow.

Does Prisma cause N+1 queries? #

include loads relations with a fixed number of queries rather than one per row. N+1 appears when you call findUnique inside a loop or resolve relations per item in GraphQL resolvers.

How many queries per request is too many? #

There’s no universal number, but a count that grows with result size is the warning sign. A list endpoint that runs the same query shape 10 or more times per request is worth a look.

Doing this with ApiTraffic #

If you’d rather not build query tracking yourself, run your app with the APM preload, node -r @apitraffic/node/register ./server.js, so OpenTelemetry instruments your database driver before it loads. Each query is recorded as a span and attributed to the request that issued it.

Top Operations ranks queries by calls per request and flags N+1 patterns, with the code location that issued them. The request waterfall shows every query on the request’s timeline, so 200 identical rows stand out at a glance, and the endpoint’s Time Breakdown shows how much of its time goes to the database.

Start free and find the N+1 queries in your API: apitraffic.io

Sources: node-postgres docs: queries; Prisma docs: logging; DataLoader README

The direct answer: Express 5 needs Node 18 or newer, changes route path syntax (named wildcards, no ? or regex in strings), makes req.query read-only, leaves req.body undefined when nothing parsed it, validates res.status(), removes old method signatures, and forwards rejected promises to your error handler. Most APIs migrate in an afternoon; the route syntax is where the time goes.

Why migrate to Express 5 now? #

Express 5.0 shipped in September 2024 after a decade on 4.x, 5.1 followed in March 2025 and became the latest tag on npm, and the current release is 5.2.1. Express 4 moved to maintenance (security and high-priority fixes only) on April 1, 2025, and the Express team’s proposed schedule puts its end of life no sooner than October 1, 2026. Not an emergency, but the clock is running. The reasons to move are practical: async error handling without wrappers, a stricter path matcher that rejects ambiguous patterns at startup, and new middleware and tutorials targeting 5.

Check your runtime first. Express 5 requires Node 18+, and Node 20 reaches end of life on April 30, 2026. If you are moving anyway, Node 24 LTS is the sensible target.

Step-by-step Express 5 migration plan #

1. Upgrade and run the codemods #

npm install express@5
npm install --save-dev @types/express@5   # TypeScript only
npx @expressjs/codemod upgrade

The official codemod rewrites the mechanical changes: res.redirect('back'), pluralized req.accepts* methods, the old res.json(obj, status) style signatures and req.param(). It does not fix route paths.

2. Find the route patterns that will throw #

Search for patterns path-to-regexp v8 rejects:

grep -rnE "\.(get|post|put|patch|delete|all|use)\(\s*['\"][^'\"]*(\*|\?|\(|\))" src/

Express 5 throws at registration time, so the app will not even start until they are fixed. That is the good news: nothing slips through to production. Here is what failed when registering common Express 4 patterns on 5.2:

THROW /*               -> Missing parameter name at index 2: /*
THROW /users/:id?      -> Unexpected ? at index 10: /users/:id?
THROW /files/:path(*)  -> Unexpected ( at index 12: /files/:path(*)
THROW /api/v(1|2)/x    -> Unexpected ( at index 6: /api/v(1|2)/x

3. Rewrite them #

Wildcard params are now arrays: /*splat on /a/b/c gives req.params.splat as ['a', 'b', 'c']. Code that did req.params[0] needs req.params.splat.join('/'). Optional parameters that are missing are omitted from req.params, not set to undefined.

What breaks at runtime after the Express 5 upgrade #

These do not crash at startup, which makes them the dangerous ones.

req.query is now a getter #

// Express 4 pattern that silently stops working
app.use((req, res, next) => {
  req.query.limit ??= '50'; // lost: req.query is re-parsed on every access
  next();
});

In ESM (strict mode), req.query = {...} throws Cannot set property query of #<IncomingMessage> which has only a getter. Mutating a property does not throw; the change just disappears. Put normalized values somewhere you own:

app.use((req, res, next) => {
  res.locals.page = {
    limit: Math.min(Number(req.query.limit ?? 50), 200),
    cursor: req.query.cursor ?? null,
  };
  next();
});

The default query parser is “simple” #

?filter[status]=open used to give { filter: { status: 'open' } }. In Express 5 it gives { 'filter[status]': 'open' }. If your API uses bracket syntax, opt back in with app.set('query parser', 'extended'). Same story for forms: express.urlencoded() now defaults to extended: false.

req.body is undefined, not {} #

When no body parser ran (no body, wrong Content-Type), req.body is undefined. Code like const { email } = req.body now throws a TypeError, which becomes a 500. Validate the body, or default it: const { email } = req.body ?? {}.

res.status() validates its argument #

res.status('201') throws Invalid status code: "201". Status code must be an integer. Values must be integers from 100 to 999. Watch for status codes read from config, upstream responses or error objects as strings.

Rejected promises go to your error handler #

app.get('/v1/orders/:id', async (req, res) => {
  const order = await db.orders.find(req.params.id); // a throw here...
  res.json(order);
});

app.use((err, req, res, next) => {                  // ...lands here
  req.log?.error({ err }, 'unhandled route error');
  res.status(err.status ?? 500).json({ error: 'internal_error' });
});

You can delete express-async-errors and hand-written asyncHandler wrappers. Check that your error handler sets a sensible status, because errors that used to become unhandled rejections now produce responses.

Smaller changes worth a search #

• app.del() is gone; use app.delete().
• res.sendfile() is now res.sendFile().
• express.static ignores dotfiles by default; serve /.well-known explicitly with dotfiles: 'allow'.
• req.host keeps the port.
• app.listen passes errors such as EADDRINUSE to its callback instead of throwing.

Express 5 migration checklist #

Forward this to whoever owns the upgrade:

• [ ] Node 18+ everywhere (ideally Node 22 or 24).
• [ ] npx @expressjs/codemod upgrade run and reviewed.
• [ ] App starts: every route pattern registers.
• [ ] No writes to req.query; bracket queries still parse.
• [ ] No unguarded destructuring of req.body.
• [ ] Every res.status() receives an integer.
• [ ] Error handler returns a sane status for async errors.
• [ ] Deployed to staging with real traffic before production.

A solo developer can do this in one sitting. A team of ten should split it: one person on route patterns, one on query and body handling, and a staging soak with production-like traffic before the cutover, ideally after you have graceful shutdown in place so the deploy itself does not drop requests.

FAQ #

Is Express 5 backwards compatible with Express 4? #

Mostly, but not fully. Most middleware works unchanged, while route path syntax, req.query, req.body defaults and several removed methods need changes. Expect your app to fail at startup until route patterns are updated.

How do I fix “Missing parameter name” in Express 5? #

Wildcards must be named. Replace /* with /*splat (or /{*splat} to also match the root) and read the matched segments from req.params.splat, which is an array.

Do I still need express-async-errors in Express 5? #

No. Express 5 forwards rejected promises and errors thrown in async handlers to your error middleware, as if you had called next(err).

Doing this with ApiTraffic #

The scary part of the Express 5 migration is the runtime changes that do not crash. If you would rather not eyeball logs to find them, ApiTraffic captures every request and response from one middleware placed after your body parsers, and records them under the matched route pattern, so a route that stopped matching shows up as traffic moving to 404.

app.use(express.json());
app.use(apitraffic.express());

After the deploy, search for status_code >= 500 per endpoint, or record a deploy event and use Regressions to compare each endpoint’s p95 before and after the upgrade.

Start free and watch your Express 5 upgrade request by request: apitraffic.io

Sources: Express: Moving to Express 5; Express 5.1.0 release announcement (March 2025); @expressjs/codemod; path-to-regexp README; Node.js release schedule.

The short version: pick two service level indicators for your API (the share of requests that do not fail with a 5xx, and the share that finish under a latency threshold), set a target over a rolling 28 days, say 99.5%, and treat the remaining 0.5% as an error budget you are allowed to spend. When the budget is healthy, ship. When it is nearly gone, the next week’s work is reliability. Everything below is how to do that in an afternoon with the request logs you already have.

What is an error budget, in one paragraph? #

A service level objective (SLO) is a target for a measurable behavior, such as “99.5% of API requests succeed over 28 days”. The error budget is the 0.5% you did not promise. If you serve 2,000,000 requests in 28 days, the budget is 10,000 failed requests. Measured in time, 0.5% of 28 days is about 3 hours 22 minutes of total outage. The budget is not a goal to hit; it is permission to take risks (deploys, migrations, experiments) until it runs low.

For a team of one to ten running on a single cloud region and a managed database, 99.9% is ambitious; your hosting provider’s own incidents can use that up. Start at 99.5% and tighten it once you have a quarter of data.

How to choose SLIs for a Node API #

An SLI (service level indicator) is a ratio: good events divided by valid events. Two are enough for most APIs.

Availability SLI #

Good: responses with a status below 500. Valid: all requests, minus the ones you exclude. Exclude:

• health checks and readiness probes (they would inflate the denominator; see our post on liveness vs readiness checks)
• requests your firewall or rate limiter rejected on purpose
• most 4xx: a 404 for a bad URL or a 401 for a wrong password is the client’s problem

Count 429 from your own limiter as excluded, but count 408, 502, 503 and 504 as bad. A 499 (client closed the connection) is a judgment call; if it happens because your API is slow, the latency SLI will catch it.

Latency SLI #

Good: requests that finished under a threshold. Valid: the same set as above. Pick the threshold per group of endpoints, not one number for everything. A GET /v1/orders/:id might have a 300 ms threshold, and a POST /v1/reports that builds a PDF might have 3 seconds.

Phrase it as a proportion (“95% of requests under 300 ms”), not as “p95 under 300 ms”. The two sound alike, but the proportion is additive across time windows and servers, and percentiles are not. Our post on reading p50, p95 and p99 without fooling yourself explains why averaging percentiles goes wrong.

Calculate your SLO and error budget from request logs #

If your app writes one JSON line per request with a route pattern, status and duration, you already have everything. This script reads NDJSON logs and reports both SLIs and the budget left:

// slo-report.mjs  -- usage: node slo-report.mjs requests.ndjson
import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const TARGET = 0.995;                       // 99.5% over the window
const LATENCY_MS = { default: 300, 'POST /v1/reports': 3000 };
const EXCLUDE = [/^GET \/healthz/, /^GET \/readyz/];

let valid = 0, failed = 0, slow = 0;

const rl = createInterface({ input: createReadStream(process.argv[2]) });
for await (const line of rl) {
  if (!line.trim()) continue;
  const r = JSON.parse(line);              // { method, route, status, durationMs }
  const key = `${r.method} ${r.route}`;
  if (EXCLUDE.some((re) => re.test(key)) || r.status === 429) continue;
  valid++;
  if (r.status >= 500) failed++;
  if (r.durationMs > (LATENCY_MS[key] ?? LATENCY_MS.default)) slow++;
}

const budget = valid * (1 - TARGET);
const report = (name, bad) => {
  const sli = 1 - bad / valid;
  const left = 1 - bad / budget;
  console.log(`${name}: SLI ${(sli * 100).toFixed(3)}%, budget left ${(left * 100).toFixed(1)}%`);
};
console.log(`valid requests: ${valid}`);
report('availability', failed);
report('latency', slow);

Run it over the last 28 days of logs once a week. The output is two lines, which is the right amount of ceremony for a small team.

Use the route pattern (/v1/orders/:id), not the raw URL, or every order ID becomes its own row.

Burn rate: when to worry before the budget is gone #

Burn rate is how fast you are spending the budget relative to plan. A burn rate of 1 spends exactly the whole budget over the window; a burn rate of 10 spends it in a tenth of the time. Google’s SRE workbook suggests alerting on combinations like these for a 30-day window:

The short window makes the alert stop soon after the problem does. For a team without a formal on-call rotation, the first row alone is a reasonable start: it fires when something is badly broken right now, and stays quiet for a single slow request at 3am.

Write an error budget policy in five lines #

The SLO only changes behavior if you agree in advance what happens when the budget runs low. Keep it short enough to fit in the README:

1. Budget above 50%: ship normally.
2. Budget between 0% and 50%: risky changes (schema migrations, dependency majors) need a second reviewer and a rollback plan.
3. Budget exhausted: the next week of planned work is reliability fixes from the incidents that spent it.
4. One incident that spends more than 20% of the budget gets a short written review.
5. Review targets once a quarter.

For a one-person team, this is a note to yourself that makes “should I ship the Postgres upgrade on Friday” an easy no. For a ten-person team, it is the thing that lets the lead say “this week is reliability” without a debate.

If you only do one thing #

• Pick one availability SLI: non-5xx share of valid requests, excluding health checks.
• Set a 99.5% target over 28 days.
• Run a weekly report and post the budget left in your team chat.
• Agree on rule 3 above before you need it.

FAQ #

What is a good SLO for a REST API? #

For a small team on managed infrastructure, 99.5% availability over 28 days is a sensible first target, with a latency SLO like 95% of reads under 300 ms. Set it from your last month’s data, not from what sounds impressive.

Should 4xx errors count against the error budget? #

Usually not. A 4xx means the client sent something wrong. The exceptions are errors your code produces by mistake, like a 400 from a validation bug after a deploy; watch for sudden 4xx jumps separately.

What is the difference between an SLO and an SLA? #

An SLA is a contract with consequences, such as credits. An SLO is your internal target, and it should be stricter than any SLA so you have room to react before you owe anyone money.

Doing this with ApiTraffic #

If you would rather not maintain the logging and the script, ApiTraffic records every request under its matched route pattern, with status and latency, from one middleware. Endpoint details show requests and p50, p95 and p99 per endpoint over time, which is the raw material for both SLIs.

When the budget drops, the request log answers “which requests spent it”: status_code >= 500 narrowed by url_path, or tag:userId == "1042" AND status_code >= 400 for one customer’s complaint (see the request search language). Regressions compares p95 before and after a deploy event, which is usually the next question.

Start free and see every request your API handles: apitraffic.io

Sources: Google SRE Workbook, “Alerting on SLOs” ; Google SRE Workbook, “Implementing SLOs” ; Google SRE Book, “Service Level Objectives”

Credential stuffing detection comes down to looking at login attempts in aggregate instead of one at a time. The strongest signals are a rising failure ratio, a spike in failures for accounts that do not exist, and single sources trying many different accounts. This post shows what to log on your login route to see those signals, plus the code and queries that turn them into an alert.

What is credential stuffing, and why is it hard to detect? #

Brute force hammers one account with many passwords. Credential stuffing tries leaked email and password pairs from other breaches, usually one or two attempts per account, spread across many IPs. Per-account lockouts barely trigger, and per-IP limits only catch the clumsy operators.

It is common. Verizon’s 2025 DBIR research found that in the median case credential stuffing made up 19% of SSO authentication attempts in its data, 12% for small businesses, with a single-day peak of 44%. The same research found that in the median case only 49% of a user’s passwords across services were distinct. You do not need to be a big target. The attacker’s list picks you.

OWASP files this as automated threat OAT-008, and its Credential Stuffing Prevention Cheat Sheet is the best reference for defenses. This post covers detection, because you cannot tune a defense you cannot see.

Credential stuffing signals in request logs #

The last row is the one that matters most. A handful of 200s from the IP ranges that produced thousands of 401s are compromised accounts.

How to log login attempts for credential stuffing detection #

Most of these signals need fields a default access log does not have. This Express 5 handler records an outcome you can aggregate, without logging the password or the raw email.

import express from 'express';
import { createHmac } from 'node:crypto';
import pino from 'pino';

const log = pino();
const app = express();
app.use(express.json({ limit: '10kb' }));
app.set('trust proxy', 1); // only behind exactly one proxy you control

const PEPPER = process.env.LOGIN_LOG_PEPPER;
const accountKey = (email) =>
  createHmac('sha256', PEPPER)
    .update(String(email).trim().toLowerCase())
    .digest('hex')
    .slice(0, 16);

app.post('/v1/auth/login', async (req, res) => {
  const { email, password } = req.body ?? {};
  const user = await findUserByEmail(email);
  const ok = user ? await verifyPassword(user, password) : false;
  const outcome = !user ? 'unknown_user' : ok ? 'success' : 'bad_password';

  log.info({
    event: 'login_attempt',
    outcome,
    account: accountKey(email ?? ''),
    ip: req.ip,
    ua: req.get('user-agent') ?? 'none',
    hasClientHeader: Boolean(req.get('x-client-version')),
  });

  if (!ok) return res.status(401).json({ error: 'invalid_credentials' });
  res.json({ token: await issueToken(user) });
});

The response is identical for an unknown user and a wrong password, so attackers cannot use your API to validate their list. Internally the log tells them apart, because “unknown user” is your strongest signal. Make sure verifyPassword still does a hash comparison when the user does not exist, or timing gives the answer away.

The HMAC with a pepper lets you count distinct accounts per IP without putting emails in logs. More on keeping personal data out of logs in what not to log.

Detecting credential stuffing in code #

A small in-process detector flags a source that tries too many distinct accounts. This is a single-instance sketch. With several instances, keep the same structure in Redis.

const WINDOW_MS = 10 * 60 * 1000;
const MAX_DISTINCT_ACCOUNTS = 20;
const sources = new Map(); // ip -> { accounts, failures, first }

export function recordAttempt({ ip, account, outcome }) {
  const now = Date.now();
  let s = sources.get(ip);
  if (!s || now - s.first > WINDOW_MS) {
    s = { accounts: new Set(), failures: 0, first: now };
    sources.set(ip, s);
  }
  s.accounts.add(account);
  if (outcome !== 'success') s.failures++;
  const suspicious =
    s.accounts.size > MAX_DISTINCT_ACCOUNTS && s.failures / s.accounts.size > 0.8;
  return { suspicious, distinct: s.accounts.size };
}

setInterval(() => {
  const cutoff = Date.now() - WINDOW_MS;
  for (const [ip, s] of sources) if (s.first < cutoff) sources.delete(ip);
}, 60_000).unref();

When suspicious is true, log and alert before you block. Stuffing often comes through residential proxies, where one IP can also carry real users.

SQL queries to find credential stuffing #

If your logs land in a queryable login_attempts table, ask two questions. Failure ratio by hour:

SELECT date_trunc('hour', ts) AS hour,
       count(*) FILTER (WHERE outcome <> 'success')::float
         / NULLIF(count(*) FILTER (WHERE outcome = 'success'), 0) AS fail_ratio,
       count(*) FILTER (WHERE outcome = 'unknown_user') AS unknown_users
FROM login_attempts
WHERE ts > now() - interval '7 days'
GROUP BY 1 ORDER BY 1;

Sources trying many accounts:

SELECT ip, count(DISTINCT account) AS accounts, count(*) AS attempts,
       count(*) FILTER (WHERE outcome = 'success') AS successes
FROM login_attempts
WHERE ts > now() - interval '1 hour'
GROUP BY ip
HAVING count(DISTINCT account) > 20
ORDER BY accounts DESC;

The successes column is your incident list: lock those accounts, force a reset, and review their activity since the login.

How to stop credential stuffing once you see it #

• Step up successful logins from suspicious sources to a second factor or email confirmation instead of blocking the source.
• Rate limit the login route per IP and per account, plus a global ceiling on unknown-user failures. Rate limiting algorithms compared covers which one to pick.
• Check new passwords against known breached passwords at signup and reset.
• Require MFA for admin and API key management.

For a one-person team, the afternoon version is the logging change, the second SQL query as a daily scheduled job, and a login rate limit. A ten-person team should add the in-process or Redis detector, alerting on the failure ratio, and a written runbook for locking compromised accounts.

If you only do one thing #

Log every login attempt with an outcome of success, bad_password or unknown_user, an HMAC of the account and the client IP. Every other step on this page depends on it.

FAQ #

How do I know if my API is under a credential stuffing attack? #

Compare today’s login failure ratio and unknown-account failures with your normal baseline, and look for single IPs or networks trying many distinct accounts. A sudden jump in any of those, especially with a few successes from the same sources, is the pattern.

What is the difference between credential stuffing and brute force? #

Brute force guesses many passwords for one account. Credential stuffing tries one known password per account across many accounts, which is why per-account lockouts rarely catch it.

Does rate limiting stop credential stuffing? #

It slows it down but rarely stops it, because attackers spread requests across many IPs. Combine rate limits with detection on distinct accounts per source and MFA.

Doing this with ApiTraffic #

If you would rather not build this pipeline yourself, ApiTraffic captures every request to your login route from one middleware, with Authorization, Cookie and X-Api-Key masked by default. Add a redaction rule for the password field in the dashboard and it is pushed to your servers without a deploy.

• Search for bursts: url_path == "/v1/auth/login" AND status_code == 401.
• Behavioral protection scores each caller on signals including 401/403 counts and distinct endpoints, and gives callers over the threshold a temporary deny on every server.
• A rate limit on the login path returns 429 with Retry-After, and access lists can deny by IP, CIDR, country or ASN. Monitor mode shows what a policy would block first.

Start free and see every request your API handles: apitraffic.io

Sources: Verizon, “Additional 2025 DBIR research on credential stuffing” (2025); OWASP Credential Stuffing Prevention Cheat Sheet; OWASP Automated Threats, OAT-008 Credential Stuffing

The quickest way to log outbound HTTP requests in Node.js without touching every call site is node:diagnostics_channel. Node’s global fetch is built on undici, which publishes undici:request:* events; the node:http client publishes http.client.* events, which covers axios and most older SDKs. Subscribe once at startup, record host, method, path, status and duration, and you get one log line per outbound call. This post builds that subscriber, then covers what to do with the data. To turn the same events into per-vendor p95, error rate and throttling numbers, see measuring third-party API latency.

What to record when you log outbound HTTP requests #

Leave out request and response bodies by default, and never log the outbound Authorization header: it’s your credential with that provider. The rules from redacting sensitive data in logs apply to outbound logs too.

How to log fetch requests with diagnostics_channel #

Undici publishes undici:request:create before dispatch, undici:request:headers when response headers arrive, undici:request:trailers when the body completes, and undici:request:error on failure. The same request object is passed to each, so a WeakMap can hold the start time.

// outbound-log.js
import diagnosticsChannel from 'node:diagnostics_channel';
import { logger } from './logger.js';
import { getRequestId } from './context.js';

const started = new WeakMap();

function pathOnly(path) {
  const i = path.indexOf('?');
  return i === -1 ? path : path.slice(0, i);
}

diagnosticsChannel.subscribe('undici:request:create', ({ request }) => {
  started.set(request, { t: performance.now(), requestId: getRequestId() });
});

diagnosticsChannel.subscribe('undici:request:headers', ({ request, response }) => {
  const s = started.get(request);
  if (!s) return;
  s.status = response.statusCode;
  s.ttfbMs = Math.round(performance.now() - s.t);
});

diagnosticsChannel.subscribe('undici:request:trailers', ({ request }) => {
  const s = started.get(request);
  if (!s) return;
  logger.info({
    outbound: true,
    host: new URL(request.origin).host,
    method: request.method,
    path: pathOnly(request.path),
    status: s.status,
    ttfbMs: s.ttfbMs,
    durationMs: Math.round(performance.now() - s.t),
    requestId: s.requestId,
  }, 'outbound request');
  started.delete(request);
});

diagnosticsChannel.subscribe('undici:request:error', ({ request, error }) => {
  const s = started.get(request);
  logger.warn({
    outbound: true,
    host: new URL(request.origin).host,
    method: request.method,
    path: pathOnly(request.path),
    error: error.code ?? error.name,
    durationMs: s ? Math.round(performance.now() - s.t) : undefined,
    requestId: s?.requestId,
  }, 'outbound request failed');
  started.delete(request);
});

Import it once, before your app starts making calls:

// server.js
import './outbound-log.js';
import { app } from './app.js';
app.listen(3000);

Three details matter. Subscribers run synchronously inside undici, so keep them fast and make sure they can’t throw. The trailers event only fires once the response body has been consumed; if your code never reads a body, you’ll get the headers event but no completion line, which is itself a bug worth finding. And the requestId comes from the AsyncLocalStorage request ID setup, captured at create time while the inbound request’s context is active.

How to log axios and http.request calls #

Libraries built on node:http, including axios’s Node adapter, don’t go through undici. Node publishes http.client.request.start, http.client.response.finish and http.client.request.error for those:

const httpStarted = new WeakMap();

diagnosticsChannel.subscribe('http.client.request.start', ({ request }) => {
  httpStarted.set(request, { t: performance.now(), requestId: getRequestId() });
});

diagnosticsChannel.subscribe('http.client.response.finish', ({ request, response }) => {
  const s = httpStarted.get(request);
  if (!s) return;
  logger.info({
    outbound: true,
    host: request.host,
    method: request.method,
    path: pathOnly(request.path),
    status: response.statusCode,
    durationMs: Math.round(performance.now() - s.t),
    requestId: s.requestId,
  }, 'outbound request');
  httpStarted.delete(request);
});

The Node.js diagnostics_channel docs mark the built-in HTTP channels and the undici channels as experimental, so pin a test that asserts a log line appears for a known call. If a Node upgrade changes an event shape, you’ll find out in CI.

The wrapper alternative #

If diagnostics channels feel too low-level, a wrapper around fetch works when all calls go through your code:

export async function loggedFetch(url, init) {
  const t = performance.now();
  const u = new URL(url);
  try {
    const res = await fetch(url, init);
    logger.info({ outbound: true, host: u.host, path: u.pathname, status: res.status,
      durationMs: Math.round(performance.now() - t) }, 'outbound request');
    return res;
  } catch (err) {
    logger.warn({ outbound: true, host: u.host, path: u.pathname, error: err.name }, 'outbound request failed');
    throw err;
  }
}

The catch is coverage: vendor SDKs make their own calls and won’t use your wrapper. For a 1-person team with two dependencies, the wrapper is fine. A 10-person team with a dozen SDKs should use the channels.

What to look for in outbound logs #

Once a few days of outbound logs exist, these questions are worth answering per host (for per-vendor metrics, timeouts and alerting built on them, see third-party API latency per vendor):

• Error rate by host. A provider at 2% 5xx explains a lot of “random” failures in your own error rate.
• p95 duration by host. Compare it with the timeout you set. If p95 is 4 seconds and your timeout is 5, you’re one bad day from cascading timeouts. Timeouts with fetch and AbortSignal covers setting them.
• 429s and Retry-After. Retrying faster than the provider allows makes the limit last longer.
• Calls per inbound request. Ten calls to the same host for one inbound request usually means a loop that should be a batch call.
• Connect errors vs response errors. ENOTFOUND and connect timeouts point at DNS or network; 5xx points at the provider.
• Time of day. Provider slowdowns often line up with their batch jobs or your traffic peaks. A chart of p95 per host per hour makes the pattern obvious, and gives you something concrete to send their support team.

If you only do one thing #

Add the undici subscriber above, log host, path, status and duration, and look at p95 and error rate per host once a week. That alone turns “the API is flaky” into “the email provider’s p95 tripled on Tuesday”.

FAQ #

Does Node’s fetch log requests by default? #

No. Node’s fetch doesn’t log anything. You need a diagnostics channel subscriber, a wrapper or an instrumentation library such as OpenTelemetry’s undici instrumentation.

Will diagnostics_channel slow down my outbound calls? #

When nobody subscribes, publishing costs almost nothing. With a subscriber, the cost is whatever your handler does, so keep it to building a small object and handing it to an async logger.

How do I log outbound request bodies safely? #

Do it per host and per path, with an allowlist of fields, and only while debugging a specific integration. Bodies sent to providers often contain personal data and credentials.

Doing this with ApiTraffic #

If you’d rather not build and maintain this, ApiTraffic logs outbound requests alongside inbound ones from the same SDK. Search them like any other traffic, for example direction == "out" AND status_code >= 500 for failing calls to providers.

With the APM preload (node -r @apitraffic/node/register ./server.js), outbound fetch calls are recorded as spans inside the inbound request that started them, so a request’s waterfall shows exactly which third-party call it was waiting on, and endpoint details show p50, p95 and p99 over time.

Start free and see every call your API makes and receives: apitraffic.io

Sources: Node.js docs: diagnostics_channel, built-in HTTP channels; undici Diagnostics Channel docs; OpenTelemetry undici instrumentation on npm

This post covers the defaults Node and undici actually use, code for outbound and inbound timeouts in Express and Fastify, and how to pick the numbers.

What are the default timeouts in Node.js fetch and undici? #

A vendor that accepts the connection and then stalls holds your handler, your client’s connection and any database connection you have checked out for minutes.

Inbound, Node’s http.Server defaults are requestTimeout 300 seconds (since Node 18), headersTimeout the lower of 60 seconds and requestTimeout, and keepAliveTimeout 5 seconds. Fastify sets requestTimeout and connectionTimeout to 0 (no limit) and keepAliveTimeout to 72 seconds.

How to set a Node.js fetch timeout with AbortSignal #

const res = await fetch('https://api.vendor.example/v1/quotes', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify(payload),
  signal: AbortSignal.timeout(3_000),
});
const data = await res.json();

The signal covers the whole exchange, including reading the body, as long as you read it before the timer fires. On timeout, fetch rejects with a DOMException named 'TimeoutError'; a manual controller.abort() gives 'AbortError':

try {
  await fetch(url, { signal });
} catch (err) {
  if (err.name === 'TimeoutError') { /* our deadline */ }
  else if (err.name === 'AbortError') { /* cancelled, e.g. client went away */ }
  else if (err.cause?.code) { /* ECONNREFUSED, UND_ERR_CONNECT_TIMEOUT, ... */ }
  throw err;
}

Network failures from built-in fetch arrive as TypeError: fetch failed with the real reason in err.cause. Log err.cause.code; the top-level message tells you nothing.

Cancel outbound calls when the client disconnects #

If your caller has gone, stop waiting on the vendor for them. AbortSignal.any() (Node 20.3+) aborts when any of its signals does:

app.get('/quote', async (req, res) => {
  const clientGone = new AbortController();
  res.on('close', () => {
    if (!res.writableFinished) clientGone.abort();
  });

  const signal = AbortSignal.any([clientGone.signal, AbortSignal.timeout(3_000)]);
  const upstream = await fetch('https://api.vendor.example/v1/quotes', { signal });
  res.json(await upstream.json());
});

Listen on res for close, not req: in current Node versions req emits close once the body has been read, which happens on every normal request.

Deadline budgets across several calls #

Three calls with 3-second timeouts can take 9 seconds. If the endpoint should answer within 5, give each call what is left of a shared budget:

export function deadline(ms, parent) {
  const end = performance.now() + ms;
  return {
    remaining: () => Math.max(0, end - performance.now()),
    signal(capMs = Infinity) {
      const t = AbortSignal.timeout(Math.max(1, Math.min(capMs, end - performance.now())));
      return parent ? AbortSignal.any([parent, t]) : t;
    },
  };
}

app.get('/checkout/summary', async (req, res) => {
  const d = deadline(5_000);
  const cart = await fetch(cartUrl, { signal: d.signal(2_000) }).then((r) => r.json());
  const tax = await fetch(taxUrl, { signal: d.signal(2_000) }).then((r) => r.json());
  const ship = d.remaining() > 500
    ? await fetch(shipUrl, { signal: d.signal() }).then((r) => r.json())
    : null; // skip the optional call rather than blow the budget
  res.json({ cart, tax, ship });
});

How to change undici timeout defaults #

AbortSignal sets a deadline per call. To change connection-level defaults for every call, install undici and configure an Agent:

import { Agent, fetch, setGlobalDispatcher } from 'undici';

const agent = new Agent({
  connectTimeout: 2_000,
  headersTimeout: 10_000,
  bodyTimeout: 10_000,
});

setGlobalDispatcher(agent); // also affects global fetch()

const res = await fetch(url, { dispatcher: agent, signal: AbortSignal.timeout(3_000) });

The npm package and the copy bundled in Node can be different versions. Pin the package to the major version your Node bundles, or use undici’s fetch consistently. Undici’s timeout errors carry codes such as UND_ERR_CONNECT_TIMEOUT, UND_ERR_HEADERS_TIMEOUT and UND_ERR_BODY_TIMEOUT.

Server-side request timeouts in Express and Fastify #

Outbound timeouts protect you from dependencies; inbound ones protect you from clients.

const server = app.listen(3000);
server.requestTimeout = 30_000;   // whole request within 30s
server.headersTimeout = 10_000;   // headers within 10s
server.keepAliveTimeout = 65_000; // longer than the load balancer's idle timeout

// Fastify
const app = Fastify({ requestTimeout: 30_000, keepAliveTimeout: 65_000 });

The keepAliveTimeout line prevents a classic intermittent 502. If the load balancer keeps idle connections open longer than Node does, Node can close one just as the load balancer sends a request on it. AWS Application Load Balancers default to a 60-second idle timeout, hence 65 above; check your own proxy or platform rather than copying the number. These settings also interact with graceful shutdown: your drain deadline should cover your longest allowed request.

For databases, set the client’s own limits too: with pg, statement_timeout and the pool’s connectionTimeoutMillis; for Redis, connect and command timeouts. For vendor SDKs, check whether they accept a signal or timeout option.

How to pick timeout values #

• Per call: a little above the dependency’s observed p99. If a vendor’s p99 is 800 ms, 2 seconds loses almost nothing. Read p95 and p99 correctly before you pick.
• Per endpoint: what your caller tolerates. If the frontend gives up at 10 seconds, finishing at 11 helps nobody.
• Retries: idempotent calls only, with backoff, inside the endpoint’s budget.
• Nesting: client > load balancer > handler > each dependency.

A solo developer can start with AbortSignal.timeout(5_000) on every outbound call and the three server settings above. A 10-person team should put a deadline() helper and a configured Agent in a shared module so every service uses the same defaults.

If you only do one thing #

• grep -rn "fetch(" src/ and add signal: AbortSignal.timeout(...) to every call that lacks one.
• Log err.cause?.code when fetch fails.
• Set keepAliveTimeout above your load balancer’s idle timeout.
• Make your readiness check use its own short timeout, so a slow dependency cannot hang it.

FAQ #

Does Node.js fetch have a default timeout? #

Not an overall one. Undici, which powers it, defaults to a 10-second connect timeout and 300 seconds each for headers and between body chunks. Add AbortSignal.timeout() to set a real deadline.

What is the difference between TimeoutError and AbortError? #