API log retention and cost: how much request logging a small team needs

Two kinds of logging surprise hit small teams. The first is the bill, when log storage quietly becomes one of your bigger infrastructure costs. The second is the support ticket about a failed payment from six weeks ago, when your logs only go back seven days.

The short answer for API log retention: keep full request logs, including bodies, for 7 to 14 days; keep request metadata without bodies for 30 to 90 days; keep security and audit events for 12 months; and delete everything else on a schedule. Sample fast, successful reads if volume is a problem, but never sample errors. The rest of this post shows how to size that and set it up in an afternoon.

Why API log retention matters for small teams #

Retention is a trade between three things: the questions you need to answer, what storing data costs, and what you are allowed to keep. Keeping everything forever fails the last two. Keeping a week fails the first.

Under GDPR, the storage limitation principle in Article 5(1)(e) says personal data should be kept no longer than necessary for its purpose, and request logs usually contain personal data such as IPs, user IDs and emails in bodies. If you handle card data, PCI DSS requirement 10.5.1 asks for audit log history for at least 12 months, with the most recent three months immediately available for analysis. Most small API teams are not in PCI scope, but it is a sensible reference point for security logs.

How long should you keep API logs? #

Work backwards from the questions logs answer:

Only the first row needs bodies. That single observation is most of the savings.

How to estimate request logging costs #

Do the arithmetic before you pick numbers. Request volume times bytes per entry times days:

// log-size.mjs
const rps = 20;                // average, not peak
const perDay = rps * 86_400;   // 1,728,000 requests
const sizes = { metadata: 1_000, withBodies: 4_000 }; // bytes per entry

for (const [kind, bytes] of Object.entries(sizes)) {
  const gbPerDay = (perDay * bytes) / 1e9;
  console.log(kind, {
    gbPerDay: gbPerDay.toFixed(2),
    gb14d: (gbPerDay * 14).toFixed(0),
    gb90d: (gbPerDay * 90).toFixed(0),
  });
}

At 20 requests per second, metadata is about 1.7 GB a day and full entries about 6.9 GB a day. Ninety days of full entries is over 600 GB. Fourteen days of full entries plus ninety days of metadata is about 250 GB. Measure your own entry sizes rather than trusting these, because a few endpoints with large JSON responses can triple the average. Then apply your provider’s price per GB ingested and stored.

Log sampling: keep errors, sample the boring parts #

Most request volume is fast, successful reads that nobody will ever look at individually. Sampling those cuts storage without losing the requests you will investigate. The rule: decide after the response, so the outcome drives the decision.

import express from 'express';
import pino from 'pino';

const log = pino();
const app = express();
const SAMPLE_RATE = 0.1;
const SLOW_MS = 1000;
const MAX_BODY = 8 * 1024;

app.use((req, res, next) => {
  const start = process.hrtime.bigint();
  res.on('finish', () => {
    const ms = Number(process.hrtime.bigint() - start) / 1e6;
    const route = req.route?.path ?? 'unmatched';
    if (route === '/livez' || route === '/readyz') return;

    const keepFull =
      res.statusCode >= 400 || ms > SLOW_MS || req.method !== 'GET' ||
      Math.random() < SAMPLE_RATE;

    const entry = {
      method: req.method, route, status: res.statusCode,
      ms: Math.round(ms), userId: req.user?.id, sampled: !keepFull,
    };
    if (keepFull && req.body) {
      entry.body = JSON.stringify(req.body).slice(0, MAX_BODY);
    }
    log.info(entry);
  });
  next();
});

Every request still gets a metadata line, so counts and latency stay accurate. Only bodies are sampled. Errors, slow requests and writes are always kept in full. Record sampled: true so nobody mistakes a sampled dataset for a complete one.

How to set up a log retention policy #

Step 1: split logs by type #

Send request metadata, full request entries and security events to separate streams or indexes. Different data, different retention.

Step 2: redact before storage #

Retention is easier to justify when logs hold no secrets. Mask credentials and personal fields at write time; what not to log has the list.

Step 3: automate deletion #

Deletion that depends on someone remembering does not happen. If you archive logs to S3, a lifecycle rule handles it:

{
  "Rules": [
    { "ID": "full-requests", "Filter": { "Prefix": "requests-full/" },
      "Status": "Enabled", "Expiration": { "Days": 14 } },
    { "ID": "metadata", "Filter": { "Prefix": "requests-meta/" },
      "Status": "Enabled", "Expiration": { "Days": 90 } },
    { "ID": "security", "Filter": { "Prefix": "security/" },
      "Status": "Enabled",
      "Transitions": [{ "Days": 90, "StorageClass": "GLACIER_IR" }],
      "Expiration": { "Days": 400 } }
  ]
}

aws s3api put-bucket-lifecycle-configuration \
  --bucket my-api-logs --lifecycle-configuration file://lifecycle.json

If logs live in Postgres, partition by day and drop old partitions instead of running large DELETE statements.

Step 4: cut duplicate logging #

Many APIs log the same request three times: the proxy access log, a framework logger and an application line. Keep one request log and make application logs carry the request ID. A clear logging policy with meaningful levels also removes a lot of debug noise from production.

Log retention by team size #

A solo developer mostly needs the last week in detail and a month of history for customer questions. A larger team has more people asking older questions, and more reason to write the policy down.

If you only do one thing #

Write down three retention numbers (full entries, metadata, security events) and set a lifecycle rule that enforces them. A policy that deletes automatically beats a perfect policy on a wiki page.

FAQ #

How long should API logs be retained? #

For most small teams, 7 to 14 days of full request logs, 30 to 90 days of metadata, and 12 months of security events. Adjust for contracts or regulations that apply to you, such as PCI DSS 10.5.1 for card data.

How can I reduce logging costs without losing visibility? #

Keep a metadata line for every request, but store bodies only for errors, slow requests, writes and a sample of successful reads. Drop health check requests and duplicate log lines.

Does GDPR set a maximum log retention period? #

No fixed number. It requires keeping personal data no longer than necessary for the purpose, so document why each retention period exists.

Doing this with ApiTraffic #

If you would rather not build the sampling middleware, the redaction and the storage pipeline yourself, ApiTraffic captures every inbound request and response from one middleware.

• Authorization, Cookie and X-Api-Key are masked by default, and redaction rules for other fields are set in the dashboard and pushed to servers without a deploy, so what you keep is safer to keep.
• Requests are recorded under the matched route pattern, so per-endpoint counts and p50, p95 and p99 stay useful over weeks without storing anything extra yourself.
• Searches like tag:userId == "1042" AND status_code >= 400 answer the six-weeks-ago customer question directly.

Start free and see every request your API handles: apitraffic.io

Sources: GDPR Article 5, principles relating to processing of personal data; Microsoft Learn, PCI DSS requirement 10 (quotes 10.5.1); AWS, S3 Lifecycle configuration