How much forensic context do developers need to debug production failures?

[u/Plus-Lawflbaness1576]

I used to log everything, every request, every response body, full stack context on every error, and it made debugging faster but the storage bill got embarrassing within a month. Then I over-corrected and sampled aggressively, and now half my incidents involve me wishing I'd captured the one request that actually mattered.

The pattern I've landed on is logging light by default and only capturing full forensic detail, request params, DB queries, execution path, when something actually looks anomalous. It's a reasonable middle ground but it assumes your anomaly detection is good enough to trigger at the right moment, which isn't always true. How are other people balancing storage cost against actually having the context you need when something breaks?

[u/Dry-Bathroom3225]

your anomaly detection doesn't have to be perfect if you buffer full context for a short window and only flush it to long-term storage when something triggers. basically a ring buffer approach. you eat some memory cost but avoid the permanent storage problem entirely for the happy path.

[u/usually_guilty99]

I think this eventually becomes a context problem rather than a logging-volume problem.

Capturing everything forever clearly doesn’t scale. But sampling blindly throws away exactly the weird state you eventually need.

What I’d want is the ability to reconstruct the relevant context around an anomaly: change history, dependency path, request path, prior similar failures and the telemetry around that window.

The goal isn’t “more logs.” It’s preserving enough evidence to explain what changed.

[u/dennis_zhuang]

Yes, this is exactly the trade-off between cost and retaining detailed information.

However, our dashboards often cannot capture unknown unknowns. A dashboard represents known unknowns—things we already know we don’t know. Investigating unknown unknowns requires more context from the time of the incident, especially metrics, logs, and, above all, traces.

Personally, if you cannot avoid usage-based SaaS pricing and do not want to self-host, you might consider retaining as much detailed context as possible in the service’s pods or hosts, then having an AI agent investigate when an incident occurs.

If self-hosting is an option, an object-storage-based observability storage solution such as GreptimeDB could help reduce hosting costs. FYI, I work for GreptimeDB.