Watching your systems costs more than running them

There is a stage most growing systems reach where the monitoring bill overtakes something it should not. Teams are now spending $40,000–120,000 a month on observability, in some cases more than on their database, CDN or compute. Around 84% of observability users report struggling with the cost.

That is a strange place to arrive, and the path there is always the same.

How it happens

Nobody decides to spend six figures watching their software. It accumulates.

You add structured logging, which is good practice. Someone adds a debug log inside a hot loop during an incident and it never comes out. You adopt tracing and sample at 100% because sampling feels like losing data. A well-meaning engineer adds a metric label containing a user ID or a request path, and the cardinality explodes — every distinct value becomes its own time series, and the bill is a function of series count, not of usefulness.

Then traffic grows, and every one of those decisions multiplies.

The vendor pricing model does the rest: you are billed on ingest and retention, so the incentive to send everything is baked into how the tools are adopted and the cost of that choice arrives months later.

The two changes that do most of the work

Tail-based sampling. Keep 100% of traces that contain an error or high latency, and discard most successful requests. This reportedly reduces storage footprint by up to 90%, and it works because the traces you actually open are almost always the anomalous ones. A representative sample of successful requests tells you as much as all of them.

The word "tail" is the important part: the decision is made after the trace completes, when you know whether it was interesting. Head-based sampling decides at the start and therefore throws away errors at the same rate as everything else, which is the worst of both.

Log-to-metric conversion at the edge. Most log volume exists to answer questions that are actually about counts and rates — how many 500s, how many retries, what is the p99. Aggregating into metrics before shipping keeps the answer and drops the volume. Keep the raw lines where you genuinely need to read individual events, which is a much smaller set than current retention suggests.

Together these are reported to take 50–72% off a bill without losing the signal that matters.

The discipline underneath

The uncomfortable framing is that observability spend is usually a proxy for not having decided what you need to know.

Sending everything is what you do when you have not worked out which questions you actually ask during an incident. It feels safe. It is expensive, and it makes incidents harder, because a search across a hundred times more data is slower and noisier at exactly the moment you are under pressure.

A team that can name its ten most common incident questions can instrument for those specifically, keep the raw data that answers them, aggregate the rest, and spend a fraction of the money while resolving faster.

Two habits help. Separate product analytics from infrastructure health — they have different retention needs and different audiences, and mixing them means paying infrastructure-grade retention for product questions. And put a collector in the middle rather than pointing every service directly at a vendor, so filtering, sampling and redaction are decisions you can change centrally instead of a deployment per service.

Why now

Two things make this more urgent than it was.

OpenTelemetry adoption is around 48.5% with another quarter planning it, which means the portability argument has largely been won. If your instrumentation is vendor-neutral, changing backends is a configuration change rather than a re-instrumentation project — and that shifts the negotiating position considerably.

The other is AI workloads. Teams planning for 5–10x current telemetry volume are planning for token counts, prompt and completion logging, tool-call traces and evaluation results. That data is verbose, and it is arriving on top of everything you already send. A pipeline that is uneconomic at today's volume will not become economic at ten times it.

The right time to decide what you keep is before the volume arrives, not after the invoice does.