# Tail Sampling Is a Stateful System: Design It Like One

Tail sampling rules are the easy part. Routing by trace ID, sizing memory, and handling late spans decide whether the OpenTelemetry Collector keeps the traces you need.

PowerTraces AI · 8 November 2026

![](https://powertraces.com/blog-assets/ea3edc64-a904-4981-91a7-3a6c53e44bd0/cover.png)

Tail sampling is usually sold as a list of rules: keep errors, keep slow traces, sample the rest. The rules are the easy part. What decides whether it works in production is that the sampler is a stateful system, and it needs to be sized, routed and monitored like one.

## Why you end up here

According to the OpenTelemetry documentation, head sampling decides as early as possible, without inspecting the whole trace, so it cannot promise to keep every trace that contains an error. Tail sampling decides after considering all or most of a trace's spans, which lets you keep errors, slow traces or traces with specific attributes. The same page says larger systems almost always need tail sampling to balance data volume against usefulness.

It also lists the cost: tail samplers must be stateful systems that accept and store a large amount of data, and they can be hard to operate. That sentence is the whole article in miniature.

## Rule one: one trace, one Collector

The tail sampling processor README is blunt: all spans for a given trace must be received by the same Collector instance for effective sampling decisions. The OpenTelemetry scaling guide describes what happens otherwise. If you simply add replicas, you get traces with missing spans, which misrepresent what happened in the transaction.

The documented answer is two layers: one with the load-balancing exporter and one with the tail sampling processor. The README recommends separating them for failure isolation. In the load-balancing exporter, traces route by `traceID` by default, and the DNS resolver can point at a Kubernetes headless service. Its documented defaults are port 4317, a 5s refresh interval and a 1s timeout.

Layer one, the routing tier. It holds no sampling state, so it can be small and replaced freely:

```
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true   # example only; use TLS between tiers in production
    resolver:
      dns:
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317
        interval: 5s

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [loadbalancing]
```

The hostname above is a placeholder for your own headless service. The scaling guide also notes that each Collector may run its DNS query at different times, so the view of the sampler set can differ briefly, and it suggests a lower refresh interval in highly elastic environments.

## Rule two: the sampling tier is a memory budget

The processor keeps spans in memory while it waits to decide. Two settings dominate that cost, and both are documented: `decision_wait` (default 30s) and `num_traces` (default 50000, the number of traces held in memory). Raising either raises memory use. The README says so directly.

Layer two, the sampling tier:

```
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  tail_sampling:
    decision_wait: 20s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    decision_cache:
      sampled_cache_size: 1000000
      non_sampled_cache_size: 1000000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

exporters:
  otlp:
    endpoint: backend.example.internal:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp]
```

Every number here is a placeholder, not a recommendation. Size them from your own trace rate and trace duration. Check the file against your Collector version (`otelcol-contrib validate --config=sampler.yaml`) before rollout, since option names can differ between versions and the processor is documented at beta stability for traces.

The policy semantics are worth reading once. A trace is sampled when a policy decides to sample it, so the three policies above act as "errors, or slow, or a 5% baseline". The `latency` policy uses the trace's earliest start and latest end time, and omitting an upper bound keeps anything above the threshold.

**The baseline policy matters more than it looks.** If you only keep errors and slow traces, your stored data is a biased sample of your system. You lose the normal traces you need for comparison. The 5% baseline is what lets you ask what a healthy request looks like. This is my recommendation; the percentage is yours to tune.

## Rule three: plan for late spans

A trace is never finished at one instant. The README warns that late spans, which arrive after the decision, can lead to different sampling decisions for different parts of one trace. It describes three cases. While the decision is still in the `num_traces` buffer, a late span inherits it. Without a decision cache, once the decision has left the buffer, the late span is buffered for `decision_wait` and judged again, on its own. With a cache configured, kept trace IDs are remembered after the span data is released.

So the decision cache is not an optimization. It is what keeps long-running or retried work from being half-kept. The README advises sizing the cache much larger than `num_traces` so decisions outlive span data.

## What breaks, and how you notice

- **Resharding.** When the sampler set changes, the load-balancing exporter documents that roughly R/N of routes move to different backends (R routes, N backends). In-flight traces can split across two samplers. Expect partial traces during scale events, and avoid frequent autoscaling of this tier.
- **Memory pressure.** Long `decision_wait` times with high trace rates are the usual cause. Watch the Collector's own memory and telemetry rather than guessing.
- **Long traces.** Anything longer than `decision_wait` is judged on partial data. Choose the wait from your real trace-duration distribution, not from the default.
- **Silent bias.** A policy that matches too little looks identical to a healthy system. Compare sampled error counts against error counts in your metrics.

The OpenTelemetry docs recommend monitoring the sampling components to confirm they have the resources to decide correctly. In practice that means alerting on the sampler tier the same way you would alert on any stateful service.

## Consider protecting the pipeline with head sampling

The sampling documentation notes that the two techniques combine: head-sample a very high-volume service first to protect the pipeline, then tail-sample before export. If one noisy service dominates your trace rate, that can be cheaper than buying more sampler memory. It does mean those services lose the "always keep errors" guarantee for the share you drop early, so make that trade-off explicitly.

## Checklist

- Routing and sampling run as two tiers, with trace-ID routing in front.
- `decision_wait` is chosen from observed trace durations.
- `num_traces` and memory limits are sized from real trace rates.
- A decision cache is configured, larger than `num_traces`.
- A baseline probabilistic policy keeps ordinary traces too.
- The sampler tier has its own alerts and a scaling runbook.
- Sampled error counts are compared with error counts from metrics.

## References

- [Sampling, OpenTelemetry documentation](https://opentelemetry.io/docs/concepts/sampling/)
- [Scaling the Collector, OpenTelemetry documentation](https://opentelemetry.io/docs/collector/scaling/)
- [Tail Sampling Processor README, OpenTelemetry Collector contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md)
- [Load Balancing Exporter README, OpenTelemetry Collector contrib](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/README.md)