PowerTraces

Allowlist, Don't Blocklist: Redacting Telemetry

Blocklist regexes miss whatever you did not think of. A fail-closed allowlist at the source and in the OpenTelemetry Collector keeps secrets and personal data out of traces, logs and metrics.

PowerTraces AI ·

Most telemetry leaks are not breaches. They are an Authorization header captured "just for debugging", an email address inside a URL path, or a query string carrying a signed token. Once such a value is in your trace backend it is searchable, replicated and retained, so the useful question is where in the pipeline you stop it.

Why blocklists fail quietly

The instinctive fix is a blocklist: a regex for card numbers, another for bearer tokens, one for emails. It feels thorough, and it fails in a predictable way. Every new attribute, library upgrade or developer-added span tag is allowed by default, and you only learn about a leak when someone finds it.

An allowlist inverts that. Anything you have not explicitly approved is dropped. A new attribute then costs one reviewed line of configuration instead of an unnoticed exposure. The trade-off is real: dropped attributes make debugging harder until someone adds them, so the allowlist needs an owner and a short path for adding keys.

My recommendation is to use both, in layers. The allowlist is the primary control and value patterns are a safety net for what slips through approved keys.

Layer one: capture less at the source

The cheapest redaction is not collecting the data. The OpenTelemetry HTTP semantic conventions make http.request.header.<key> an opt-in attribute and say instrumentations should require explicit configuration of which headers are captured, because including all request headers can be a security risk.

The same conventions say url.full must not contain credentials passed via the URL, and that the values of certain query keys (X-Amz-Signature, X-Amz-Credential, X-Amz-Security-Token, AWSAccessKeyId, Signature, sig, X-Goog-Signature) should be replaced with REDACTED by default, with the key preserved. That list is a floor, not a guarantee. Your own ?token= or ?api_key= is not on it, and not every instrumentation version implements it identically.

In practice: capture request headers only from a short list you chose, never put user identifiers in span names, and prefer route templates (/users/{id}) over raw paths.

Layer two: an allowlist in the Collector

You will not control every SDK, especially across many teams, so enforce the policy centrally. The Collector's redaction processor, in the contrib distribution, is built for this. According to its README, it deletes span, log and metric datapoint attributes that are not on an allowed list, and masks values that match a blocked list. Its traces support is beta and logs and metrics are alpha, so check the stability level for your signals before depending on it.

Two details matter. First, the allowlist fails closed: if allowed_keys is empty, every attribute is removed. Second, allowed_values takes precedence over blocked_values, so a broad allow pattern can silently disable a block. Keep allow patterns narrow and anchored.

Test the failure mode, not just the happy path. Send a span with an attribute you expect to be dropped and one with a fake token in an allowed key, then check what reaches the exporter. A redaction rule you have never seen fire is an assumption.

A starting configuration

This sketch uses only keys documented in the processor README. The attribute names and patterns are examples to adapt, and you should validate it against your Collector version with otelcol-contrib validate --config=config.yaml before rollout.

processors:
  redaction:
    allow_all_keys: false
    allowed_keys:
      - http.request.method
      - http.response.status_code
      - http.route
      - url.path
      - server.address
      - db.system
      - service.name
      - service.version
      - deployment.environment
    blocked_key_patterns:
      - ".*token.*"
      - ".*secret.*"
      - ".*password.*"
    blocked_values:
      - "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}"
      - "4[0-9]{12}(?:[0-9]{3})?"
    summary: info
    url_sanitizer:
      enabled: true
      attributes: ["http.url", "url.full"]

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [redaction]
      exporters: [otlp]

Notes on the choices. summary: info adds counts of redacted attributes, which gives you a signal for noisy teams; the README also describes debug (which adds the redacted key names) and silent. The card pattern is deliberately the simple Visa-style example from the project's own documentation, and it will produce false positives and misses, since a regex cannot validate a card number. Put redaction ahead of batching and any tail-sampling step, so that nothing buffered or exported ever holds the raw value. Because the allowlist fails closed, run a test that confirms the attributes you rely on, such as service.name, still arrive after the processor, since a missing key is dropped without any error.

Things redaction will not save you from

  • Log bodies. Free-text messages are where most real leaks live. Attribute allowlists do not help; fix the logging call or add value patterns and test them.
  • High-cardinality identifiers. A user ID is not a secret, but it may be personal data under your policy. Decide what your policy says, then enforce it by key.
  • Data already stored. Redaction is forward-looking. If a leak happened, you need deletion or retention expiry at the backend, and a note in your incident process.
  • The Collector config itself. OpenTelemetry's configuration guidance warns that Collector configuration may contain API tokens and TLS private keys, so keep it in a secret store.

Make the policy reviewable

A redaction policy that only one person understands will decay. Teams add attributes, someone widens a pattern to stop an alert, and a year later nobody knows why a key is allowed.

Treat the allowlist as code. Keep it in version control, require review from someone who owns data handling, and add a test fixture of known-bad spans that must come out clean in CI. When an engineer needs a new attribute, the pull request is the conversation about whether it is safe.

Checklist

  • Request headers are captured only from an explicit, short list.
  • Span names use route templates, never raw paths with identifiers.
  • The Collector runs an allowlist that fails closed, placed before batching and tail sampling.
  • Allow patterns are narrow, and you have tested that they do not override blocks.
  • A CI fixture proves known-bad spans are scrubbed.
  • Redaction counts are watched, and there is a documented path to purge stored data.

References