Trace sampling — the multi-tier decision tree

orbitalreg-api uses a custom OpenTelemetry sampler that wraps the env-driven base sampler with two always-on overrides for paths that matter more than the default sampling ratio admits. This page covers who decides what, how to tune it, and what the trade-offs are.

The implementation lives in api/internal/telemetry/sampling.go; the env contract lives in api/internal/telemetry/tracing.go.

Why a custom sampler

At a 10% root-sampling rate (the default we recommend in Observability), two classes of trace are systematically under-represented:

1. Health and readiness probes. Liveness + readiness probes fire roughly once per second per container. A 10% sample means ~6 traces per minute — enough for cardinality, not enough to alarm on when Postgres is slow or S3 is rate-limiting. The OTLP backend ends up with a trickle of probe traces and operators have no continuous heartbeat to alert against.

2. Artifact uploads. PUT/POST under /api/v1/artifacts/* and the per-repo upload mounts (/api/v1/repos/{id}/artifacts/*) are the slow lane of the registry: byte streaming → S3 PutObject → metadata extraction → scan-dispatcher fan-out. Uploads are rare relative to reads (a busy registry sees ~50:1 read:write), so a 10% sample drops 9 out of every 10 of the spans operators most want when a build pipeline complains.

The multi-tier sampler forces 100% sampling on both classes and delegates everything else to the env-configured base. The result is that operators can keep the default 10% ratio for the common-case read path (where 10% is statistically rich and budget-friendly) while still getting a complete trace timeline for the two surfaces that debugging usually needs.

Decision table

The sampler decision uses the initial span kind and attributes, so it runs before the handler does any work.

Span kind	Method	Target prefix	Decision
Server	any	/health/...	Force record
Server	any	/healthz	Force record
Server	POST / PUT	/api/v1/artifacts/...	Force record
Server	POST / PUT	/api/v1/repos/{id}/artifacts/...	Force record
Server	any other	(anything else)	Inner sampler
Client / Internal / Producer / Consumer	—	—	Inner sampler

Forced spans carry the attribute orbitalreg.sampler.rule=force-path, so a backend search like attribute("orbitalreg.sampler.rule") = "force-path" returns only the spans that bypassed the ratio gate. Useful when reviewing whether the inner ratio is set correctly: if the dashboard is dominated by force-path rows, the ratio is too low for the read workload.

What the inner sampler does

The inner sampler is the standard one configured by OTel Sampler env vars:

• OTEL_TRACES_SAMPLER — name, one of always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, parentbased_traceidratio. Default parentbased_traceidratio.
• OTEL_TRACES_SAMPLER_ARG — argument for the ratio variants. Default 0.1 (10%).

The wrap is unconditional: even with OTEL_TRACES_SAMPLER=always_off, the operator still gets /health/* and upload traces. The reasoning is that an operator setting always_off is opting out of background trace volume (cost or signal-to-noise), not out of the two surfaces that are usually the reason tracing was enabled in the first place. If a deployment genuinely wants to drop every trace, the cleaner choice is to leave OTEL_EXPORTER_OTLP_ENDPOINT unset.

How to tune

Lower the cost without losing health visibility

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.02   # 2% reads

Health probes still go 100%; everything else drops to 2%. A reasonable choice for a high-RPS deployment with an OTLP backend that bills by ingested span.

Maximise debug coverage

OTEL_TRACES_SAMPLER=always_on

Every span records. Combine with a collector-side tail sampler if the backend can't keep up. Recommended for short-lived staging environments and during incident triage.

Air-gapped install

Leave OTEL_EXPORTER_OTLP_ENDPOINT unset. The v220 air-gapped gate also short-circuits Init when the egress channel is denied, so an operator running the standard hardened bundle gets no OTel I/O regardless of the rest of the variables.

What we don't sample on (and why)

HTTP status code. The status is known only after the handler runs, but the sampler is called before the span exists. The OTel SDK has no "decide later" hook — that's what tail sampling at the collector is for. If you need "every 5xx must be sampled," configure a tail_sampling processor in your collector pipeline rather than asking the SDK to do something it can't.

User-agent / build-id. Sampling decisions need to be cheap; iterating the attribute slice on every span has measurable overhead at high RPS, so we only inspect the two attributes the rule needs (http.method / http.request.method and http.target / url.path). The slice scan is bounded to the otelhttp-emitted initial attributes (typically ≤ 10).

Backward compatibility

The env-driven sampler shape from Item 106 Phase A is unchanged: same variable names, same defaults, same switch table. A deployment running with OTEL_TRACES_SAMPLER=traceidratio and OTEL_TRACES_SAMPLER_ARG=0.05 after upgrading will see the same 5% sampling rate for the read path it had before, plus the new force-path guarantee for /health and uploads.

The wrapper surfaces in the bootstrap log line as the sampler description, e.g.:

opentelemetry tracing enabled  sampler=OrbitalRegMultiTier{always:health,artifact-upload;else:ParentBased{...}}

so a grep sampler= /var/log/orbitalreg.log confirms which sampler is wired at startup without digging through the binary's --version.