Add OpenTelemetry SDK as a parallel option to ddtrace

[prim-8]

Summary

• Adds a parallel OTEL_TRACING_ENABLED runtime flag and INSTALL_OTEL Dockerfile build arg for shipping traces to any OTLP/HTTP backend (Tempo, Jaeger, Honeycomb, Grafana Cloud, etc.).
• ddtrace path is unchanged. Deployers who run Datadog keep the native experience; deployers running OSS observability stacks now have a first-class path.
• The two SDKs are mutually exclusive at runtime — ddtrace wins if both flags are set. Both can coexist as installed packages.

Motivation

Tracing in primitivemail has so far been a single path: install ddtrace, set DATADOG_TRACING_ENABLED=true, ship to a Datadog Agent. Self-hosters who don't run Datadog inherited a dead trace pipeline. Adding vanilla OpenTelemetry alongside (not replacing) ddtrace gives the OSS deployment story a real tracing option without forcing existing Datadog users onto an OTLP relay.

Configuration

Build:

• --build-arg INSTALL_DDTRACE=true — install ddtrace (existing)
• --build-arg INSTALL_OTEL=true — install opentelemetry-{api,sdk,exporter-otlp-proto-http}

Runtime — pick one (don't set both):

```bash

Datadog (unchanged):

DATADOG_TRACING_ENABLED=true
DD_SERVICE=milter
DD_ENV=...
DD_TRACE_AGENT_URL=http://localhost:8126

OpenTelemetry:

OTEL_TRACING_ENABLED=true
OTEL_SERVICE_NAME=milter
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
OTEL_PROPAGATORS=tracecontext,baggage # default
```

Both SDKs speak W3C tracecontext on the wire, so cross-service trace continuity works regardless of which is selected.

Implementation note

Rather than branching every `tracer.trace()` / `span.set_tag()` / `HTTPPropagator.inject()` call site, the OTel path is wrapped in a thin shim that exposes the same surface ddtrace does. This keeps the diff confined to module-level setup and one `entrypoint.sh` block — no behavior change for the ddtrace path.

The shim manages OTel context attach/detach manually so child spans (storage, webhook) inherit the parent email span — ddtrace's `tracer.trace()` makes the new span active; OTel's `start_span()` does not.

Backwards compatibility

• `INSTALL_DDTRACE=false` (default) and `DATADOG_TRACING_ENABLED=false` (default) — no change.
• `INSTALL_DDTRACE=true` only and `DATADOG_TRACING_ENABLED=true` — identical to today.
• Both flags set — ddtrace wins, OTel branch skipped, log notes which provider loaded.

Test plan

• `pytest` — existing tests pass (tracing branches default off; module-level changes are import-gated).
• Build with `INSTALL_DDTRACE=true INSTALL_OTEL=true`, run with `DATADOG_TRACING_ENABLED=true` — confirm Datadog APM shows the milter spans (no behavior change).
• Same image, run with `OTEL_TRACING_ENABLED=true` + `OTEL_EXPORTER_OTLP_ENDPOINT=...` pointing at a local OTel collector — confirm traces arrive with same span names (`milter.process_email`, `milter.storage_upload`, `milter.webhook_call`) and proper parent/child structure.
• Outbound webhook from the OTel run carries a `traceparent` header and the downstream service joins the trace.
• Log line `APM tracing enabled (provider=...)` reports the correct provider.

[greptile-apps]

Greptile Summary

This PR introduces a parallel OpenTelemetry tracing path alongside the existing ddtrace integration, controlled by a new OTEL_TRACING_ENABLED runtime flag and INSTALL_OTEL build arg. A ddtrace-shaped shim (_OtelTracerShim / _OtelSpanShim) wraps the OTel SDK so that call sites (tracer.trace, span.set_tag, span.finish, HTTPPropagator.inject) remain unchanged. All concerns raised in previous review threads — atexit + SIGTERM span flushing, OTel error status propagation, and misconfiguration logging — are addressed in this revision.

Confidence Score: 5/5

Safe to merge — all previous P1 findings are addressed and no new blocking issues found.

All three P1 issues from previous review threads (BatchSpanProcessor daemon-thread flush on exit, SIGTERM not running atexit, and OTel error status not propagated to backends) are explicitly resolved in this revision. The signal-chaining logic uses a factory closure to correctly capture per-signal previous handlers. The deferred error-status pattern in _OtelSpanShim.finish() correctly handles the OTel set_status lock-after-first-call constraint. No new P1 issues were identified.

No files require special attention.

Important Files Changed

Filename	Overview
milter/primitivemail_milter.py	Core OTel shim implementation: TracerProvider setup, BatchSpanProcessor with atexit + SIGTERM flush chain, _OtelSpanShim with deferred error status, and comprehensive misconfiguration logging — all well-implemented.
Dockerfile	Adds optional INSTALL_OTEL build arg that pip-installs opentelemetry-api/sdk/exporter-otlp-proto-http with a pinned minor-version range; default is false so existing images are unaffected.
milter/entrypoint.sh	Adds OTEL_TRACING_ENABLED shell block that sets OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, and OTEL_PROPAGATORS defaults; comment clarifies ddtrace-wins precedence rule.

Sequence Diagram

sequenceDiagram
    participant EP as entrypoint.sh
    participant PY as primitivemail_milter.py (import)
    participant OTel as OTel SDK
    participant Main as main()
    participant Milter as PrimitiveMailMilter
    participant Collector as OTLP Collector

    EP->>PY: OTEL_TRACING_ENABLED=true
    PY->>OTel: TracerProvider + BatchSpanProcessor + OTLPSpanExporter
    PY->>PY: atexit.register(provider.shutdown)
    PY->>PY: define _install_otel_shutdown_signals()

    PY->>Main: module loaded
    Main->>Main: signal.signal(SIGHUP, reload_config)
    Main->>Main: _install_otel_shutdown_signals() [chains SIGTERM/SIGINT]

    Milter->>OTel: tracer.trace("milter.process_email") → attach ctx_token
    Milter->>OTel: tracer.trace("milter.storage_upload") → child span
    Milter->>Milter: HTTPPropagator.inject(span.context, headers) [W3C traceparent]
    Milter->>OTel: storage_span.finish() → set_status(ERROR?) + span.end() + detach
    Milter->>OTel: tracer.trace("milter.webhook_call") → child span
    Milter->>OTel: webhook_span.finish() → set_status(ERROR?) + span.end() + detach
    Milter->>OTel: process_email span.finish() → detach root ctx_token

    OTel-->>Collector: BatchSpanProcessor flush (background thread)

    Note over Main,OTel: On SIGTERM (docker stop)
    Main->>OTel: signal handler → provider.shutdown() [force flush]
    Main->>Main: re-raise SIG_DFL → sys.exit
    Main->>OTel: atexit fires → provider.shutdown() [idempotent]

Loading

_{Reviews (6): Last reviewed commit: "Address Greptile round 5: SpanKind on ou..." | Re-trigger Greptile}