feat: trace infrastructure — spans + request ID propagation [DIS-1643]

[nnshah1]

Summary

• Generate x_dynamo_request_id UUID in make_inference_request_span() when client doesn't provide one — ensures ID exists from first log and propagates to workers
• Validate x-dynamo-request-id header is valid UUID (returns 400 if invalid)
• Separate span targets: request_span (inference, always on) vs system_span (health/metrics, debug level)
• Add request_span=trace filter directive so inference spans survive any DYN_LOG setting
• Simplify get_or_create_request_id() to return Result<String, String> — callers format errors for their API (OpenAI vs Anthropic)

Design Decisions

Custom span targets instead of error_span!: The tracing ecosystem has no always_on_span!. We use info_span!(target: "request_span", ...) with a request_span=trace directive in filters() — same pattern as existing span_event=trace. Overridable via DYN_LOG=request_span=off.

System vs inference separation: service_v2.rs router split into two groups with separate TraceLayer. System endpoints (health, metrics, models) use debug_span!(target: "system_span") — invisible at DYN_LOG=info, visible at DYN_LOG=debug.

UUID validation returns Result<String, String>: The error is a plain message string. OpenAI handlers convert via ErrorMessage::from_http_error(HttpError{...}), Anthropic via anthropic_error(...). Each endpoint returns its native error format.

Files Changed

File	Change
lib/runtime/src/logging.rs	make_inference_request_span + make_system_request_span + request_span=trace directive + worker span targets
lib/llm/src/http/service/openai.rs	get_or_create_request_id() simplified, all call sites updated
lib/llm/src/http/service/anthropic.rs	Call site updated with Anthropic error format
lib/llm/src/http/service/service_v2.rs	Router split: system routes + inference routes with separate TraceLayer
lib/runtime/src/system_status_server.rs	Uses make_system_request_span
lib/runtime/src/.../push_endpoint.rs	Fallback span uses request_span target

Test plan

• cargo fmt + cargo clippy clean
• Request without x-dynamo-request-id → UUID auto-generated, visible in all logs
• Request with valid UUID → client's UUID used throughout
• Request with invalid UUID → HTTP 400
• DYN_LOG=info → no system endpoint logs, inference logs visible
• DYN_LOG=debug → both visible
• Worker logs show x_dynamo_request_id matching frontend

🤖 Generated with Claude Code

Linear: DIS-1643

Summary by CodeRabbit

Release Notes

• Bug Fixes

  • Invalid request ID headers now return 400 Bad Request responses instead of being silently accepted.

• Improvements

  • Enhanced request identifier validation and error handling for HTTP requests.
  • Improved request tracing with distinct logging for inference and system endpoints.

[coderabbitai]

Walkthrough

Request-id resolution was made fallible: handlers now validate x-dynamo-request_id and return 400 on invalid values. Tracing span factories were refactored: inference vs system spans introduced, spans now set x_dynamo_request_id and declare model, input_tokens, output_tokens fields; TraceLayer usages were switched to the new factories.

Changes

Cohort / File(s)	Summary
Request ID / HTTP handlers lib/llm/src/http/service/openai.rs, lib/llm/src/http/service/anthropic.rs	get_or_create_request_id signature changed to accept only &HeaderMap and return Result<String, String>; request-id resolution now strictly validates x-dynamo-request-id (returns Err on invalid UTF-8 or non-UUID). Handlers map validation errors to HTTP 400 responses; Anthropic handler returns Anthropic-formatted 400 on failure.
Tracing span factories & fields lib/runtime/src/logging.rs, lib/llm/src/http/service/service_v2.rs, lib/runtime/src/system_status_server.rs	Renamed/refactored span factory: make_request_span → make_inference_request_span; added make_system_request_span. Spans set target: "request_span" (inference) or target: "system_span" (system), populate x_dynamo_request_id (trace-provided or new UUID), and declare new span fields: model, input_tokens, output_tokens. Axum TraceLayer usages updated to call the new factories. EnvFilter now unconditionally allows request_span=trace.
Ingress / payload handling span target lib/runtime/src/pipeline/network/ingress/push_endpoint.rs	When NATS request headers are missing, the created handle_payload span is now created with target: "request_span" (was previously created without that target attribute).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: introducing trace infrastructure with spans and request ID propagation, including a ticket reference.
Description check	✅ Passed	The description covers all required template sections with comprehensive details about changes, design decisions, affected files, and test plan—well-structured and complete.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@lib/llm/src/http/service/anthropic.rs`:
- Line 126: The call to get_or_create_request_id currently maps errors directly
to into_response(), which returns the OpenAI-shaped (StatusCode,
Json<ErrorMessage>) body; instead catch the Err from get_or_create_request_id
and map it through anthropic_error (or the shared helper that returns
Anthropic-style status+message) before returning so invalid x-dynamo-request-id
yields an Anthropic-formatted 400; update the expression around
get_or_create_request_id(&headers).map_err(...) to call anthropic_error(...) (or
the helper) rather than into_response(), referencing get_or_create_request_id,
anthropic_error, and into_response to find the callsite.

In `@lib/llm/src/http/service/openai.rs`:
- Around line 292-317: The function get_or_create_request_id currently validates
the header but discards it unless get_distributed_tracing_context() supplies an
ID, and to_str() failures silently fall through; change the logic so that if
headers.get(DYNAMO_REQUEST_ID_HEADER) is present you call raw.to_str() and
return a BAD_REQUEST error on to_str() failure, then parse the string with
uuid::Uuid::parse_str and return BAD_REQUEST if parsing fails, otherwise keep
that validated header value as a candidate; then return
trace_context.x_dynamo_request_id if present, else the validated header value if
present, else generate a new UUID. Make these changes in
get_or_create_request_id and reference DYNAMO_REQUEST_ID_HEADER and
get_distributed_tracing_context accordingly.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: fee1cb8a-7e67-4468-b61d-adfb0b4eb4ca

📥 Commits

Reviewing files that changed from the base of the PR and between 457db71 and d76dceb.

📒 Files selected for processing (4)

• lib/llm/src/http/service/anthropic.rs
• lib/llm/src/http/service/openai.rs
• lib/runtime/src/logging.rs
• lib/runtime/src/pipeline/network/ingress/push_endpoint.rs

[nnshah1]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

lib/llm/src/http/service/openai.rs (1)

299-300: Update stale span helper name in comments.

Line 299 and Line 324 still reference make_request_span(), but the codebase now uses make_inference_request_span(). This can mislead future maintainers.

Also applies to: 324-324

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@lib/llm/src/http/service/openai.rs` around lines 299 - 300, Update the
outdated helper name in the comments: replace references to make_request_span()
with make_inference_request_span() in the comment blocks around the OpenAI HTTP
service (e.g., the comment that starts "The request ID comes from the trace
context..." and the later reference at the other occurrence). Ensure both
comments now mention make_inference_request_span() so they match the actual
helper function name used in the code.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@lib/llm/src/http/service/openai.rs`:
- Around line 307-316: The request-ID validation error messages returned from
the UUID and UTF-8 checks should be prefixed with VALIDATION_PREFIX so they are
classified as validation (400) errors by classify_error_for_metrics; update the
error returns in the OpenAI request-ID parsing branch (the arms that currently
return "{} header must be a valid UTF-8 string" and "{} header must be a valid
UUID, got: {}") to prepend VALIDATION_PREFIX (the same constant used elsewhere)
to the message so downstream ErrorMessage handling and
classify_error_for_metrics treat them as validation errors.

---

Nitpick comments:
In `@lib/llm/src/http/service/openai.rs`:
- Around line 299-300: Update the outdated helper name in the comments: replace
references to make_request_span() with make_inference_request_span() in the
comment blocks around the OpenAI HTTP service (e.g., the comment that starts
"The request ID comes from the trace context..." and the later reference at the
other occurrence). Ensure both comments now mention
make_inference_request_span() so they match the actual helper function name used
in the code.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 39c4d1ed-4a42-4770-96cb-cc0da885c26c

📥 Commits

Reviewing files that changed from the base of the PR and between d76dceb and 3f5d48b.

📒 Files selected for processing (6)

• lib/llm/src/http/service/anthropic.rs
• lib/llm/src/http/service/openai.rs
• lib/llm/src/http/service/service_v2.rs
• lib/runtime/src/logging.rs
• lib/runtime/src/pipeline/network/ingress/push_endpoint.rs
• lib/runtime/src/system_status_server.rs

✅ Files skipped from review due to trivial changes (1)

• lib/llm/src/http/service/service_v2.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• lib/runtime/src/pipeline/network/ingress/push_endpoint.rs

[nnshah1]

Implementation Plan — DIS-1643: Consistent Error Tracing

Overview

4 stacked PRs implementing consistent structured logging across all frontend error paths and worker trace propagation.

PR1: Trace Infrastructure (#7733)

Problem: Request spans use info_span! which gets filtered out by DYN_LOG=warn/error, losing request context on error logs. No UUID validation on x-dynamo-request-id. System and inference endpoints share the same span/log treatment.

Solution:

• Custom span targets: request_span (inference, always on via request_span=trace directive) and system_span (health/metrics, debug level)
• make_inference_request_span() generates UUID when client doesn't provide one — captured by DistributedTraceIdLayer and propagated to workers
• get_or_create_request_id() returns Result<String, String> — validates UUID header, callers format errors for their API (OpenAI via from_http_error, Anthropic via anthropic_error)
• Router split in service_v2.rs: system routes get make_system_request_span, inference routes get make_inference_request_span

Files: logging.rs, openai.rs, anthropic.rs, service_v2.rs, system_status_server.rs, push_endpoint.rs

---

PR2: Request Lifecycle Logging (#7734)

Problem: No consistent "request received" / "request completed" log. Error paths have different log formats. Worker has no request-level logs at INFO.

Solution:

• InflightGuard is single source of truth: logs "request received" (INFO) on creation, "request completed" (INFO/ERROR) on Drop
• create_inflight_guard() takes request_id, records model on span — no separate setup calls
• All inference errors at ERROR level with status=error, error_type, error_detail
• on_response renamed to "http response sent" — system at DEBUG, inference at INFO/ERROR
• Worker push_handler.rs logs "request received" / "request completed" at INFO

Files: metrics.rs, openai.rs, anthropic.rs, service_v2.rs, push_handler.rs, grpc/{openai,tensor}.rs

---

PR3: Token Counts, TTFT, ITL, Worker IDs (#7735)

Problem: No token counts, latency metrics, or worker identification on request completion logs.

Solution:

• ResponseMetricCollector::Drop records on span: input_tokens, output_tokens, ttft_ms, avg_itl_ms, prefill_worker_id, decode_worker_id
• TTFT stored from already-computed value; ITL accumulated from per-chunk computation (2 field additions, no new calculations)
• WARN log at cancellation point in disconnect.rs
• Connection monitor upgraded from TRACE to WARN

Performance: All additions are on cleanup path (Drop), not streaming hot path. Two f64/u64 accumulations per chunk (negligible alongside existing histogram publish).

Files: metrics.rs, disconnect.rs, logging.rs (Empty fields)

---

PR4: E2E Tests (#7766)

11 parallel-safe pytest tests (25s with -n auto):

Category	Tests
Aggregated success	unary, streaming, request ID propagation
Aggregated errors	404, 400 invalid UUID, cancellation, worker crash
Disaggregated success	unary, streaming (both workers verified)
Disaggregated crashes	prefill crash, decode crash

---

Request ID Propagation Flow

Client → x-dynamo-request-id header (optional, 400 if invalid)
  ↓
make_inference_request_span() → generates UUID if absent
  ↓
DistributedTraceIdLayer::on_new_span() → captures into trace context
  ↓
get_or_create_request_id() → validates + reads from trace context
  ↓
create_inflight_guard() → logs "request received", records model on span
  ↓
addressed_router.rs → inject_trace_headers_into_map() → worker gets UUID
  ↓
Worker span → x_dynamo_request_id, trace_id correlation
  ↓
InflightGuard::Drop → logs "request completed" with all fields

"request completed" Fields (streaming)

{
  "level": "INFO",
  "message": "request completed",
  "status": "success",
  "request_id": "...",
  "model": "qwen/qwen3-0.6b",
  "endpoint": "chat_completions",
  "request_type": "stream",
  "elapsed_ms": "20",
  "input_tokens": "9",
  "output_tokens": "50",
  "ttft_ms": "5.85",
  "avg_itl_ms": "0.29",
  "trace_id": "..."
}

Follow-up Issues

• DIS-1652 — Propagate model name to worker via transport headers
• DIS-1653 — Token counts/TTFT missing on unary requests (collector Drop ordering)

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-ffbdf0ec-04a3-41e9-a58f-3766f0b225dc.docs.buildwithfern.com/dynamo/dev

[tmonty12]

Start frontend like so:

DYN_LOGGING_JSONL=true DYN_LOG=debug DYN_NAMESPACE=test DYN_EVENT_PLANE=zmq python -m dynamo.frontend

Start mock worker like so:

DYN_LOGGING_JSONL=true DYN_LOG=debug DYN_NAMESPACE=test DYN_EVENT_PLANE=zmq python -m dynamo.mocker --model-path Qwen/Qwen3-0.6B --block-size 2

Send a chat completions request

Frontend logs for request:

{
  "time": "2026-04-02T01:14:11.069507Z",
  "level": "DEBUG",
  "file": "/Users/tmontfort/Dynamo/repos/dynamo/lib/runtime/src/pipeline/network/egress/push_router.rs",
  "line": 552,
  "target": "dynamo_runtime::pipeline::network::egress::push_router",
  "message": "Using TCP transport for instance",
  "instance_id": 7587893895631993984,
  "method": "POST",
  "span_id": "cafd5bb2662ad1aa",
  "span_name": "http-request",
  "tcp_endpoint": "192.168.1.164:56530/694d9d4bb27ffc80/generate",
  "trace_id": "646a97a0ca50f33dfe7269a341ec53a8",
  "uri": "/v1/chat/completions",
  "version": "HTTP/1.1",
  "x_dynamo_request_id": "0f4ac56e-5bd1-4657-bf8b-39cdcde9036b"
}
{
  "time": "2026-04-02T01:14:11.069606Z",
  "level": "DEBUG",
  "file": "/Users/tmontfort/Dynamo/repos/dynamo/lib/runtime/src/pipeline/network/tcp/server.rs",
  "line": 243,
  "target": "dynamo_runtime::pipeline::network::tcp::server",
  "message": "Registering new TcpStream on 192.168.1.164:56573",
  "parent_id": "cafd5bb2662ad1aa",
  "request_id": "0f4ac56e-5bd1-4657-bf8b-39cdcde9036b",
  "router_mode": "RoundRobin",
  "span_id": "6e09efb48634df43",
  "span_name": "router.route_request",
  "trace_id": "646a97a0ca50f33dfe7269a341ec53a8",
  "worker_id": "7587893895631993984"
}
{
  "time": "2026-04-02T01:14:11.069738Z",
  "level": "DEBUG",
  "file": "/Users/tmontfort/Dynamo/repos/dynamo/lib/runtime/src/pipeline/network/egress/tcp_client.rs",
  "line": 510,
  "target": "dynamo_runtime::pipeline::network::egress::tcp_client",
  "message": "TCP client sending request to address: 192.168.1.164:56530/694d9d4bb27ffc80/generate",
  "parent_id": "cafd5bb2662ad1aa",
  "request_id": "0f4ac56e-5bd1-4657-bf8b-39cdcde9036b",
  "router_mode": "RoundRobin",
  "span_id": "6e09efb48634df43",
  "span_name": "router.route_request",
  "trace_id": "646a97a0ca50f33dfe7269a341ec53a8",
  "worker_id": "7587893895631993984"
}
{
  "time": "2026-04-02T01:14:11.075612Z",
  "level": "DEBUG",
  "file": "/Users/tmontfort/Dynamo/repos/dynamo/lib/llm/src/http/service/service_v2.rs",
  "line": 548,
  "target": "dynamo_llm::http::service::service_v2",
  "message": "request completed",
  "latency_ms": "8",
  "method": "POST",
  "span_id": "cafd5bb2662ad1aa",
  "span_name": "http-request",
  "status": "200",
  "trace_id": "646a97a0ca50f33dfe7269a341ec53a8",
  "uri": "/v1/chat/completions",
  "version": "HTTP/1.1",
  "x_dynamo_request_id": "0f4ac56e-5bd1-4657-bf8b-39cdcde9036b"
}

We see both request_id and x_dynamo_request_id. This should be consolidated to the latter.