feat(tracing): Support sending traces to a generic OTLP trace endpoint

[ringerc]

Add support for sending OpenTelemetry traces to a generic OTLP trace destination using the standard OpenTelemetry trace configuration environment variables. Implements #12117

Changes

(Changes are split into logical commits for easier review.)

Logic common to the various otel-based tracing exporters is extracted from the existing providers traceloop.py, arize_phoenix.py and langwatch.py into a new otlp_base.py skeleton provider. This handles attribute transformation, trace-context propagation and other common logic.

A new provider otlp.py is added and registered with tracing service.py. The generic OpenTelemetry provider activates if either of the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT env-var is set. Configuration follows the conventions set out in OTLP Exporter Configuration
and the go otel SDK docs . Some test cover is added to exercise the tracer.

An additional test is added to check that trace context propagation works in the expected manner as requests flow through Langflow.

(Subsequent changes arising from automated review, rebases and merges):

Cache and reuse OpenTelemetry based tracers. The Arize/Phoenix tracer and the new generic OpenTelemetry tracer were constructing a new trace engine, batch span processor etc for every graph evaluation, then orphaning them. These would leak and remain running in the background, as well as failing to properly flush on shutdown. Address by using a module-level singleton to initialize the traceprovider only once, and only construct a new tracer for each invocation. Other providers are unaffected; either they internally use a singleton already, or they don't have a long-lived trace engine to preserve.

Update langwatch trace provider initialization to use the updated singleton pattern. The langwatch tracer already used a singleton, but it didn't have a shutdown hook or any locking to prevent races between concurrent initializations. Update it to work the same way as the other otel-based tracers.

Rebase on top of #12962 which added trace context propagation for the otel-flavoured exporters and de-duplicate functionality.

Update generic OTLP instrumentation to ensure it follows the otel semantic conventions for GenAI/LLM.

Extend test cover to validate that the expected attributes are emitted by each provider.

Usage

1. Add at least OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT env-var, pointing to your OTLP endpoint
2. Set env-var OTEL_EXPORTER_OTLP_PROTOCOL to the exporter protocol (grpc, http/protobuf or http/json)
3. Preferably set OTEL_SERVICE_NAME to something suitable (langflow will do if nothing more specific applies in your environment).

Optionally define additional SDK env-vars to control server and client certificates, additional headers, timeouts, sampling, etc.

Related issues

Fixes #12117

Context

This patch series was developed with assistance from Claude Code, but all non-test code changes have been manually reviewed and inspected to the best of my (not amazing) ability and modified where appropriate.

The instrumentation is explicitly created in code because the Langflow multi-provider tracing abstraction doesn't fit very well with the python otel SDK's auto-instrumentation and opentelemetry-instrument too. I also didn't want to impose the requirement of wrapping the langflow backend execution in the opentelemetry-instrument command.

I omitted various additional, verbose test cover for things like asserting that all the otel env-vars are respected as they didn't seem useful enough or langflow-specific enough.

Testing

Using the existing tests for the current tracers and some additions made to ensure they properly cover the emitted attributes etc, run the tests against release-1.10.0 and against this PR branch HEAD. Compare. REGRESSION_TEST_RESULTS.md

Summary by CodeRabbit

• New Features

  • Added OTLP (OpenTelemetry) tracing support configurable via environment variables.
  • Improved W3C Trace Context propagation for reliable distributed tracing and concurrent-request isolation.
  • Centralized tracer provider lifecycle and explicit shutdown for more reliable startup/shutdown and global instrumentation.

• Bug Fixes / Reliability

  • Safer correlation handling and reduced per-instance resource churn to avoid cross-run interference.

• Tests

  • New comprehensive tests covering OTLP setup, exporter selection, propagation, concurrency isolation, and provider shutdown.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c1356476-abf7-461f-acfc-bea5d954fa61

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Walkthrough

Adds a new OTLP tracing stack (OTLPTracer + OTLPTracerBase) for generic OTLP endpoints, refactors existing tracers to reuse the OTLP base and a shared provider lifecycle, centralizes provider shutdown/reset helpers, and adds extensive OTLP and W3C trace-context tests and minor setup cleanup.

Changes

Cohort / File(s)	Summary
OTLP Core src/backend/base/langflow/services/tracing/otlp_base.py, src/backend/base/langflow/services/tracing/otlp.py	Introduce OTLPTracerBase (timestamp/JSON conversion, type normalization, span-attributes helpers, readiness) and OTLPTracer (env-based OTLP exporter selection grpc/http+protobuf, shared provider/tracer singletons, root/child span lifecycle, propagation carrier, provider shutdown/_reset helpers).
Refactored Tracers src/backend/base/langflow/services/tracing/arize_phoenix.py, src/backend/base/langflow/services/tracing/langwatch.py, src/backend/base/langflow/services/tracing/traceloop.py	Switch tracers to inherit from OTLPTracerBase, remove per-instance provider/exporter setup and duplicate conversion helpers, adopt module-level shared provider/tracer singletons, move correlation-id to per-root-span assignment, remove per-instance destructor/flush logic.
Service Integration src/backend/base/langflow/services/tracing/service.py	Add lazy _get_otlp_tracer(), _initialize_otlp_tracer() to attach OTLP tracer into TracingService trace context, and TracingService.teardown() to call provider shutdown helpers.
Tests — OTLP & Tracing Service src/backend/tests/unit/services/tracing/test_tracing_service.py	Add autouse fixture to reset shared provider, extensive OTLP unit tests (env readiness, protocol selection/fallback, resource attrs, span lifecycles/attributes, error handling, provider shutdown/reset, integration with TracingService) and concurrency adjustment in test orchestration.
Tests — W3C Trace Context src/backend/tests/unit/services/tracing/test_w3c_trace_context.py	New integration-style tests validating W3C Trace Context extraction/injection, OTLPTracer root-span inheritance from active context, outbound httpx propagation with/without instrumentation, and concurrency isolation. Adds in-memory CollectingExporter and provider-reset fixture.
Minor Setup Change src/backend/base/langflow/initial_setup/setup.py	Micro-optimization: replace nested loop with a dictionary comprehension when flattening component type mappings.

Sequence Diagram(s)

sequenceDiagram
    participant App as Application
    participant OTLP as OTLPTracer
    participant Provider as TracerProvider
    participant Exporter as OTLP Exporter
    participant Collector as OTLP Endpoint

    App->>OTLP: __init__(trace_name, trace_type, ...)
    OTLP->>OTLP: _validate_otlp_env()
    OTLP->>Provider: get/create shared TracerProvider
    OTLP->>Provider: start root span (store carrier)

    App->>OTLP: add_trace(trace_id, name, inputs, metadata)
    OTLP->>OTLP: _convert_to_otlp_dict(inputs/metadata)
    OTLP->>Provider: start child span (using injected context)

    App->>OTLP: end_trace(trace_id, outputs, logs, error)
    OTLP->>OTLP: _convert_to_otlp_dict(outputs/logs)
    OTLP->>Provider: set attributes/record_exception/end child span

    App->>OTLP: end(inputs, outputs, ...)
    OTLP->>Provider: set root attributes/record_exception/end root span
    OTLP->>Provider: force_flush()/shutdown (shutdown_* helper)
    Provider->>Exporter: export spans
    Exporter->>Collector: send (gRPC or HTTP/protobuf)

Loading

sequenceDiagram
    participant Client as HTTP Client
    participant FastAPI as Server
    participant Propagator as TraceContext Propagator
    participant OTLP as OTLPTracer
    participant Exporter as OTLP Exporter

    Client->>FastAPI: Request + traceparent header
    FastAPI->>Propagator: extract(traceparent)
    Propagator->>FastAPI: set active context

    FastAPI->>OTLP: instantiate tracer
    OTLP->>OTLP: root span created inheriting active context

    FastAPI->>OTLP: add_trace(...)
    OTLP->>OTLP: child span created with parent linkage

    FastAPI->>OTLP: end_trace(...), end(...)
    OTLP->>Exporter: spans exported with preserved trace_id/parent_id

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 6 | ❌ 3

❌ Failed checks (2 warnings, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 74.34% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Quality And Coverage	⚠️ Warning	Test suite contains race condition in concurrent test (test_w3c_trace_context.py lines 453-465) where patches are applied inside worker threads instead of globally, and unreachable code exists in otlp_base.py lines 105-108.	Move patch.dict and exporter patches outside worker threads in test_concurrent_otlp_tracers_have_isolated_contexts. Remove unreachable return statement from otlp_base.py. Add direct unit tests for OTLPTracer.add_trace(), end_trace(), and end() methods.
Out of Scope Changes check	❓ Inconclusive	Most changes are in-scope (new OTLPTracer, OTLPTracerBase, trace context propagation tests, service integration). However, refactoring existing providers (ArizePhoenixTracer, LangWatchTracer, TraceloopTracer) to inherit from OTLPTracerBase and use shared singleton providers, while improving code reuse, appears partially tangential to the core OTLP endpoint support objective.	Clarify whether refactoring existing tracers to OTLPTracerBase and shared singletons was necessary for the core OTLP feature, or whether it can be separated into a follow-up consolidation PR.

✅ Passed checks (6 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title clearly summarizes the main change: adding support for sending traces to a generic OTLP trace endpoint, which aligns with the primary objective in the changeset.
Linked Issues check	✅ Passed	The PR addresses all primary coding objectives from issue #12117: implementing a generic OTLP tracer using standard OpenTelemetry environment variables, supporting both HTTP and gRPC protocols, respecting OTEL_SERVICE_NAME and other standard config vars, enabling distributed trace context propagation, and avoiding dependency on auto-instrumentation wrappers.
Test Coverage For New Implementations	✅ Passed	PR includes 32 new/modified test functions (23 OTLPTracer-specific tests + 9 W3C trace context tests) with 1,963 lines of test code covering configuration, protocol selection, trace lifecycle, error handling, concurrent isolation, and distributed trace propagation.
Test File Naming And Structure	✅ Passed	Pull request includes properly structured backend test files following pytest conventions with descriptive names and comprehensive coverage.
Excessive Mock Usage Warning	✅ Passed	Test suite demonstrates appropriate mock usage with real implementations for core tracing logic. Mock-to-test ratio is reasonable (0.47-0.33), 155+ assertions across tests validate authentic behavior.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/backend/base/langflow/services/tracing/traceloop.py (1)

111-117: ⚠️ Potential issue | 🟠 Major

Add Traceloop-specific metadata key prefixes for association properties.

Metadata spread into span attributes at lines 111-117 and 159-168 lacks the traceloop.association.properties.* prefix required by Traceloop. The _convert_to_otlp_dict() method only normalizes values and converts keys to strings—it does not add provider-specific prefixes. Custom metadata will be sent as plain span attributes and will not appear as association properties in Traceloop.

Wrap metadata keys with the traceloop.association.properties. prefix before spreading them into attributes:

# Instead of:
**self._convert_to_otlp_dict(metadata or {})

# Use something like:
**{f"traceloop.association.properties.{k}": v for k, v in self._convert_to_otlp_dict(metadata or {}).items()}

Applies to lines 111-117 (child spans) and 159-168 (root span).

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/backend/base/langflow/services/tracing/traceloop.py` around lines 111 -
117, The metadata being spread into span attributes (the attributes dict built
around trace_id/trace_name/trace_type/inputs) is not being prefixed for
Traceloop; change the spread of self._convert_to_otlp_dict(metadata or {}) so
each key is renamed with the "traceloop.association.properties." prefix before
merging (e.g., map the dict returned by _convert_to_otlp_dict to new keys with
that prefix), and apply the same change at both places where metadata is merged
(the child-span attributes block around trace_id/trace_name/inputs and the
root-span attributes block later); keep the conversion logic in
_convert_to_otlp_dict but wrap its output keys with the required prefix when
building the attributes dict.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/backend/base/langflow/services/tracing/otlp_base.py`:
- Around line 79-83: The metadata conversion currently leaves nested dicts/lists
intact in _convert_langflow_type and _convert_to_otlp_dict, which later get
passed to Span.set_attribute and will fail; update _convert_langflow_type to
detect nested containers (dict or list) and return a JSON-stringified
representation (e.g., json.dumps(value, ensure_ascii=False)) instead of the raw
container, or alternatively convert them into sequences/primitives only so that
_convert_to_otlp_dict always returns only str/bool/int/float or sequences of
primitives before anything is passed to Span.set_attribute; make sure to
import/use json and apply this change in _convert_langflow_type and any code
paths used by _convert_to_otlp_dict.

In `@src/backend/base/langflow/services/tracing/otlp.py`:
- Around line 109-120: The protocol selection currently reads only
OTEL_EXPORTER_OTLP_PROTOCOL and accepts an unsupported "http/json"; change it to
first read OTEL_EXPORTER_OTLP_TRACES_PROTOCOL (falling back to
OTEL_EXPORTER_OTLP_PROTOCOL) into the protocol variable, and restrict accepted
values to "grpc" and "http/protobuf" when importing OTLPSpanExporter; remove any
"http/json" branch, and when protocol is unrecognized, log a warning and default
to "http/protobuf" before importing
opentelemetry.exporter.otlp.proto.http.trace_exporter.
- Around line 123-137: The current manual parsing of OTEL_RESOURCE_ATTRIBUTES
should be removed and replaced by delegating parsing to the OpenTelemetry SDK:
call the SDK's environment-based resource detector (obtain an env_resource using
the SDK's environment parsing helper) and then merge it with your explicit base
resource that sets "service.name" from OTEL_SERVICE_NAME and
"langflow.project_name" (use Resource.merge with the base resource first so base
values take precedence over env values); update the code in otlp.py where
Resource.create and manual OTEL_RESOURCE_ATTRIBUTES parsing occur (use
Resource.merge and the SDK's env resource detector instead of
split(",")/split("=") logic).
- Around line 144-150: The close() method currently calls force_flush() on the
TracerProvider which leaves BatchSpanProcessor worker threads running; change
close() to call tracer_provider.shutdown() instead of force_flush() to ensure
processors and background threads are stopped, and update the end() method to
invoke close() after ending the root span so each OTLPTracer instance
(tracer_provider, BatchSpanProcessor, span_exporter) is properly shut down;
refer to the tracer_provider attribute, close() and end() methods, and the
BatchSpanProcessor/OTLPSpanExporter creation in the OTLP tracer class to locate
where to replace force_flush() with shutdown() and add the close() call in
end().

In `@src/backend/tests/unit/services/tracing/test_tracing_service.py`:
- Around line 652-662: The test flakes because it only patches a single OTEL env
var and leaves other OTEL_* keys that alter OTLPTracer behavior; update these
tests in test_tracing_service.py (the blocks creating OTLPTracer) to run with a
clean OTEL-related environment by using patch.dict("os.environ", {}, clear=True)
and then re-inject only the required OTEL_EXPORTER_OTLP_TRACES_ENDPOINT (or
alternatively explicitly remove/sanitize all keys matching OTEL_* before
constructing OTLPTracer) so the tracer.ready assertion always exercises the
intended code path.

In `@src/backend/tests/unit/services/tracing/test_w3c_trace_context.py`:
- Around line 156-166: The propagation tests patch.dict calls should create a
clean OTEL environment before setting the test-specific OTLP vars; change the
patch.dict(...) usage in test_w3c_trace_context.py (the blocks that currently
call patch.dict(os.environ, { "OTEL_EXPORTER_OTLP_ENDPOINT": ... })) to either
use patch.dict(os.environ, {}, clear=True) followed by another patch.dict that
sets only the required OTEL_* keys, or add clear=True to the existing patch.dict
call and include all OTEL_* keys you need (e.g., OTEL_EXPORTER_OTLP_ENDPOINT,
OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_EXPORTER_OTLP_HEADERS,
OTEL_RESOURCE_ATTRIBUTES) so the CollectingExporter-based tests run
hermetically; apply the same change to the other two blocks noted (around the
ranges mentioned).

---

Outside diff comments:
In `@src/backend/base/langflow/services/tracing/traceloop.py`:
- Around line 111-117: The metadata being spread into span attributes (the
attributes dict built around trace_id/trace_name/trace_type/inputs) is not being
prefixed for Traceloop; change the spread of self._convert_to_otlp_dict(metadata
or {}) so each key is renamed with the "traceloop.association.properties."
prefix before merging (e.g., map the dict returned by _convert_to_otlp_dict to
new keys with that prefix), and apply the same change at both places where
metadata is merged (the child-span attributes block around
trace_id/trace_name/inputs and the root-span attributes block later); keep the
conversion logic in _convert_to_otlp_dict but wrap its output keys with the
required prefix when building the attributes dict.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3d02fec8-bc02-4bd6-8320-53d5d735795b

📥 Commits

Reviewing files that changed from the base of the PR and between cacb54d and 63c0a5f.

📒 Files selected for processing (8)

• src/backend/base/langflow/services/tracing/arize_phoenix.py
• src/backend/base/langflow/services/tracing/langwatch.py
• src/backend/base/langflow/services/tracing/otlp.py
• src/backend/base/langflow/services/tracing/otlp_base.py
• src/backend/base/langflow/services/tracing/service.py
• src/backend/base/langflow/services/tracing/traceloop.py
• src/backend/tests/unit/services/tracing/test_tracing_service.py
• src/backend/tests/unit/services/tracing/test_w3c_trace_context.py

[ringerc]

I've rebased this onto release-1.9.0 per guidance recently added to CONTRIBUTING.md

Some test failures exist, but they were pre-existing on the release-1.9.0 branch.

[ringerc]

CI job failures appear unrelated; an error for TestConfigureComponent.test_configure_dynamic_field on very old, deprecated Python version

[ringerc]

@erichare @viktoravelino @Jkavia @schuellerf I've prepared a PR to add a generic OpenTelemetry destination for trace events. Is there any chance anyone would be willing to review this and consider it for merge?

I've pinged the Discord a couple of times but didn't get a response so I thought I'd try tagging a few people I can see have been involved with tracing in Langflow based on #11689 and/or are active contributors.

[ringerc]

I think there's a workaround for this (as I've been able to get no traction at all on this PR): Configure Langflow to use Arize Phoenix or Langwatch but with a fake API key, and an API endpoint env-var that overrides the endpoint to point to a native OTLP trace API endpoint on an otel collector.

There are plenty of problems with this of course; there's no sensible way to configure custom mutual TLS, for one thing, and it's a hack. But it's ... something.

[ringerc]

After further digging I have been able to determine that there are two partial workarounds for getting Langflow traces, but neither will deliver working trace-context propagation through downstreams due to missing support in Langflow itself.

The Phoenix exporter cannot be used as it uses a custom span exporter, but the Arize exporter will work, and the Langflow exporter will work when workaround is applied on the otel-collector side.

Arize exporter

The Arize exporter can be used without otel-collector configuration changes if the otel-collector does not require mutual TLS (or TLS is handled via a service mesh like Istio).

Configure a fake Arize endpoint to point to an otel-collector gRPC endpoint, like:

ARIZE_API_KEY=local
ARIZE_SPACE_ID=local
ARIZE_COLLECTOR_ENDPOINT=http://SERVICE.NAMESPACE.svc.cluster.local:4317

Omit the http:// prefix if TLS is desired.

The otel collector will ignore the suffix appended by the Arize exporter and accept the gRPC span pushes.

Limitations:

• Trace context propagation is broken
• No custom TLS configuration; cannot set client cert/key, cannot set a custom CA cert
• Missing otel resource attributes
• No sampling configuration

Langwatch exporter

A similar approach works with Langwatch but the otel-collector configuration must be customized to accept traces on the Langwatch endpoint:

receivers:
      # ...
      otlp/langwatch:
        protocols:
          http:
            endpoint: ${env:MY_POD_IP}:4319
            traces_url_path: /api/otel/v1/traces
      pipelines:
          # ...
          receivers:
            # ...
            - otlp/langwatch

The k8s workload and Service will also need updating to add port 4319.

Then the Langflow environment can be updated to add env-vars:

LANGWATCH_API_KEY="local"
LANGWATCH_ENDPOINT="http://SERVICE.NAMESPACE.svc.cluster.local:4319"

Limitations are the same as for Arize.

[ringerc]

Rebased onto release-1.10.0 @ cb10f1d, on top of #12962

[ringerc]

Currently revising this a little to tidy up integration with the prior trace context PR and add some more testing to guard against regressions in existing tracers.

Edit: done. If you want I can rebase the changes into the original commit series to make it easier to review.