Migrate observability to OpenTelemetry and pipeline to LangGraph

[rafacm]

Summary

• OpenTelemetry: Replace Langfuse-specific instrumentation with OTel SDK. Traces export to any OTLP-compatible backend (Langfuse, Sentry, Jaeger). New RAGTIME_OTEL_* env vars replace RAGTIME_LANGFUSE_*.
• LangGraph: Replace Django Q2 signal-based pipeline dispatch with a StateGraph that supports autonomous step skipping, recovery routing, and resume-from-failure. Remove django-q2 dependency entirely.
• LangGraph Studio: Add langgraph.json and server module for local graph visualization via Studio desktop app.

Changes

Phase 1 — OpenTelemetry

• Created episodes/telemetry.py (@trace_step, @trace_provider, record_llm_input/output)
• Deleted episodes/observability.py (256 lines of Langfuse-specific code)
• Updated all step files, providers, agents, settings, .env.sample, configure wizard

Phase 2 — LangGraph Pipeline

• Created episodes/graph/ package (state, nodes, edges, pipeline, run — 7 new files)
• 10-node graph: route → scrape → download → transcribe → summarize → chunk → extract → resolve → embed + recovery
• Entry router inspects episode data for skip-already-done and resume-from-failure
• Added handle_step_failure_from_graph() for graph-initiated recovery
• Removed queue_next_step signal handler, Django Q2, Q_CLUSTER config
• Updated admin to use threading.Thread + run_pipeline()

Phase 3 — LangGraph Studio

• langgraph.json + episodes/graph/server.py for Studio integration

Documentation

• Updated README.md, doc/README.md, CLAUDE.md for new architecture
• Plan, feature doc, session transcripts, and changelog entry included

Test plan

• uv run python manage.py test — all 211 tests pass (9 skipped for optional deps)
• uv run python manage.py check — no issues
• Run pipeline with RAGTIME_OTEL_EXPORTER=console and verify spans in terminal
• langgraph dev starts and Studio shows the graph
• Process an episode end-to-end through the graph

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR migrates RAGtime’s pipeline observability from Langfuse-specific instrumentation to OpenTelemetry, and replaces the Django Q2 + post_save signal-driven pipeline dispatch with a LangGraph StateGraph orchestration layer (including LangGraph Studio support).

Changes:

• Replace Langfuse observability (episodes/observability.py) with an OpenTelemetry-based module (episodes/telemetry.py) and update step/provider instrumentation accordingly.
• Replace Django Q2 pipeline dispatch (signals + async_task) with a LangGraph pipeline (episodes/graph/*) and update admin-triggered pipeline runs to execute via background threads.
• Update configuration/docs/tests to reflect new OTel env vars (RAGTIME_OTEL_*) and new orchestration model (LangGraph).

Reviewed changes

Copilot reviewed 49 out of 51 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
uv.lock	Updates resolved dependencies: removes django-q2/langfuse, adds langgraph + langchain-core deps and OTel libs.
pyproject.toml	Removes django-q2, adds langgraph, opentelemetry-* (and OTLP exporter in observability extra).
ragtime/settings.py	Removes django_q/Q_CLUSTER; replaces RAGTIME_LANGFUSE_* with RAGTIME_OTEL_*.
.env.sample	Replaces Langfuse env vars with OpenTelemetry env vars.
core/management/commands/_configure_helpers.py	Updates configure wizard sections/fields to OpenTelemetry settings.
core/tests/test_configure.py	Updates wizard tests to match new OTel prompts/inputs.
episodes/telemetry.py	New OTel tracing setup + decorators + span event helpers for LLM input/output.
episodes/observability.py	Deleted Langfuse-specific observability implementation.
episodes/providers/openai.py	Switches to plain openai.OpenAI and OTel span events/decorators.
episodes/scraper.py	Replaces @observe_step with @trace_step.
episodes/downloader.py	Not directly changed in diff shown, but now invoked by graph nodes instead of signals/Q2.
episodes/transcriber.py	Replaces @observe_step with @trace_step.
episodes/summarizer.py	Replaces @observe_step with @trace_step.
episodes/extractor.py	Replaces @observe_step with @trace_step.
episodes/resolver.py	Replaces @observe_step with @trace_step.
episodes/signals.py	Removes post_save pipeline dispatch; keeps only step_completed/step_failed.
episodes/apps.py	Stops auto-connecting recovery handler; calls telemetry setup on app ready.
episodes/recovery.py	Adds handle_step_failure_from_graph() for graph-initiated recovery.
episodes/admin.py	Replaces Q2 async_task usage with threading.Thread to run graph + recovery tasks.
episodes/agents/agent.py	Reworks recovery agent tracing to use OTel spans and flush OTel provider.
episodes/agents/tools.py	Records screenshots as OTel span events (replacing Langfuse media attachment).
episodes/graph/state.py	Defines EpisodeState schema for LangGraph state passing.
episodes/graph/edges.py	Entry routing + after-step conditional routing (including recovery routing).
episodes/graph/nodes.py	Wraps existing step functions as graph nodes + adds embed/recovery nodes.
episodes/graph/pipeline.py	Defines and compiles the ingestion StateGraph.
episodes/graph/run.py	Adds run_pipeline() entry point that creates runs and invokes the compiled graph.
episodes/graph/server.py	Ensures django.setup() is run before LangGraph Studio imports the graph.
langgraph.json	Adds LangGraph Studio/server configuration pointing to the ingestion graph.
episodes/tests/test_telemetry.py	New tests for OTel helper behaviors and calling conventions.
episodes/tests/test_observability.py	Deleted Langfuse observability tests.
episodes/tests/test_signals.py	Deleted tests for now-removed Q2/signal dispatch behavior.
episodes/tests/test_admin.py	Updates admin tests to expect thread-based background execution.
episodes/tests/test_scraper.py	Updates tests for new run creation assumptions and status-based behavior.
episodes/tests/test_transcribe.py	Removes Q2 signal patching; calls step function directly.
episodes/tests/test_summarize.py	Removes Q2 signal patching; calls step function directly.
episodes/tests/test_chunk.py	Removes Q2 signal patching; calls step function directly.
episodes/tests/test_extract.py	Removes Q2 signal patching; calls step function directly.
episodes/tests/test_resolve.py	Removes Q2 signal patching; calls step function directly.
episodes/tests/test_download.py	Removes Q2 signal patching; calls step function directly.
episodes/tests/test_events.py	Removes Q2 signal patching; validates event emission directly.
episodes/tests/test_models.py	Removes Q2 signal patching from model tests.
episodes/tests/test_recovery.py	Updates recovery tests to manually connect signals where needed.
episodes/tests/test_agent_resume.py	Removes Q2 patching assumptions around resume behaviors.
README.md	Updates documentation to reference OpenTelemetry + LangGraph; removes qcluster instructions.
doc/README.md	Updates pipeline + observability docs for LangGraph and OpenTelemetry.
CHANGELOG.md	Adds a 2026-04-13 entry describing the migrations.
AGENTS.md	Updates agent guidance to reflect LangGraph + OpenTelemetry architecture.
doc/plans/2026-04-13-otel-langgraph-migration.md	Adds migration plan documentation.
doc/features/2026-04-13-otel-langgraph-migration.md	Adds feature documentation for the migration.
doc/sessions/2026-04-13-otel-langgraph-migration-implementation-session.md	Adds implementation session transcript.
doc/sessions/2026-04-13-otel-langgraph-migration-planning-session.md	Adds planning session transcript.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[rafacm]

High: this change appears to remove the primary ingestion trigger when a new episode is created.

episodes/signals.py now only defines step_completed / step_failed and no longer contains the old post_save-driven queue_next_step hook. I also don’t see a replacement on the normal add path in EpisodeAdmin — the only new pipeline entry point I found is the reprocess action’s background thread in episodes/admin.py:306-316.

Impact: creating an Episode from the admin now looks like it will leave the row in pending indefinitely instead of starting the pipeline. That regresses the documented workflow in README.md:134-136 ("submit episode URLs, monitor pipeline progress...").

The focused Django test slice passed, but this behavior is no longer covered because episodes/tests/test_signals.py was deleted in this PR, so there is currently no test protecting automatic ingestion on create.

[rafacm]

Medium: admin-triggered pipeline/recovery work is now launched via raw daemon threads, which makes the "queued" behavior non-durable.

The reprocess action starts threading.Thread(..., daemon=True) in episodes/admin.py:312-316, and recovery retry does the same in episodes/admin.py:686-690.

Impact: if the Django process reloads, exits, or the worker is recycled after the admin response returns, daemon threads can be terminated immediately and the background work is silently lost. That is a reliability regression from the previous task-queue model, because the UI reports success (Queued ...) without any persistence or retry semantics.

The current tests only mock threading.Thread, so they verify that a thread is started, not that the work survives process restarts.

[rafacm]

Fixed in 9134c90.

1. Auto-ingestion restored — added save_model() override to EpisodeAdmin that starts the pipeline in a background thread when a new episode is created (change=False). This restores the auto-start behavior that was lost when the post_save signal handler was removed during the Django Q2 → LangGraph migration.

2. Test added — test_create_episode_auto_starts_pipeline verifies that creating an episode via the admin triggers _run_pipeline_task.

[rafacm]

Fixed in 9134c90.

Removed daemon=True from all three threading.Thread usages (reprocess, recovery retry, and the new auto-start on create). Non-daemon threads block process exit until they complete, so work in progress survives graceful shutdown (e.g., runserver reload, gunicorn graceful restart) instead of being silently killed.

This doesn't provide full persistence across hard kills or crashes — that would require re-introducing a task queue. But it eliminates the most common failure mode (work lost on reload) while keeping the implementation simple.

[rafacm]

High: the auto-start-on-create fix still has a transaction race.

In episodes/admin.py:182-189, save_model() starts a background thread immediately after super().save_model(...):

threading.Thread(
    target=_run_pipeline_task,
    args=(obj.pk,),
).start()

This thread uses a separate DB connection, so it can run before the admin request's transaction commits and fail to see the newly created Episode row at all. In that case _run_pipeline_task() / run_pipeline() can raise Episode.DoesNotExist, and the new episode is left stuck without ingestion starting.

The safe pattern here is to schedule the thread from transaction.on_commit(...) so the worker only starts after the row is committed and visible to other connections.

The new test does not catch this because it mocks threading.Thread rather than exercising the real post-commit behavior.

[rafacm]

Fixed in b16c617.

Wrapped the background thread start in transaction.on_commit() so the worker only starts after the Episode row is committed and visible to other DB connections. Updated the test to use captureOnCommitCallbacks(execute=True) to verify the on_commit behavior.