UI offline evaluation mishandles OTLP JSON/JSONL traces and crashes with `Cannot read properties of undefined (reading 'map')`

[erauner12]

The Web UI's offline evaluation flow appears to mishandle OTLP JSON traces exported as .json/.jsonl.

A Tempo/OTLP trace with top-level resourceSpans works in parts of the app only after renaming from .json to .jsonl, but offline evaluation/upload still has UI-side assumptions that appear to either reject .jsonl or crash client-side with:

Cannot read properties of undefined (reading 'map')

No useful backend error is emitted in this failure mode.

Environment

• agentevals started with:

agentevals serve --dev

• Server output:

OTLP HTTP: http://0.0.0.0:4318  (OTEL_EXPORTER_OTLP_ENDPOINT default)
OTLP gRPC: 0.0.0.0:4317  (OTEL_EXPORTER_OTLP_PROTOCOL=grpc)
WebSocket: ws://0.0.0.0:8001/ws/traces
API:       http://0.0.0.0:8001/api
Web UI:    http://localhost:5173

• Trace source: Tempo v2 / OTLP JSON export
• Trace shape: top-level resourceSpans

Reproduction

1. Export an OTLP trace JSON from Tempo or another OTLP source.
2. The file has top-level shape:

{
  "resourceSpans": [...]
}

3. Try uploading as trace.otlp.json.

Observed:

Failed to load 'trace.otlp.json': Invalid Jaeger JSON format: expected top-level 'data' key

Server log excerpt:

WARNING:agentevals.api.routes:Failed to load 'trace.otlp.json': Invalid Jaeger JSON format: expected top-level 'data' key in /var/folders/nh/5ff88n4577sfrrqrsf71mkkc0000gp/T/tmpz1mwl3cp/0_trace.otlp.json
INFO:     127.0.0.1:59218 - "POST /api/convert HTTP/1.1" 400 Bad Request
WARNING:agentevals.api.routes:Failed to load 'trace.otlp.json': Invalid Jaeger JSON format: expected top-level 'data' key in /var/folders/nh/5ff88n4577sfrrqrsf71mkkc0000gp/T/tmprk1fftge/0_trace.otlp.json
INFO:     127.0.0.1:59237 - "POST /api/convert HTTP/1.1" 400 Bad Request
WARNING:agentevals.api.routes:Failed to load 'trace.otlp.json': Invalid Jaeger JSON format: expected top-level 'data' key in /var/folders/nh/5ff88n4577sfrrqrsf71mkkc0000gp/T/tmp98dia95n/0_trace.otlp.json
INFO:     127.0.0.1:59277 - "POST /api/convert HTTP/1.1" 400 Bad Request

This suggests the upload path defaults .json to Jaeger even when the content is clearly OTLP.

4. Rename the same file to:

trace.otlp.jsonl

5. The trace now works in at least one path because .jsonl selects otlp-json.

Observed successful backend conversion after renaming:

INFO:agentevals.loader.otlp:Loaded 1 trace(s) from /var/folders/nh/5ff88n4577sfrrqrsf71mkkc0000gp/T/tmp5pzdcsg0/0_trace.otlp.jsonl
INFO:agentevals.converter:Auto-detected trace format: adk for trace mdQk80xJsNjJzZnIYcvllg==
INFO:     127.0.0.1:59942 - "POST /api/convert HTTP/1.1" 200 OK

6. Try using the same .jsonl trace in the offline evaluation UI.

Observed:

Cannot read properties of undefined (reading 'map')

In my run, there was no corresponding useful backend error, suggesting the failure is frontend-side.

One UI path also appeared to mutate the filename in a way that caused it to be treated as Jaeger again:

WARNING:agentevals.api.routes:Failed to load 'trace.otlp.jsonl.json': Invalid Jaeger JSON format: expected top-level 'data' key in /var/folders/nh/5ff88n4577sfrrqrsf71mkkc0000gp/T/tmpxucohlbs/0_trace.otlp.jsonl.json
INFO:     127.0.0.1:59831 - "POST /api/convert HTTP/1.1" 400 Bad Request

Expected behavior

The UI should support OTLP trace uploads consistently across conversion, inspection, and offline evaluation.

At minimum:

• Trace file upload should accept both .json and .jsonl.
• Full OTLP JSON files with top-level resourceSpans should be detected as otlp-json, even when the extension is .json.
• Offline evaluation should pass trace_format: "otlp-json" when the selected trace file is .jsonl or when content detection identifies OTLP.
• The UI should not crash if the streaming evaluation result shape is missing traceResults; it should show a useful error instead.

Actual behavior

There appear to be several inconsistent assumptions:

1. .json uploads are treated as Jaeger JSON and fail on OTLP payloads with top-level resourceSpans.
2. Renaming to .jsonl works around backend format detection.
3. Offline evaluation UI appears to accept only .json in at least one upload/dropzone path, or otherwise does not handle .jsonl cleanly.
4. Frontend crashes with:

Cannot read properties of undefined (reading 'map')

Relevant code pointers

• FileDropZone defaults accept = '.json', which may block or discourage .jsonl trace files in UI upload paths.
• convertTraces(traceFiles, traceFormat?) supports an optional traceFormat, but TraceProvider.setTraceFiles calls convertTraces(files) without passing one.
• runEvaluation calls evaluateTracesStreaming(...) with config containing metrics, judge model, threshold, and trajectory match type, but does not include trace_format.
• The backend accepts .json and .jsonl trace files and infers .jsonl as otlp-json; eval set files are correctly restricted to .json.
• TraceProvider completion handling does result.traceResults.map(...) without guarding for missing/alternate result shape, which likely explains the frontend crash.

Suggested fix

1. Add trace-format detection in the frontend or backend based on content:

   • top-level resourceSpans => otlp-json
   • top-level data => jaeger-json
   • Tempo v2 wrapper { "trace": { "resourceSpans": [...] } } could either be supported or emit a targeted error suggesting jq '.trace // .'.

2. Update trace file upload dropzones to accept:

.json,.jsonl

while keeping eval-set upload restricted to .json.

3. Ensure offline evaluation passes trace_format: "otlp-json" when appropriate.

4. Add a defensive guard around:

result.traceResults.map(...)

for example:

const traceResults =
  result.traceResults ??
  (result as any).trace_results ??
  [];

if (!Array.isArray(traceResults)) {
  // surface a useful UI error instead of crashing
}

5. Add regression tests for:

   • .json OTLP file with top-level resourceSpans
   • .jsonl OTLP file containing full OTLP JSON, not necessarily newline-delimited individual spans
   • offline evaluation UI flow with an OTLP trace
   • malformed/partial SSE completion payload that lacks traceResults

Workaround

For CLI/API usage, explicitly pass OTLP format:

agentevals run trace.otlp.jsonl \
  --format otlp-json \
  --eval-set eval_set.json \
  -m response_match_score

or call the API with:

{
  "trace_format": "otlp-json"
}

For the UI, renaming trace.otlp.json to trace.otlp.jsonl works in some paths but not consistently in offline evaluation.

Human confirmation

• I am a human user filing this issue.

[krisztianfekete]

Hi @erauner12, thank you for opening this issue!
I just fixed this and released v0.7.3 that should not have this problem anymore. Can you please test it and see if it works for you as well?

Also, wondering if you can share some details on your use-cases and setup, e.g. are you considering doing manually offline evals via the UI vs. using agentevals in an OTel pipeline, etc? We are always happy to learn more about our user's preferences/requirements. :)

You can also hop on our Discord and chat there if it's more convenient for you.

[erauner12]

Thanks @krisztianfekete, I really appreciate the quick fix and the v0.7.3 release.
I tested v0.7.3 yesterday against a small validation setup in my repo. The traces are OTLP JSON traces from a Kubernetes query-agent scenario, with one known-good trace fixture and one deliberate mismatch fixture used to verify both
pass and fail behavior.
What worked:

• CLI/file scoring successfully loaded the OTLP JSON fixtures.
• The known-good fixture scored tool_trajectory_avg_score=1.0 / PASSED.
• The deliberate mismatch fixture scored tool_trajectory_avg_score=0.0 / FAILED.
• Multipart API uploads worked via:
  • POST /api/validate/eval-set
  • POST /api/evaluate
• The API path worked with trace_format=otlp-json, the same eval set, threshold 1.0, and trajectoryMatchType=EXACT.
• Live receiver/session validation also worked for one bounded query-agent run: I was able to select a completed session, create an eval set from it, and evaluate that selected session successfully.
  So for the CLI/API/backend paths I needed, v0.7.3 appears to resolve the blocker. I have not done a full browser UI regression pass yet, so I can’t fully confirm every frontend path from the original issue.
  For context, my main use case is moving toward repeatable validation lanes for agent behavior: deterministic file/fixture scoring, multipart upload/API scoring, and live OTel receiver/session validation in local Kubernetes before rolling
  the pattern out to real Kubernetes clusters. I’m using Garden for the repo-owned validation workflow: https://docs.garden.io/overview/what-is-garden

The UI is useful for inspection, and I’d like to incorporate it into the workflow so someone can interactively inspect/test the agent setup without touching code. The source shape for one scenario currently looks roughly like:

evals/scenarios/query-agent-kubernetes-readonly-smoke/
├── agentevals/
│   ├── eval-config.yaml
│   └── eval-set.json
├── fixtures/traces/
│   ├── positive.otlp.json
│   └── mismatch.otlp.json
├── scenario-metadata.yaml
└── user-prompt.txt

One UX note from the live receiver flow: once the receiver connects to a Kubernetes environment, the Live Agent Sessions list can accumulate many completed OTLP sessions. Some of those sessions are not useful as eval-set/golden
candidates because they appear to have no extracted invocations or useful eval-set source. In one test run I saw hundreds of completed sessions and had to manually identify the bounded query-agent session that provided usable
invocations.

Longer term, filtering/grouping by trace attributes, scenario/eval-set identifiers, session name, invocation count, or “can be used as eval set” status would make the UI much easier to use for this workflow.
ex:

I also checked the live sessions API directly. In my current run, /api/streaming/sessions returns 291 completed sessions. All have spans, but only 1 exposes a non-empty invocations list. That one usable session has
service.name=query_agent, spanCount=839, and invocations.length=2. The sessions crowding the UI mostly have service.name=kubernetes-mcp-server, spanCount=1-2, and invocations=null.

So the UX issue is less “completed vs incomplete” and more “completed trace sessions that are usable as eval-set/golden candidates vs completed low-level OTLP sessions that have spans but no extracted invocations.” A filter or badge for
invocations.length > 0 / “can be used as eval set” would be very helpful. I can create another issue if it would help with more details!

Would love to get together on discord! I'll join in and introduce myself.

[krisztianfekete]

@erauner12, would you mind opening an issue about the noisy and unrelated stream chunks on the UI so we can fix it?

[erauner12]

@krisztianfekete will do! Will try to do some investigation first on my end for context.