Add structured-output migration repro test

[fergusfinn]

Summary

• add a focused fault-tolerance repro for structured-output request migration with real local dynamo.frontend and real dynamo.vllm workers
• extend the migration test helper to accept a payload override so the repro can inject an OpenAI response_format JSON schema request
• kill the serving worker mid-stream and assert the resumed response still parses as JSON

What this reproduces

On the current v1.0.1-based environment, request migration is happening, but the resumed structured-output stream becomes invalid JSON.

Observed in the targeted test run on gotenks:

• frontend logs Stream disconnected... recreating stream...
• migration metrics report ongoing_request: 1, new_request: 0
• the resumed response contains nested restarted JSON content, for example:

{
  "animals": [
    {
      "name": "Lion",
      "habitat": "{
        "animals": [
          {

• json.loads(...) then fails with:

json.decoder.JSONDecodeError: Invalid control character at: line 7 column 20 (char 66)

Repro command

source /tmp/dynamo-install-mv3o75/.venv/bin/activate
cd /tmp/dynamo-install-mv3o75
python -m pytest -q tests/fault_tolerance/migration/test_vllm_structured.py::test_request_migration_vllm_aggregated_structured_output -s

Latest result

FAILED tests/fault_tolerance/migration/test_vllm_structured.py::test_request_migration_vllm_aggregated_structured_output[tcp]
1 failed in 49.31s

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a10f9700e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Mark the repro test non-blocking until migration fix lands

This new test is tagged post_merge, and the post-merge workflow selects all vllm and gpu_1 tests with (pre_merge or post_merge) markers, so it will run in nightly CI; because the test asserts json.loads(response_text) succeeds after forced migration (the known repro path), it will keep the post-merge pipeline red on environments where the migration bug is still present. Please gate it with xfail/skip (or remove the post_merge marker) until the product fix is merged.

Useful? React with 👍 / 👎.

[fergusfinn]

Reran the repro on the rebased main branch in a fresh Python 3.11 environment (/tmp/dynamo-install-mv3o75/.venv-main) with current vllm deps and rebuilt local bindings.

Command:

source /tmp/dynamo-install-mv3o75/.venv-main/bin/activate
cd /tmp/dynamo-install-mv3o75
python -m pytest -q tests/fault_tolerance/migration/test_vllm_structured.py::test_request_migration_vllm_aggregated_structured_output -s

Latest result on main:

FAILED tests/fault_tolerance/migration/test_vllm_structured.py::test_request_migration_vllm_aggregated_structured_output[tcp]
1 failed in 60.17s

Observed behavior differs from the earlier v1.0.1 snapshot run:

• worker 2 is killed as intended
• frontend logs Stream disconnected... recreating stream...
• worker 1 then hits a vLLM engine crash / segfault during the resumed request
• frontend logs Cannot recreate stream: no instances found for endpoint dynamo/backend/generate
• the test fails in validate_response(...) because the client sees an event: error SSE and a 12.293s delay instead of a completed migrated response

So on rebased main, this repro is still failing, but the failure mode is now migration collapse / backend crash rather than malformed resumed JSON.

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information.