test: skip test_sglang_indexers_sync in nightly only (DYN-2784)

[nv-tusharma]

Overview:

Skip tests/router/test_router_e2e_with_sglang.py::test_sglang_indexers_sync in nightly only so the nightly sglang-pipeline / Test (amd64) matrix stops burning 5+ hours on a known root-caused hang (DYN-2784). The test continues to run — and pass — in pre_merge and post_merge CI.

Parallel to #8468 which does an unconditional skip for the same test under DYN-2784 but is currently red on sglang Multi-GPU checks. Whichever lands first unblocks the next nightly; this one is scoped narrower and stays green in the other pipelines.

Details:

With #8441 (skip test_router_decisions_sglang_multiple_workers) merged yesterday, today's nightly (run 24769044442) shows the hang has shifted downstream by one test, exactly as DYN-2784's root-cause chain predicts:

• test_router_decisions_sglang_multiple_workers[tcp] now reports SKIPPED [96%] (the prior skip is taking effect)
• pytest never emits a header for the next unskipped test — the hang is in its fixture setup
• The next unskipped test in file order is test_sglang_indexers_sync[nats_core] (since test_router_decisions_sglang_dp is already skip-marked under DYN-2265)
• Jobs run 5h06m and hit GitHub's 6h runner cap, triggering the Terminate orphan process: pid (580) (docker) cleanup at ~14:11 UTC

Evidence:

• sglang 12.9 amd64 — job 72537019984, 5h06m, hang
• sglang 13.0 amd64 — job 72537020134, 5h06m, hang

Root cause (from DYN-2784): worker #2 dies in SGLangProcess.__init__ launch; ManagedEngineProcessMixin.__enter__ never proc.poll()s so it proceeds as if both workers are alive; KvRouter then blocks forever waiting for min_initial_workers=2 since only one registered. @pytest.mark.timeout(150) doesn't rescue because signal delivery is swallowed by a C-level syscall in the fixture. The test passes reliably in pre_merge/post_merge — it only wedges in nightly, where the broader test suite accumulates state that makes worker #2's death more likely.

Approach

Rather than an unconditional @pytest.mark.skip, introduce a narrow opt-in:

1. New pytest marker skip_in_nightly — registered in pyproject.toml. Tests flagged with it are excluded from nightly but continue running in pre_merge and post_merge. Applied to test_sglang_indexers_sync.
2. Nightly filter tweak — .github/workflows/nightly-ci.yml sglang single_gpu_test_markers changed from 'sglang and gpu_1' to 'sglang and gpu_1 and not skip_in_nightly'. Scoped to sglang single-GPU, where the hang lives; vllm/trtllm filters are left untouched and other matrices can opt in if/when needed.

This keeps coverage in the PR/post-merge pipelines (where the test passes) while unblocking the nightly amd64 matrix.

Where should the reviewer start?

• tests/router/test_router_e2e_with_sglang.py — @pytest.mark.skip_in_nightly added above the existing decorators on test_sglang_indexers_sync, with a comment explaining the fixture-setup hang and the re-enable condition.
• pyproject.toml — skip_in_nightly registered in the markers list (required by --strict-markers).
• .github/workflows/nightly-ci.yml — one-line filter change on sglang's single-GPU matrix with an inline comment documenting the pattern.

Related Issues:

• Relates to DYN-2784: https://linear.app/nvidia/issue/DYN-2784
• Complements test(router): skip test_sglang_indexers_sync under DYN-2784 #8468 (unconditional skip for the same test; closing either once the other merges is fine)

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Walkthrough

A pytest.mark.skip(...) decorator was added to the test_sglang_indexers_sync test function with reason "DYN-2784: hangs in fixture setup; see linked ticket", preventing the test from executing. An accompanying comment block was moved to appear immediately before the decorator stack.

Changes

Cohort / File(s)	Summary
Test Skip Marker tests/router/test_router_e2e_with_sglang.py	Added @pytest.mark.skip(...) decorator to test_sglang_indexers_sync and repositioned DYN-2784 comment block for clarity regarding fixture hang issue.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly describes the main change: skipping a specific test in nightly CI for a known issue (DYN-2784). It is concise, specific, and directly related to the changeset.
Description check	✅ Passed	The description follows the template with all required sections (Overview, Details, Where should the reviewer start, Related Issues) comprehensively completed. It provides thorough context about the problem, root cause, approach, and implementation details.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[nv-tusharma]

/ok to test ba734dc