refactor(test): auto-stub missing modules via meta-path finder#9250
Draft
furionw wants to merge 1 commit into
Draft
refactor(test): auto-stub missing modules via meta-path finder#9250furionw wants to merge 1 commit into
furionw wants to merge 1 commit into
Conversation
Replace the hand-maintained STUB_MODULES list (~190 entries) with a
sys.meta_path import hook that auto-stubs any third-party module the
real loaders can't resolve. The list was drift-prone — stale entries
clutter and missed entries surface only as red CI runs.
Mechanism (Python 3.12 modern API): _AutoStubFinder.find_spec returns
a stub ModuleSpec backed by _StubLoader, which instantiates the
existing _StubModule. Appended to the END of sys.meta_path so real
finders win first; auto-stubbing only fires on ModuleNotFoundError.
is_package=True + empty __path__ makes the cascade automatic — no
need to enumerate vllm.* / sglang.* / kubernetes.* upfront.
Three guard rails address the worry that auto-stub silently masks
real bugs:
- First-party namespaces (dynamo.*, dynamo_run.*, tests.*,
components.*, _pytest.*) are excluded — a typo or real bug in our
own code surfaces as ImportError. ALWAYS_STUB_EXACT lets
dynamo._core / nixl._api fall through (native bindings, .so not
built in pre-commit env).
- sys.stdlib_module_names skips the entire stdlib namespace at the
top level. Catches platform-specific imports (winreg, _winapi,
msilib, _posixsubprocess) that stdlib code wraps in try/except —
auto-stubbing those makes blocks take the wrong branch (Windows
code path on Linux, mimetypes registry read against a stub).
- importlib.util.find_spec presence-checks are detected by stack
walk and return None. Otherwise patterns like dash/dash.py's
`if find_spec("dash_design_kit"): metadata.version(...)` would
false-positive and fail at the metadata lookup.
The two specials previously inlined (vllm.__version__,
pytest_benchmark.logger.PytestBenchmarkWarning) move into a
_LOADER_SPECIALS dict keyed by module name; loader applies them in
exec_module after the bare stub is created.
Visibility: stderr summary line `[auto-stub] N modules across M
packages: ...` printed after collection so reviewers can spot-check
what got stubbed (surprising names like "pandas" surface immediately).
Net diff: tests/report_pytest_markers.py shrinks from 711 to ~605
lines (-103). pre-commit hook still passes; auto-stub catches ~200
modules across ~60 packages on a clean tree (cascade discovers the
actual transitive set, vs the manual list's 190 hand-picked entries).
Also adds pytest.mark.unit to test_trtllm_request_handler_factory.py
— the validator now collects this file (which previously failed
under STUB_MODULES due to tensorrt_llm import errors) and correctly
flags the missing Test Type marker. Sibling tests in the same dir
already had it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
tests/report_pytest_markers.py's hand-maintainedSTUB_MODULESlist (~190 entries) drifts: stale entries clutter, missed entries surface only as failed CI runs hours later. Reviewer feedback (Keiven) on PR #7796 asked for an import hook that auto-stubs anything that fails to resolve, eliminating the list entirely. A prior attempt at this used the deprecatedfind_module/load_moduleAPI that Python 3.12 ignores (52bec7279d removed it as dead code) — the right fix is the modernfind_spec+Loaderpair, with guards against the failure modes auto-stubbing introduces (stdlib platform-specific imports wrapped in try/except,find_spec()presence checks).What Change
_AutoStubFinderappended to end ofsys.meta_path; cascades submodules viais_package=True.sys.stdlib_module_names, andimportlib.util.find_specpresence checks._LOADER_SPECIALSdict replaces inline patches forvllm.__version__andpytest_benchmark.logger.PytestBenchmarkWarning.pytest.mark.unittotest_trtllm_request_handler_factory.py(newly surfaced by the validator).Test Plan
pre-commit run pytest-marker-report --all-files— passes;[auto-stub] N modules across M packageslog shows ~200 modules cascade-stubbed.winreg/_winapiskipped,dash/dash.py find_spec("dash_design_kit")returns None,dynamo.runtime.fooraisesModuleNotFoundError.