[CI][Diag] Dump numpy/scipy/openblas state pre-pytest#5626
Closed
hujc7 wants to merge 1 commit into
Closed
Conversation
CI is hitting non-deterministic SIGSEGV crashes that look like the OpenBLAS atfork regression tracked at numpy#30092 and scipy#23686. The hypothesis is that the Isaac Sim base image rotated on 2026-05-12 picked up a scipy>=1.16.2 wheel whose bundled OpenBLAS registers a buggy atfork handler. Without knowing the actual numpy/scipy versions inside the pinned test image, we can't confirm the hypothesis or pick a remediation version. This adds a small diagnostic print right before pytest that emits: - pip show numpy scipy - numpy.__version__ / scipy.__version__ - the on-disk filename of the bundled libscipy_openblas*.so so the GitHub Actions log captures the actual deployed versions for every test job. The output is a few lines of plain text, no artifacts, no impact on test execution.
This was referenced May 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Pure-diagnostic companion to the CI instability investigation in
#C06HLQ6CB41
and OMPE-92261.
Why
The hypothesis being explored is that the Isaac Sim base image rotated on 2026-05-12 picked up a NumPy/SciPy wheel whose bundled OpenBLAS triggers the well-documented atfork SIGSEGV (see numpy#30092, scipy#23686). SciPy 1.16.1 is reported clean, 1.16.2+ has the regression.
But we don't actually know what versions are inside the currently pinned Isaac Sim image (
latest-develop@sha256:0dd49a11…from PR #5600). Without that, we can't:What
Adds a 4-line diagnostic print right before pytest runs, emitting:
pip show numpy scipy(Name / Version / Location)numpy.__version__andscipy.__version__libscipy_openblas*.so(the same name that shows up in the crash backtrace)Output goes to the GitHub Actions log; no artifact upload, no test impact. ~1 second per job.
Relationship to companion PR
This PR is paired with the env-var fix-attempt PR which tests the OpenBLAS thread-pool suppression hypothesis. Decision tree:
Type of change
Checklist