Refactor: run sim CI in single subprocess with parallel workers#493
Merged
ChaoWao merged 1 commit intohw-native-sys:mainfrom Apr 10, 2026
Merged
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
hw-native-sys-bot
force-pushed
the
refactor/sim-single-subprocess
branch
from
421d042 to
d051ff2
Compare
hw-native-sys-bot
changed the title
hw-native-sys-bot
force-pushed
the
refactor/sim-single-subprocess
branch
7 times, most recently
from
da3e69e to
8a5e6c9
Compare
Previously sim launched one subprocess per runtime group to avoid host SO symbol collisions. With the handle-based DeviceRunner API (hw-native-sys#483), multiple runtimes can coexist in a single process. - Replace run_sim_tasks_subprocess (per-runtime subprocesses) with a single _run_device_worker_subprocess call for all tasks - Add parallel sim execution: tasks distributed across cpu_count/20 virtual device IDs, each with its own ChipWorker in a thread - ChipWorker::run() uses std::thread internally so the real work runs outside the Python GIL, enabling true parallelism - Add timeout parameter to _run_device_worker_subprocess using subprocess.run(timeout=) for clean process kill on deadlock - Thread-safe progress: [devN] [M/total] PASS/FAIL: task (Xs)
hw-native-sys-bot
force-pushed
the
refactor/sim-single-subprocess
branch
from
8a5e6c9 to
0089d5a
Compare
ChaoWao
approved these changes
Apr 10, 2026
ChaoWao
added a commit
to ChaoWao/simpler-fork
that referenced
this pull request
Apr 11, 2026
On macOS, `python ci.py -p a2a3sim` (or a5sim) aborts every task with "OMP: Error hw-native-sys#15: Initializing libomp.dylib, but found libomp.dylib already initialized" (SIGABRT) before any DeviceRunner code runs. Two distinct libomp.dylib copies get mapped into the single CI process: homebrew's /opt/homebrew/opt/libomp/lib/libomp.dylib (via numpy -> openblas) and pip torch's .venv/.../torch/lib/libomp.dylib. They have different install names, so dyld loads them both and Intel's libomp aborts on the second init. Surfaced after hw-native-sys#493 collapsed sim CI into one long-lived Python process; each golden's `import numpy`/`import torch` now accumulates conflicting libomps in the same address space. - Set KMP_DUPLICATE_LIB_OK=TRUE at the top of ci.py on darwin, before any import that can transitively pull in numpy or torch. This is Intel's documented escape hatch; safe for our workload where numpy and torch are only used for golden reference math, not parallel OMP regions. - Document the full root cause, debugging steps, and explicit "what not to do" list in docs/macos-libomp-collision.md so future contributors don't re-investigate. Link it from docs/ci.md. - Rewrite the two remaining numpy-based goldens (a2a3/{aicpu,host}_build_graph/bgemm) in torch for style consistency with the rest of examples/. Note this does not avoid the libomp collision on its own -- `import torch` transitively imports numpy. Verified: `python ci.py` passes 32/32 sim tests (20 a2a3sim + 12 a5sim) on macOS without KMP_DUPLICATE_LIB_OK needing to be set manually.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cpu_count // 20virtual device IDs, each with its own ChipWorker in a separate threadrun_runtimeexecutes insideDeviceRunner::create_thread()so each invocation gets proper device binding (sim:pto_cpu_sim_bind_device, onboard:rtSetDevice) without holding Python GILreset_device_context()on onboard after each run to destroy streams +rtDeviceReset, enabling clean re-creation on the next run's threadset_deviceon onboard is now a no-op — device/stream init moved torun_runtime's worker thread viaensure_device_setsubprocess.run(timeout=)for clean kill on deadlock; sim subprocess runs quiet withPTO_LOG_LEVEL=warnTesting