[CI unblock] Fix two bugs causing isaaclab (core) per-test timeouts#5595
[CI unblock] Fix two bugs causing isaaclab (core) per-test timeouts#5595hujc7 wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🤖 Isaac Lab Review Bot — PR #5595
Verdict: ✅ LGTM — clean, low-risk CI fix.
Summary
Three isaacsim experimental extensions (isaacsim.core.experimental.primdata, isaacsim.robot.wheeled_robots.nodes, isaacsim.sensors.experimental.rtx) are marked { optional = true } in apps/isaaclab.python.kit because they don't ship in publicly-released isaacsim 6.0.x. This unblocks CI by avoiding Kit's slow remote-registry fallback (~55s locally, 1000s+ on runners).
Review Notes
- Correctness:
{ optional = true }is Kit's standard pattern for conditionally-available extensions (already used in isaacsim's own kit files likeisaacsim.app.compatibility_check). Extensions auto-load when present, skip silently when absent. - No functional regression: PR body confirms no Isaac Lab code directly imports these by name — they register OmniGraph nodes that will auto-register once the extensions ship publicly.
- Forward-compatible: No follow-up PR needed; once upstream isaacsim ships these extensions, they'll load automatically.
- Changelog: Well-structured RST fragment in
changelog.d/, explains root cause and fix clearly. - Scope: Minimal — only the 3 confirmed-missing extensions are changed; neighboring hard deps are untouched.
No issues found. Good to merge.
Greptile SummaryThis PR fixes a CI regression caused by three Isaac Sim extensions being declared as hard dependencies in
Confidence Score: 5/5Safe to merge — narrowly scoped to one kit experience file with a local test confirmation (57s fail → 13.82s pass). The change swaps No files require special attention. Both changed files are straightforward: a config tweak and its changelog entry. Important Files Changed
Sequence DiagramsequenceDiagram
participant Test as CI Test
participant Kit as Kit Resolver
participant Local as Local Ext Cache
participant Registry as Remote Registry
Note over Test,Registry: Before this PR (broken)
Test->>Kit: resolve isaaclab.python.kit deps
Kit->>Local: find isaacsim.core.experimental.primdata
Local-->>Kit: not found
Kit->>Registry: remote sync (10k+ packages, 4 registries)
Registry-->>Kit: slow / timeout (~55s local, 1000s+ CI)
Kit-->>Test: resolver failure / exit-1
Note over Test,Registry: After this PR (fixed)
Test->>Kit: resolve isaaclab.python.kit deps
Kit->>Local: "find isaacsim.core.experimental.primdata (optional=true)"
Local-->>Kit: not found — skip
Kit->>Local: "find isaacsim.robot.wheeled_robots.nodes (optional=true)"
Local-->>Kit: not found — skip
Kit->>Local: "find isaacsim.sensors.experimental.rtx (optional=true)"
Local-->>Kit: not found — skip
Kit-->>Test: resolved (~13s, PASSED)
Reviews (1): Last reviewed commit: "[CI unblock] Mark missing isaacsim exper..." | Re-trigger Greptile |
d3088fe to
8eb3a3b
Compare
…saaclab.python.kit
The `apps/isaaclab.python.kit` experience declared three Isaac Sim
extensions as hard dependencies that don't exist in publicly-released
`isaacsim 6.0.x`:
isaacsim.core.experimental.primdata
isaacsim.robot.wheeled_robots.nodes
isaacsim.sensors.experimental.rtx
These were introduced in PR isaac-sim#5293 as part of the deprecated-extension
migration ahead of an upstream Isaac Sim release, but the renamed
targets ship only in unreleased Isaac Sim builds. When Kit's resolver
can't satisfy a hard dep locally it falls back to a remote registry
sync that walks 10k+ packages across 4 registries -- which silently
burns ~55s on a fast network and reaches per-test timeouts of
1000s/1700s on the IsaacLab CI runners, where the registry endpoints
appear to be slow or rate-limited.
That timeout cascade has been hitting `isaaclab (core) [1/3]` for
~7 days, blowing up `test_non_headless_launch.py`,
`test_outdated_sensor.py`, and `test_color_randomization.py` -- all
three import `AppLauncher` at module top and share the same kit-load
boot path. CI then retries each test 3x, burning ~150 min of
wall-time per run.
Marking the three as `{ optional = true }` lets Kit skip them if
absent (matching the convention used elsewhere, e.g.
`omni.kit.notification_manager = { optional = true }` in
`isaacsim.app.compatibility_check`). When the upstream extensions
eventually ship publicly, they auto-load -- no further change needed.
Verified locally: `test_non_headless_launch` goes from 57s exit-1
to 13.82s PASSED.
Refs: isaac-sim/IsaacLab-Internal#... (CI slowness tracking)
8eb3a3b to
357141e
Compare
Summary
Two independent bugs that both show up in the
isaaclab (core) [1/3]CI matrix as per-test timeouts on:source/isaaclab/test/app/test_non_headless_launch.pysource/isaaclab/test/sensors/test_outdated_sensor.pysource/isaaclab/test/envs/test_color_randomization.pyCI evidence:
Fix 1 — Missing isaacsim experimental extensions in
apps/isaaclab.python.kitThe experience declares three Isaac Sim extensions as hard deps that don't exist in publicly-released
isaacsim 6.0.x:isaacsim.core.experimental.primdataisaacsim.robot.wheeled_robots.nodesisaacsim.sensors.experimental.rtxThey were introduced in #5293 as part of the deprecated-extension migration ahead of an upstream Isaac Sim release, but the renamed targets only ship in unreleased Isaac Sim builds.
When Kit's resolver can't satisfy a hard dep locally, it falls back to a remote registry sync that walks 10k+ packages across 4 registries (
kit/default,kit/sdk,kit/prod/default,kit/prod/sdk). On a fast network this still burns ~55s before failing; on the CI self-hosted runners the registry endpoints appear slow / rate-limited, which is what's pushing per-test wall-time to 1000s / 1700s. CI then retries each test 3×.Marks the three as
{ optional = true }. Kit will skip them when absent (the convention used by isaacsim's own extensions, e.g.omni.kit.notification_manager = { optional = true }inisaacsim.app.compatibility_check). When upstream Isaac Sim ships these extensions publicly, they auto-load — no further IL change.Fix 2 —
randomize_visual_color(and three siblings) crash onrep.__file__ = Nonesource/isaaclab/isaaclab/envs/mdp/events.pyhad four sites doing:This assumes
omni.replicator.corealways exposes a string__file__. But Kit's extension manager loadsomni.replicator.coreas a namespace package, leavingrep.__file__ = None. Calling.split("/")onNoneraisesAttributeError: 'NoneType' object has no attribute 'split', killing the event term beforecompare_versionsruns. Locally reproduced —test_color_randomizationfails with exactly that traceback atevents.py:2249.Replaces the four occurrences with a small helper:
rep.__path__[0]is always populated for namespace packages and embeds the extscache directory name (e.g.omni.replicator.core-1.13.4+110.0.0.lx64.r.cp312), so the version is recoverable.Are all three tests fixed by these?
test_non_headless_launchtest_outdated_sensorisaaclab.python.headless.rendering.kit, has all deps locally)test_color_randomizationrep.__file__ is Nonecrash__file__crash; locally the test still hits a separaterep.functionalissue caused by my local Kit 110.0.0 having a staleomni.replicator.corethat fails onwp.context.Kernelimport. CI uses Kit 110.1.1 with a differentomni.replicator.corethat doesn't have that issue, so CI should pass after both fixes hereHonest caveat: I cannot reproduce tests 2 & 3's CI hangs locally — locally they fail in different ways (or pass). The
kit/defaultregistry-sync path is the most plausible shared mechanism, since:AppLauncherboot path.If the camera tests still time out on this PR's CI, the residual cause is likely separate and worth a follow-up bisect against the suspect commits (#5523 Newton 1.2.0rc2, #5492 OVRTX, #5473 Newton viz markers, etc.).
Test plan
isaacsim==6.0.0.0(PyPI) + Kit 110.0.0 + L40, no MIGtest_non_headless_launch: 57s exit-1 → 14s PASSEDtest_outdated_sensor: 18s PASSED before fix, 17s PASSED after (kit fix doesn't change behavior locally)test_color_randomization: separaterep.functionalissue from local Kit version skew; events.py fix removes therep.__file__crash that would otherwise hit in CI tooisaaclab (core)matrix turns greenRelated
kellyg/fix-installationbranch as a follow-up for "installation, prebundle, and extension-exclusion fixes"; that branch is no longer on the remote, so this PR fills part of that gap. cc @kellyguo11