Skip to content

[CI unblock] Fix two bugs causing isaaclab (core) per-test timeouts#5595

Closed
hujc7 wants to merge 3 commits into
isaac-sim:developfrom
hujc7:jichuanh/ci-unblock-kit-optional-experimental-exts
Closed

[CI unblock] Fix two bugs causing isaaclab (core) per-test timeouts#5595
hujc7 wants to merge 3 commits into
isaac-sim:developfrom
hujc7:jichuanh/ci-unblock-kit-optional-experimental-exts

Conversation

@hujc7
Copy link
Copy Markdown
Collaborator

@hujc7 hujc7 commented May 12, 2026

Summary

Two independent bugs that both show up in the isaaclab (core) [1/3] CI matrix as per-test timeouts on:

  • source/isaaclab/test/app/test_non_headless_launch.py
  • source/isaaclab/test/sensors/test_outdated_sensor.py
  • source/isaaclab/test/envs/test_color_randomization.py

CI evidence:

Fix 1 — Missing isaacsim experimental extensions in apps/isaaclab.python.kit

The experience declares three Isaac Sim extensions as hard deps that don't exist in publicly-released isaacsim 6.0.x:

  • isaacsim.core.experimental.primdata
  • isaacsim.robot.wheeled_robots.nodes
  • isaacsim.sensors.experimental.rtx

They were introduced in #5293 as part of the deprecated-extension migration ahead of an upstream Isaac Sim release, but the renamed targets only ship in unreleased Isaac Sim builds.

When Kit's resolver can't satisfy a hard dep locally, it falls back to a remote registry sync that walks 10k+ packages across 4 registries (kit/default, kit/sdk, kit/prod/default, kit/prod/sdk). On a fast network this still burns ~55s before failing; on the CI self-hosted runners the registry endpoints appear slow / rate-limited, which is what's pushing per-test wall-time to 1000s / 1700s. CI then retries each test 3×.

Marks the three as { optional = true }. Kit will skip them when absent (the convention used by isaacsim's own extensions, e.g. omni.kit.notification_manager = { optional = true } in isaacsim.app.compatibility_check). When upstream Isaac Sim ships these extensions publicly, they auto-load — no further IL change.

Fix 2 — randomize_visual_color (and three siblings) crash on rep.__file__ = None

source/isaaclab/isaaclab/envs/mdp/events.py had four sites doing:

version = re.match(r"^(\d+\.\d+\.\d+)", rep.__file__.split("/")[-5][21:]).group(1)

This assumes omni.replicator.core always exposes a string __file__. But Kit's extension manager loads omni.replicator.core as a namespace package, leaving rep.__file__ = None. Calling .split("/") on None raises AttributeError: 'NoneType' object has no attribute 'split', killing the event term before compare_versions runs. Locally reproduced — test_color_randomization fails with exactly that traceback at events.py:2249.

Replaces the four occurrences with a small helper:

def _get_replicator_version(rep) -> str:
    rep_path = rep.__file__ or (rep.__path__[0] if getattr(rep, "__path__", None) else None)
    if rep_path is None:
        raise RuntimeError("omni.replicator.core has no resolvable __file__ or __path__")
    match = re.search(r"omni\.replicator\.core-(\d+\.\d+\.\d+)", rep_path)
    if match is None:
        raise RuntimeError(f"Could not parse omni-replicator-core version from {rep_path!r}")
    return match.group(1)

rep.__path__[0] is always populated for namespace packages and embeds the extscache directory name (e.g. omni.replicator.core-1.13.4+110.0.0.lx64.r.cp312), so the version is recoverable.

Are all three tests fixed by these?

Test Local repro of CI symptom? Verified fix locally?
test_non_headless_launch yes — Kit dep error in 9-57s vs CI's 1000s+ passes (14s) after the kit-file fix
test_outdated_sensor no — passes locally regardless of the kit fix (its kit experience, isaaclab.python.headless.rendering.kit, has all deps locally) (cannot verify locally; needs CI)
test_color_randomization yes — rep.__file__ is None crash events.py fix removes the __file__ crash; locally the test still hits a separate rep.functional issue caused by my local Kit 110.0.0 having a stale omni.replicator.core that fails on wp.context.Kernel import. CI uses Kit 110.1.1 with a different omni.replicator.core that doesn't have that issue, so CI should pass after both fixes here

Honest caveat: I cannot reproduce tests 2 & 3's CI hangs locally — locally they fail in different ways (or pass). The kit/default registry-sync path is the most plausible shared mechanism, since:

  1. All three tests share the AppLauncher boot path.
  2. CI's runner network behavior + Kit's registry-resolver retry logic could plausibly cascade the failure cost across the suite, even for camera-enabled tests that use a different kit experience.
  3. Both fixes here are real bug fixes, narrowly scoped, and tested for non-regression on test 1.

If the camera tests still time out on this PR's CI, the residual cause is likely separate and worth a follow-up bisect against the suspect commits (#5523 Newton 1.2.0rc2, #5492 OVRTX, #5473 Newton viz markers, etc.).

Test plan

  • Local: isaacsim==6.0.0.0 (PyPI) + Kit 110.0.0 + L40, no MIG
    • test_non_headless_launch: 57s exit-1 → 14s PASSED
    • test_outdated_sensor: 18s PASSED before fix, 17s PASSED after (kit fix doesn't change behavior locally)
    • test_color_randomization: separate rep.functional issue from local Kit version skew; events.py fix removes the rep.__file__ crash that would otherwise hit in CI too
  • CI isaaclab (core) matrix turns green
  • No new failures elsewhere

Related

@github-actions github-actions Bot added isaac-sim Related to Isaac Sim team isaac-lab Related to Isaac Lab team labels May 12, 2026
Copy link
Copy Markdown

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Isaac Lab Review Bot — PR #5595

Verdict: ✅ LGTM — clean, low-risk CI fix.

Summary

Three isaacsim experimental extensions (isaacsim.core.experimental.primdata, isaacsim.robot.wheeled_robots.nodes, isaacsim.sensors.experimental.rtx) are marked { optional = true } in apps/isaaclab.python.kit because they don't ship in publicly-released isaacsim 6.0.x. This unblocks CI by avoiding Kit's slow remote-registry fallback (~55s locally, 1000s+ on runners).

Review Notes

  • Correctness: { optional = true } is Kit's standard pattern for conditionally-available extensions (already used in isaacsim's own kit files like isaacsim.app.compatibility_check). Extensions auto-load when present, skip silently when absent.
  • No functional regression: PR body confirms no Isaac Lab code directly imports these by name — they register OmniGraph nodes that will auto-register once the extensions ship publicly.
  • Forward-compatible: No follow-up PR needed; once upstream isaacsim ships these extensions, they'll load automatically.
  • Changelog: Well-structured RST fragment in changelog.d/, explains root cause and fix clearly.
  • Scope: Minimal — only the 3 confirmed-missing extensions are changed; neighboring hard deps are untouched.

No issues found. Good to merge.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 12, 2026

Greptile Summary

This PR fixes a CI regression caused by three Isaac Sim extensions being declared as hard dependencies in apps/isaaclab.python.kit despite not existing in publicly-released isaacsim 6.0.x. Kit's resolver stalled on a remote registry walk when it couldn't satisfy these deps locally, inflating per-test wall-time to 1000s+ in CI.

  • isaacsim.core.experimental.primdata, isaacsim.robot.wheeled_robots.nodes, and isaacsim.sensors.experimental.rtx are changed to { optional = true } — matching the convention already used for optional extensions in isaacsim's own kit configs.
  • A changelog fragment is added documenting the root cause and affected tests (test_non_headless_launch.py, test_outdated_sensor.py, test_color_randomization.py).

Confidence Score: 5/5

Safe to merge — narrowly scoped to one kit experience file with a local test confirmation (57s fail → 13.82s pass).

The change swaps {} for { optional = true } on three deps confirmed absent from the public release. Only isaaclab.python.kit references these extensions across all six kit experience files, the headless and rendering variants are untouched, and the pattern is already established convention in isaacsim's own configs. No logic, imports, or tests are modified.

No files require special attention. Both changed files are straightforward: a config tweak and its changelog entry.

Important Files Changed

Filename Overview
apps/isaaclab.python.kit Three Isaac Sim extensions absent from public isaacsim 6.0.x are changed from hard deps to { optional = true }, unblocking Kit's resolver from stalling on remote registry sync.
source/isaaclab/changelog.d/jichuanh-ci-unblock-kit-optional-experimental-exts.rst New changelog fragment accurately describing the fix, affected extensions, root cause, and affected test files.

Sequence Diagram

sequenceDiagram
    participant Test as CI Test
    participant Kit as Kit Resolver
    participant Local as Local Ext Cache
    participant Registry as Remote Registry

    Note over Test,Registry: Before this PR (broken)
    Test->>Kit: resolve isaaclab.python.kit deps
    Kit->>Local: find isaacsim.core.experimental.primdata
    Local-->>Kit: not found
    Kit->>Registry: remote sync (10k+ packages, 4 registries)
    Registry-->>Kit: slow / timeout (~55s local, 1000s+ CI)
    Kit-->>Test: resolver failure / exit-1

    Note over Test,Registry: After this PR (fixed)
    Test->>Kit: resolve isaaclab.python.kit deps
    Kit->>Local: "find isaacsim.core.experimental.primdata (optional=true)"
    Local-->>Kit: not found — skip
    Kit->>Local: "find isaacsim.robot.wheeled_robots.nodes (optional=true)"
    Local-->>Kit: not found — skip
    Kit->>Local: "find isaacsim.sensors.experimental.rtx (optional=true)"
    Local-->>Kit: not found — skip
    Kit-->>Test: resolved (~13s, PASSED)
Loading

Reviews (1): Last reviewed commit: "[CI unblock] Mark missing isaacsim exper..." | Re-trigger Greptile

@hujc7 hujc7 force-pushed the jichuanh/ci-unblock-kit-optional-experimental-exts branch from d3088fe to 8eb3a3b Compare May 12, 2026 19:13
@hujc7 hujc7 requested a review from ooctipus as a code owner May 12, 2026 19:13
@hujc7 hujc7 changed the title [CI unblock] Mark missing isaacsim experimental exts as optional in isaaclab.python.kit [CI unblock] Fix two bugs causing isaaclab (core) per-test timeouts May 12, 2026
…saaclab.python.kit

The `apps/isaaclab.python.kit` experience declared three Isaac Sim
extensions as hard dependencies that don't exist in publicly-released
`isaacsim 6.0.x`:

    isaacsim.core.experimental.primdata
    isaacsim.robot.wheeled_robots.nodes
    isaacsim.sensors.experimental.rtx

These were introduced in PR isaac-sim#5293 as part of the deprecated-extension
migration ahead of an upstream Isaac Sim release, but the renamed
targets ship only in unreleased Isaac Sim builds. When Kit's resolver
can't satisfy a hard dep locally it falls back to a remote registry
sync that walks 10k+ packages across 4 registries -- which silently
burns ~55s on a fast network and reaches per-test timeouts of
1000s/1700s on the IsaacLab CI runners, where the registry endpoints
appear to be slow or rate-limited.

That timeout cascade has been hitting `isaaclab (core) [1/3]` for
~7 days, blowing up `test_non_headless_launch.py`,
`test_outdated_sensor.py`, and `test_color_randomization.py` -- all
three import `AppLauncher` at module top and share the same kit-load
boot path. CI then retries each test 3x, burning ~150 min of
wall-time per run.

Marking the three as `{ optional = true }` lets Kit skip them if
absent (matching the convention used elsewhere, e.g.
`omni.kit.notification_manager = { optional = true }` in
`isaacsim.app.compatibility_check`). When the upstream extensions
eventually ship publicly, they auto-load -- no further change needed.

Verified locally: `test_non_headless_launch` goes from 57s exit-1
to 13.82s PASSED.

Refs: isaac-sim/IsaacLab-Internal#... (CI slowness tracking)
@hujc7 hujc7 force-pushed the jichuanh/ci-unblock-kit-optional-experimental-exts branch from 8eb3a3b to 357141e Compare May 12, 2026 20:01
@hujc7 hujc7 closed this May 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

isaac-lab Related to Isaac Lab team isaac-sim Related to Isaac Sim team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants