Skip to content

fix(sim): set ImageFormat.RGB for MuJoCo video frames#972

Merged
Nabla7 merged 1 commit intodevfrom
fix/mujoco-rgb-channel-order
Jan 9, 2026
Merged

fix(sim): set ImageFormat.RGB for MuJoCo video frames#972
Nabla7 merged 1 commit intodevfrom
fix/mujoco-rgb-channel-order

Conversation

@Nabla7
Copy link
Contributor

@Nabla7 Nabla7 commented Jan 9, 2026

MuJoCo's renderer outputs RGB, but Image.from_numpy() defaults to BGR. This caused red/blue channel swap in Rerun camera visualization when running in simulation mode.

MuJoCo's renderer outputs RGB, but Image.from_numpy() defaults to BGR.
This caused red/blue channel swap in Rerun camera visualization when
running in simulation mode.
@Nabla7 Nabla7 requested a review from a team January 9, 2026 10:26
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greptile Overview

Greptile Summary

This PR fixes a color channel swap bug in MuJoCo video frames by explicitly specifying ImageFormat.RGB when converting frames to Image messages. MuJoCo's renderer outputs RGB data, but Image.from_numpy() defaults to BGR format, causing red/blue channels to be swapped in Rerun visualizations.

Key changes:

  • Added ImageFormat import to mujoco_connection.py
  • Specified format=ImageFormat.RGB parameter in Image.from_numpy() call
  • Added explanatory comment about MuJoCo renderer format and Image.from_numpy() default

Related issue discovered:
While reviewing, I found a similar bug in manipulation_module.py line 908 where RGB data is passed to Image.from_numpy() without specifying the format parameter.

Confidence Score: 5/5

  • This PR is safe to merge with no risk
  • The fix is minimal, well-documented, and directly addresses the root cause. The change only affects MuJoCo video frame processing and correctly specifies the RGB format that MuJoCo's renderer outputs. The fix follows the same pattern used in other parts of the codebase (webcam.py, dji_video_stream.py, unitree connection.py) where RGB data is explicitly tagged with ImageFormat.RGB.
  • manipulation_module.py has a similar bug (not in this PR) that should be fixed separately

Important Files Changed

File Analysis

Filename Score Overview
dimos/robot/unitree_webrtc/mujoco_connection.py 5/5 Correctly fixes RGB/BGR channel swap by specifying ImageFormat.RGB when creating Image from MuJoCo renderer output

Sequence Diagram

sequenceDiagram
    participant MP as MuJoCo Process
    participant Renderer as mujoco.Renderer
    participant SHM as Shared Memory
    participant MC as MujocoConnection
    participant Image as Image.from_numpy
    participant Rerun as Rerun Visualization

    MP->>Renderer: render() camera frame
    Renderer-->>MP: RGB uint8 pixels (H×W×3)
    MP->>SHM: write_video(pixels)
    
    MC->>SHM: read_video()
    SHM-->>MC: RGB frame array
    MC->>Image: from_numpy(frame, format=ImageFormat.RGB)
    Note over Image: Correctly tagged as RGB<br/>(previously defaulted to BGR)
    Image-->>MC: Image object with RGB format
    MC->>Rerun: log image
    Rerun->>Rerun: Display with correct colors
Loading

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Jan 9, 2026

Additional Comments (1)

dimos/manipulation/visual_servoing/manipulation_module.py
Image.from_numpy() defaults to BGR format, but viz_rgb is RGB. Specify format explicitly:

            msg = Image.from_numpy(viz_rgb, format=ImageFormat.RGB)

@Nabla7 Nabla7 merged commit 17df193 into dev Jan 9, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants