fix(sim): set ImageFormat.RGB for MuJoCo video frames#972
Merged
Conversation
MuJoCo's renderer outputs RGB, but Image.from_numpy() defaults to BGR. This caused red/blue channel swap in Rerun camera visualization when running in simulation mode.
Contributor
There was a problem hiding this comment.
Greptile Overview
Greptile Summary
This PR fixes a color channel swap bug in MuJoCo video frames by explicitly specifying ImageFormat.RGB when converting frames to Image messages. MuJoCo's renderer outputs RGB data, but Image.from_numpy() defaults to BGR format, causing red/blue channels to be swapped in Rerun visualizations.
Key changes:
- Added
ImageFormatimport tomujoco_connection.py - Specified
format=ImageFormat.RGBparameter inImage.from_numpy()call - Added explanatory comment about MuJoCo renderer format and
Image.from_numpy()default
Related issue discovered:
While reviewing, I found a similar bug in manipulation_module.py line 908 where RGB data is passed to Image.from_numpy() without specifying the format parameter.
Confidence Score: 5/5
- This PR is safe to merge with no risk
- The fix is minimal, well-documented, and directly addresses the root cause. The change only affects MuJoCo video frame processing and correctly specifies the RGB format that MuJoCo's renderer outputs. The fix follows the same pattern used in other parts of the codebase (webcam.py, dji_video_stream.py, unitree connection.py) where RGB data is explicitly tagged with
ImageFormat.RGB. manipulation_module.pyhas a similar bug (not in this PR) that should be fixed separately
Important Files Changed
File Analysis
| Filename | Score | Overview |
|---|---|---|
| dimos/robot/unitree_webrtc/mujoco_connection.py | 5/5 | Correctly fixes RGB/BGR channel swap by specifying ImageFormat.RGB when creating Image from MuJoCo renderer output |
Sequence Diagram
sequenceDiagram
participant MP as MuJoCo Process
participant Renderer as mujoco.Renderer
participant SHM as Shared Memory
participant MC as MujocoConnection
participant Image as Image.from_numpy
participant Rerun as Rerun Visualization
MP->>Renderer: render() camera frame
Renderer-->>MP: RGB uint8 pixels (H×W×3)
MP->>SHM: write_video(pixels)
MC->>SHM: read_video()
SHM-->>MC: RGB frame array
MC->>Image: from_numpy(frame, format=ImageFormat.RGB)
Note over Image: Correctly tagged as RGB<br/>(previously defaulted to BGR)
Image-->>MC: Image object with RGB format
MC->>Rerun: log image
Rerun->>Rerun: Display with correct colors
Contributor
Additional Comments (1)
|
spomichter
approved these changes
Jan 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MuJoCo's renderer outputs RGB, but Image.from_numpy() defaults to BGR. This caused red/blue channel swap in Rerun camera visualization when running in simulation mode.