Perception Full Refactor and Cleanup, deprecated Manipulation AIO Pipeline and replaced with Object Scene Registration #936
Merged
spomichter merged 19 commits intodevfrom Jan 15, 2026
Merged
Conversation
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
alexlin2
force-pushed
the
alexl_realsense_integration
branch
from
e87af86 to
381e5ab
Compare
alexlin2
force-pushed
the
alexl_object_registration
branch
from
b2ee958 to
ca95025
Compare
alexlin2
force-pushed
the
alexl_realsense_integration
branch
from
381e5ab to
e5dd14d
Compare
alexlin2
force-pushed
the
alexl_object_registration
branch
4 times, most recently
from
e675808 to
f19f8b3
Compare
alexlin2
force-pushed
the
alexl_object_registration
branch
2 times, most recently
from
64f275a to
27f9643
Compare
alexlin2
force-pushed
the
alexl_object_registration
branch
2 times, most recently
from
10222f2 to
14cb654
Compare
alexlin2
force-pushed
the
alexl_object_registration
branch
2 times, most recently
from
6b82e5a to
800cd61
Compare
alexlin2
changed the title
alexlin2
force-pushed
the
alexl_object_registration
branch
from
af86eb5 to
722b51e
Compare
spomichter
force-pushed
the
alexl_object_registration
branch
from
710e226 to
2649493
Compare
alexlin2
force-pushed
the
alexl_object_registration
branch
from
2649493 to
81e72db
Compare
spomichter
force-pushed
the
alexl_object_registration
branch
from
8cc43ee to
fe45b75
Compare
alexlin2
force-pushed
the
alexl_object_registration
branch
from
b5cdcef to
ffa2833
Compare
Contributor
|
@greptile final final review |
Contributor
|
@codex final final review |
spomichter
requested changes
Jan 15, 2026
| orientation=self.orientation, | ||
| ) | ||
|
|
||
| def to_ros_detection3d(self) -> Detection3D: |
Contributor
There was a problem hiding this comment.
@alexlin2 Didn't we say we're removing all ROS stuff
Contributor
Author
There was a problem hiding this comment.
Yeah this is confusing, but it's only ros in the method name, because there is a Detection3D type under detections and a Detection3D message under dimos/msgs, this to_ros_detection3d() actually just turns a perception/detection/Detection3D type into a LCM Detection3D (under dimos/msgs), it is confusing I know, but this has nothing to do with ROS, I have renamed it to to_detection3d_msg()
dimos/msgs/sensor_msgs/Image.py
Outdated
| def frame_id(self, value: str) -> None: | ||
| self._impl.frame_id = str(value) | ||
|
|
||
| def from_ros_header(self, header) -> None: # type: ignore[no-untyped-def] |
Contributor
There was a problem hiding this comment.
ros stuff here is this used?
Contributor
There was a problem hiding this comment.
Yeah just checked and this is not used
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9284cea26a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+212
to
+216
| if self._target_frame != color_image.frame_id: | ||
| camera_transform = self.tf.get( | ||
| self._target_frame, | ||
| color_image.frame_id, | ||
| color_image.ts, |
There was a problem hiding this comment.
When target_frame differs from the camera frame, this calls tf.get(target_frame, color_image.frame_id), which returns a transform from target→camera. But Object.from_2d_to_list expects a camera→target transform (it applies PointCloud2.transform directly to points already in the camera frame), so the points get moved in the wrong direction and 3D object poses end up in the wrong location. This will consistently misplace detections whenever target_frame != color_image.frame_id (default map). Invert the returned transform or request tf.get(color_image.frame_id, self._target_frame) instead.
Useful? React with 👍 / 👎.
Contributor
Greptile SummaryThis PR accomplishes a major architectural cleanup, removing ~10,900 lines of deprecated perception and manipulation code and replacing the legacy Manipulation AIO Pipeline with a modern, type-safe Object Scene Registration system. Key Improvements:
Deprecated removals:
New documentation:
The refactor correctly addresses temporal alignment concerns from previous reviews by using image timestamps for transform lookups (line 216 in Confidence Score: 4/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant Camera
participant OSR as ObjectSceneRegistration
participant YOLOE as Yoloe2DDetector
participant ObjDB as ObjectDB
participant TF as Transform Lookup
Camera->>OSR: color_image, depth_image
Camera->>OSR: camera_info (subscribed)
OSR->>OSR: align_timestamped(color, depth)
OSR->>YOLOE: process_image(color_image)
YOLOE-->>OSR: ImageDetections2D[Detection2DSeg]
OSR->>OSR: Publish Detection2DArray
OSR->>OSR: Publish overlay annotations
OSR->>TF: get(target_frame, camera_frame, timestamp)
TF-->>OSR: camera_transform
OSR->>OSR: Object.from_2d_to_list(detections_2d, color, depth, camera_info, transform)
Note over OSR: RGBD projection with<br/>mask erosion, voxel downsample,<br/>outlier removal
OSR->>ObjDB: add_objects(objects)
ObjDB->>ObjDB: Match by track_id (TTL 2s)
ObjDB->>ObjDB: Match by distance (threshold 0.1m)
ObjDB->>ObjDB: Update or create pending objects
ObjDB->>ObjDB: Promote to permanent (6+ detections)
ObjDB-->>OSR: Updated objects list
OSR->>OSR: to_detection3d_array(objects)
OSR->>OSR: aggregate_pointclouds(objects)
OSR->>OSR: Publish Detection3DArray
OSR->>OSR: Publish aggregated pointcloud
|
Comment on lines
+152
to
+153
| self._detector.set_prompts(text=list(prompts)) | ||
| time.sleep(2.0) |
Contributor
There was a problem hiding this comment.
style: The 2-second sleep in the detect skill blocks the event loop. Consider using an async approach or background task if this skill is called frequently from agents.
Contributor
There was a problem hiding this comment.
Skills are executed in a thread. time.sleep does not block any threads.
That said, I'm not a fan of magic sleep constants. If possible, it's much better to wait for a condition to happen.
Comment on lines
+234
to
+237
| pcd_filtered, _ = pc0.pointcloud.remove_statistical_outlier( | ||
| nb_neighbors=statistical_nb_neighbors, | ||
| std_ratio=statistical_std_ratio, | ||
| ) |
Contributor
There was a problem hiding this comment.
style: Consider making statistical_nb_neighbors=10 and statistical_std_ratio=0.5 configurable parameters - aggressive outlier removal could filter valid points from sparse objects or small masks
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
spomichter
reviewed
Jan 15, 2026
Comment on lines
+194
to
+231
| for det in detections_2d.detections: | ||
| if isinstance(det, Detection2DSeg): | ||
| mask = det.mask | ||
| store_mask = det.mask | ||
| else: | ||
| mask = np.zeros((h, w), dtype=np.uint8) | ||
| x1, y1, x2, y2 = map(int, det.bbox) | ||
| x1, y1 = max(0, x1), max(0, y1) | ||
| x2, y2 = min(w, x2), min(h, y2) | ||
| mask[y1:y2, x1:x2] = 255 | ||
| store_mask = mask | ||
|
|
||
| if mask_erode_pixels > 0: | ||
| mask_uint8 = mask.astype(np.uint8) | ||
| if mask_uint8.max() == 1: | ||
| mask_uint8 = mask_uint8 * 255 # type: ignore[assignment] | ||
| kernel_size = 2 * mask_erode_pixels + 1 | ||
| erode_kernel = cv2.getStructuringElement( | ||
| cv2.MORPH_ELLIPSE, (kernel_size, kernel_size) | ||
| ) | ||
| mask = cv2.erode(mask_uint8, erode_kernel) # type: ignore[assignment] | ||
|
|
||
| depth_masked = depth_cv.copy() | ||
| depth_masked[mask == 0] = 0 | ||
|
|
||
| rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth( | ||
| o3d.geometry.Image(color_cv.astype(np.uint8)), | ||
| o3d.geometry.Image(depth_masked.astype(np.float32)), | ||
| depth_scale=depth_scale, | ||
| depth_trunc=depth_trunc, | ||
| convert_rgb_to_intensity=False, | ||
| ) | ||
| pcd = o3d.geometry.PointCloud.create_from_rgbd_image(rgbd, intrinsic_o3d) | ||
|
|
||
| pc0 = PointCloud2( | ||
| pcd, | ||
| frame_id=depth_image.frame_id, | ||
| ts=depth_image.ts, |
Contributor
There was a problem hiding this comment.
@alexlin2 can we review this for next pass. looks inefficient
spomichter
approved these changes
Jan 15, 2026
Nabla7
pushed a commit
that referenced
this pull request
Jan 19, 2026
…eline and replaced with Object Scene Registration (#936) * added rate limiting and backpressure to pointcloud publishing CI code cleanup updated ZED module to the same standard as realsense CI code cleanup fixed stash's comments CI code cleanup mypy fixes + comments removed property of camera_info should pass CI now added detection3d pointcloud types from depth image added yoloe support and 3D object segmentation CI code cleanup use yoloe-s instead for nuc CI code cleanup removed deprecated perception code some pointcloud color changes major refactor and added object class for object scene registration CI code cleanup refactored, added objectDB for persistent object memory CI code cleanup made objectDB a normal class instead of a module CI code cleanup revert to dev reverted more files CI code cleanup completely refactored object scene registration to work natively in dimos instead of using ROS as transport. Made everything super clean and working CI code cleanup bug fixed + use yoloe-l by default added yolo object exlusion list CI code cleanup added zed camera to the object registration demo CI code cleanup added image and pointclou2 fixes and as_numpy function working promptable object scene registration CI code cleanup bug fixes bug fix + remove ros imports should not fail CI now CI code cleanup more CI fixes, somehow local CI did not catch changed prompt fixed bug CI code cleanup reverted some changes Cleanup very dead code and fixed mypy errors CI code cleanup fixed more mypy CI code cleanup * one last mypy fix * added default to imagedetection2d to not set off mypy * fixed bug and default to open vocab for detection * mypy fixes * fixed one last mypy error * fixed all of Stash's comments * should pass mypy now * added uv lock * sync uv.lock with dev * fixed the last mypy error * fixed mypy errors from source * reverted mypy import error fixes * fixed Ivan's comment * fixed last of ivan's comment * remove all to_ros_msgs stuff in this commit * passed Ivan's detector tests * added README for depth camera integration * fixed last of Stash's comments
Nabla7
added a commit
that referenced
this pull request
Jan 20, 2026
* feat(sim): add MJLab G1 velocity policy profile
Introduce a 'mujoco_profile' concept allowing self-contained MuJoCo
simulation bundles (MJCF + ONNX policy + assets) to be loaded by name.
Key changes:
- GlobalConfig: new 'mujoco_profile' field (--mujoco-profile CLI flag)
- model.py: profile-scoped asset loading and bundle.json support
- mujoco_process.py: read camera names from bundle.json per profile
- policy.py: MjlabVelocityOnnxController reads joint_names,
default_joint_pos, action_scale from ONNX metadata for exact
MJLab action contract (per-joint scaling & named ordering)
- mujoco_connection.py: skip menagerie download when profile is set
- blueprints.py / rerun_init.py: gate Rerun init on rerun_enabled
Bundle added to data/.lfs/mujoco_sim.tar.gz (LFS-tracked):
data/mujoco_sim/unitree_g1_mjlab/
├── model.xml (MJLab-compiled G1 MJCF with correct actuators)
├── policy.onnx (trained velocity policy with metadata)
├── bundle.json (camera name mappings)
└── assets/ (STL meshes from MJLab asset_zoo)
Usage:
dimos --simulation \
--mujoco-profile unitree_g1_mjlab \
run unitree-g1-basic-sim
* CI code cleanup
* fix(sim): resolve meshdir/profile asset conflicts for GO2 and G1
- mujoco_process.py: only use mujoco_profile when explicitly set
(fixes GO2 accidentally being treated as a profile bundle)
- model.py: rewrite scene XML to remove meshdir/texturedir attrs
and prefix mesh/texture filenames explicitly, preventing scene
compiler settings from hijacking robot mesh resolution
* configure unitree go2 mapper to use 10 cm voxels (#1032)
* feat(sim): add MuJoCo subprocess profiler for performance debugging
Adds a built-in timing breakdown for the MuJoCo simulation subprocess.
When enabled, logs rolling averages of time spent in each component:
- physics_ms: mj_step loop
- viewer_sync_ms: MuJoCo viewer synchronization
- rgb_render_ms, depth_render_ms: camera rendering
- pcd_ms: depth-to-pointcloud + voxel downsample
- *_shm_ms: shared memory writes
- ctrl_calls, ctrl_obs_ms, ctrl_onnx_ms: policy cost breakdown
This helps diagnose performance issues (e.g. 'molasses' effect) by
showing exactly where frame time is being spent.
Usage:
# Standard G1 sim with profiler:
dimos --simulation --mujoco-profiler --mujoco-profiler-interval-s 2 run unitree-g1-basic-sim
# MJLab bundle with profiler:
dimos --simulation --mujoco-profile unitree_g1_mjlab --mujoco-profiler --mujoco-profiler-interval-s 2 run unitree-g1-basic-sim
New GlobalConfig flags:
- mujoco_profiler: bool (default False)
- mujoco_profiler_interval_s: float (default 2.0)
* pre commit
* small docs clarification (#1043)
* Fix split view on wide monitors (#1048)
* fix print to be correct URL based on rerun web or not
* make width of rerun/command center adjustable
* swap sides
* Docs: Install & Develop (#1022)
* minimal edit
* rice the readme
* grammar
* formatting
* fix examples
* change links to reduce change count
* improve wording
* wording
* remove acknowledgements
* improve the humancli example
* formatting
* Update README.md
* switch to dev branch for development
* changes for paul
* Update README.md
* fix broken link
* update broken link
* Add uv to nix and fix resulting problems (#1021)
* add uv to nix and fix resulting problems
* fix for linux
* v0.0.8 version update (#1050)
* Style changes in docs (#1051)
* capitalization
* punctuation
* more small fixes
* Revert "Add uv to nix and fix resulting problems (#1021)" (#1053)
This reverts commit 8af8f8f.
* Transport benchmarks & Raw ROS transport (#1038)
* raw rospubsub and benchmarks
* typefixes, shm added to the benchmark
* SHM is not so important to tell us every time when it starts
* greptile comments
* Add co-authorship line to commit message filter patterns
* Remove unused contextmanager import
* lcmservice correct kernel settings reintroduced
* mixin mixin resolved
* lcmservice tests fix
* macos lcm rmem fix
* feat: default to rerun-web and auto-open browser on startup (#1019)
- Changed GlobalConfig.viewer_backend default from rerun-native to rerun-web
- WebsocketVisModule now opens dashboard in browser automatically on start
- Requested by Jeff
Co-authored-by: s <pomichterstash@gmail.com>
* chore: fix indentation in blueprints ambiguity check
* CI code cleanup
* use p controller to stop oscillations on unitree go2 (#1014)
* Dynamic session providers for onnxruntime (#983)
* refactor(policy): update inference session initialization
* refactor(policy): simplify inference session provider initialization
* Log the policy directory and provider
* Perception Full Refactor and Cleanup, deprecated Manipulation AIO Pipeline and replaced with Object Scene Registration (#936)
* added rate limiting and backpressure to pointcloud publishing
CI code cleanup
updated ZED module to the same standard as realsense
CI code cleanup
fixed stash's comments
CI code cleanup
mypy fixes + comments
removed property of camera_info
should pass CI now
added detection3d pointcloud types from depth image
added yoloe support and 3D object segmentation
CI code cleanup
use yoloe-s instead for nuc
CI code cleanup
removed deprecated perception code
some pointcloud color changes
major refactor and added object class for object scene registration
CI code cleanup
refactored, added objectDB for persistent object memory
CI code cleanup
made objectDB a normal class instead of a module
CI code cleanup
revert to dev
reverted more files
CI code cleanup
completely refactored object scene registration to work natively in dimos instead of using ROS as transport. Made everything super clean and working
CI code cleanup
bug fixed + use yoloe-l by default
added yolo object exlusion list
CI code cleanup
added zed camera to the object registration demo
CI code cleanup
added image and pointclou2 fixes and as_numpy function
working promptable object scene registration
CI code cleanup
bug fixes
bug fix + remove ros imports
should not fail CI now
CI code cleanup
more CI fixes, somehow local CI did not catch
changed prompt fixed bug
CI code cleanup
reverted some changes
Cleanup very dead code and fixed mypy errors
CI code cleanup
fixed more mypy
CI code cleanup
* one last mypy fix
* added default to imagedetection2d to not set off mypy
* fixed bug and default to open vocab for detection
* mypy fixes
* fixed one last mypy error
* fixed all of Stash's comments
* should pass mypy now
* added uv lock
* sync uv.lock with dev
* fixed the last mypy error
* fixed mypy errors from source
* reverted mypy import error fixes
* fixed Ivan's comment
* fixed last of ivan's comment
* remove all to_ros_msgs stuff in this commit
* passed Ivan's detector tests
* added README for depth camera integration
* fixed last of Stash's comments
* feat(cli): type-free topic echo via /topic#pkg.Msg inference, this mi… (#988)
* feat(cli): type-free topic echo via /topic#pkg.Msg inference, this mirrors ros topic echo functionality.
- Make type_name optional in 'dimos topic echo'
- Infer message type from LCM channel suffix (e.g. /odom#nav_msgs.Odometry)
- Dynamically import dimos.msgs.<pkg> and call cls.lcm_decode(data)
- Keep existing explicit-type mode working
- Update transports.md docs
* fix(cli): use LCMPubSubBase instead of raw lcm.LCM for topic echo, my bad
* verify blueprints (#1018)
* verify blueprints
* Fix geometry msgs check failure in CI
---------
Co-authored-by: stash <pomichterstash@gmail.com>
* Experimental Streamed Temporal Memory with SpatioTemporal & Entity based RAG (#973)
* temporal memory + vlm agent + blueprints
* fixing module issue and style
* fix skill registration
* removing state functions unpickable
* inheritancefixes and memory management
* docstring for query
* microcommit: fixing memory buffer
* sharpness filter and simplified frame filtering
* CI code cleanup
* initial graph database implementation
* db implementation, working and stylized, best reply is unitree_go2_office_walk2
* type checking issues
* final edits, move into experimental, revert non-memory code edits, typechecking
* persistent db flag enabled in config
* Fix test to not run in CI due to LFS pull
* Fix CLIP filter to use dimensional clip
* Add path to temporal memory
* revert video operators
* Revert moondream
* added temporal memory docs
* Refactor move to /experimental/temporal_memory
---------
Co-authored-by: Paul Nechifor <paul@nechifor.net>
Co-authored-by: Stash Pomichter <pomichterstash@gmail.com>
Co-authored-by: shreyasrajesh0308 <shreyasrajesh0308@users.noreply.github.com>
Co-authored-by: spomichter <12108168+spomichter@users.noreply.github.com>
* Control Orchestrator - Unified Controller for multi-arm and full body controller (#970)
* archive old driver to manipulators_old for redesign
* spec.py defining minimal protocol for an arm driver
* xarm driver driver added - driver owns control thread and robot state threads also invokes rpc calls to arm specific SDK backends
* xarm SDK specific wrapper to interface with dimos RPC calls from the driver
* removed type checking for old armdriver spec from the cartesian controller
* replicated piper driver to meet the new architecture
* added mock backend
* updated all blueprints to add new arm module
* Added readme explaining new driver architecture overview
* config now parsed in backend init instead of connect method
* addded dual arm control blueprint using trajectory controller
* adding a control orchestrator for single control loop for multiple arms and joint control - added dataclasses for orchestrator and protocol for ControlTask
* hardware interface protocol that wraps specific arm SDK to work with orchestrator. Also solves namespace for multiple arm and hardware
* main orchestrator module and control loop that claims resources computes next commands, and arbitrates priority of different tasks and controllers
* added a trajectory task implementation that performs trajecotry control
* added blueprints to launch orchestratory module with differnt arms for testing
* updated blueprints to add piper + xarm blueprint
* orchestrator client that can send tasks to the control orchestrator module
* added a readme
* added pytest and e2e test
* Update dimos/control/hardware_interface.py
explicit false added to the Torque Mode command sent, to avoid silent failing scenario
* CI code cleanup
* Fixed issues flagged by greptile
Mode conflict detection in routing: Added check in _route_to_hardware
Preemption tracking: Changed structure to {preempted_task: {joint: winning_task}}
Mode conflict preemption: Tasks dropped due to mode conflict at same priority
Trajectory completion edge case: Returns final position instead of None on completion
Dead code removal: and Piper backend cleanup
* Renamed deprecated old manipuialtion test file and Mypy type fixes
* fix mypy test
* mypy test fix added explicit type
* Remove deprecated manipulators_old folder
* fixed redef error in dual trajectory setter
* Fixed bugs identified by greptile overview:
1. tick_loop.py - Race condition in _route_to_hardware
2. orchestrator.py - Added hardware_added tracking list and rollback in outer except block
3. hardware_interface.py - Added disconnect() to both HardwareInterface protocol and BackendHardwareInterface
4. Added disconnect() to both HardwareInterface protocol and BackendHardwareInterface
5. orchestrator.py - Start order fix Moved super().start() to end, after tick loop starts successfully
6. trajectory_task.py - Added Empty joint_names validation
* addressed greptile suggestion:
hardware_interface.py - Torque mode logging fix
orchestrator.py - Fail hardware removal if joints in use
tick_loop.py - Rate control drift fix
* undo change to pyproject.toml
* Replaced _running bool with threading.Event (_stop_event) for thread safety
Removed duplicate _auto_start() call from __init__ - connection now only happens in start()
orchestrator_client.py IPython conversion
* added type ignore for ipythin
* removed check for has attribute in hardware interface
Moved super.start() at the beginning
replaced running bool with stop_event in tick_loop to improve thread safety
removed default ip from init
removed simple dataclasses test
* orchestrator.py: Use match statement for backend factory, restructure backend cleanup
task.py: Use match statement in get_values()
tick_loop.py: Add JointWinner NamedTuple for cleaner arbitration logic
xarm/backend.py: Extract unit conversions into static helper methods
* tick_loop.py: Notify preemption when lower-priority task loses to existing winner
hardware_interface.py: Call set_control_mode() before mode-specific writes, Convert if/elif to match statement for control mode dispatch
* tick_loop.py: Notify preemption when lower-priority task loses to existing winner
hardware_interface.py: Call set_control_mode() before mode-specific writes, convert if/elif to match statement
trajectory_task.py: Defer start time to first compute() for consistent timing
orchestrator.py: Extract _setup_hardware() helper for cleaner config setup
piper and xarm/backend.py: Fail fast on read_joint_positions(), map SERVO_POSITION to mode 1
hardware_interface.py: Retry initialization with proper error propagation,
spec.py: Add SERVO_POSITION control mode for confusion between position planning and position servo
task.py: added SERVO_POSITION to JointCommandOutput helper
* cleaned up legacy blueprints for manipulator drivers
* enforce ManipulatorBackend Protocol on the backend.py
* feat: add runtime protocol checks for manipulator backends
* added runtime checking for controlTask protocol
* Add TaskStatus dataclass, refactor get_trajectory_status and Explicitly inherit from ControlTask protocol
---------
Co-authored-by: stash <pomichterstash@gmail.com>
* configure unitree go2 mapper to use 10 cm voxels (#1032)
* Create DDSPubSubBase, DDSTopic
* Create PickleDDS
* Fix hash/equality inconsistency in DDSTopic
* Add DDSMsg
* Create DDSTransport
* Add broadcast and subscribe methods to DDSTransport
* Create DDSService
* Add CycloneDDS package
* Remove unnecessary attributes
* Add threading and serialization methods to DDSService
* Ensure broadcast and subscribe methods initialize DDS if not started
* Add Transport benchmarking capabilities to CycloneDDS (#1055)
* raw rospubsub and benchmarks
* typefixes, shm added to the benchmark
* SHM is not so important to tell us every time when it starts
* greptile comments
* Add co-authorship line to commit message filter patterns
* Remove unused contextmanager import
---------
Co-authored-by: Ivan Nikolic <lesh@sysphere.org>
* Fix DDS segmentation fault using bytearray for binary data storage
Replace base64 string encoding with native IDL bytearray type to eliminate
buffer overflow issues. The original base64 encoding exceeded CycloneDDS's
default string size limit (~256 bytes) and caused crashes on messages >= 1KB.
Key changes:
- Use make_idl_struct with bytearray field instead of string
- Convert bytes to bytearray when publishing to DDS
- Convert bytearray back to bytes when receiving from DDS
- Add _DDSMessageListener for async message dispatch
- Implement thread-safe DataWriter/DataReader management
- Add pickle support via __getstate__/__setstate__
Result: All 12 DDS benchmark tests pass (64B to 10MB messages).
* Refactor DDS PubSub implementation to use CycloneDDS Topic
* Remove DDS pickling
* CI code cleanup
* bugfix
* CI code cleanup
---------
Co-authored-by: leshy <lesh@sysphere.org>
Co-authored-by: Jeff Hykin <jeff.hykin@gmail.com>
Co-authored-by: Paul Nechifor <paul@nechifor.net>
Co-authored-by: s <pomichterstash@gmail.com>
Co-authored-by: Miguel Villa Floran <miguel.villafloran@gmail.com>
Co-authored-by: alexlin2 <44330195+alexlin2@users.noreply.github.com>
Co-authored-by: claire wang <clara32356@gmail.com>
Co-authored-by: shreyasrajesh0308 <shreyasrajesh0308@users.noreply.github.com>
Co-authored-by: spomichter <12108168+spomichter@users.noreply.github.com>
Co-authored-by: Mustafa Bhadsorawala <39084056+mustafab0@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Perception Full Refactor and Cleanup
Summary
Major refactor of the perception and manipulation pipelines, removing ~10,900 lines of deprecated code and replacing the legacy Manipulation AIO Pipeline with a cleaner, type-safe Object Scene Registration system.
Deprecated Code Removed
Old Detection2d, Pointcloud, Segmentation Folders and Utility Functions
Manipulation Visual Servoing Pipeline (
dimos/manipulation/visual_servoing/)detection3d.py,manipulation_module.py,pbvs.py,utils.pyDeprecated Detectors & Streams
detic.py- Removed unused Detic detectorobject_detection_stream.py- Replaced by ObjectSceneRegistrationModuleperson_tracker.py(perception/) - Legacy tracker superseded by new architectureDeprecated PiperArm Driver and Manipulation Module
piper_arm.py\New Architecture: Object Scene Registration
ObjectSceneRegistrationModule (
dimos/perception/object_scene_registration.py)align_timestampedto synchronize color, depth, and camera infoResourcepattern (start/stop lifecycle) instead of manual Disposable managementObjectSceneRegistrationConfigdataclassObjectDB (
dimos/perception/detection/objectDB.py)Object Class (
dimos/perception/detection/type/detection3d/object.py)from_2d()classmethod for projecting 2D detections to 3D using RGBDto_ros_detection3d()YOLOE Integration (
dimos/perception/detection/detectors/yoloe.py)Type Safety Improvements
ImageDetections2Dnow properly parameterized asImageDetections2D[Detection2DBBox],ImageDetections2D[Detection2DSeg], etc.Testing