Person follow skill with EdgeTAM

[paul-nechifor]

Usage

uv sync --all-extras
uv run dimos --robot-ip 192.168.X.XXX run unitree-go2-agentic
uv run humancli

Write "follow the man in blue" or some description of yourself.

E2E

Run the end-to-end test:

uv run pytest dimos/e2e_tests/test_person_follow.py

How it works

• Uses EdgeTAM for detection.
• Estimates the person's distance based on the width of the detection. (I'm using width instead of height since people's height isn't always visible from the low angle of the robot.)
• Rotates or moves forward/back to keep the detection in the center.
• Does not check for free space. This is especially bad when you make it move backwards.
• Cannot follow you around tight corners, either because it loses track,or because it tries to move through the corner.

[greptile-apps]

Greptile Summary

This PR adds a person-following skill using EdgeTAM (SAM2-based video tracking) for visual servoing control on the Unitree GO2 robot. The implementation includes a new skill container, EdgeTAM processor for frame tracking, and integration into the agentic blueprint system.

Key Changes:

• PersonFollowSkillContainer: Implements visual servoing with proportional control for distance and heading
• EdgeTAMProcessor: Wraps SAM2 video predictor with frame buffer management to prevent OOM
• Dependencies: Added edgetam, omegaconf, hydra-core for model configuration and video tracking
• Blueprint integration: Wired person_follow_skill into agentic configurations

Known Limitations (Per PR Description):

• No obstacle avoidance or free space checking
• Cannot follow around tight corners
• May lose track at unexpected angles

Issues Found:

• Line 202: Assert will crash the control loop if image streaming pauses (should use error handling)
• Line 215: Potential ValueError if detections.detections is empty despite len(detections) == 0 check
• Line 316: Magic number for angular gain should be parameterized for tuning

Confidence Score: 3/5

• This PR introduces person-following functionality with video tracking, but has critical logic issues that could cause runtime crashes during control loop execution.
• The score reflects two critical logic issues in the control loop: (1) An assert statement that will crash if image streaming pauses, and (2) A potential ValueError when calling max() on detections without proper list validation. These issues will cause the follow_person skill to fail during normal operation. The EdgeTAM processor and blueprint integration are solid, but the skill container has insufficient error handling for asynchronous image streams.
• dimos/agents/skills/person_follow.py requires attention for error handling in the control loop, particularly around image availability assertions and detection list validation.

Important Files Changed

Filename	Overview
dimos/agents/skills/person_follow.py	New skill container implementing person-following using EdgeTAM for tracking. Uses visual servoing with proportional control. Has potential issues: assert statement at line 202 could fail without proper handling, and no validation that detections.detections is not empty before calling max() at line 215.
dimos/models/segmentation/edge_tam.py	New EdgeTAM processor for video object tracking. Properly handles CUDA requirement, model loading with state dict validation, and frame buffer management. Memory cleanup logic is sound.
dimos/robot/unitree_webrtc/unitree_go2_blueprints.py	Integrated person_follow_skill into the agentic blueprints. Passes camera_info_static correctly. No issues found.
pyproject.toml	Added EdgeTAM, omegaconf, and hydra-core dependencies. Dependencies are appropriate for the new ML features.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant PersonFollowSkill
    participant EdgeTAMProcessor
    participant GO2Connection
    participant ColorImageStream
    
    Agent->>PersonFollowSkill: "follow_person(query)"
    PersonFollowSkill->>PersonFollowSkill: "detect person via QwenVL"
    PersonFollowSkill->>PersonFollowSkill: "spawn daemon thread"
    
    Note over PersonFollowSkill: "_follow_loop thread"
    PersonFollowSkill->>EdgeTAMProcessor: "init_track(image, bbox)"
    EdgeTAMProcessor->>EdgeTAMProcessor: "SAM2 initializes"
    
    loop "Control Loop (20 Hz)"
        ColorImageStream->>PersonFollowSkill: "new frame"
        PersonFollowSkill->>EdgeTAMProcessor: "process_image()"
        
        alt "Tracking Success"
            EdgeTAMProcessor-->>PersonFollowSkill: "detections"
            PersonFollowSkill->>PersonFollowSkill: "compute twist"
            PersonFollowSkill->>GO2Connection: "move(twist)"
        else "Lost Track"
            EdgeTAMProcessor-->>PersonFollowSkill: "empty detections"
            PersonFollowSkill->>GO2Connection: "move(zero)"
        end
        
        PersonFollowSkill->>PersonFollowSkill: "sleep"
    end
    
    Agent->>PersonFollowSkill: "stop_following()"
    PersonFollowSkill->>GO2Connection: "move(zero)"

Loading

[greptile-apps]

_{5 files reviewed, 3 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

logic: Calling max(detections.detections, ...) without checking if the list is empty will raise ValueError. Even though len(detections) == 0 is checked, the len() behavior on ImageDetections2D may differ from checking detections.detections length. Add explicit check: if detections.detections: before calling max().

[leshy]

looks ok as a first sketch, would like to understand global planner issues etc.
will need to plug in reid also, Alex found some good model

[leshy]

any reason we aren't just emitting twist from this as OUT? since this would makae this work cross embodyment

[paul-nechifor]

It slipped my mind. 😅 Fixed now.

[leshy]

isn't this some new model, for tracking only? so would be good to have maybe new dir for it, since it's not segmentation? and new type for trackers, since this is not strictly a Detector so we can change trackers? not sure, what do you think?

[alexlin2]

Actually the way I implemented before was I conformed it to the same structure as YOLO and YOLOE, also in detection/types/detection3d/seg.py, there is a from_sam2_result() that is for this edgeTAM output. It should output ImageDetection2D[Detection2DSeg). I think it's better to put the general EdgeTAMTracker code (conforms to the same detector type) under perception/, keep the inference and model loading code as separate in models/ like I did before.

[leshy]

we have this fancy system of hot/colr getters

https://github.com/dimensionalOS/dimos/blob/dev/docs/api/sensor_streams/advanced_streams.md#getter_hot---background-subscription-instant-reads

so you can do getter = geter_hot(self.image().observable())

idk if you want this, just letting you know it exists. also getter is not disposable, should make it one

[leshy]

generally I'd much prefer this as a reactive pipeline in which you

self.image_frame.observable().pipe(
  ops.map(track)
  ops.map(detection_to_twist)
).subscribe(self.twist.publish)

so you don't need to do this self._image thing, thread management etc etc. much cleaner, just letting you know, this is an ok first test

[paul-nechifor]

But doing it that way ties the frequency of the cmd_vel to the frequency of the camera, right? And you still need some state like counting how many frames are without a detection, or knowing when to stop.

[leshy]

there are nice tools for all of this - like sample() maybe? samples latest value at fixed hz, so you can emit twist at a faster fixed rate then input image

https://github.com/dimensionalOS/dimos/blob/dev/docs/api/sensor_streams/reactivex.md#sampleinterval---emit-latest-value-every-n-seconds

you still need some state like counting how many frames are without a detection, or knowing when to stop.

yeah you either start propagading more complex types (worst case but still nice)

{
  Image :image
  Detection2d: ...
  frames: n
}

and this gives you full power, and clear typing also, since you can transform between those complex types at each step

def bla(in: ComplexType1) -> ComplexType2

ops.map(bla)

but as an also easier shortcut before going for complex types you can have a map that has a sideeffect that counts empty Detection2Ds and starts returning false/exiting early.

detection_to_twist in above example can have a sideeffect of counting empty detections it got, and starting to output twist(0,0,0) - after initial few twist(0,0,0) you can also choose to terminate tracking loop as well, observable just finishes

to be clear Im not asking for a rewrite, just mentioning how this can be structured in a way in which it's easy and fun to quickly iterate (add twist easing etc), rotation towards a person and walking towards a person can be separate maps() that layer onto the same Twist, etc

[leshy]

again first sketch so all good, we should make sure that API for merging this with lidar and estimating segmentation distance (turning into detection3d) is super easy easy. detections/ stuff implements this for projecting 2d into 3d

perception/detection/reid/module.py

this is for re-assembling fancy Detection2D types from ros msgs + image, but you'd do the same for aligning lidar + camera image

    def detections_stream(self) -> Observable[ImageDetections2D]:
        return backpressure(
            align_timestamped(
                self.image.pure_observable(),
                self.detections.pure_observable().pipe(
                    ops.filter(lambda d: d.detections_length > 0)  # type: ignore[attr-defined]
                ),
                match_tolerance=0.0,
                buffer_size=2.0,
            ).pipe(ops.map(lambda pair: ImageDetections2D.from_ros_detection2d_array(*pair)))  # type: ignore[misc, arg-type]
        )

documented at https://github.com/dimensionalOS/dimos/blob/dev/docs/api/sensor_streams/temporal_alignment.md

also this might be interesting
https://github.com/dimensionalOS/dimos/blob/dev/docs/api/sensor_streams/quality_filter.md

[paul-nechifor]

Yeah, makes sense. I'll use the lidar to estimate the distance.

[leshy]

no need for first merge as well, your call how to proceed with this, it could turn out that lidar method is less robust also

[paul-nechifor]

@leshy

will need to plug in reid also, Alex found some good model

EdgeTAM does not require ReID.