DimOS Manipulation Framework, ObjectDetectionStream Changes

[spomichter]

Changes

• Added manipulation_constraint.py with AbstractConstraint base class and three concrete constraint types (Translation, Rotation, Force)
• Implemented RotationConstraintSkill for controlling rotation axes, angles, and pivot points
• Implemented TranslationConstraintSkill for bounded/unbounded translations with reference points
• Implemented ForceConstraintSkill for force-limited manipulations with min/max forces and direction
• Created ManipulateSkill that combines constraint types for executing complex manipulation tasks
• Added testfile with Manipulation development interface with streaming of VLM ManipulationPoint locations on RGB images
• Created ManipulationInterface
• Created ManipulationHistory with pytest unit test

DimOS Manipulation Framework

A comprehensive framework for defining, executing, and tracking robot manipulation tasks with constraint-based control.

Overview

The DimOS Manipulation Framework structure robot manipulation tasks with:

• Type-safe constraint definitions (Translation, Rotation, Force)
• Detailed task history with advanced search capabilities
• Integration with object detection systems
• Comprehensive metadata tracking

Key Components

Core Types

Type	Description
ManipulationTask	Defines a manipulation operation with target objects, points, and constraints
AbstractConstraint	Base class for all constraint types
TranslationConstraint	Controls movement along specific axes with bounds
RotationConstraint	Manages rotation around specific axes with angle limits
ForceConstraint	Specifies force parameters for manipulation
ManipulationTaskConstraint	Container for multiple constraints
ManipulationMetadata	Stores context information about the manipulation environment

History Tracking

Type	Description
ManipulationHistory	Stores and retrieves manipulation task history
ManipulationHistoryEntry	Records individual task execution with results

Integration Classes

Type	Description
ManipulationInterface	Connects robot hardware to manipulation logic
MockManipulationRobot	Test Robot class with webcam camera

Usage Examples

Creating a Basic Manipulation Task

from dimos.types.manipulation import ManipulationTask
from dimos.types.vector import Vector
import time

# Create a simple task to pick up a cup
task = ManipulationTask(
    description="Pick up the cup",
    target_object="cup",
    target_point=(100, 200),  # 2D point from image detection
    task_id="task1",
    metadata={
        "timestamp": time.time(),
        "objects": {
            "cup1": {
                "object_id": 1,
                "label": "cup",
                "confidence": 0.95,
                "position": {"x": 1.5, "y": 2.0, "z": 0.5},
            }
        }
    }
)

Adding Constraints to a Task

from dimos.types.manipulation import TranslationConstraint, RotationConstraint, ForceConstraint

# Translation constraint (movement only along Y-axis)
translation_constraint = TranslationConstraint(
    translation_axis="y",
    reference_point=Vector(2.5, 1.0, 0.3),
    bounds_min=Vector(2.0, 0.5, 0.3),
    bounds_max=Vector(3.0, 1.5, 0.3),
    target_point=Vector(2.7, 1.2, 0.3),
    description="Constrained translation along Y-axis only"
)
task.add_constraint(translation_constraint)

# Rotation constraint (rotation only around X-axis/roll)
rotation_constraint = RotationConstraint(
    rotation_axis="roll",
    start_angle=Vector(0, 0, 0),
    end_angle=Vector(90, 0, 0),
    pivot_point=Vector(2.5, 1.0, 0.3),
    description="Constrained rotation around X-axis (roll only)"
)
task.add_constraint(rotation_constraint)

# Force constraint (apply moderate downward force)
force_constraint = ForceConstraint(
    min_force=2.0,
    max_force=5.0,
    force_direction=Vector(0, 0, -1),
    description="Apply moderate downward force during manipulation"
)
task.add_constraint(force_constraint)

Using a Robot to Execute Manipulation

from dimos.robot.robot import MockManipulationRobot
from dimos.skills.skills import SkillLibrary
from dimos.skills.manipulation.manipulate_skill import ManipulateSkill

# Initialize robot and skills
skill_library = SkillLibrary()
skill_library.register(ManipulateSkill)
robot = MockManipulationRobot(skill_library=skill_library)

# Create and execute a manipulation task
manipulation_skill = robot.get_skill(ManipulateSkill)

# Create constraint objects
translation_constraint = TranslationConstraint(
    translation_axis="y",
    reference_point=Vector(2.5, 1.0, 0.3),
    bounds_min=Vector(2.0, 0.5, 0.3),
    bounds_max=Vector(3.0, 1.5, 0.3)
)

rotation_constraint = RotationConstraint(
    rotation_axis="roll",
    start_angle=Vector(0, 0, 0),
    end_angle=Vector(90, 0, 0),
    pivot_point=Vector(2.5, 1.0, 0.3)
)

# Execute the skill with proper constraint objects
result = manipulation_skill(
    target_object="bottle",
    target_point=(150, 250),
    constraints=[translation_constraint, rotation_constraint]
)

Manipulation History

Creating a ManipulationHistory instance and Adding an Entry

from dimos.manipulation.manipulation_history import ManipulationHistory
from dimos.types.manipulation import ManipulationTask

# Initialize history (will load existing history if available)
history = ManipulationHistory(output_dir="/path/to/history")

# Create a task entry directly through history
entry = history.create_task_entry(
    task=task: ManipulationTask,
    result={"status": "success", "execution_time": 2.5},
    agent_response="Successfully picked up the cup"
)

# Basic methods for retrieving entries
cup_entries = history.get_entries_by_object("cup")

Searching Manipulation History

The ManipulationHistory class provides powerful search capabilities for finding manipulation tasks based on various criteria:

from dimos.manipulation.manipulation_history import ManipulationHistory

# Initialize history
history = ManipulationHistory(output_dir="/path/to/history")

# Example 1: Search by object properties and metadata
# Find all tasks involving cups with position.z greater than 0.4
cup_entries = history.search(**{
    "task.target_object": "cup",
    "task.metadata.objects.*.position.z": (">", 0.4)
})

# Example 2: Search by constraint properties
# Find tasks with rotation constraints around roll axis with end angle of 90 degrees
rotation_tasks = history.search(**{
    "task.constraints.*.rotation_axis": "roll",
    "task.constraints.*.end_angle.x": 90
})

# Example 3: Search by time range and execution status
# Find successful tasks executed in the last hour
import time
current_time = time.time()
one_hour_ago = current_time - 3600

recent_success = history.search(**{
    "task.metadata.timestamp": (">", one_hour_ago),
    "result.status": "success"
})

Additional ManipulationHistory Search Parameters

The ManipulationHistory class provides powerful search capabilities using dot notation to access any field in the task history:

Time-Based Searches

• Search for tasks after a specific time: search(**{"task.metadata.timestamp": ('>', start_time)})
• Find tasks from the last 30 minutes: search(**{"task.metadata.timestamp": ('>=', time.time() - 1800)})
• Tasks between two timestamps: search(**{"task.metadata.timestamp": ('>', start_time), "task.metadata.timestamp": ('<', end_time)})

Constraint Property Searches

• Tasks with specific reference points: search(**{"task.constraints.*.reference_point.x": 2.5})
• Tasks with specific rotation angles: search(**{"task.constraints.*.end_angle.x": 90})
• Tasks with locked axes: search(**{"task.constraints.*.lock_x": True})
• Tasks using a specific constraint type: search(**{"task.constraints.*.rotation_axis": "roll"})

Object and Result Searches

• Tasks involving specific objects: search(**{"task.metadata.objects.*.label": "cup"})
• Tasks with specific outcome: search(**{"result.status": "success"})
• Tasks with specific errors: search(**{"result.error": "Collision"})
• Tasks with execution time: search(**{"result.execution_time": ("<", 3.0)})

Wildcard and Nested Property Searches

• Using * to match any object in a collection: search(**{"task.metadata.objects.*.position.z": ('>', 0.5)})
• String contains matching (automatic for string values): search(**{"manipulation_response": "successfully"})
• Deep nested property searches: search(**{"task.metadata.objects.bottle1.position.x": 2.5})

Combined Multi-Criteria Searches

• All criteria must match: search(**{"task.target_object": "cup", "result.status": "success", "task.constraints.*.translation_axis": "y"})

How Manipulation Integrates with ObjectDetectionStream

The manipulation system integrates with object detection systems to track and manipulate detected objects:

# Get detected objects from the robot's vision system
detected_objects: List[ObjectData] = robot.manipulation_interface.get_latest_objects()

# Create manipulation metadata with detected objects
objects_data = {}
for obj in detected_objects:
    obj_id = obj.get("object_id", str(time.time()))
    objects_data[obj_id] = dict(obj)
    
metadata = {"timestamp": time.time(), "objects": objects_data}

# Create task with the detected objects
task = ManipulationTask(
    description="Pick up detected cup",
    target_object="cup",
    target_point=detected_objects[0]["center_point"],
    metadata=metadata
)

[spomichter]

@yashas-salankimatt @atharva620 Where y'all would integrate with constraint executor / motion planning