Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 47 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,16 @@ We are shipping a first look at the DIMOS x Unitree Go2 integration, allowing fo
- Camera feeds (image_raw, compressed_image, etc.)
- IMU data
- State information
- Lidar / PointCloud primitives 🚧
- Lidar / PointCloud primitives
- Any other generic Unitree ROS2 topics

### Features

- **DimOS Agents**
- Agent() classes with planning, spatial reasoning, and Robot.Skill() function calling abilities.
- Integrate with any off-the-shelf model: OpenAIAgent, GeminiAgent 🚧, DeepSeekAgent 🚧, HuggingfaceAgent 🚧, etc.
- Integrate with any off-the-shelf hosted or local model: OpenAIAgent, ClaudeAgent, GeminiAgent 🚧, DeepSeekAgent 🚧, HuggingFaceRemoteAgent, HuggingFaceLocalAgent, etc.
- Modular agent architecture for easy extensibility and chaining of Agent output --> Subagents input.
- Agent spatial / language memory for location grounded reasoning and recall. 🚧
- Agent spatial / language memory for location grounded reasoning and recall.

- **DimOS Infrastructure**
- A reactive data streaming architecture using RxPY to manage real-time video (or other sensor input), outbound commands, and inbound robot state between the DimOS interface, Agents, and ROS2.
Expand All @@ -69,6 +69,8 @@ We are shipping a first look at the DIMOS x Unitree Go2 integration, allowing fo
Configure your environment variables in `.env`
```bash
OPENAI_API_KEY=<OPENAI_API_KEY>
ALIBABA_API_KEY=<ALIBABA_API_KEY>
ANTHROPIC_API_KEY=<ANTHROPIC_API_KEY>
ROBOT_IP=<ROBOT_IP>
CONN_TYPE=webrtc
WEBRTC_SERVER_HOST=0.0.0.0
Expand All @@ -89,7 +91,7 @@ docker compose -f docker/unitree/agents_interface/docker-compose.yml up --build

- A Unitree Go2 robot accessible on your network
- The robot's IP address
- OpenAI API Key
- OpenAI/Claude/Alibaba API Key

### Python Installation (Ubuntu 22.04)

Expand All @@ -106,25 +108,44 @@ source venv/bin/activate

sudo apt install portaudio19-dev python3-pyaudio

# Install torch and torchvision if not already installed
pip install -r base-requirements.txt

# Install dependencies
pip install -r requirements.txt

# Copy and configure environment variables
cp default.env .env
```

### Agent API keys

Full functionality will require API keys for the following:

Requirements:
- OpenAI API key (required for all LLMAgents due to OpenAIEmbeddings)
- Claude API key (required for ClaudeAgent)
- Alibaba API key (required for Navigation skills)

These keys can be added to your .env file or exported as environment variables.
```
export OPENAI_API_KEY=<your private key>
export CLAUDE_API_KEY=<your private key>
export ALIBABA_API_KEY=<your private key>
```

### ROS2 Unitree Go2 SDK Installation

#### System Requirements
- Ubuntu 22.04
- ROS2 Distros: Iron, Humble, Rolling

See [Unitree Go2 ROS2 SDK](https://github.com/abizovnuralem/go2_ros2_sdk) for additional installation instructions.
See [Unitree Go2 ROS2 SDK](https://github.com/dimensionalOS/go2_ros2_sdk) for additional installation instructions.

```bash
mkdir -p ros2_ws
cd ros2_ws
git clone --recurse-submodules https://github.com/abizovnuralem/go2_ros2_sdk.git src
git clone --recurse-submodules https://github.com/dimensionalOS/go2_ros2_sdk.git src
sudo apt install ros-$ROS_DISTRO-image-tools
sudo apt install ros-$ROS_DISTRO-vision-msgs

Expand Down Expand Up @@ -157,7 +178,7 @@ ros2 launch go2_robot_sdk robot.launch.py
```bash
# Change path to your Go2 ROS2 SDK installation
source /ros2_ws/install/setup.bash
python test/test_planning_agent_web_interface.py
python tests/run.py
```

#### DimOS Interface:
Expand All @@ -173,17 +194,24 @@ Non-production directories excluded
```
.
├── dimos/
│ ├── agents/ # Agent implementation and behaviors
│ ├── robot/ # Robot control and hardware interface
│ │ └── unitree/ # Unitree Go2 specific control and skills implementations
│ ├── stream/ # WebRTC and data streaming
│ ├── web/ # DimOS development interface and API
│ │ └── dimos_interface/ # DimOS web interface
│ ├── simulation/ # Robot simulation environments
│ ├── utils/ # Utility functions and helpers
│ └── types/ # Type definitions and interfaces
├── tests/ # Test files
└── docker/ # Docker configuration files and compose definitions
│ ├── agents/ # Agent implementations
│ │ ├── planning/ # Planning-specific agents
│ │ └── execution/ # Execution-specific agents
│ ├── robot/
│ │ ├── skills/ # AbstractRobotSkill
│ │ ├── control/ # Robot control interfaces
│ │ └── unitree/ # Unitree Go2 specific code
│ │ ├── global_planner/ # Global planners
│ │ └── local_planner/ # Local navigation planners
│ ├── stream/ # Reactive data streams
│ │ ├── video/ # Video streaming
│ │ └── commands/ # Command streaming
│ ├── simulation/ # Simulation environments
│ ├── types/ # Type definitions
│ ├── utils/ # Utility functions
│ └── web/ # Web interface
├── tests/ # Examples
└── docker/ # Docker configs
```

## Building
Expand Down Expand Up @@ -434,3 +462,4 @@ Huge thanks to!
## Known Issues
- Agent() failure to execute Nav2 action primitives (move, reverse, spinLeft, spinRight) is almost always due to the internal ROS2 collision avoidance, which will sometimes incorrectly display obstacles or be overly sensitive. Look for ```[behavior_server]: Collision Ahead - Exiting DriveOnHeading``` in the ROS logs. Reccomend restarting ROS2 or moving robot from objects to resolve.
- ```docker-compose up --build``` does not fully initialize the ROS2 environment due to ```std::bad_alloc``` errors. This will occur during continuous docker development if the ```docker-compose down``` is not run consistently before rebuilding and/or you are on a machine with less RAM, as ROS is very memory intensive. Reccomend running to clear your docker cache/images/containers with ```docker system prune``` and rebuild.