RL_Dog

---

Reinforced-Learning for autonomous walking and suddenly-stopping for Legged Robot (AlienGo by Unitree)

Project by Pietro Dardano, advised by prof. A. Del Prete - UniTn - Summer 2024 + Spring 2025

Since the amount of example code for IsaacLab is still quite limited, this project can be seen as and example (even if still work in progress) on how to create Manager_Based_RL_Envs with different algorithms.

Remark: The available public code does not include the Safety and Robust features for harsh terrains!

If you find it useful, please consider giving a star to the repository. 😄 ⭐

Update 2026: I have started a new open-project on robot manipulation using UR5e and FR3 robots in IsaacLab and Mujoco: RL_Arm. ⭐

Feel free to check it out—contributions are very welcome!

Methodology and Latest Updates

• Algorithms and Agents:

  • Proximal Policy Optimization (PPO)
    • SKRL: Works ✅
  • Deep Deterministic Policy Gradient (DDPG)
    • SKRL: To fix NaN generation ⚙
    • SB3: To init and derive from TD3 ⚙
  • Twind_Delayed_DDPG (TD3)
    • SB3: To fix and do first train ⚙

• Architecture Inspired by ANYmal (ETH-RSL): ANYmal paper.

• SKRL: Wrapper for algorithms and agents. Documentation.

• SB3: Using it since SKRL is giving problems with DDPG. Documentation

• Python + PyTorch: Programming languages and framework for development and deep learning.

Repository

Branches:

• main: Currently mantained and working code: IsaacSim 4.5, IsaacLab 2.1.0
• IsaacLab_v1.1: Code from summer 2024, Legacy versions: IsaacSim 4.1, IsaacLab 1.1.0

Project Structure

RL_Dog
├── Isaac_aliengo         # Envs and tasks for training using NVIDIA's Isaac Sim
│   ├── aliengo_DDPG              # DDPG try (SKRL)
│   ├── aliengo_DDPG_SB3          # DDPG/TD3 try (Stable Baselines3)
│   ├── aliengo_safety            # DDPG to try Safety
│   ├── aliengo_Stop_FullState    # Inherited from v1.1, not good
│   ├── aliengo_Stop_FullState_2  # Ideal config stop
│   ├── aliengo_Stop_Real         # Real config stop
│   ├── aliengo_Walk_Real         # Real config walk
│   ├── aliengo_Walk_Ideal        # Ideal config walk
├── assets                # Assets such as models and textures
├── isaaclab              # Just to access IsaacLab code, like if forked
├── isaaclab_assets       # Just to access IsaacLab code, like if forked
├── isaaclab_tasks        # Just to access IsaacLab code, like if forked
├── runs                  # Logs, results & checkpoints from trainings
├── configs.py            # Config file for training and environment setup
├── run_training.py       # Script to launch training sessions
├── run_training.sh       # Shell script to run trainings
├── TRAINING_GUIDE.md     # How to run trainings
├── LICENSE
├── README.md

Train and Play

For detailed instructions on training the agent, see the TRAINING_GUIDE.md.

Agent Status

• PPO-based agents: Fully implemented, tested, and ready for training and deployment. ✅
• DDPG and TD3 agents: Currently under development and undergoing refinement. ⚙

Running the Simulation

The simulation is a work in progress.
(As of August 2025: Full implementation is planned for completion by the end of the year, as it is not the current focus.)

System Configurations

Nvidia & CUDA

• Driver Version: 570.124.06
• CUDA Version: 12.8
• For Nvidia Blackwell: Driver 570.133.20 (server-open) | CUDA Version 12.8

Workstation Specifications

Workstation	CPU	GPU	RAM	OS
WS 1	AMD® Ryzen 9 7950x	2x NVIDIA RTX A6000 Ada Gen, 48GB GDDR6, 300W	192GB	Ubuntu 22.04.4 LTS
WS 2	Intel Xeon® Gold 6226R	NVIDIA RTX A6000, 48GB GDDR6, 300W	128GB	Ubuntu 20.04 LTS
WS 3	Intel Xeon® Gold 5415+	NVIDIA RTX A4000, 14GB GDDR6, 140W	128GB	Ubuntu 20.04 LTS
WS 4 📌	AMD® Ryzen Threadripper 7970x	NVIDIA RTX PRO Blackwell A6000, 96GB GDDR7	128GB	Ubuntu 22.04.4 LTS

Understanding the Project

For a comprehensive understanding of the principles and techniques used in this project, refer to the following resources:

• A detailed review of related methodologies can be found in reference 1.
• Insights into recent advancements are discussed in reference 2.

Installation

To set up the project, follow these steps:

1. Follow the installation GUIDE of IsaacSim and IsaacLab
2. Clone this repository: (Remark! main branch is for IsaacSim > v4.5 and IsaacLab > v2.0.0)

   git clone https://github.com/pietrodardano/RL_Dog.git
   

3. Check that your assets (URDF, config) are installed locally, in your IsaacLab folder in isaaclab_assets directory.
4. I am using Miniconda, be sure to change or use the same environment name.
5. Launch the simulation (headless or not) with the scripts that you can find at the beginning of each main.py

Extra

NVIDIA's Isaac Lab: allows for parallel simulation of multiple environments necessary for training our models. Refer to the Orbit and Isaac Sim pages for more information.

Please note that IsaacLab contains many OpenAI Gym and Gymnasium features. It is common to find attributes, methods and classes related to them.