Implemented an action manager to process action and replace RLEnv by EmbodiEnv by yangchen73 · Pull Request #164 · DexForce/EmbodiChain

yangchen73 · 2026-03-08T07:25:11Z

Description

Implemented action manager like IsaacLab to process action
Remove useless RLEnv, replaced by EmbodiEnv
Make corresponding changes in RL algorithms and environments
Fix a bug: Increment _elapsed_steps immediately after _post_step so that the max_episode_steps truncation check uses the newest step count and ends the episode at the correct step.
Add unit tests for action manager

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (existing functionality will not work without user modification)
Documentation update

Checklist

I have run the black . command to format the code base.
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
Dependencies have been updated, if applicable.

Copilot

Pull request overview

This PR refactors the action handling in the EmbodiChain RL stack by introducing a modular ActionManager (similar to IsaacLab) to replace the deleted RLEnv base class. It also includes a bug fix that moves _elapsed_steps increment to occur before the episode truncation check.

Changes:

New ActionManager + concrete ActionTerm subclasses (DeltaQposTerm, QposTerm, QposNormalizedTerm, EefPoseTerm, QvelTerm, QfTerm) added to the manager system
RLEnv deleted; its compute_task_state, get_info, evaluate, and _preprocess_action methods moved into EmbodiedEnv; all RL task environments updated to inherit from EmbodiedEnv directly
_elapsed_steps increment moved before the max_episode_steps truncation check in BaseEnv.step() to fix off-by-one episode boundary behavior

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`embodichain/lab/gym/envs/managers/action_manager.py`	New file: ActionManager and ActionTerm base + concrete implementations
`embodichain/lab/gym/envs/managers/cfg.py`	Adds `ActionTermCfg` dataclass
`embodichain/lab/gym/envs/managers/__init__.py`	Exports new action manager classes
`embodichain/lab/gym/envs/embodied_env.py`	Integrates ActionManager; moves RL utility methods from deleted RLEnv
`embodichain/lab/gym/envs/base_env.py`	Fixes `_elapsed_steps` increment ordering
`embodichain/lab/gym/envs/rl_env.py`	Deleted: entire file removed
`embodichain/lab/gym/envs/__init__.py`	Removes `RLEnv` imports
`embodichain/lab/gym/envs/tasks/rl/push_cube.py`	Updates to inherit from `EmbodiedEnv`
`embodichain/lab/gym/envs/tasks/rl/basic/cart_pole.py`	Updates to inherit from `EmbodiedEnv`
`embodichain/lab/gym/utils/gym_utils.py`	Adds actions config parsing in `config_to_cfg`
`embodichain/agents/rl/algo/ppo.py`	Uses `action_manager.action_type` when available
`embodichain/agents/rl/algo/grpo.py`	Uses `action_manager.action_type` when available
`embodichain/agents/rl/utils/trainer.py`	Uses `action_manager.action_type` when available
`configs/agents/rl/push_cube/gym_config.json`	Migrates action config to `actions` section
`configs/agents/rl/basic/cart_pole/gym_config.json`	Migrates action config to `actions` section
`docs/source/tutorial/rl.rst`	Updates docs for new Action Manager API
`docs/source/overview/gym/env.md`	Updates docs for new Action Manager API
`CONTRIBUTING.md` / `CLAUDE.md`	Updates project structure descriptions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

embodichain/lab/gym/envs/managers/action_manager.py

embodichain/lab/gym/envs/managers/__init__.py

docs/source/overview/gym/env.md

embodichain/lab/gym/envs/embodied_env.py

embodichain/lab/gym/envs/managers/action_manager.py

embodichain/lab/gym/utils/gym_utils.py

embodichain/lab/gym/envs/embodied_env.py

embodichain/lab/gym/envs/managers/action_manager.py

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tests/gym/envs/managers/test_action_manager.py

embodichain/lab/gym/utils/gym_utils.py

embodichain/lab/gym/envs/managers/action_manager.py

Copilot

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

yuecideng · 2026-03-08T14:35:40Z

embodichain/lab/gym/envs/managers/action_manager.py

+        """Dimension of the action term (policy output dimension)."""
+        raise NotImplementedError
+
+    def process_action(self, action: torch.Tensor) -> EnvAction:


Add abstractmethod to this method to enforce the inherent class to implment it.

embodichain/lab/gym/envs/managers/action_manager.py

yuecideng · 2026-03-08T14:45:57Z

embodichain/lab/gym/envs/embodied_env.py

-                    action_to_store.to(buffer_device), non_blocking=True
-                )
+                else:
+                    logger.log_error(


log_error will raise exception and the progrom will stop. If you still want to keep the program to run, use log_warning instead. Otherwise we should keep use log_error

Okay, I'v changed it to log_error instead of stopping the program.

…il case for penalty

yangchen73 added 6 commits March 7, 2026 18:27

Feat: implement ActionManager

53f64d0

Update enviornment to fit with action manager

c101fb0

Update rl algorithm to fit with action manager

24edb19

Remove useless RLEnv

e6c5071

Update docs

9276cf4

Fix:_elapsed_steps += 1 then truncated

32648f3

Copilot AI review requested due to automatic review settings March 8, 2026 07:25

Copilot started reviewing on behalf of yangchen73 March 8, 2026 07:25 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

yuecideng self-requested a review March 8, 2026 08:06

yangchen73 added 3 commits March 8, 2026 13:49

Add unit tests for action manager

f5be828

Update docs and comments

3f323a8

Reformat files

3e8cae1

yangchen73 requested a review from Copilot March 8, 2026 13:55

Copilot started reviewing on behalf of yangchen73 March 8, 2026 13:55 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

yangchen73 added 2 commits March 8, 2026 14:11

Add unit tests for EefPoseTerm

ea913e6

Update comments and docs

798dad3

yangchen73 requested a review from Copilot March 8, 2026 14:16

Copilot started reviewing on behalf of yangchen73 March 8, 2026 14:17 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

yuecideng requested changes Mar 8, 2026

View reviewed changes

yangchen73 added 2 commits March 8, 2026 15:39

Use log_warning to keep program running

5c5edc7

Add is_success to the TensorDict returned by EdfPoseTerm for using fa…

bdb8b29

…il case for penalty

yuecideng approved these changes Mar 9, 2026

View reviewed changes

yangchen73 merged commit 157a0c9 into main Mar 9, 2026
5 checks passed

yangchen73 deleted the yc/action_manager branch March 9, 2026 02:05

Conversation

yangchen73 commented Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

yuecideng Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yuecideng Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

yangchen73 Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yangchen73 commented Mar 8, 2026 •

edited

Loading