Adds Teleop UI Feedback Visualization: Framework and Application

[lotusl-code]

Framework

Architecture

1. XRVisualization
   1. Provide APIs. (For API explanation, see next section “API Definition)
   2. Manage a VisualizationManager instance and a DataCollector instance, and pass them the user commands.
2. DataCollector
   1. Collects and manages data for visualization purposes.
   2. Performs pre-processing and post-processing for certain data fields.
   3. Trigger VisualizationManager’s callbacks when data is changed.
3. VisualizationManager
   1. Responsible for the visualization.
   2. Use callbacks to manage visualizers. As a part of system default, the VisualizationManager class comes with some callbacks and visualizers. Users can use API to register new callbacks or disable default callbacks.
   3. Common visualizers are widgets and markers.

TeleopVisualizationManager is a child-class of VisualizationManager. It provides a series of visualization logics by defining callbacks.
If users don't need the logics, they can also use the plain VisualizationManager instead of TeleopVisualizationManager, or even customize their own child-class.

5. IsaacLab Systems: Any other systems within IsaacLab.
6. User Scripts: Users can control visualization through APIs anywhere in their codes.
7. Main Loop (To Be Determined)
   1. To call update_loop() every frame to update the whole system.
   2. Compare the positions to call update_loop():

Where to call update	Pros	Cons
User Script	Stable	Need users to control
Async Thread	Easy-to-use for users	Vulnerable to single-point failure
In IsaacLab System	Both stable and easy-to-use	No such always-update system

Sample

scene_visualization_sample.py includes 3 visualizers:

1. Panel on the left
   1. Display all data in the DataStorage
   2. Updated periodically every 1 second
2. Panel on the right
   1. (also) Display all data in the DataStorage
   2. Only update when the data changes.
   3. Data is posted by user script (record_demos.py), and will change for 5 times.
3. Message widget in the center
   1. Display yellow text “IK Error Detected”. Will disappear automatically after 5 secondes
   2. Triggered on event “ik_error”. The event is posted by user script (record_demos.py)

Application

teleop_visualization_manager.py contains 2.5 visualizers:

1. IK Error Widget
   1. At event "ik_solver_failed", a widget will show at the center of view saying "IK Error Detected".
   2. The widget will keep showing until the event is no longer received within last 3 seconds.
2. Torque Skeleton
   1. Drawn above operator's hands (not robot hands)
   2. Torque on the robot finger joints will be mapped to the skeleton. The mapping chart is sent in by the retargeter.
   3. Color indicates the ratio of the torque to the limit. From small to large is corresponding to Green-Yellow-Orange-Red.
   4. The skeleton is slightly above the hands position for better visibility.
   5. [Known Issue] Current torque data is accumulated number instead of actual torque on each joint. Need further investigation.
3. Whiteboard
   1. The function _send_visualization_data_to_client() sends the following critical robot telemetry to client for visualization:
      • manipulability_ellipsoid
      • joints_distance_percentage_to_limit
      • joints_torque_percentage_of_limit
      • joints_names
   2. Currently disabled because sending large data (>1KB per frame) will cause the client-side StreamSDK to crash.
   3. Will be turned on after finding a solution with StreamSDK solution team, and get the data from Mimic team.

[jtigue-bdai]

Closed. New MR: #3178

This is not closed and 3178 is this PR

Also you don't need to put code API definitions in the description, please remove.

[lotusl-code]

Original MR: https://github.com/isaac-sim/IsaacLab-Internal/pull/629

[rwiltz]

Can we not get this from the hand_joint_names in the GR1T2RetargeterCfg?

[rwiltz]

This is not the right place to put this I think. We only care about this for imitation learning but this is a base class specifically for RL tasks.

[rwiltz]

@lotusl-code Are you still getting 45fps with these changes specifically with the hand skeleton enabled? In the past we have been bitten by non instanced visualizations in the scene causing performance issues.

[lotusl-code]

@lotusl-code Are you still getting 45fps with these changes specifically with the hand skeleton enabled? In the past we have been bitten by non instanced visualizations in the scene causing performance issues.

I'll test

[life1ess]

nit: Can this be a private static member of TeleopVisualizationManager for better encapsulation?

[life1ess]

Why is this commented out?

[lotusl-code]

Newly opened #3356