Skip to content

feat(ui): 3D stick figure visualization (COCO-17 + smooth animations)#490

Open
schwarztim wants to merge 1 commit intoruvnet:mainfrom
schwarztim:pr/3d-stick-figure-viz
Open

feat(ui): 3D stick figure visualization (COCO-17 + smooth animations)#490
schwarztim wants to merge 1 commit intoruvnet:mainfrom
schwarztim:pr/3d-stick-figure-viz

Conversation

@schwarztim
Copy link
Copy Markdown

Motivation

Spatial dashboards currently show voxel "blobs" from CSI tomography — non-intuitive for end users. This PR adds a COCO-17 stick figure renderer to ui/spatial.html that maps body cluster centroids to anatomical poses, with smooth lerp animations between ticks and posture-aware idle motion (breathing, gentle sway).

Changes

  • ui/spatial.html — Three.js / Canvas-based 3D renderer with:
    • COCO-17 joint definitions inline (17 keypoints, 18 bone pairs)
    • Three pose variants driven by pose_hint: standing (green), sitting (yellow), lying (blue)
    • FigureState persistent figure map — no recreate-on-every-frame
    • Position lerp (factor 0.1) for smooth interpolation between cluster ticks
    • Posture-transition keypoint blending over ~1 s (poseBlend 0→1 at +0.03/frame)
    • Breathing animation: torso Y-scale at detected BR rate from /api/v1/vital-signs
    • Activity-based sway: still=0.01 m, moderate=0.03 m, active=0.08 m
    • Opacity fadeout for disappearing bodies (quality decrements at −0.05/tick)
    • Fallback: single standing figure from voxel centroid when no cluster data
  • CHANGELOG.md — entry under [Unreleased]

Backend dependency

This UI consumes:

For best results merge this AFTER #487. Filing now to surface for parallel review.

WebSocket schema

The renderer subscribes to the existing /ws/sensing endpoint and handles spatial_update messages that include a clusters array:

{
  "type": "spatial_update",
  "data": {
    "clusters": [
      {
        "id": 0,
        "centroid": [x, y, z],
        "bbox_min": [x, y, z],
        "bbox_max": [x, y, z],
        "pose_hint": "standing | sitting | lying"
      }
    ]
  }
}

Falls back to polling /api/v1/spatial/clusters every 2 s if WebSocket is unavailable.

COCO-17 joint definitions

0: nose       1: left_eye    2: right_eye   3: left_ear    4: right_ear
5: left_shoulder            6: right_shoulder
7: left_elbow               8: right_elbow
9: left_wrist              10: right_wrist
11: left_hip               12: right_hip
13: left_knee              14: right_knee
15: left_ankle             16: right_ankle

18 bone pairs connecting adjacent joints per the COCO topology. Definitions are documented inline in spatial.html as STANDING_KEYPOINTS and BONES constants.

Visual notes

  • Dark theme: background #050508, overlays rgba(10,10,15,0.85) — MD3-compatible dark surface
  • Stick figures render at 60 fps via requestAnimationFrame; lerp keeps motion smooth at 2 s poll intervals
  • No external 3D dependencies beyond Three.js (loaded via importmap from unpkg CDN, same pattern as other UI pages)
  • Legend updated with pose-colored entries (standing/sitting/lying swatches)

Test results

  • cargo check --workspace --no-default-features (v2/): PASS (1 pre-existing warning, 0 errors)
  • No hardcoded IPs, hostnames, or credentials in the HTML file (verified via grep)
  • Manual review: all four activity states (absent/gentle/moderate/active) produce distinct animation amplitudes

Notes

  • Pure frontend addition — zero impact on sensing pipeline, signal processing, or Rust crates
  • COCO-17 joint definitions documented inline in HTML for future maintenance
  • Activity-state animations tuned for adult human bodies; non-human subjects would render unnaturally (acceptable trade-off, noted in subtitle disclaimer)

…animations

- Add ui/spatial.html: Three.js 3D renderer for spatial dashboard
- COCO-17 keypoint definitions inline; 18 bone pairs (STANDING_KEYPOINTS)
- Three pose variants: standing (green), sitting (yellow), lying (blue)
- Persistent figure map (no recreate-on-every-frame) via FigureState class
- Position lerp (factor 0.1) for smooth interpolation between ticks
- Posture-transition keypoint blending over ~1s (poseBlend 0->1 at +0.03/frame)
- Breathing animation: torso Y-scale at detected BR rate from /api/v1/vital-signs
- Activity-based sway: still=0.01m, moderate=0.03m, active=0.08m
- Opacity fadeout for disappearing bodies (quality field decrements at -0.05/tick)
- WS subscription to spatial_update + body_cluster_update; falls back to polling
- Fallback: single standing figure from voxel centroid when no cluster data

Depends on /api/v1/spatial/clusters endpoint from body tracking PR (ruvnet#487).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant