feat(human): mascot tab with viseme-driven lipsync (staging only)

[senamakel]

Summary

• New /human tab — Ghosty SVG mascot with idle bob, blink, and viseme-driven mouth, paired with a floating chat sidebar that reuses Conversations in variant=\"sidebar\" mode.
• Three-phase lipsync flow: inference_start → thinking, text_delta → speaking with letter-driven pseudo-lipsync while the response streams, chat_done → ElevenLabs TTS via the new backend POST /openai/v1/audio/speech (tinyhumansai/backend#677) with with_visemes:true, then real Oculus 15-set viseme timeline keyed off audio.currentTime.
• Tab and route are gated to non-production builds (staging + dev) — the feature is still WIP.

Problem

We want the agent to feel like a person, not a chat log. Voice and a face you can watch are the entry points. This PR ships the visual+lipsync foundation so the next iteration (microphone input, persistent personality state) has somewhere to land.

Solution

• Mascot port (app/src/features/human/Mascot/) — Ghosty SVG ported from remotion/src/Ghosty, with useCurrentFrame swapped for a RAF-driven useMascotClock. Mouth is parametric: 8 viseme presets (REST/A/E/I/O/U/M/F) generated from {openness, width}, with the resting smile preserved when openness < 0.05.
• Lipsync hook (useHumanMascot) — subscribes to chat events. During streaming, picks a viseme from the trailing letter of each text_delta and decays back to rest (180 ms). On chat_done, fires the TTS request, plays the returned mp3, and reads audio.currentTime against the Oculus viseme timeline via a sticky-cursor findActiveFrame so playback stays O(1) per frame.
• TTS client (features/human/voice/) — thin apiClient.post('/openai/v1/audio/speech', ...) wrapper, base64 → blob → HTMLAudio player, Oculus → mascot shape map.
• Layout — split layout: full-bleed mascot stage with a floating, vertically-centered chat card on the right that clears the BottomTabBar. "Speak replies" toggle persists in localStorage.
• Gating — APP_ENVIRONMENT !== 'production' hides both the tab and the route in shipped builds.

Submission Checklist

• Tests added or updated (happy path + at least one failure / edge case) per docs/TESTING-STRATEGY.md
• N/A: diff coverage threshold — feature is staging-only and the chat-event-driven hook + audio playback are inherently DOM-bound; unit-tested the deterministic pieces (path interpolation, viseme picker, Oculus map, monotonic frame search).
• N/A: coverage matrix update — entry will land when the feature graduates out of staging-only.
• N/A: feature IDs — not yet matrix-tracked (see above).
• No new external network dependencies introduced (mock backend used per docs/TESTING-STRATEGY.md)
• N/A: release smoke — gated to staging/dev builds; no production surface.
• N/A: linked issue — exploratory feature, no tracking issue.

Impact

• Runtime: desktop only (web pnpm dev will show the tab too in dev). No new native dependencies; audio playback uses HTMLAudio.
• Network: new dependency on POST /openai/v1/audio/speech against the configured backend (VITE_BACKEND_URL, expected to point at staging-admin during testing). Backend endpoint added in tinyhumansai/backend#677.
• Performance: a single RAF loop while "speaking". Sticky cursor on the viseme timeline avoids per-frame scans.
• Migration / compatibility: production builds see no change (route + tab hidden via APP_ENVIRONMENT).

Related

• Closes:
• Follow-up PR(s)/TODOs: real microphone STT path, persistent listening/thinking visual states (currently face is computed but only viseme drives the SVG), surface a per-message replay button so users can re-hear a reply.

Summary by CodeRabbit

• New Features

  • Added a Human feature with an animated mascot, adjustable face/viseme states, and a centered stage background.
  • Integrated TTS playback with lip‑syncing and a persisted "Speak replies" toggle.
  • Exposed a Human route and a Human tab in navigation (enabled only outside production).
  • Added a floating Conversations sidebar above the tab area.

• Tests

  • Added tests covering viseme generation, mapping, timing, and mascot behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: eccf6754-f51c-4903-a2db-3e581fe11c8f

📥 Commits

Reviewing files that changed from the base of the PR and between ee4e725 and ba962b3.

📒 Files selected for processing (4)

• app/src/features/human/useHumanMascot.ts
• app/src/features/human/voice/audioPlayer.ts
• app/src/features/human/voice/visemeMap.test.ts
• app/src/features/human/voice/visemeMap.ts

✅ Files skipped from review due to trivial changes (2)

• app/src/features/human/voice/visemeMap.test.ts
• app/src/features/human/voice/visemeMap.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• app/src/features/human/voice/audioPlayer.ts
• app/src/features/human/useHumanMascot.ts

---

📝 Walkthrough

Walkthrough

Adds a non-production "Human" feature: a Ghosty SVG mascot with viseme-driven mouth animation, TTS playback and viseme timeline handling, a useHumanMascot hook wired to chat events, a HumanPage UI, and conditional routing/tab visibility when APP_ENVIRONMENT !== 'production'.

Changes

Human Mascot Feature

Layer / File(s)	Summary
Viseme Domain Model app/src/features/human/Mascot/visemes.ts, app/src/features/human/Mascot/visemes.test.ts	Defines VisemeId, VisemeShape, VISEMES, REST_SMILE_PATH, lerpViseme, and visemePath. Tests validate path shape, interpolation, clamping, and width effects.
SVG Paths & Shapes app/src/features/human/Mascot/paths.ts	Exports BODY_PATH, LEFT_LEG_PATH, RIGHT_LEG_PATH, ARM_PATH, and VIEWBOX for mascot geometry.
SVG Definitions app/src/features/human/Mascot/Defs.tsx	Adds GhostyDefs component rendering SVG <defs> (gradients, filters, clipPath) with ID prefixing.
Mascot Rendering & API app/src/features/human/Mascot/Ghosty.tsx, app/src/features/human/Mascot/index.ts	Adds Ghosty component (animation: bob, dot drift, arm wave, blink, viseme mouth) and exports/types via barrel file.
Animation Clock Hook app/src/features/human/Mascot/useMascotClock.ts	Adds useMascotClock(active = true) returning RAF-driven elapsed seconds.
Chat Integration Hook app/src/features/human/useHumanMascot.ts, app/src/features/human/useHumanMascot.test.ts	Adds useHumanMascot(options) and pickViseme(delta); subscribes to chat lifecycle events (onInferenceStart, onTextDelta, onDone, onError), manages face state, pseudo-lipsync, optional TTS playback and viseme timeline handling, and an RAF loop for re-renders. Tests cover viseme picking.
TTS API Client app/src/features/human/voice/ttsClient.ts	Adds synthesizeSpeech(text, opts) POSTing { text, with_visemes: true } to /openai/v1/audio/speech and types for viseme/alignment frames and TtsResponse.
Audio Playback app/src/features/human/voice/audioPlayer.ts	Adds playBase64Audio(base64, mime?) returning PlaybackHandle with currentMs(), stop(), and ended promise; manages object URL and Audio element lifecycle.
Viseme Mapping & Timeline app/src/features/human/voice/visemeMap.ts, app/src/features/human/voice/visemeMap.test.ts	Maps ElevenLabs/Oculus viseme tokens to VisemeShape (oculusVisemeToShape) and adds findActiveFrame(frames, ms, cursor) with sticky cursor semantics; tests verify mappings and cursor behavior.
Page, Routing & Tab app/src/features/human/HumanPage.tsx, app/src/AppRoutes.tsx, app/src/components/BottomTabBar.tsx	Adds HumanPage (speakReplies persisted to localStorage, uses useHumanMascot, renders Ghosty and sidebar). Conditionally mounts /human route and shows a Human tab when APP_ENVIRONMENT !== 'production'.

sequenceDiagram
    participant Chat as Chat/Stream
    participant Hook as useHumanMascot
    participant TTS as TTS API
    participant Audio as Audio Player
    participant Mascot as Ghosty Component

    Chat->>Hook: onInferenceStart
    Hook->>Mascot: face = 'thinking'
    Chat->>Hook: onTextDelta(text segment)
    Hook->>Hook: pickViseme(delta)
    Hook->>Mascot: viseme = heuristic shape
    Chat->>Hook: onDone(full response)
    alt speakReplies enabled
        Hook->>TTS: synthesizeSpeech(text, with_visemes: true)
        TTS-->>Hook: { audio_base64, visemes[] }
        Hook->>Audio: playBase64Audio(audio_base64)
        Audio-->>Hook: PlaybackHandle
        Hook->>Mascot: face = 'speaking'
        par Playback / Viseme tracking
            Audio->>Hook: time updates (currentMs)
            Hook->>Hook: findActiveFrame(visemes, currentMs)
            Hook->>Mascot: viseme = active frame shape
        end
        Audio->>Hook: onEnded/onError
        Hook->>Mascot: face = 'normal', viseme = rest
    else
        Hook->>Mascot: viseme decays to rest
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

• feat(remotion): Ghosty character library with transparent MOV variants #1059: Adds overlapping Ghosty mascot implementation (SVG defs, paths, and component structure) with direct code-level similarity.

Suggested reviewers

• graycyrus

"A rabbit hops in code so spry and quick,
Ghosty bobs and lipsyncs with a whiskered flick,
Non‑prod stage lights set for playful replies,
I chew a carrot and clap at the mascot's eyes!" 🐇✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding a new mascot tab with viseme-driven lipsync, limited to staging only.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Review rate limit: 1/5 review remaining, refill in 39 minutes and 3 seconds.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (2)

app/src/features/human/Mascot/Defs.tsx (1)

1-8: ⚡ Quick win

Use type-only React imports and a named props interface.

This component uses a runtime React import only for typing and defines props inline. Converting to import type + a GhostyDefsProps interface will align with repo TS rules and improve reuse/readability.

♻️ Suggested refactor

-import React from 'react';
+import type { FC } from 'react';

 import { BODY_PATH } from './paths';

-export const GhostyDefs: React.FC<{ idPrefix: string; bodyColor: string }> = ({
+interface GhostyDefsProps {
+  idPrefix: string;
+  bodyColor: string;
+}
+
+export const GhostyDefs: FC<GhostyDefsProps> = ({
   idPrefix,
   bodyColor,
 }) => {

As per coding guidelines, "**/*.{ts,tsx}: Use import type for type-only imports in TypeScript" and "Prefer interface for defining object shapes in TypeScript."

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@app/src/features/human/Mascot/Defs.tsx` around lines 1 - 8, The GhostyDefs
component currently imports React at runtime and uses an inline prop type;
change the runtime import to a type-only import (e.g., import type { FC } from
'react') and introduce a named interface GhostyDefsProps for the props
(idPrefix: string; bodyColor: string), then update the component signature to
use GhostyDefsProps and the type-only FC where applicable (preserve existing
references like GhostyDefs and BODY_PATH).

app/src/features/human/HumanPage.tsx (1)

10-17: 🏗️ Heavy lift

Move speakReplies persistence to Redux/persist instead of localStorage.

Lines 10-17 introduce ad hoc localStorage-backed state inside app/src. Please route this through the existing Redux/persist pattern for consistency and centralized state behavior.

As per coding guidelines app/src/**/*.{ts,tsx}: Use Redux and persist where configured for app state; prefer Redux over ad hoc localStorage for state management.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@app/src/features/human/HumanPage.tsx` around lines 10 - 17, The local
localStorage-backed state in HumanPage (speakReplies, setSpeakReplies,
SPEAK_REPLIES_KEY and the useEffect) should be removed and replaced with a
Redux-persisted piece of state: create or extend a slice (e.g., uiPreferences or
humanPage) with a speakReplies boolean, wire it into the store persistence
config, then in HumanPage use useSelector to read speakReplies and useDispatch
to update it (dispatching an action like setSpeakReplies) instead of
useState/useEffect and direct localStorage access; ensure the slice action and
persist config names match existing store conventions so the value is persisted
automatically.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@app/src/features/human/useHumanMascot.ts`:
- Around line 116-135: The startTtsPlayback function uses a .then() chain and
has a race where an earlier audio's completion callback can clear state for a
newer playback; rewrite the completion handling to use async/await (avoid
.then()) and guard by comparing playbackRef.current === handle before clearing
visemeFramesRef, visemeCursorRef and resetting face; ensure you await
playBase64Audio to get the handle, set playbackRef.current = handle, then in a
try/finally (or after awaiting handle.ended) only clear state if
playbackRef.current === handle so stale callbacks don't override newer playbacks
(symbols: startTtsPlayback, synthesizeSpeech, playBase64Audio, playbackRef,
visemeFramesRef, visemeCursorRef, setFace).

In `@app/src/features/human/voice/audioPlayer.ts`:
- Line 47: audio.play() can reject before playback starts and leak the created
object URL and leave callers without a PlaybackHandle; wrap the await
audio.play() call in a try/catch, and in the catch revoke the created object URL
(the variable used as the audio.src/objectUrl) and clean up the audio element
before rethrowing the error so callers still receive the rejection and no blob
URL is leaked; update the code paths that would have returned a PlaybackHandle
to only do so after successful play() and ensure the catch does not return a
handle.

In `@app/src/features/human/voice/visemeMap.ts`:
- Around line 53-57: The code uses cursor directly to compute i which can be
negative and cause frames[i] to be undefined; fix by clamping cursor before
using it (e.g., compute i = Math.min(Math.max(0, cursor), frames.length - 1)) so
i is always within [0, frames.length-1], then proceed with the existing
rewind/forward loops that reference frames, affecting the logic around variables
cursor, i, frames, and ms.

---

Nitpick comments:
In `@app/src/features/human/HumanPage.tsx`:
- Around line 10-17: The local localStorage-backed state in HumanPage
(speakReplies, setSpeakReplies, SPEAK_REPLIES_KEY and the useEffect) should be
removed and replaced with a Redux-persisted piece of state: create or extend a
slice (e.g., uiPreferences or humanPage) with a speakReplies boolean, wire it
into the store persistence config, then in HumanPage use useSelector to read
speakReplies and useDispatch to update it (dispatching an action like
setSpeakReplies) instead of useState/useEffect and direct localStorage access;
ensure the slice action and persist config names match existing store
conventions so the value is persisted automatically.

In `@app/src/features/human/Mascot/Defs.tsx`:
- Around line 1-8: The GhostyDefs component currently imports React at runtime
and uses an inline prop type; change the runtime import to a type-only import
(e.g., import type { FC } from 'react') and introduce a named interface
GhostyDefsProps for the props (idPrefix: string; bodyColor: string), then update
the component signature to use GhostyDefsProps and the type-only FC where
applicable (preserve existing references like GhostyDefs and BODY_PATH).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3f8ceb9b-022e-48c1-bbff-e21a57fe2ab9

📥 Commits

Reviewing files that changed from the base of the PR and between 744344e and ee4e725.

📒 Files selected for processing (16)

• app/src/AppRoutes.tsx
• app/src/components/BottomTabBar.tsx
• app/src/features/human/HumanPage.tsx
• app/src/features/human/Mascot/Defs.tsx
• app/src/features/human/Mascot/Ghosty.tsx
• app/src/features/human/Mascot/index.ts
• app/src/features/human/Mascot/paths.ts
• app/src/features/human/Mascot/useMascotClock.ts
• app/src/features/human/Mascot/visemes.test.ts
• app/src/features/human/Mascot/visemes.ts
• app/src/features/human/useHumanMascot.test.ts
• app/src/features/human/useHumanMascot.ts
• app/src/features/human/voice/audioPlayer.ts
• app/src/features/human/voice/ttsClient.ts
• app/src/features/human/voice/visemeMap.test.ts
• app/src/features/human/voice/visemeMap.ts