Skip to content

feat: adding ten-vad wasm to project, configuration options on audio#3822

Open
emmkay440 wants to merge 1 commit into
element-hq:livekitfrom
emmkay440:feat/ten-vad
Open

feat: adding ten-vad wasm to project, configuration options on audio#3822
emmkay440 wants to merge 1 commit into
element-hq:livekitfrom
emmkay440:feat/ten-vad

Conversation

@emmkay440
Copy link
Copy Markdown

@emmkay440 emmkay440 commented Mar 29, 2026

Adding TEN-VAD voice activity detection gate for local microphone

What and why

This PR adds an optional voice activity detection (VAD) gate applied to the local microphone track before it is published to the SFU. When enabled, audio is silenced during non-speech segments, reducing unwanted noises from being transmitted.

The gate is implemented using TEN-VAD and it runs entirely inside an AudioWorklet on the audio thread, no IPC round-trip to the main thread giving approximately 16 ms detection latency(customizeable).

Before this change there was no client-side VAD gate.
After this change users can opt into a VAD gate from Audio Settings.

The setting is off by default and persisted in localStorage.

Architecture

getUserMedia -> LocalAudioTrack
                   └─ TenVadTransformer (LiveKit TrackProcessor)
                         └─ AudioWorkletNode (TenVadProcessor)
                               ├─ TenVADRuntime (WASM, synchronous)
                               └─ gain ramp -> processedTrack -> SFU
  • public/vad/ten_vad.wasm + ten_vad.js TEN-VAD Emscripten build.
  • src/livekit/TenVadProcessor.worklet.ts AudioWorklet; wraps the WASM synchronously, decimates 48 kHz -> 16 kHz at 3:1, runs VAD every hop (10-16 ms), applies asymmetric gain ramp.
  • src/livekit/TenVadTransformer.ts LiveKit TrackProcessor adapter; fetches and compiles the WASM once (module is cached and reused across restarts), wires the Web Audio graph.
  • src/settings/settings.ts seven new Setting entries for VAD state and parameters.
  • src/settings/SettingsModal.tsx VAD section added to the Audio tab with Disabled / Simple / Advanced radio buttons and the corresponding controls.
  • src/state/CallViewModel/localMember/Publisher.ts applyTenVad() subscribes to the local mic track and the vadEnabled setting, attaches or detaches TenVadTransformer reactively, and pushes parameter changes to the live worklet.

Testing

  1. Run node /tmp/yarn4.js backend
  2. Open two instances of the same call to talk to yourself.
  3. Open a call and go to Settings -> Audio.
  4. Gently scratch microphone in a way you can hear it, breath heavily next to it and try hitting your table or putting a big cup on the desk next to the mic.
  5. Under Voice Activity Detection, select Simple.
  6. Perform step 3 again and you shouldn't be able to hear those noises anymore.
  7. Switch back to Disabled
  8. Reproduce step 3 again and confirm you can hear the unwanted noises again.

Notes

Claude helped me with the implementation but going forward with this PR I can do all changes by myself if asked to.
An enhancement to this PR would be adding some sort of voice feedback loop so users can test for themselves.

@emmkay440 emmkay440 requested a review from a team as a code owner March 29, 2026 00:09
@emmkay440 emmkay440 requested a review from toger5 March 29, 2026 00:09
@CLAassistant
Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


mk seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants