From a311b388f373958d222628267175c2667f535590 Mon Sep 17 00:00:00 2001 From: Chris Wilson Date: Wed, 4 Mar 2026 10:24:46 -0600 Subject: [PATCH 1/5] Restructure readme and adds a couple of docs to split up video and the up comming audio processor work --- README.md | 82 ++++------------ processor-docs/audio-processors.md | 149 +++++++++++++++++++++++++++++ processor-docs/video-processors.md | 93 ++++++++++++++++++ 3 files changed, 263 insertions(+), 61 deletions(-) create mode 100644 processor-docs/audio-processors.md create mode 100644 processor-docs/video-processors.md diff --git a/README.md b/README.md index 79b3e57..8a8139f 100644 --- a/README.md +++ b/README.md @@ -1,91 +1,51 @@ # LiveKit track processors +Prebuilt audio and video track processors for [LiveKit](https://livekit.io), implementing the [`TrackProcessor`](https://docs.livekit.io/home/client/tracks/manipulate/#track-processors) interface from `livekit-client`. + ## Install ``` npm add @livekit/track-processors ``` -## Usage of prebuilt processors - -### Available processors - -This package exposes the `BackgroundProcessor` pre-prepared processor pipeline, which can be used in a few ways: +## Video processors -- `BackgroundProcessor({ mode: 'background-blur', blurRadius: 10 /* (optional) */ })` -- `BackgroundProcessor({ mode: 'virtual-background', imagePath: "http://path.to/image.png" })` -- `BackgroundProcessor({ mode: 'disabled' })` - -### Usage example +Background blur and virtual background for video tracks: ```ts -import { BackgroundProcessor, supportsBackgroundProcessors, supportsModernBackgroundProcessors } from '@livekit/track-processors'; - -if(!supportsBackgroundProcessors()) { - throw new Error("this browser does not support background processors") -} +import { BackgroundProcessor } from '@livekit/track-processors'; -if(supportsModernBackgroundProcessors()) { - console.log("this browser supports modern APIs that are more performant"); -} - -const videoTrack = await createLocalVideoTrack(); -const processor = BackgroundProcessor({ mode: 'background-blur' }); +const processor = BackgroundProcessor({ mode: 'background-blur', blurRadius: 10 }); await videoTrack.setProcessor(processor); -room.localParticipant.publishTrack(videoTrack); - -async function disableBackgroundBlur() { - await videoTrack.stopProcessor(); -} - -async function updateBlurRadius(radius) { - return processor.switchTo({ mode: 'background-blur', blurRadius: radius }); -} ``` -In a real application, it's likely you will want to only sometimes apply background effects. You -could accomplish this by calling `videoTrack.setProcessor(...)` / `videoTrack.stopProcessor(...)` on -demand, but these functions can sometimes result in output visual artifacts as part of the switching -process, which can result in a poor user experience. +Available modes: `background-blur`, `virtual-background`, and `disabled` (passthrough). -A better option which won't result in any visual artifacts while switching is to initialize the -`BackgroundProcessor` in its "disabled" mode, and then later on switch to the desired mode. For -example: -```ts -const videoTrack = await createLocalVideoTrack(); -const processor = BackgroundProcessor({ mode: 'disabled' }); -await videoTrack.setProcessor(processor); -room.localParticipant.publishTrack(videoTrack); - -async function enableBlur(radius) { - await processor.switchTo({ mode: 'background-blur', blurRadius: radius }); -} - -async function disableBlur() { - await processor.switchTo({ mode: 'disabled' }); -} -``` +See [processor-docs/video-processors.md](processor-docs/video-processors.md) for full usage, browser support checks, and how to avoid visual artifacts when switching modes. -## Developing your own processors +## Audio processors -A track processor is instantiated with a Transformer. +Gain control for audio tracks, and a reference implementation for building custom audio processors: ```ts -// src/index.ts -export const VirtualBackground = (imagePath: string) => { - const pipeline = new ProcessorWrapper(new BackgroundTransformer({ imagePath })); - return pipeline; -}; +import { GainAudioProcessor } from '@livekit/track-processors'; + +const processor = new GainAudioProcessor({ gainValue: 1.5 }); +await audioTrack.setProcessor(processor); ``` -### Available base transformers +See [processor-docs/audio-processors.md](processor-docs/audio-processors.md) for full usage, the `TrackProcessor` interface for audio, and a guide to building your own audio processor with the Web Audio API. + +## Developing your own processors -- BackgroundTransformer (can blur background, use a virtual background, or be put into a disabled state); +This package implements the `TrackProcessor` interface from `livekit-client`. Video and audio processors take different approaches: +- **Video processors** use `ProcessorWrapper` with a transformer pipeline — see [processor-docs/video-processors.md](processor-docs/video-processors.md#developing-your-own-video-processor) +- **Audio processors** implement `TrackProcessor` directly using the Web Audio API — see [processor-docs/audio-processors.md](processor-docs/audio-processors.md#building-your-own-audio-processor) ## Running the sample app -This repository includes a small example app built on [Vite](https://vitejs.dev/). Run it with: +This repository includes a small example app built on [Vite](https://vitejs.dev/) that demonstrates both video and audio processors. Run it with: ``` # install pnpm: https://pnpm.io/installation diff --git a/processor-docs/audio-processors.md b/processor-docs/audio-processors.md new file mode 100644 index 0000000..485f33a --- /dev/null +++ b/processor-docs/audio-processors.md @@ -0,0 +1,149 @@ +# Audio Processors + +This document covers the audio track processors available in `@livekit/track-processors` and how to build your own. + +## GainAudioProcessor + +The `GainAudioProcessor` is a minimal audio processor that applies a Web Audio [`GainNode`](https://developer.mozilla.org/en-US/docs/Web/API/GainNode) to a local audio track. It serves both as a ready-to-use volume control and as a reference implementation for building custom audio processors. + +### Basic usage + +```ts +import { GainAudioProcessor } from '@livekit/track-processors'; + +const audioTrack = await createLocalAudioTrack(); +const processor = new GainAudioProcessor({ gainValue: 1.5 }); +await audioTrack.setProcessor(processor); +room.localParticipant.publishTrack(audioTrack); + +// Update gain on the fly +processor.setGain(0.5); + +// Remove the processor +await audioTrack.stopProcessor(); +``` + +### Constructor options + +| Option | Type | Default | Description | +|--------|------|---------|-------------| +| `gainValue` | `number` | `1.0` | Initial gain multiplier. `1.0` = unity (no change), `0.0` = silence, `> 1.0` = amplify. | + +## The TrackProcessor interface for audio + +Audio processors implement the `TrackProcessor` interface from `livekit-client`: + +```ts +interface TrackProcessor { + name: string; + init(opts: AudioProcessorOptions): Promise; + restart(opts: AudioProcessorOptions): Promise; + destroy(): Promise; + processedTrack?: MediaStreamTrack; + onPublish?(room: Room): Promise; + onUnpublish?(): Promise; +} +``` + +When you call `audioTrack.setProcessor(processor)`, the SDK: + +1. Creates an `AudioContext` and passes it to your processor via `AudioProcessorOptions` +2. Calls `processor.init()` with the options +3. Reads `processor.processedTrack` and uses it as the track sent to the SFU +4. Calls `sender.replaceTrack()` to swap the raw track for the processed one + +### AudioProcessorOptions + +The SDK provides these options when calling `init()` and `restart()`: + +```ts +interface AudioProcessorOptions { + kind: Track.Kind.Audio; + track: MediaStreamTrack; // The raw microphone MediaStreamTrack + audioContext: AudioContext; // A shared AudioContext managed by the SDK + element?: HTMLMediaElement; // The media element, if one exists +} +``` + +Key points: + +- **Use the provided `AudioContext`** rather than creating your own. This avoids hitting browser limits on AudioContext instances and ensures the context is in the correct state. +- **`track`** is the raw `MediaStreamTrack` from the user's microphone. On device switch, the SDK calls `restart()` with a new track. +- **Set `this.processedTrack`** to the output `MediaStreamTrack` from your processing pipeline. The SDK reads this property after `init()` returns. + +### Lifecycle methods + +| Method | When called | What to do | +|--------|-------------|------------| +| `init(opts)` | `audioTrack.setProcessor(processor)` | Build your Web Audio graph, set `this.processedTrack` | +| `restart(opts)` | Device switch or track change | Tear down old graph, rebuild with the new `opts.track` | +| `destroy()` | `audioTrack.stopProcessor()` | Disconnect all nodes, clean up resources | +| `onPublish(room)` | Track is published to a room | Optional — use if you need room context | +| `onUnpublish()` | Track is unpublished | Optional — use for cleanup tied to room lifecycle | + +## Building your own audio processor + +The general pattern for a custom audio processor is: + +1. Create a `MediaStreamSource` from the input track +2. Connect it through your processing nodes +3. Connect the final node to a `MediaStreamDestination` +4. Expose `destination.stream.getAudioTracks()[0]` as `processedTrack` + +Here's a skeleton: + +```ts +import { Track } from 'livekit-client'; +import type { AudioProcessorOptions, TrackProcessor } from 'livekit-client'; + +class MyAudioProcessor implements TrackProcessor { + name = 'my-audio-processor'; + processedTrack?: MediaStreamTrack; + + private source?: MediaStreamAudioSourceNode; + private destination?: MediaStreamAudioDestinationNode; + // ... your processing nodes + + async init(opts: AudioProcessorOptions): Promise { + const { track, audioContext } = opts; + + // Create source from the raw microphone track + this.source = audioContext.createMediaStreamSource(new MediaStream([track])); + + // Create your processing chain + // const myNode = audioContext.create...(...); + + // Create destination + this.destination = audioContext.createMediaStreamDestination(); + + // Wire it up: source → [your nodes] → destination + this.source.connect(/* myNode */); + // myNode.connect(this.destination); + + // Expose the processed track + this.processedTrack = this.destination.stream.getAudioTracks()[0]; + } + + async restart(opts: AudioProcessorOptions): Promise { + await this.destroy(); + await this.init(opts); + } + + async destroy(): Promise { + this.source?.disconnect(); + // Disconnect your other nodes... + this.destination?.disconnect(); + this.processedTrack = undefined; + } +} +``` + +### Things to keep in mind + +**Device switching.** When a user switches microphones, the SDK calls `restart()` with a new `MediaStreamTrack`. Your processor must tear down the old Web Audio graph and rebuild with the new track. The simplest approach (shown above) is to call `destroy()` then `init()` inside `restart()`. + +**AudioContext lifecycle.** The SDK provides an `AudioContext` via the options. Always use it rather than creating your own — this avoids browser limits on AudioContext instances and ensures the context state is managed correctly. + +**Browser compatibility.** The Web Audio API nodes used in this pattern (`MediaStreamSource`, `GainNode`, `MediaStreamDestination`) are well-supported across modern browsers. No special fallbacks are needed, unlike the video processor path which requires `canvas.captureStream()` fallbacks. + +**Advanced processing.** Since you receive a full `AudioContext`, you can wire in any Web Audio processing chain — including `AudioWorkletNode` for off-main-thread processing, or WASM-backed worklets for computationally intensive tasks. The pattern is the same: route audio through your nodes and connect the final output to the `MediaStreamDestination`. diff --git a/processor-docs/video-processors.md b/processor-docs/video-processors.md new file mode 100644 index 0000000..8f715b5 --- /dev/null +++ b/processor-docs/video-processors.md @@ -0,0 +1,93 @@ +# Video Processors + +This document covers the video track processors available in `@livekit/track-processors`. + +## BackgroundProcessor + +The `BackgroundProcessor` is a prebuilt video processor that supports blurring the background of a user's local video or replacing it with a virtual background image. It can be switched between modes on the fly. + +### Available modes + +- `BackgroundProcessor({ mode: 'background-blur', blurRadius: 10 })` — Blur the background with an optional blur radius (defaults to 10) +- `BackgroundProcessor({ mode: 'virtual-background', imagePath: "http://path.to/image.png" })` — Replace the background with an image +- `BackgroundProcessor({ mode: 'disabled' })` — Passthrough mode, no effect applied (useful for avoiding switching artifacts, see below) + +### Browser support + +Before using `BackgroundProcessor`, check for browser compatibility: + +```ts +import { + BackgroundProcessor, + supportsBackgroundProcessors, + supportsModernBackgroundProcessors, +} from '@livekit/track-processors'; + +if (!supportsBackgroundProcessors()) { + throw new Error('This browser does not support background processors'); +} + +if (supportsModernBackgroundProcessors()) { + console.log('This browser supports modern APIs that are more performant'); +} +``` + +### Basic usage + +```ts +import { BackgroundProcessor } from '@livekit/track-processors'; + +const videoTrack = await createLocalVideoTrack(); +const processor = BackgroundProcessor({ mode: 'background-blur' }); +await videoTrack.setProcessor(processor); +room.localParticipant.publishTrack(videoTrack); + +async function disableBackgroundBlur() { + await videoTrack.stopProcessor(); +} + +async function updateBlurRadius(radius) { + return processor.switchTo({ mode: 'background-blur', blurRadius: radius }); +} +``` + +### Avoiding visual artifacts when switching + +In a real application, you'll likely want to toggle background effects on and off. You could call `videoTrack.setProcessor()` / `videoTrack.stopProcessor()` on demand, but these functions can sometimes produce visual artifacts during the switching process, resulting in a poor user experience. + +A better approach is to initialize the `BackgroundProcessor` in `disabled` mode and then switch to the desired mode later. This avoids artifacts entirely: + +```ts +const videoTrack = await createLocalVideoTrack(); +const processor = BackgroundProcessor({ mode: 'disabled' }); +await videoTrack.setProcessor(processor); +room.localParticipant.publishTrack(videoTrack); + +async function enableBlur(radius) { + await processor.switchTo({ mode: 'background-blur', blurRadius: radius }); +} + +async function disableBlur() { + await processor.switchTo({ mode: 'disabled' }); +} +``` + +## Developing your own video processor + +Video processors in this package are built on two layers: + +1. **`ProcessorWrapper`** — Handles the plumbing of intercepting a video track's frames, passing them through a transformer, and producing a processed output track. It manages browser compatibility (using `MediaStreamTrackProcessor`/`MediaStreamTrackGenerator` where available, with a `canvas.captureStream()` fallback). + +2. **A Transformer** (e.g., `BackgroundTransformer`) — Implements the actual frame-by-frame processing logic. + +To create a custom video processor, you can instantiate a `ProcessorWrapper` with your own transformer: + +```ts +import { ProcessorWrapper } from '@livekit/track-processors'; + +const pipeline = new ProcessorWrapper(new MyCustomTransformer(options)); +``` + +### Available base transformers + +- **BackgroundTransformer** — Can blur the background, replace it with a virtual background image, or operate in a disabled passthrough state. From a525ed0739fdeee420ae8248d6727c8718bd905d Mon Sep 17 00:00:00 2001 From: Chris Wilson Date: Wed, 4 Mar 2026 10:38:15 -0600 Subject: [PATCH 2/5] Adds a sample audio processor example --- example/index.html | 31 +++++++++ example/sample.ts | 54 ++++++++++++++- src/audio/GainAudioProcessor.ts | 114 ++++++++++++++++++++++++++++++++ src/index.ts | 1 + 4 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 src/audio/GainAudioProcessor.ts diff --git a/example/index.html b/example/index.html index c9278de..5e39c46 100644 --- a/example/index.html +++ b/example/index.html @@ -159,6 +159,37 @@

LiveKit track processor sample

+ +
+
+ +
+ + +
diff --git a/example/sample.ts b/example/sample.ts index 8b61483..fc15eff 100644 --- a/example/sample.ts +++ b/example/sample.ts @@ -24,7 +24,7 @@ import { facingModeFromLocalTrack, setLogLevel, } from 'livekit-client'; -import { BackgroundProcessor, BackgroundProcessorOptions } from '../src'; +import { BackgroundProcessor, BackgroundProcessorOptions, GainAudioProcessor } from '../src'; const $ = (id: string) => document.getElementById(id) as T; @@ -36,6 +36,8 @@ const state = { bitrateInterval: undefined as any, isBackgroundProcessorEnabled: false, backgroundProcessor: BackgroundProcessor({ mode: 'background-blur', blurRadius: BLUR_RADIUS }), + isAudioProcessorEnabled: false, + gainProcessor: new GainAudioProcessor({ gainValue: 1.0 }), }; let currentRoom: Room | undefined; @@ -339,6 +341,42 @@ const appActions = { } }, + toggleAudioProcessorEnabled: async () => { + if (!currentRoom) return; + + setButtonDisabled('toggle-audio-processor', true); + + try { + const micPub = currentRoom.localParticipant.getTrackPublication(Track.Source.Microphone); + if (!micPub || !micPub.track) { + appendLog('ERROR: No microphone track found. Enable audio first.'); + return; + } + const micTrack = micPub.track as LocalAudioTrack; + + if (state.isAudioProcessorEnabled) { + await micTrack.stopProcessor(); + state.isAudioProcessorEnabled = false; + } else { + await micTrack.setProcessor(state.gainProcessor); + state.isAudioProcessorEnabled = true; + } + } catch (e: any) { + appendLog(`ERROR: ${e.message}`); + } finally { + updateAudioProcessorButtons(); + setButtonDisabled('toggle-audio-processor', false); + } + }, + + updateGain: (value: number) => { + state.gainProcessor.setGain(value); + const display = $('gain-value'); + if (display) { + display.textContent = value.toFixed(1); + } + }, + startAudio: () => { currentRoom?.startAudio(); }, @@ -409,7 +447,9 @@ function handleRoomDisconnect(reason?: DisconnectReason) { appendLog('disconnected from room', { reason }); setButtonsForState(false); state.isBackgroundProcessorEnabled = false; + state.isAudioProcessorEnabled = false; updateTrackProcessorModeButtons(); + updateAudioProcessorButtons(); renderParticipant(currentRoom.localParticipant, true); currentRoom.remoteParticipants.forEach((p) => { renderParticipant(p, true); @@ -662,6 +702,7 @@ function setButtonsForState(connected: boolean) { 'switch-to-background-blur-button', 'switch-to-virtual-background-button', 'switch-to-disabled-button', + 'toggle-audio-processor', ]; const disconnectedSet = ['connect-button']; @@ -766,6 +807,17 @@ function updateTrackProcessorModeButtons() { } } +function updateAudioProcessorButtons() { + const toggleButtonEnabled = currentRoom?.state === ConnectionState.Connected; + if (state.isAudioProcessorEnabled) { + setButtonState('toggle-audio-processor', 'Remove Audio Processor', false, !toggleButtonEnabled); + $('audio-processor-controls').style.display = 'block'; + } else { + setButtonState('toggle-audio-processor', 'Insert Audio Processor', false, !toggleButtonEnabled); + $('audio-processor-controls').style.display = 'none'; + } +} + async function acquireDeviceList() { handleDevicesChanged(); } diff --git a/src/audio/GainAudioProcessor.ts b/src/audio/GainAudioProcessor.ts new file mode 100644 index 0000000..662b71f --- /dev/null +++ b/src/audio/GainAudioProcessor.ts @@ -0,0 +1,114 @@ +import type { Track, Room } from 'livekit-client'; +import type { AudioProcessorOptions, TrackProcessor } from 'livekit-client'; + +export interface GainAudioProcessorOptions { + /** + * Initial gain value. Defaults to 1.0 (unity gain). + * - 0.0 = silence + * - 1.0 = no change + * - > 1.0 = amplify + */ + gainValue?: number; +} + +/** + * A minimal audio track processor that applies a Web Audio GainNode to the audio pipeline. + * + * Serves as both a ready-to-use gain control and a reference implementation for building + * custom audio processors using the TrackProcessor interface. + * + * @example + * ```ts + * const processor = new GainAudioProcessor({ gainValue: 1.5 }); + * await audioTrack.setProcessor(processor); + * + * // Update gain on the fly + * processor.setGain(0.5); + * + * // Remove the processor + * await audioTrack.stopProcessor(); + * ``` + */ +export class GainAudioProcessor + implements TrackProcessor +{ + name = 'gain-audio-processor'; + + processedTrack?: MediaStreamTrack; + + private gainValue: number; + + private sourceNode?: MediaStreamAudioSourceNode; + + private gainNode?: GainNode; + + private destinationNode?: MediaStreamAudioDestinationNode; + + constructor(options: GainAudioProcessorOptions = {}) { + this.gainValue = options.gainValue ?? 1.0; + } + + async init(opts: AudioProcessorOptions): Promise { + const { track, audioContext } = opts; + + // Create source from the raw microphone track + this.sourceNode = audioContext.createMediaStreamSource(new MediaStream([track])); + + // Create gain node + this.gainNode = audioContext.createGain(); + this.gainNode.gain.value = this.gainValue; + + // Create destination + this.destinationNode = audioContext.createMediaStreamDestination(); + + // Wire up: source → gain → destination + this.sourceNode.connect(this.gainNode); + this.gainNode.connect(this.destinationNode); + + // Expose the processed track for the SDK + this.processedTrack = this.destinationNode.stream.getAudioTracks()[0]; + } + + async restart(opts: AudioProcessorOptions): Promise { + // Tear down old graph and rebuild with the new track + await this.destroy(); + await this.init(opts); + } + + async destroy(): Promise { + this.sourceNode?.disconnect(); + this.gainNode?.disconnect(); + this.destinationNode?.disconnect(); + this.sourceNode = undefined; + this.gainNode = undefined; + this.destinationNode = undefined; + this.processedTrack = undefined; + } + + /** + * Update the gain value. Can be called while the processor is active. + * @param value - Gain multiplier (0.0 = silence, 1.0 = unity, > 1.0 = amplify) + */ + setGain(value: number): void { + this.gainValue = value; + if (this.gainNode) { + this.gainNode.gain.value = value; + } + } + + /** + * Get the current gain value. + */ + getGain(): number { + return this.gainValue; + } + + // Optional lifecycle hooks — included for completeness as a reference implementation + async onPublish?(room: Room): Promise { + // No-op: override in subclasses if you need room context + } + + async onUnpublish?(): Promise { + // No-op: override in subclasses for room lifecycle cleanup + } +} diff --git a/src/index.ts b/src/index.ts index 6cdc867..4951dea 100644 --- a/src/index.ts +++ b/src/index.ts @@ -7,6 +7,7 @@ import BackgroundTransformer, { export * from './transformers/types'; export { default as VideoTransformer } from './transformers/VideoTransformer'; +export { GainAudioProcessor, type GainAudioProcessorOptions } from './audio/GainAudioProcessor'; export { ProcessorWrapper, type BackgroundOptions, From f5b48d6ddd6e3f65e826391cd3a75217702d0cda Mon Sep 17 00:00:00 2001 From: Chris Wilson Date: Wed, 4 Mar 2026 12:00:02 -0600 Subject: [PATCH 3/5] Adds some changes based on PR review --- processor-docs/audio-processors.md | 15 ++++++++++++++- src/audio/GainAudioProcessor.ts | 26 +++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/processor-docs/audio-processors.md b/processor-docs/audio-processors.md index 485f33a..411bf92 100644 --- a/processor-docs/audio-processors.md +++ b/processor-docs/audio-processors.md @@ -6,9 +6,22 @@ This document covers the audio track processors available in `@livekit/track-pro The `GainAudioProcessor` is a minimal audio processor that applies a Web Audio [`GainNode`](https://developer.mozilla.org/en-US/docs/Web/API/GainNode) to a local audio track. It serves both as a ready-to-use volume control and as a reference implementation for building custom audio processors. +### Browser support + +The Web Audio API used by `GainAudioProcessor` is [widely supported](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API#browser_compatibility) in modern browsers. You can check support before use: + +```ts +import { GainAudioProcessor } from '@livekit/track-processors'; + +if (!GainAudioProcessor.isSupported) { + console.warn('GainAudioProcessor is not supported in this environment'); +} +``` + ### Basic usage ```ts +import { createLocalAudioTrack } from 'livekit-client'; import { GainAudioProcessor } from '@livekit/track-processors'; const audioTrack = await createLocalAudioTrack(); @@ -27,7 +40,7 @@ await audioTrack.stopProcessor(); | Option | Type | Default | Description | |--------|------|---------|-------------| -| `gainValue` | `number` | `1.0` | Initial gain multiplier. `1.0` = unity (no change), `0.0` = silence, `> 1.0` = amplify. | +| `gainValue` | `number` | `1.0` | Initial gain multiplier, clamped to [0, 10]. `1.0` = unity (no change), `0.0` = silence, `> 1.0` = amplify. | ## The TrackProcessor interface for audio diff --git a/src/audio/GainAudioProcessor.ts b/src/audio/GainAudioProcessor.ts index 662b71f..71c0546 100644 --- a/src/audio/GainAudioProcessor.ts +++ b/src/audio/GainAudioProcessor.ts @@ -1,9 +1,14 @@ import type { Track, Room } from 'livekit-client'; import type { AudioProcessorOptions, TrackProcessor } from 'livekit-client'; +/** Gain is clamped to this range to avoid accidental silence or extreme amplification. */ +const MIN_GAIN = 0; +const MAX_GAIN = 10; + export interface GainAudioProcessorOptions { /** * Initial gain value. Defaults to 1.0 (unity gain). + * Clamped to [0, 10]. Values outside this range are clamped. * - 0.0 = silence * - 1.0 = no change * - > 1.0 = amplify @@ -38,6 +43,19 @@ export class GainAudioProcessor private gainValue: number; + /** + * Whether the current environment supports GainAudioProcessor (Web Audio API). + * Use this for consistency with video processors before attaching the processor. + */ + static get isSupported(): boolean { + return ( + typeof AudioContext !== 'undefined' && + typeof GainNode !== 'undefined' && + typeof MediaStreamAudioSourceNode !== 'undefined' && + typeof MediaStreamAudioDestinationNode !== 'undefined' + ); + } + private sourceNode?: MediaStreamAudioSourceNode; private gainNode?: GainNode; @@ -45,7 +63,8 @@ export class GainAudioProcessor private destinationNode?: MediaStreamAudioDestinationNode; constructor(options: GainAudioProcessorOptions = {}) { - this.gainValue = options.gainValue ?? 1.0; + const raw = options.gainValue ?? 1.0; + this.gainValue = Math.max(MIN_GAIN, Math.min(MAX_GAIN, raw)); } async init(opts: AudioProcessorOptions): Promise { @@ -87,12 +106,13 @@ export class GainAudioProcessor /** * Update the gain value. Can be called while the processor is active. + * Value is clamped to [0, 10]. * @param value - Gain multiplier (0.0 = silence, 1.0 = unity, > 1.0 = amplify) */ setGain(value: number): void { - this.gainValue = value; + this.gainValue = Math.max(MIN_GAIN, Math.min(MAX_GAIN, value)); if (this.gainNode) { - this.gainNode.gain.value = value; + this.gainNode.gain.value = this.gainValue; } } From 8f4b7d4613ee843f3f781facc0ed7319d1dcd6fc Mon Sep 17 00:00:00 2001 From: Chris Wilson Date: Wed, 4 Mar 2026 13:28:20 -0600 Subject: [PATCH 4/5] Fixes items suggested by copilot. --- example/sample.ts | 4 +++- processor-docs/audio-processors.md | 9 ++++++--- src/audio/GainAudioProcessor.ts | 8 ++++++-- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/example/sample.ts b/example/sample.ts index fc15eff..6f3a092 100644 --- a/example/sample.ts +++ b/example/sample.ts @@ -365,7 +365,9 @@ const appActions = { appendLog(`ERROR: ${e.message}`); } finally { updateAudioProcessorButtons(); - setButtonDisabled('toggle-audio-processor', false); + if (currentRoom?.state === ConnectionState.Connected) { + setButtonDisabled('toggle-audio-processor', false); + } } }, diff --git a/processor-docs/audio-processors.md b/processor-docs/audio-processors.md index 411bf92..68c0079 100644 --- a/processor-docs/audio-processors.md +++ b/processor-docs/audio-processors.md @@ -47,15 +47,18 @@ await audioTrack.stopProcessor(); Audio processors implement the `TrackProcessor` interface from `livekit-client`: ```ts -interface TrackProcessor { +// Generic signature from livekit-client +interface TrackProcessor> { name: string; - init(opts: AudioProcessorOptions): Promise; - restart(opts: AudioProcessorOptions): Promise; + init(opts: U): Promise; + restart(opts: U): Promise; destroy(): Promise; processedTrack?: MediaStreamTrack; onPublish?(room: Room): Promise; onUnpublish?(): Promise; } + +// For audio processors, T = Track.Kind.Audio and U = AudioProcessorOptions ``` When you call `audioTrack.setProcessor(processor)`, the SDK: diff --git a/src/audio/GainAudioProcessor.ts b/src/audio/GainAudioProcessor.ts index 71c0546..76371e9 100644 --- a/src/audio/GainAudioProcessor.ts +++ b/src/audio/GainAudioProcessor.ts @@ -98,6 +98,7 @@ export class GainAudioProcessor this.sourceNode?.disconnect(); this.gainNode?.disconnect(); this.destinationNode?.disconnect(); + this.processedTrack?.stop(); this.sourceNode = undefined; this.gainNode = undefined; this.destinationNode = undefined; @@ -110,6 +111,9 @@ export class GainAudioProcessor * @param value - Gain multiplier (0.0 = silence, 1.0 = unity, > 1.0 = amplify) */ setGain(value: number): void { + if (!Number.isFinite(value)) { + return; + } this.gainValue = Math.max(MIN_GAIN, Math.min(MAX_GAIN, value)); if (this.gainNode) { this.gainNode.gain.value = this.gainValue; @@ -124,11 +128,11 @@ export class GainAudioProcessor } // Optional lifecycle hooks — included for completeness as a reference implementation - async onPublish?(room: Room): Promise { + async onPublish(room: Room): Promise { // No-op: override in subclasses if you need room context } - async onUnpublish?(): Promise { + async onUnpublish(): Promise { // No-op: override in subclasses for room lifecycle cleanup } } From e4df303e232f98861ef58d447373c399ba3cde91 Mon Sep 17 00:00:00 2001 From: Chris Wilson Date: Thu, 5 Mar 2026 07:49:02 -0600 Subject: [PATCH 5/5] Adds changes based on feedback from 1egoman PR review. --- .changeset/add-audio-processor.md | 5 +++++ README.md | 6 +++--- processor-docs/audio-processors.md | 11 ++++++++++ processor-docs/video-processors.md | 33 +++++++++++++++++------------- src/audio/GainAudioProcessor.ts | 4 ++-- 5 files changed, 40 insertions(+), 19 deletions(-) create mode 100644 .changeset/add-audio-processor.md diff --git a/.changeset/add-audio-processor.md b/.changeset/add-audio-processor.md new file mode 100644 index 0000000..123db25 --- /dev/null +++ b/.changeset/add-audio-processor.md @@ -0,0 +1,5 @@ +--- +"@livekit/track-processors": minor +--- + +Add GainAudioProcessor — a reference audio TrackProcessor implementation using the Web Audio API. Includes gain control, browser support detection via `isSupported`, and a complete example in the sample app. Also restructures documentation into separate video and audio processor guides. diff --git a/README.md b/README.md index 8a8139f..a54c230 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # LiveKit track processors -Prebuilt audio and video track processors for [LiveKit](https://livekit.io), implementing the [`TrackProcessor`](https://docs.livekit.io/home/client/tracks/manipulate/#track-processors) interface from `livekit-client`. +Prebuilt audio and video track processors for [LiveKit](https://livekit.io), implementing the [`TrackProcessor`](https://docs.livekit.io/reference/client-sdk-js/interfaces/TrackProcessor.html) interface from `livekit-client`. ## Install @@ -10,7 +10,7 @@ npm add @livekit/track-processors ## Video processors -Background blur and virtual background for video tracks: +Video track processors intercept a local video track's frames and transform them before they are sent to other participants. This package provides a prebuilt `BackgroundProcessor` that supports background blur and virtual backgrounds: ```ts import { BackgroundProcessor } from '@livekit/track-processors'; @@ -25,7 +25,7 @@ See [processor-docs/video-processors.md](processor-docs/video-processors.md) for ## Audio processors -Gain control for audio tracks, and a reference implementation for building custom audio processors: +Audio track processors work similarly — they intercept the local audio track and pipe it through a Web Audio API processing graph before publishing. The included `GainAudioProcessor` provides gain control and serves as a reference implementation for building custom audio processors: ```ts import { GainAudioProcessor } from '@livekit/track-processors'; diff --git a/processor-docs/audio-processors.md b/processor-docs/audio-processors.md index 68c0079..acaf121 100644 --- a/processor-docs/audio-processors.md +++ b/processor-docs/audio-processors.md @@ -99,6 +99,17 @@ Key points: ## Building your own audio processor +### Architecture overview + +```mermaid +flowchart LR + A[Microphone\nMediaStreamTrack] --> B[MediaStreamSource] + B --> C[Your Processing Nodes
e.g. GainNode, BiquadFilter,
AudioWorklet] + C --> D[MediaStreamDestination] + D --> E[processedTrack
MediaStreamTrack] + E --> F[Published to SFU] +``` + The general pattern for a custom audio processor is: 1. Create a `MediaStreamSource` from the input track diff --git a/processor-docs/video-processors.md b/processor-docs/video-processors.md index 8f715b5..2b33a0e 100644 --- a/processor-docs/video-processors.md +++ b/processor-docs/video-processors.md @@ -32,7 +32,9 @@ if (supportsModernBackgroundProcessors()) { } ``` -### Basic usage +### Usage + +The simplest approach is to create a processor and attach it to a local video track: ```ts import { BackgroundProcessor } from '@livekit/track-processors'; @@ -41,21 +43,11 @@ const videoTrack = await createLocalVideoTrack(); const processor = BackgroundProcessor({ mode: 'background-blur' }); await videoTrack.setProcessor(processor); room.localParticipant.publishTrack(videoTrack); - -async function disableBackgroundBlur() { - await videoTrack.stopProcessor(); -} - -async function updateBlurRadius(radius) { - return processor.switchTo({ mode: 'background-blur', blurRadius: radius }); -} ``` -### Avoiding visual artifacts when switching +### Avoiding visual artifacts when toggling -In a real application, you'll likely want to toggle background effects on and off. You could call `videoTrack.setProcessor()` / `videoTrack.stopProcessor()` on demand, but these functions can sometimes produce visual artifacts during the switching process, resulting in a poor user experience. - -A better approach is to initialize the `BackgroundProcessor` in `disabled` mode and then switch to the desired mode later. This avoids artifacts entirely: +Calling `videoTrack.setProcessor()` / `videoTrack.stopProcessor()` on demand can produce visual artifacts during the switch. A better approach is to initialize the processor in `disabled` mode up front and use `switchTo()` to toggle effects. This avoids artifacts entirely: ```ts const videoTrack = await createLocalVideoTrack(); @@ -74,13 +66,26 @@ async function disableBlur() { ## Developing your own video processor +### Architecture overview + +```mermaid +flowchart LR + A[Camera\nMediaStreamTrack] --> B[ProcessorWrapper] + B -->|VideoFrame| C[Transformer
e.g. BackgroundTransformer] + C -->|Transformed
VideoFrame| B + B --> D[Processed
MediaStreamTrack] + D --> E[Published to SFU] +``` + Video processors in this package are built on two layers: 1. **`ProcessorWrapper`** — Handles the plumbing of intercepting a video track's frames, passing them through a transformer, and producing a processed output track. It manages browser compatibility (using `MediaStreamTrackProcessor`/`MediaStreamTrackGenerator` where available, with a `canvas.captureStream()` fallback). 2. **A Transformer** (e.g., `BackgroundTransformer`) — Implements the actual frame-by-frame processing logic. -To create a custom video processor, you can instantiate a `ProcessorWrapper` with your own transformer: +> **Note:** You don't have to follow this `Transformer` + `ProcessorWrapper` pattern. You can implement the `TrackProcessor` interface directly if you prefer. However, using `ProcessorWrapper` is convenient because it abstracts away the `MediaStreamTrack` → `VideoFrame` → transformer → `VideoFrame` → `MediaStreamTrack` conversion, which most use cases don't need to worry about. + +To create a custom video processor using `ProcessorWrapper`, instantiate it with your own transformer: ```ts import { ProcessorWrapper } from '@livekit/track-processors'; diff --git a/src/audio/GainAudioProcessor.ts b/src/audio/GainAudioProcessor.ts index 76371e9..6f8a064 100644 --- a/src/audio/GainAudioProcessor.ts +++ b/src/audio/GainAudioProcessor.ts @@ -129,10 +129,10 @@ export class GainAudioProcessor // Optional lifecycle hooks — included for completeness as a reference implementation async onPublish(room: Room): Promise { - // No-op: override in subclasses if you need room context + console.debug(`[${this.name}] onPublish — room: ${room.name}`); } async onUnpublish(): Promise { - // No-op: override in subclasses for room lifecycle cleanup + console.debug(`[${this.name}] onUnpublish`); } }