fix(tts): unblock FallbackAdapter when primary provider fails silently#1218
Merged
toubatbrian merged 4 commits intolivekit:mainfrom Apr 9, 2026
Merged
Conversation
🦋 Changeset detectedLatest commit: d31ccb9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 22 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
mrniket
force-pushed
the
emdash/fix-tts-fallback-450
branch
from
b677c63 to
79c864c
Compare
…pushText The base SynthesizeStream only closed `this.output` via `monitorMetrics`, which was only started on the first `pushText`. If a plugin's `run()` threw before any text was pushed (e.g. invalid API key on a streaming provider), `output` stayed open forever and FallbackSynthesizeStream's processOutput hung on `stream.output.next()`. The fallback loop never reached its catch block, never called `markUnAvailable`, and never moved to the next provider. Also forward connOptions through the ElevenLabs plugin's stream() so that FallbackAdapter.maxRetryPerTTS is actually honoured (previously silently dropped, falling back to the default maxRetry of 3).
When the primary TTS in FallbackAdapter has a different sample rate from the adapter's aggregated output rate, FallbackSynthesizeStream.run() wraps each inner frame in a resampler before forwarding it. Previously, if the inner stream yielded no real audio at all (e.g. invalid API key on a streaming provider), the code would still call `resampler.flush()` on a resampler that had nothing pushed into it. On @livekit/rtc-node@0.13.25 an unused resampler's `flush()` returns a ghost frame, which flipped `this.audioPushed = true` and made the fallback adapter treat a completely silent primary as a success — so it never marked the provider unavailable and never tried the secondary. Track `sawRawAudio` separately from `audioPushed` and: - only call `resampler.flush()` if real audio was actually iterated - gate the "no audio received" check on `sawRawAudio` instead of `audioPushed`, so phantom resampler output can never mask a silent failure Add a regression test with mismatched sample rates to exercise the resampler branch.
mrniket
force-pushed
the
emdash/fix-tts-fallback-450
branch
from
79c864c to
8829174
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
FallbackChunkedStream.run() had the identical phantom AudioResampler.flush() vulnerability that was fixed in FallbackSynthesizeStream. When the primary TTS has a different sample rate from the adapter's output rate and emits no real audio (e.g. invalid API key on a streaming provider), resampler.flush() on the unused resampler can return a ghost frame on @livekit/rtc-node@0.13.25, flipping 'audioReceived' to true and making adapter.synthesize() incorrectly treat a silent failure as a success — so the non-streaming path never falls back to the secondary provider. Apply the same sawRawAudio tracking pattern and drop the now-redundant audioReceived bookkeeping. Add a regression test covering the synthesize() path with mismatched sample rates.
toubatbrian
approved these changes
Apr 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
tts.FallbackAdaptersilently hangs when the primary provider fails without emitting any audio (e.g. ElevenLabs closing its WebSocket with1008 Invalid API key).markUnAvailable(0)is never called, the secondary provider is never tried, and the call stays silent until the caller disconnects. This PR fixes three bugs that combined to produce that symptom, plus adds regression tests.Changes Made
agents/src/tts/tts.ts— BaseSynthesizeStreamnow closesthis.outputwhenmainTasksettles, not justthis.queue. Previouslythis.outputwas only closed inside#monitorMetricsTask.finally(...), which is only attached on the firstpushText()call.FallbackSynthesizeStreamdrives the inner stream by callingpushTextfrom its own scheduler, so if the inner plugin'srun()threw before anypushTexthad been scheduled,this.outputstayed open forever and the consumer'sPromise.allSettlednever resolved. The fire-and-forget task also swallows the rejection explicitly (the error is already emitted viaemitError) to avoid unhandled rejections.agents/src/tts/fallback_adapter.ts—FallbackSynthesizeStream.run()now trackssawRawAudioseparately fromaudioPushed. When the primary's sample rate differs from the adapter's output rate a resampler is created, and on@livekit/rtc-node@0.13.25an unusedAudioResampler.flush()can return a phantom frame — enough to flipaudioPushedtotrueand make the adapter treat a completely silent failure as a success.resampler.flush()is now only called when real audio actually went in, and the silent-failure check is gated onsawRawAudio.plugins/elevenlabs/src/tts.ts—TTS.stream()now accepts{ connOptions }and threads it throughSynthesizeStream's constructor to the base class. Previously the plugin silently dropped the argument, soFallbackAdapter.maxRetryPerTTSwas ignored and fallback always took ~6 s (3 inner retries × 2 s backoff) regardless of configuration.agents/src/tts/fallback_adapter.test.ts(new) — Two regression tests. The first uses matched sample rates and exercises the deadlock path; without thetts.tsfix it hangs and a 3 s timeout fires. The second uses mismatched sample rates (22 050 → 24 000) so a resampler is created, exercising the phantom-flush path; without thefallback_adapter.tsfix the adapter never falls back to the secondary.Pre-Review Checklist
pnpm build,pnpm vitest run agents/src(596/596 pass),pnpm -F @livekit/agents lint,pnpm format:checkall clean locally.FallbackAdaptersilently hanging on primary failure) and were discovered sequentially while investigating.Testing
agents/src/tts/fallback_adapter.test.ts).pnpm vitest run agents/src→ 38 test files, 596 passed, 2 skipped.