feat: add Telnyx STT and TTS plugins #4665
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
Contributor
📝 WalkthroughWalkthroughAdds a Telnyx plugin and example voice agent: streaming STT and TTS backends, plugin registration, session management utilities, project packaging, and an examples/voice_agents/telnyx_voice_agent.py demonstrating RTC session wiring, VAD prewarm, function tools, metrics collection, and a CLI entrypoint. Changes
Sequence Diagram(s)sequenceDiagram
participant Client as AgentServer/Agent
participant STT as Telnyx STT
participant Stream as SpeechStream
participant WS as Telnyx WebSocket
Client->>STT: recognize(audio_stream)
STT->>Stream: create stream()
Stream->>WS: connect (wss + auth)
loop sending audio
Client->>Stream: push(audio_frame)
Stream->>WS: send WAV header / chunk
end
Client->>Stream: end_input
loop receive events
WS-->>Stream: JSON event (interim/final)
Stream->>Client: emit INTERIM/FINAL transcript
end
WS-->>Stream: stream_finished
Stream->>STT: close()
sequenceDiagram
participant Client as AgentServer/Agent
participant TTS as Telnyx TTS
participant Stream as SynthesizeStream
participant WS as Telnyx WebSocket
participant Decoder as MP3 Decoder
Client->>TTS: stream()
TTS->>Stream: new SynthesizeStream()
loop send text segments
Client->>Stream: push_text(segment)
end
Client->>Stream: flush()
Stream->>WS: connect (wss + auth) and send text
loop receive audio chunks
WS-->>Stream: JSON with base64 MP3
Stream->>Decoder: feed MP3 bytes
Decoder-->>Stream: PCM frames
Stream->>Client: emit PCM frame
end
WS-->>Stream: stream_finished
Stream->>TTS: close()
Estimated code review effort🎯 4 (Complex) | ⏱️ ~50 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Fix all issues with AI agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py`:
- Around line 1-5: Add the standard plugin registration after the existing
__all__ by importing Plugin from livekit.agents and the module logger, define a
TelnyxPlugin class that calls super().__init__(__name__, __version__,
__package__, logger), register it with Plugin.register_plugin(TelnyxPlugin()),
and then build the NOT_IN_ALL list and __pdoc__ cleanup so unexported names
(beyond STT, TTS, __version__ in __all__) are hidden from docs; use the symbols
TelnyxPlugin, Plugin.register_plugin, logger, NOT_IN_ALL and __pdoc__ to locate
where to insert this code.
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py`:
- Around line 1-303: Ruff reports formatting issues in this module; run ruff
format on livekit/plugins/telnyx/stt.py and commit the changes to satisfy CI —
the formatting fixes will target whitespace, import and function/class spacing
and line breaks around symbols such as STT, SpeechStream, and
_create_streaming_wav_header; run `ruff format livekit/plugins/telnyx/stt.py`
(or the project-wide ruff formatter), review the diff, and commit the formatted
file.
- Around line 89-92: The code in _recognize_impl mutates shared self._opts by
assigning config.language = language which causes race conditions across
concurrent streams; fix by making a local copy of the options (e.g., shallow
copy via copy.copy or construct a new options object) into a local variable
(keep the name config) and then set config.language on that copy (do not modify
self._opts), ensuring any necessary import (copy) and using the copy when
calling downstream functions.
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py`:
- Around line 1-27: This file (livekit.plugins.telnyx.tts.py) fails Ruff
formatting; run ruff format on the file or apply the formatting changes Ruff
would make so CI passes. Specifically reformat the module-level imports and
docstring and ensure spacing around imports (e.g., the block importing
APIConnectionError, APIConnectOptions, APIStatusError, APITimeoutError, tts,
utils) and the local imports (TTS_ENDPOINT, SessionManager, get_api_key,
NUM_CHANNELS, SAMPLE_RATE, logger) follow Ruff's style; run `ruff format
livekit/plugins/telnyx/tts.py` (or `ruff format --diff` to preview) and commit
the resulting changes.
In `@livekit-plugins/livekit-plugins-telnyx/pyproject.toml`:
- Around line 9-12: Add an explicit aiohttp dependency to the pyproject.toml
dependencies list so the plugin declares its direct use of aiohttp (matching the
transitive version used, e.g., "aiohttp~=3.10"); update the existing
dependencies array (which currently contains "livekit-agents>=0.8.0" and
"livekit-api>=0.6.0") to include the aiohttp spec to make the dependency
relationship explicit and consistent with other plugins.
🧹 Nitpick comments (7)
pyproject.toml (1)
58-58: Consider maintaining alphabetical ordering.The new
livekit-plugins-telnyxentry is appended at the end, but existing entries appear to follow alphabetical order. It should be placed betweentavusandturn-detectorfor consistency.livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.py (1)
15-19: Consider adding a Google-style docstring.Per coding guidelines, public functions should have Google-style docstrings documenting parameters and return values.
Example docstring
def get_api_key(api_key: str | None = None) -> str: + """Resolve the Telnyx API key from argument or environment. + + Args: + api_key: Explicit API key, or None to read from environment. + + Returns: + The resolved API key. + + Raises: + ValueError: If no API key is provided or found in environment. + """ resolved_key = api_key or os.environ.get("TELNYX_API_KEY")livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (2)
204-206: Arbitrary 1-second delay before closing WebSocket.The hard-coded
asyncio.sleep(1.0)delay is unexplained and may cause unnecessary latency. Consider documenting why this delay exists or using a more robust approach like waiting for a server acknowledgment.Suggestion
- await asyncio.sleep(1.0) + # Allow server time to process final audio before closing + # TODO: Consider waiting for explicit server acknowledgment instead + await asyncio.sleep(0.5) closing_ws = True await ws.close()
227-228: Bare exception catch swallows all errors.The bare
except Exceptioncatches and logs all exceptions but continues processing. This could hide important errors. Consider being more specific about expected exceptions.Proposed fix
try: data = json.loads(msg.data) logger.debug("Telnyx STT received: %s", data) self._process_stream_event(data) - except Exception: - logger.exception("Failed to process Telnyx STT message") + except json.JSONDecodeError: + logger.exception("Failed to parse Telnyx STT JSON message") + except KeyError as e: + logger.exception("Missing expected field in Telnyx STT message: %s", e)examples/voice_agents/telnyx_voice_agent.py (2)
54-55: Add return type annotation for mypy strict mode compliance.The
on_entermethod is missing a return type annotation.Suggested fix
- async def on_enter(self): + async def on_enter(self) -> None: self.session.generate_reply(allow_interruptions=False)
106-113: Consider adding type annotations to inner functions.For mypy strict mode compliance, the event handler and callback could benefit from type annotations.
Suggested improvements
`@session.on`("metrics_collected") - def _on_metrics_collected(ev): + def _on_metrics_collected(ev: metrics.MetricsEvent) -> None: metrics.log_metrics(ev.metrics) usage_collector.collect(ev.metrics) - async def log_usage(): + async def log_usage() -> None: summary = usage_collector.get_summary() logger.info(f"Usage: {summary}")livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (1)
142-143: Consider URL-encoding the voice parameter.If the
voiceparameter contains special characters (spaces, ampersands, etc.), the URL could be malformed. Consider usingurllib.parse.quote()for safety.Suggested fix
+from urllib.parse import quote + async def _run_ws(self, text: str, output_emitter: tts.AudioEmitter) -> None: segment_id = utils.shortuuid() output_emitter.start_segment(segment_id=segment_id) - url = f"{self._tts._opts.base_url}?voice={self._tts._opts.voice}" + url = f"{self._tts._opts.base_url}?voice={quote(self._tts._opts.voice)}" headers = {"Authorization": f"Bearer {self._tts._opts.api_key}"}
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
examples/voice_agents/telnyx_voice_agent.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/log.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/version.pylivekit-plugins/livekit-plugins-telnyx/pyproject.tomlpyproject.toml
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Format code with ruff
Run ruff linter and auto-fix issues
Run mypy type checker in strict mode
Maintain line length of 100 characters maximum
Ensure Python 3.9+ compatibility
Use Google-style docstrings
Files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/version.pyexamples/voice_agents/telnyx_voice_agent.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/log.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py
🧠 Learnings (3)
📚 Learning: 2026-01-30T12:53:12.738Z
Learnt from: milanperovic
Repo: livekit/agents PR: 4660
File: livekit-plugins/livekit-plugins-personaplex/livekit/plugins/personaplex/__init__.py:19-21
Timestamp: 2026-01-30T12:53:12.738Z
Learning: In plugin __init__.py files under the livekit-plugins or similar plugin directories, place internal imports (for example, from .log import logger) after the __all__ definition. These imports are used for plugin registration and are not part of the public API. This pattern is used across plugins (e.g., openai, deepgram, ultravox) and helps avoid E402 violations while keeping the public API surface clean.
Applied to files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/version.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/log.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py
📚 Learning: 2026-01-30T12:53:12.738Z
Learnt from: milanperovic
Repo: livekit/agents PR: 4660
File: livekit-plugins/livekit-plugins-personaplex/livekit/plugins/personaplex/__init__.py:19-21
Timestamp: 2026-01-30T12:53:12.738Z
Learning: In the livekit/agents repository, plugin __init__.py files follow a convention where `from livekit.agents import Plugin` and `from .log import logger` imports are placed after the `__all__` definition. These are internal imports for plugin registration and are not part of the public API. This pattern is used consistently across plugins like openai, deepgram, and ultravox, and does not trigger ruff E402 violations.
Applied to files:
pyproject.toml
📚 Learning: 2026-01-16T07:44:56.353Z
Learnt from: CR
Repo: livekit/agents PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-16T07:44:56.353Z
Learning: Implement Model Interface Pattern for STT, TTS, LLM, and Realtime models with provider-agnostic interfaces, fallback adapters for resilience, and stream adapters for different streaming patterns
Applied to files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py
🧬 Code graph analysis (5)
examples/voice_agents/telnyx_voice_agent.py (2)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (2)
STT(44-133)model(75-76)livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (2)
model(61-62)TTS(37-87)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (5)
livekit-agents/livekit/agents/_exceptions.py (2)
APIConnectionError(84-88)APIStatusError(45-81)livekit-agents/livekit/agents/utils/misc.py (1)
is_given(25-26)livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.py (3)
SessionManager(22-36)get_api_key(15-19)ensure_session(27-31)livekit-agents/livekit/agents/stt/stt.py (4)
STTCapabilities(78-84)SpeechEvent(70-74)SpeechEventType(32-49)SpeechData(53-61)livekit-agents/livekit/agents/utils/audio.py (1)
AudioByteStream(41-157)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py (2)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (1)
STT(44-133)livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (1)
TTS(37-87)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.py (1)
livekit-agents/livekit/agents/utils/http_context.py (1)
http_session(40-51)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (5)
livekit-agents/livekit/agents/_exceptions.py (3)
APIConnectionError(84-88)APIStatusError(45-81)APITimeoutError(91-95)livekit-agents/livekit/agents/types.py (1)
APIConnectOptions(54-88)livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.py (4)
SessionManager(22-36)get_api_key(15-19)close(33-36)ensure_session(27-31)livekit-agents/livekit/agents/utils/aio/channel.py (1)
Chan(49-178)livekit-agents/livekit/agents/utils/codecs/decoder.py (1)
AudioStreamDecoder(119-339)
🪛 GitHub Actions: CI
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py
[error] 1-1: Ruff format check failed. This file would be reformatted by 'ruff format'.
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py
[error] 1-1: Ruff format check failed. This file would be reformatted by 'ruff format'.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: unit-tests
🔇 Additional comments (11)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/version.py (1)
1-1: LGTM!Standard version module following existing plugin patterns.
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/log.py (1)
1-3: LGTM!Standard logger setup following the existing plugin conventions.
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (1)
193-199: Review comment is incorrect:write()method exists and is valid.The
AudioByteStreamclass inlivekit-agents/livekit/agents/utils/audio.pydefineswriteas an alias topush()on line 120:write = push. Both methods work identically and returnlist[rtc.AudioFrame]. The code in stt.py usingwrite()is correct and will not cause anAttributeError.Likely an incorrect or invalid review comment.
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.py (1)
27-31: The synchronous session creation pattern is appropriate for this use case.
aiohttp.ClientSession()uses lazy event loop binding, so creating it synchronously inensure_session()works correctly when called from async context (as it is in stt.py and tts.py). The current pattern is consistent with other plugins in this repository that also create sessions directly (e.g., lemonslice, openai realtime). No change is needed.Likely an incorrect or invalid review comment.
examples/voice_agents/telnyx_voice_agent.py (3)
1-40: LGTM! Clean module setup with proper documentation.The docstring provides clear usage instructions with required environment variables. Imports are well-organized.
117-125: LGTM!The session startup and CLI entry point follow established patterns from other voice agent examples.
97-97: No action needed. The model"gpt-4.1-mini"is a valid OpenAI model that launched on April 14, 2025, and is available in the API as of January 2026. The code is correct.Likely an incorrect or invalid review comment.
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (4)
37-88: LGTM! Well-structured TTS class.The implementation properly follows the Model Interface Pattern with clear lifecycle management. The use of
WeakSetfor stream tracking and proper cleanup inaclose()is good practice. Based on learnings, this aligns with the STT, TTS, LLM, and Realtime model patterns with provider-agnostic interfaces.
90-137: LGTM! Solid streaming orchestration.The
_runmethod properly orchestrates segment collection and WebSocket processing with appropriate error mapping to framework exceptions (APITimeoutError,APIStatusError,APIConnectionError). Thefinallyblock ensures tasks are gracefully cancelled.
151-156: LGTM! Proper Telnyx TTS protocol implementation.The send pattern (placeholder text → actual content → empty string to signal end) follows the Telnyx streaming TTS API protocol correctly.
182-210: LGTM! Proper async coordination and cleanup.The three concurrent tasks (send, recv, decode) are well-coordinated with proper cancellation in the inner
finallyblock. The outerfinallyensures the decoder is always closed. The error handling correctly maps transport errors to framework-specific exceptions.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
Comment on lines
+1
to
+5
| from .stt import STT | ||
| from .tts import TTS | ||
| from .version import __version__ | ||
|
|
||
| __all__ = ["STT", "TTS", "__version__"] |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check plugin __init__.py files from other plugins mentioned in learnings
echo "=== OpenAI Plugin ==="
cat livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/__init__.py 2>/dev/null | head -30
echo -e "\n=== Deepgram Plugin ==="
cat livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/__init__.py 2>/dev/null | head -30
echo -e "\n=== Ultravox Plugin ==="
cat livekit-plugins/livekit-plugins-ultravox/livekit/plugins/ultravox/__init__.py 2>/dev/null | head -30
echo -e "\n=== Personaplex Plugin (from learning) ==="
cat livekit-plugins/livekit-plugins-personaplex/livekit/plugins/personaplex/__init__.py 2>/dev/null | head -30
echo -e "\n=== Telnyx Plugin (current) ==="
cat livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py 2>/dev/nullRepository: livekit/agents
Length of output: 3507
🏁 Script executed:
#!/bin/bash
# Check the full Deepgram and Ultravox __init__.py files to see plugin registration
echo "=== Deepgram Plugin (full file) ==="
wc -l livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/__init__.py
echo -e "\n=== Ultravox Plugin (full file) ==="
wc -l livekit-plugins/livekit-plugins-ultravox/livekit/plugins/ultravox/__init__.py
# Get full contents
echo -e "\n=== Deepgram Plugin Content ==="
cat livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/__init__.py
echo -e "\n=== Ultravox Plugin Content ==="
cat livekit-plugins/livekit-plugins-ultravox/livekit/plugins/ultravox/__init__.pyRepository: livekit/agents
Length of output: 3205
🏁 Script executed:
#!/bin/bash
# Check the full telnyx __init__.py file
echo "=== Telnyx Plugin file size ==="
wc -l livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py
echo -e "\n=== Telnyx Plugin full content ==="
cat livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py
echo -e "\n=== Check if log.py exists in telnyx ==="
ls -la livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/log.py 2>&1Repository: livekit/agents
Length of output: 475
Add plugin registration following the established pattern.
The telnyx plugin is missing the standard plugin registration that all other plugins implement. Based on the consistent pattern in deepgram and ultravox, add the following after the __all__ definition:
Plugin registration pattern
from livekit.agents import Plugin
from .log import logger
class TelnyxPlugin(Plugin):
def __init__(self) -> None:
super().__init__(__name__, __version__, __package__, logger)
Plugin.register_plugin(TelnyxPlugin())
# Cleanup docs of unexported modules
_module = dir()
NOT_IN_ALL = [m for m in _module if m not in __all__]
__pdoc__ = {}
for n in NOT_IN_ALL:
__pdoc__[n] = FalseThis ensures proper plugin registration with the LiveKit agents framework, consistent with the plugin system design.
🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py`
around lines 1 - 5, Add the standard plugin registration after the existing
__all__ by importing Plugin from livekit.agents and the module logger, define a
TelnyxPlugin class that calls super().__init__(__name__, __version__,
__package__, logger), register it with Plugin.register_plugin(TelnyxPlugin()),
and then build the NOT_IN_ALL list and __pdoc__ cleanup so unexported names
(beyond STT, TTS, __version__ in __all__) are hidden from docs; use the symbols
TelnyxPlugin, Plugin.register_plugin, logger, NOT_IN_ALL and __pdoc__ to locate
where to insert this code.
Comment on lines
1
to
303
| """ | ||
| * Telnyx STT API documentation: | ||
| <https://developers.telnyx.com/docs/voice/programmable-voice/stt-standalone>. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
| import json | ||
| import struct | ||
| import weakref | ||
| from dataclasses import dataclass | ||
| from typing import Literal | ||
|
|
||
| import aiohttp | ||
|
|
||
| from livekit import rtc | ||
| from livekit.agents import ( | ||
| APIConnectionError, | ||
| APIConnectOptions, | ||
| APIStatusError, | ||
| stt, | ||
| utils, | ||
| ) | ||
| from livekit.agents.types import DEFAULT_API_CONNECT_OPTIONS, NOT_GIVEN, NotGivenOr | ||
| from livekit.agents.utils import AudioBuffer, is_given | ||
|
|
||
| from .common import NUM_CHANNELS, SAMPLE_RATE, STT_ENDPOINT, SessionManager, get_api_key | ||
| from .log import logger | ||
|
|
||
| TranscriptionEngine = Literal["telnyx", "google", "deepgram", "azure"] | ||
|
|
||
|
|
||
| @dataclass | ||
| class _STTOptions: | ||
| api_key: str | ||
| language: str | ||
| transcription_engine: TranscriptionEngine | ||
| interim_results: bool | ||
| base_url: str | ||
| sample_rate: int | ||
|
|
||
|
|
||
| class STT(stt.STT): | ||
| def __init__( | ||
| self, | ||
| *, | ||
| language: str = "en", | ||
| transcription_engine: TranscriptionEngine = "telnyx", | ||
| interim_results: bool = True, | ||
| api_key: str | None = None, | ||
| base_url: str = STT_ENDPOINT, | ||
| sample_rate: int = SAMPLE_RATE, | ||
| http_session: aiohttp.ClientSession | None = None, | ||
| ) -> None: | ||
| super().__init__( | ||
| capabilities=stt.STTCapabilities( | ||
| streaming=True, | ||
| interim_results=interim_results, | ||
| ) | ||
| ) | ||
|
|
||
| self._opts = _STTOptions( | ||
| api_key=get_api_key(api_key), | ||
| language=language, | ||
| transcription_engine=transcription_engine, | ||
| interim_results=interim_results, | ||
| base_url=base_url, | ||
| sample_rate=sample_rate, | ||
| ) | ||
| self._session_manager = SessionManager(http_session) | ||
| self._streams = weakref.WeakSet[SpeechStream]() | ||
|
|
||
| @property | ||
| def model(self) -> str: | ||
| return self._opts.transcription_engine | ||
|
|
||
| @property | ||
| def provider(self) -> str: | ||
| return "telnyx" | ||
|
|
||
| async def _recognize_impl( | ||
| self, | ||
| buffer: AudioBuffer, | ||
| *, | ||
| language: NotGivenOr[str] = NOT_GIVEN, | ||
| conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS, | ||
| ) -> stt.SpeechEvent: | ||
| config = self._opts | ||
| if is_given(language): | ||
| config.language = language | ||
|
|
||
| stream = self.stream(language=language, conn_options=conn_options) | ||
| for frame in buffer: | ||
| stream.push_frame(frame) | ||
| stream.end_input() | ||
|
|
||
| final_text = "" | ||
| async for event in stream: | ||
| if event.type == stt.SpeechEventType.FINAL_TRANSCRIPT: | ||
| if event.alternatives: | ||
| final_text += event.alternatives[0].text | ||
|
|
||
| return stt.SpeechEvent( | ||
| type=stt.SpeechEventType.FINAL_TRANSCRIPT, | ||
| alternatives=[ | ||
| stt.SpeechData( | ||
| language=config.language, | ||
| text=final_text, | ||
| ) | ||
| ], | ||
| ) | ||
|
|
||
| def stream( | ||
| self, | ||
| *, | ||
| language: NotGivenOr[str] = NOT_GIVEN, | ||
| conn_options: APIConnectOptions = DEFAULT_API_CONNECT_OPTIONS, | ||
| ) -> SpeechStream: | ||
| resolved_language = language if is_given(language) else self._opts.language | ||
| stream = SpeechStream( | ||
| stt=self, | ||
| conn_options=conn_options, | ||
| language=resolved_language, | ||
| ) | ||
| self._streams.add(stream) | ||
| return stream | ||
|
|
||
| async def aclose(self) -> None: | ||
| for stream in list(self._streams): | ||
| await stream.aclose() | ||
| self._streams.clear() | ||
| await self._session_manager.close() | ||
|
|
||
|
|
||
| def _create_streaming_wav_header(sample_rate: int, num_channels: int) -> bytes: | ||
| """Create a WAV header for streaming with maximum possible size.""" | ||
| bytes_per_sample = 2 | ||
| byte_rate = sample_rate * num_channels * bytes_per_sample | ||
| block_align = num_channels * bytes_per_sample | ||
| data_size = 0x7FFFFFFF | ||
| file_size = 36 + data_size | ||
|
|
||
| header = struct.pack( | ||
| "<4sI4s4sIHHIIHH4sI", | ||
| b"RIFF", | ||
| file_size, | ||
| b"WAVE", | ||
| b"fmt ", | ||
| 16, | ||
| 1, | ||
| num_channels, | ||
| sample_rate, | ||
| byte_rate, | ||
| block_align, | ||
| 16, | ||
| b"data", | ||
| data_size, | ||
| ) | ||
| return header | ||
|
|
||
|
|
||
| class SpeechStream(stt.RecognizeStream): | ||
| def __init__( | ||
| self, | ||
| *, | ||
| stt: STT, | ||
| conn_options: APIConnectOptions, | ||
| language: str, | ||
| ) -> None: | ||
| super().__init__(stt=stt, conn_options=conn_options, sample_rate=stt._opts.sample_rate) | ||
| self._stt: STT = stt | ||
| self._language = language | ||
| self._speaking = False | ||
|
|
||
| async def _run(self) -> None: | ||
| closing_ws = False | ||
|
|
||
| @utils.log_exceptions(logger=logger) | ||
| async def send_task(ws: aiohttp.ClientWebSocketResponse) -> None: | ||
| nonlocal closing_ws | ||
|
|
||
| wav_header = _create_streaming_wav_header(self._stt._opts.sample_rate, NUM_CHANNELS) | ||
| await ws.send_bytes(wav_header) | ||
|
|
||
| samples_per_chunk = self._stt._opts.sample_rate // 20 | ||
| audio_bstream = utils.audio.AudioByteStream( | ||
| sample_rate=self._stt._opts.sample_rate, | ||
| num_channels=NUM_CHANNELS, | ||
| samples_per_channel=samples_per_chunk, | ||
| ) | ||
|
|
||
| async for data in self._input_ch: | ||
| if isinstance(data, rtc.AudioFrame): | ||
| for frame in audio_bstream.write(data.data.tobytes()): | ||
| await ws.send_bytes(frame.data.tobytes()) | ||
| elif isinstance(data, self._FlushSentinel): | ||
| for frame in audio_bstream.flush(): | ||
| await ws.send_bytes(frame.data.tobytes()) | ||
|
|
||
| for frame in audio_bstream.flush(): | ||
| await ws.send_bytes(frame.data.tobytes()) | ||
|
|
||
| await asyncio.sleep(1.0) | ||
| closing_ws = True | ||
| await ws.close() | ||
|
|
||
| @utils.log_exceptions(logger=logger) | ||
| async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None: | ||
| nonlocal closing_ws | ||
| while True: | ||
| msg = await ws.receive() | ||
| if msg.type in ( | ||
| aiohttp.WSMsgType.CLOSED, | ||
| aiohttp.WSMsgType.CLOSE, | ||
| aiohttp.WSMsgType.CLOSING, | ||
| ): | ||
| if closing_ws: | ||
| return | ||
| raise APIStatusError(message="Telnyx STT WebSocket closed unexpectedly") | ||
|
|
||
| if msg.type == aiohttp.WSMsgType.TEXT: | ||
| try: | ||
| data = json.loads(msg.data) | ||
| logger.debug("Telnyx STT received: %s", data) | ||
| self._process_stream_event(data) | ||
| except Exception: | ||
| logger.exception("Failed to process Telnyx STT message") | ||
| elif msg.type == aiohttp.WSMsgType.ERROR: | ||
| logger.error("Telnyx STT WebSocket error: %s", ws.exception()) | ||
|
|
||
| ws: aiohttp.ClientWebSocketResponse | None = None | ||
| try: | ||
| ws = await self._connect_ws() | ||
| tasks = [ | ||
| asyncio.create_task(send_task(ws)), | ||
| asyncio.create_task(recv_task(ws)), | ||
| ] | ||
| try: | ||
| await asyncio.gather(*tasks) | ||
| finally: | ||
| await utils.aio.gracefully_cancel(*tasks) | ||
| finally: | ||
| if ws is not None: | ||
| await ws.close() | ||
|
|
||
| async def _connect_ws(self) -> aiohttp.ClientWebSocketResponse: | ||
| opts = self._stt._opts | ||
| params = { | ||
| "transcription_engine": opts.transcription_engine, | ||
| "language": self._language, | ||
| "input_format": "wav", | ||
| } | ||
| query_string = "&".join(f"{k}={v}" for k, v in params.items()) | ||
| url = f"{opts.base_url}?{query_string}" | ||
| headers = {"Authorization": f"Bearer {opts.api_key}"} | ||
|
|
||
| try: | ||
| ws = await asyncio.wait_for( | ||
| self._stt._session_manager.ensure_session().ws_connect(url, headers=headers), | ||
| self._conn_options.timeout, | ||
| ) | ||
| logger.debug("Established Telnyx STT WebSocket connection") | ||
| return ws | ||
| except (aiohttp.ClientConnectorError, asyncio.TimeoutError) as e: | ||
| raise APIConnectionError("Failed to connect to Telnyx STT") from e | ||
|
|
||
| def _process_stream_event(self, data: dict) -> None: | ||
| transcript = data.get("transcript", "") | ||
| is_final = data.get("is_final", False) | ||
|
|
||
| if not transcript: | ||
| return | ||
|
|
||
| if not self._speaking: | ||
| self._speaking = True | ||
| self._event_ch.send_nowait(stt.SpeechEvent(type=stt.SpeechEventType.START_OF_SPEECH)) | ||
|
|
||
| alternatives = [ | ||
| stt.SpeechData( | ||
| language=self._language, | ||
| text=transcript, | ||
| confidence=data.get("confidence", 0.0), | ||
| ) | ||
| ] | ||
|
|
||
| if is_final: | ||
| self._event_ch.send_nowait( | ||
| stt.SpeechEvent( | ||
| type=stt.SpeechEventType.FINAL_TRANSCRIPT, | ||
| alternatives=alternatives, | ||
| ) | ||
| ) | ||
| self._speaking = False | ||
| self._event_ch.send_nowait(stt.SpeechEvent(type=stt.SpeechEventType.END_OF_SPEECH)) | ||
| else: | ||
| self._event_ch.send_nowait( | ||
| stt.SpeechEvent( | ||
| type=stt.SpeechEventType.INTERIM_TRANSCRIPT, | ||
| alternatives=alternatives, | ||
| ) | ||
| ) | ||
|
|
Contributor
There was a problem hiding this comment.
Pipeline failure: Ruff format check failed.
The CI pipeline indicates this file would be reformatted by ruff format. Please run ruff format on this file to fix formatting issues.
#!/bin/bash
# Check what formatting changes ruff would make
cd livekit-plugins/livekit-plugins-telnyx && ruff format --check --diff livekit/plugins/telnyx/stt.py🧰 Tools
🪛 GitHub Actions: CI
[error] 1-1: Ruff format check failed. This file would be reformatted by 'ruff format'.
🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py` around
lines 1 - 303, Ruff reports formatting issues in this module; run ruff format on
livekit/plugins/telnyx/stt.py and commit the changes to satisfy CI — the
formatting fixes will target whitespace, import and function/class spacing and
line breaks around symbols such as STT, SpeechStream, and
_create_streaming_wav_header; run `ruff format livekit/plugins/telnyx/stt.py`
(or the project-wide ruff formatter), review the diff, and commit the formatted
file.
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py
Outdated
Show resolved
Hide resolved
Comment on lines
+1
to
+27
| """ | ||
| * Telnyx TTS API documentation: | ||
| <https://developers.telnyx.com/docs/voice/programmable-voice/tts-standalone>. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import asyncio | ||
| import base64 | ||
| import json | ||
| import weakref | ||
| from dataclasses import dataclass | ||
|
|
||
| import aiohttp | ||
|
|
||
| from livekit.agents import ( | ||
| APIConnectionError, | ||
| APIConnectOptions, | ||
| APIStatusError, | ||
| APITimeoutError, | ||
| tts, | ||
| utils, | ||
| ) | ||
| from livekit.agents.types import DEFAULT_API_CONNECT_OPTIONS | ||
|
|
||
| from .common import NUM_CHANNELS, SAMPLE_RATE, TTS_ENDPOINT, SessionManager, get_api_key | ||
| from .log import logger |
Contributor
There was a problem hiding this comment.
Fix Ruff formatting to pass CI.
The pipeline indicates this file would be reformatted by ruff format. Run ruff format on this file to resolve the CI failure.
#!/bin/bash
# Check what formatting changes ruff would make
cd livekit-plugins/livekit-plugins-telnyx
ruff format --diff livekit/plugins/telnyx/tts.py🧰 Tools
🪛 GitHub Actions: CI
[error] 1-1: Ruff format check failed. This file would be reformatted by 'ruff format'.
🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py` around
lines 1 - 27, This file (livekit.plugins.telnyx.tts.py) fails Ruff formatting;
run ruff format on the file or apply the formatting changes Ruff would make so
CI passes. Specifically reformat the module-level imports and docstring and
ensure spacing around imports (e.g., the block importing APIConnectionError,
APIConnectOptions, APIStatusError, APITimeoutError, tts, utils) and the local
imports (TTS_ENDPOINT, SessionManager, get_api_key, NUM_CHANNELS, SAMPLE_RATE,
logger) follow Ruff's style; run `ruff format livekit/plugins/telnyx/tts.py` (or
`ruff format --diff` to preview) and commit the resulting changes.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Fix all issues with AI agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py`:
- Around line 221-225: The current debug log prints the entire STT payload
(logger.debug("Telnyx STT received: %s", data)), which can expose PII; change
the logging in the WebSocket TEXT handler to avoid dumping raw transcripts by
creating a redacted summary before logging: inspect the parsed data (variable
data from json.loads(msg.data) used before calling _process_stream_event),
replace or omit fields like "transcript", "text", "alternatives", or "raw" with
either a redaction token (e.g. "[REDACTED]") or metadata such as their lengths,
and then call logger.debug with that summary (only metadata/keys/timestamps),
not the full payload. Ensure _process_stream_event continues to receive the
original data unmodified.
- Around line 203-219: The recv_task can falsely raise APIStatusError because
closing_ws is only set after the 1s sleep and close; set closing_ws = True
before awaiting the delay/closing so the recv_task sees the flag if the server
closes the connection in response to our shutdown. Update the shutdown sequence
in the function that calls ws.close() (the block that currently does await
asyncio.sleep(1.0); closing_ws = True; await ws.close()) to assign closing_ws =
True first, then await asyncio.sleep(1.0) and finally await ws.close(), so
recv_task (which checks closing_ws) will treat server-side closes as expected
shutdowns.
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py`:
- Around line 125-134: The generic exception handler in _run currently wraps all
errors from _run_ws as APIConnectionError, losing APIStatusError and
APITimeoutError details; change the except Exception as e block in _run to
re-raise those known API exceptions (APITimeoutError, APIStatusError) untouched
(raise) and only wrap other unknown exceptions as APIConnectionError (raise
APIConnectionError() from e), so that _run preserves diagnostics and retry
semantics coming from _run_ws.
- Around line 139-212: The code currently calls
output_emitter.start_segment(segment_id=segment_id) before the WebSocket work
but only calls output_emitter.end_segment() after the try/except that can be
bypassed on exceptions; move the end_segment() call into a finally that always
runs (alongside the existing decoder.aclose() cleanup) so that
output_emitter.end_segment() executes regardless of WS/connect/handler failures
(update references around ensure_session().ws_connect, the ws async-with block,
and the finally that currently awaits decoder.aclose()).
🧹 Nitpick comments (2)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (1)
44-301: Add Google-style docstrings for the public STT surface.
STT,SpeechStream, and_create_streaming_wav_headerare public/central APIs but lack Google-style docstrings. Please add concise docstrings with Args/Returns where applicable.
As per coding guidelines: Use Google-style docstrings.livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (1)
37-212: Add Google-style docstrings for the public TTS surface.
TTSandSynthesizeStreamare public APIs but lack Google-style docstrings. Please add docstrings with Args/Returns where applicable.
As per coding guidelines: Use Google-style docstrings.
📜 Review details
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/py.typedlivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.pylivekit-plugins/livekit-plugins-telnyx/pyproject.toml
🚧 Files skipped from review as they are similar to previous changes (1)
- livekit-plugins/livekit-plugins-telnyx/pyproject.toml
🧰 Additional context used
📓 Path-based instructions (1)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Format code with ruff
Run ruff linter and auto-fix issues
Run mypy type checker in strict mode
Maintain line length of 100 characters maximum
Ensure Python 3.9+ compatibility
Use Google-style docstrings
Files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py
🧠 Learnings (5)
📚 Learning: 2026-01-30T12:53:12.738Z
Learnt from: milanperovic
Repo: livekit/agents PR: 4660
File: livekit-plugins/livekit-plugins-personaplex/livekit/plugins/personaplex/__init__.py:19-21
Timestamp: 2026-01-30T12:53:12.738Z
Learning: In plugin __init__.py files under the livekit-plugins or similar plugin directories, place internal imports (for example, from .log import logger) after the __all__ definition. These imports are used for plugin registration and are not part of the public API. This pattern is used across plugins (e.g., openai, deepgram, ultravox) and helps avoid E402 violations while keeping the public API surface clean.
Applied to files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py
📚 Learning: 2026-01-16T07:44:56.353Z
Learnt from: CR
Repo: livekit/agents PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-16T07:44:56.353Z
Learning: Applies to **/*.py : Run ruff linter and auto-fix issues
Applied to files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py
📚 Learning: 2026-01-16T07:44:56.353Z
Learnt from: CR
Repo: livekit/agents PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-16T07:44:56.353Z
Learning: Applies to **/*.py : Format code with ruff
Applied to files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py
📚 Learning: 2026-01-16T07:44:56.353Z
Learnt from: CR
Repo: livekit/agents PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-16T07:44:56.353Z
Learning: Implement Model Interface Pattern for STT, TTS, LLM, and Realtime models with provider-agnostic interfaces, fallback adapters for resilience, and stream adapters for different streaming patterns
Applied to files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.pylivekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py
📚 Learning: 2026-01-16T07:44:56.353Z
Learnt from: CR
Repo: livekit/agents PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-01-16T07:44:56.353Z
Learning: Follow the Plugin System pattern where plugins in livekit-plugins/ are separate packages registered via the Plugin base class
Applied to files:
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py
🧬 Code graph analysis (3)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (6)
livekit-agents/livekit/agents/_exceptions.py (3)
APIConnectionError(84-88)APIStatusError(45-81)APITimeoutError(91-95)livekit-agents/livekit/agents/types.py (1)
APIConnectOptions(54-88)livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.py (4)
SessionManager(22-36)get_api_key(15-19)close(33-36)ensure_session(27-31)livekit-agents/livekit/agents/tts/tts.py (3)
TTSCapabilities(47-51)num_channels(121-122)exception(213-214)livekit-agents/livekit/agents/utils/aio/channel.py (1)
Chan(49-178)livekit-agents/livekit/agents/utils/codecs/decoder.py (1)
AudioStreamDecoder(119-339)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py (3)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (1)
STT(44-132)livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py (1)
TTS(37-87)livekit-agents/livekit/agents/plugin.py (2)
Plugin(13-56)register_plugin(31-36)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py (5)
livekit-agents/livekit/agents/types.py (1)
APIConnectOptions(54-88)livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/common.py (4)
SessionManager(22-36)get_api_key(15-19)close(33-36)ensure_session(27-31)livekit-agents/livekit/agents/stt/stt.py (4)
SpeechEvent(70-74)SpeechEventType(32-49)SpeechData(53-61)RecognizeStream(252-469)livekit-agents/livekit/agents/utils/log.py (1)
log_exceptions(9-41)livekit-agents/livekit/agents/utils/audio.py (1)
AudioByteStream(41-157)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: type-check (3.9)
- GitHub Check: unit-tests
- GitHub Check: type-check (3.13)
🔇 Additional comments (1)
livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/__init__.py (1)
1-18: LGTM — plugin registration and exports are wired correctly.
✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.
Comment on lines
+203
to
+219
| await asyncio.sleep(1.0) | ||
| closing_ws = True | ||
| await ws.close() | ||
|
|
||
| @utils.log_exceptions(logger=logger) | ||
| async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None: | ||
| nonlocal closing_ws | ||
| while True: | ||
| msg = await ws.receive() | ||
| if msg.type in ( | ||
| aiohttp.WSMsgType.CLOSED, | ||
| aiohttp.WSMsgType.CLOSE, | ||
| aiohttp.WSMsgType.CLOSING, | ||
| ): | ||
| if closing_ws: | ||
| return | ||
| raise APIStatusError(message="Telnyx STT WebSocket closed unexpectedly") |
Contributor
There was a problem hiding this comment.
Prevent false “unexpected close” errors on normal shutdown.
closing_ws is set after a 1s sleep; if the server closes earlier in response to end-of-audio, recv_task raises APIStatusError even on a normal completion. Set the flag before the delay/close so expected server closes aren’t treated as errors.
✅ Safer close sequencing
- await asyncio.sleep(1.0)
- closing_ws = True
- await ws.close()
+ closing_ws = True
+ await asyncio.sleep(1.0)
+ await ws.close()📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| await asyncio.sleep(1.0) | |
| closing_ws = True | |
| await ws.close() | |
| @utils.log_exceptions(logger=logger) | |
| async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None: | |
| nonlocal closing_ws | |
| while True: | |
| msg = await ws.receive() | |
| if msg.type in ( | |
| aiohttp.WSMsgType.CLOSED, | |
| aiohttp.WSMsgType.CLOSE, | |
| aiohttp.WSMsgType.CLOSING, | |
| ): | |
| if closing_ws: | |
| return | |
| raise APIStatusError(message="Telnyx STT WebSocket closed unexpectedly") | |
| closing_ws = True | |
| await asyncio.sleep(1.0) | |
| await ws.close() | |
| `@utils.log_exceptions`(logger=logger) | |
| async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None: | |
| nonlocal closing_ws | |
| while True: | |
| msg = await ws.receive() | |
| if msg.type in ( | |
| aiohttp.WSMsgType.CLOSED, | |
| aiohttp.WSMsgType.CLOSE, | |
| aiohttp.WSMsgType.CLOSING, | |
| ): | |
| if closing_ws: | |
| return | |
| raise APIStatusError(message="Telnyx STT WebSocket closed unexpectedly") |
🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py` around
lines 203 - 219, The recv_task can falsely raise APIStatusError because
closing_ws is only set after the 1s sleep and close; set closing_ws = True
before awaiting the delay/closing so the recv_task sees the flag if the server
closes the connection in response to our shutdown. Update the shutdown sequence
in the function that calls ws.close() (the block that currently does await
asyncio.sleep(1.0); closing_ws = True; await ws.close()) to assign closing_ws =
True first, then await asyncio.sleep(1.0) and finally await ws.close(), so
recv_task (which checks closing_ws) will treat server-side closes as expected
shutdowns.
Comment on lines
+221
to
+225
| if msg.type == aiohttp.WSMsgType.TEXT: | ||
| try: | ||
| data = json.loads(msg.data) | ||
| logger.debug("Telnyx STT received: %s", data) | ||
| self._process_stream_event(data) |
Contributor
There was a problem hiding this comment.
Avoid logging full STT payloads (PII risk).
logger.debug("... %s", data) can dump raw transcripts and metadata into logs. Please redact or log only metadata/lengths to avoid sensitive data leakage.
🔒 Suggested redaction
- logger.debug("Telnyx STT received: %s", data)
+ logger.debug(
+ "Telnyx STT received (is_final=%s, transcript_len=%d)",
+ data.get("is_final"),
+ len(data.get("transcript", "")),
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if msg.type == aiohttp.WSMsgType.TEXT: | |
| try: | |
| data = json.loads(msg.data) | |
| logger.debug("Telnyx STT received: %s", data) | |
| self._process_stream_event(data) | |
| if msg.type == aiohttp.WSMsgType.TEXT: | |
| try: | |
| data = json.loads(msg.data) | |
| logger.debug( | |
| "Telnyx STT received (is_final=%s, transcript_len=%d)", | |
| data.get("is_final"), | |
| len(data.get("transcript", "")), | |
| ) | |
| self._process_stream_event(data) |
🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/stt.py` around
lines 221 - 225, The current debug log prints the entire STT payload
(logger.debug("Telnyx STT received: %s", data)), which can expose PII; change
the logging in the WebSocket TEXT handler to avoid dumping raw transcripts by
creating a redacted summary before logging: inspect the parsed data (variable
data from json.loads(msg.data) used before calling _process_stream_event),
replace or omit fields like "transcript", "text", "alternatives", or "raw" with
either a redaction token (e.g. "[REDACTED]") or metadata such as their lengths,
and then call logger.debug with that summary (only metadata/keys/timestamps),
not the full payload. Ensure _process_stream_event continues to receive the
original data unmodified.
Comment on lines
+125
to
+134
| try: | ||
| await asyncio.gather(*tasks) | ||
| except asyncio.TimeoutError: | ||
| raise APITimeoutError() from None | ||
| except aiohttp.ClientResponseError as e: | ||
| raise APIStatusError( | ||
| message=e.message, status_code=e.status, request_id=request_id, body=None | ||
| ) from None | ||
| except Exception as e: | ||
| raise APIConnectionError() from e |
Contributor
There was a problem hiding this comment.
Preserve APIStatusError/APITimeoutError from _run_ws.
_run wraps any exception as APIConnectionError, which overwrites specific errors raised by _run_ws (e.g., status code/timeouts). Re-raise known API errors to preserve diagnostics and retry semantics.
🔧 Preserve API-specific errors
try:
await asyncio.gather(*tasks)
+ except (APITimeoutError, APIStatusError, APIConnectionError):
+ raise
except asyncio.TimeoutError:
raise APITimeoutError() from None
except aiohttp.ClientResponseError as e:
raise APIStatusError(
message=e.message, status_code=e.status, request_id=request_id, body=None
) from None
except Exception as e:
raise APIConnectionError() from e🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py` around
lines 125 - 134, The generic exception handler in _run currently wraps all
errors from _run_ws as APIConnectionError, losing APIStatusError and
APITimeoutError details; change the except Exception as e block in _run to
re-raise those known API exceptions (APITimeoutError, APIStatusError) untouched
(raise) and only wrap other unknown exceptions as APIConnectionError (raise
APIConnectionError() from e), so that _run preserves diagnostics and retry
semantics coming from _run_ws.
Comment on lines
+139
to
+212
| segment_id = utils.shortuuid() | ||
| output_emitter.start_segment(segment_id=segment_id) | ||
|
|
||
| url = f"{self._tts._opts.base_url}?voice={self._tts._opts.voice}" | ||
| headers = {"Authorization": f"Bearer {self._tts._opts.api_key}"} | ||
|
|
||
| decoder = utils.codecs.AudioStreamDecoder( | ||
| sample_rate=SAMPLE_RATE, | ||
| num_channels=NUM_CHANNELS, | ||
| format="audio/mp3", | ||
| ) | ||
|
|
||
| async def send_task(ws: aiohttp.ClientWebSocketResponse) -> None: | ||
| await ws.send_str(json.dumps({"text": " "})) | ||
| self._mark_started() | ||
| await ws.send_str(json.dumps({"text": text})) | ||
| await ws.send_str(json.dumps({"text": ""})) | ||
|
|
||
| async def recv_task(ws: aiohttp.ClientWebSocketResponse) -> None: | ||
| async for msg in ws: | ||
| if msg.type == aiohttp.WSMsgType.TEXT: | ||
| try: | ||
| data = json.loads(msg.data) | ||
| audio_data = data.get("audio") | ||
| if audio_data: | ||
| audio_bytes = base64.b64decode(audio_data) | ||
| if audio_bytes: | ||
| decoder.push(audio_bytes) | ||
| except json.JSONDecodeError: | ||
| logger.warning("Telnyx TTS: Received invalid JSON") | ||
|
|
||
| elif msg.type in ( | ||
| aiohttp.WSMsgType.CLOSE, | ||
| aiohttp.WSMsgType.CLOSED, | ||
| aiohttp.WSMsgType.CLOSING, | ||
| ): | ||
| break | ||
| elif msg.type == aiohttp.WSMsgType.ERROR: | ||
| logger.error(f"Telnyx TTS WebSocket error: {ws.exception()}") | ||
| break | ||
|
|
||
| decoder.end_input() | ||
|
|
||
| async def decode_task() -> None: | ||
| async for frame in decoder: | ||
| output_emitter.push(frame.data.tobytes()) | ||
|
|
||
| try: | ||
| ws = await asyncio.wait_for( | ||
| self._tts._session_manager.ensure_session().ws_connect(url, headers=headers), | ||
| self._conn_options.timeout, | ||
| ) | ||
| async with ws: | ||
| tasks = [ | ||
| asyncio.create_task(send_task(ws)), | ||
| asyncio.create_task(recv_task(ws)), | ||
| asyncio.create_task(decode_task()), | ||
| ] | ||
| try: | ||
| await asyncio.gather(*tasks) | ||
| finally: | ||
| await utils.aio.gracefully_cancel(*tasks) | ||
| except asyncio.TimeoutError: | ||
| raise APITimeoutError() from None | ||
| except aiohttp.ClientResponseError as e: | ||
| raise APIStatusError( | ||
| message=e.message, status_code=e.status, request_id=None, body=None | ||
| ) from None | ||
| except Exception as e: | ||
| raise APIConnectionError() from e | ||
| finally: | ||
| await decoder.aclose() | ||
|
|
||
| output_emitter.end_segment() |
Contributor
There was a problem hiding this comment.
Ensure end_segment() is emitted even on WS failures.
start_segment() is called before the WS connect. If an exception is raised, end_segment() is skipped and downstream consumers may wait indefinitely. Move end_segment() into a finally block.
🧹 Always end the segment
- try:
- ws = await asyncio.wait_for(
- self._tts._session_manager.ensure_session().ws_connect(url, headers=headers),
- self._conn_options.timeout,
- )
- async with ws:
- tasks = [
- asyncio.create_task(send_task(ws)),
- asyncio.create_task(recv_task(ws)),
- asyncio.create_task(decode_task()),
- ]
- try:
- await asyncio.gather(*tasks)
- finally:
- await utils.aio.gracefully_cancel(*tasks)
- except asyncio.TimeoutError:
- raise APITimeoutError() from None
- except aiohttp.ClientResponseError as e:
- raise APIStatusError(
- message=e.message, status_code=e.status, request_id=None, body=None
- ) from None
- except Exception as e:
- raise APIConnectionError() from e
- finally:
- await decoder.aclose()
-
- output_emitter.end_segment()
+ try:
+ ws = await asyncio.wait_for(
+ self._tts._session_manager.ensure_session().ws_connect(url, headers=headers),
+ self._conn_options.timeout,
+ )
+ async with ws:
+ tasks = [
+ asyncio.create_task(send_task(ws)),
+ asyncio.create_task(recv_task(ws)),
+ asyncio.create_task(decode_task()),
+ ]
+ try:
+ await asyncio.gather(*tasks)
+ finally:
+ await utils.aio.gracefully_cancel(*tasks)
+ except asyncio.TimeoutError:
+ raise APITimeoutError() from None
+ except aiohttp.ClientResponseError as e:
+ raise APIStatusError(
+ message=e.message, status_code=e.status, request_id=None, body=None
+ ) from None
+ except Exception as e:
+ raise APIConnectionError() from e
+ finally:
+ await decoder.aclose()
+ output_emitter.end_segment()🤖 Prompt for AI Agents
In `@livekit-plugins/livekit-plugins-telnyx/livekit/plugins/telnyx/tts.py` around
lines 139 - 212, The code currently calls
output_emitter.start_segment(segment_id=segment_id) before the WebSocket work
but only calls output_emitter.end_segment() after the try/except that can be
bypassed on exceptions; move the end_segment() call into a finally that always
runs (alongside the existing decoder.aclose() cleanup) so that
output_emitter.end_segment() executes regardless of WS/connect/handler failures
(update references around ensure_session().ws_connect, the ws async-with block,
and the finally that currently awaits decoder.aclose()).
|
@theomonnom Hi Théo, would love it if you or someone from your team could take a look at this PR and help review/approve it so we can get it merged and make the integration official. We’re contributing this from Telnyx and are excited to formally plug into the LiveKit ecosystem. There are already several companies in the ecosystem here (e.g. Twilio, Deepgram, Gladia), and we think this fits naturally alongside those and makes the platform more complete. Happy to make any changes or iterate based on feedback - just let us know |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR Description
Summary
Adds support for Telnyx as a supported vendor for Speech-to-Text (STT) and Text-to-Speech (TTS) within the LiveKit Agents framework. This includes a new dedicated plugin package and an example voice agent demonstrating the integration.
New Features
examples/voice_agents/telnyx_voice_agent.pyusing Telnyx for both STT and TTS alongside OpenAI for reasoning.Implementation Details
livekit-plugins-telnyxpackage underlivekit-plugins/.SessionManagerincommon.pyto handle sharedaiohttpclient sessions and API key resolution.pyproject.tomlworkspace.Documentation
Summary by CodeRabbit
New Features
Integrations
Chores
✏️ Tip: You can customize this high-level summary in your review settings.