Skip to content

feat: implement language detection for Microsoft Voice TTS #5582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hengstchon wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat: implement language detection for Microsoft Voice TTS #5582

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
25a8097
feat: implement language detection for Microsoft Voice TTS
hengstchon Oct 3, 2024
554901d
fix: eld module
hengstchon Oct 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion app/components/chat.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { eld } from "eld";
import { useDebouncedCallback } from "use-debounce";
import React, {
useState,
Expand Down Expand Up @@ -104,6 +105,7 @@ import {
REQUEST_TIMEOUT_MS,
UNFINISHED_INPUT,
ServiceProvider,
MS_EDGE_TTS_VOICES,
} from "../constant";
import { Avatar } from "./emoji";
import { ContextPrompts, MaskAvatar, MaskConfig } from "./mask";
Expand Down Expand Up @@ -1214,7 +1216,11 @@ function _Chat() {
const { markdownToTxt } = require("markdown-to-txt");
const textContent = markdownToTxt(text);
if (config.ttsConfig.engine !== DEFAULT_TTS_ENGINE) {
const edgeVoiceName = accessStore.edgeVoiceName();
const detectLang = eld.detect(text).language;
const edgeVoiceName =
detectLang in MS_EDGE_TTS_VOICES
? MS_EDGE_TTS_VOICES[detectLang as keyof typeof MS_EDGE_TTS_VOICES]
: accessStore.edgeVoiceName();
Comment on lines +1219 to +1223
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Oct 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Add Error Handling for Language Detection

The call to eld.detect(text).language may return undefined or throw an error if the language cannot be detected. To ensure stability, add error handling for cases where language detection fails.

Apply the following changes to handle potential errors:

 const { markdownToTxt } = require("markdown-to-txt");
 const textContent = markdownToTxt(text);
+let detectLang = 'default';
+try {
+  const detectionResult = eld.detect(text);
+  if (detectionResult && detectionResult.language) {
+    detectLang = detectionResult.language;
+  }
+} catch (error) {
+  console.error('Language detection failed:', error);
+}
 const edgeVoiceName =
   detectLang in MS_EDGE_TTS_VOICES
     ? MS_EDGE_TTS_VOICES[detectLang as keyof typeof MS_EDGE_TTS_VOICES]
     : accessStore.edgeVoiceName();

Committable suggestion was skipped due to low confidence.

const tts = new MsEdgeTTS();
await tts.setMetadata(
edgeVoiceName,
Expand Down
60 changes: 60 additions & 0 deletions app/constant.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,66 @@ export const DEFAULT_TTS_VOICES = [
"nova",
"shimmer",
];
export enum MS_EDGE_TTS_VOICES {
am = "am-ET-MekdesNeural",
ar = "ar-AE-FatimaNeural",
az = "az-AZ-BanuNeural",
be = "en-US-AlloyMultilingualNeural",
bg = "bg-BG-KalinaNeural",
bn = "bn-IN-TanishaaNeural",
ca = "ca-ES-JoanaNeural",
cs = "cs-CZ-VlastaNeural",
da = "da-DK-ChristelNeural",
de = "de-DE-ConradNeural",
el = "el-GR-AthinaNeural",
en = "en-US-AndrewNeural",
es = "es-ES-ElviraNeural",
et = "et-EE-AnuNeural",
eu = "eu-ES-AinhoaNeural",
fa = "fa-IR-DilaraNeural",
fi = "fi-FI-SelmaNeural",
fr = "fr-FR-DeniseNeural",
gu = "gu-IN-DhwaniNeural",
he = "he-IL-HilaNeural",
hi = "hi-IN-AaravNeural",
hr = "hr-HR-GabrijelaNeural",
hu = "hu-HU-NoemiNeural",
hy = "hy-AM-AnahitNeural",
is = "is-IS-GudrunNeural",
it = "it-IT-ElsaNeural",
ja = "ja-JP-NanamiNeural",
ka = "ka-GE-EkaNeural",
kn = "kn-IN-SapnaNeural",
ko = "ko-KR-SunHiNeural",
lo = "lo-LA-KeomanyNeural",
lt = "lt-LT-OnaNeural",
lv = "lv-LV-EveritaNeural",
ml = "ml-IN-SobhanaNeural",
mr = "mr-IN-AarohiNeural",
ms = "ms-MY-YasminNeural",
nl = "nl-NL-FennaNeural",
no = "nb-NO-PernilleNeural",
or = "or-IN-SubhasiniNeural",
pa = "pa-IN-OjasNeural",
pl = "pl-PL-AgnieszkaNeural",
pt = "pt-PT-RaquelNeural",
ro = "ro-RO-AlinaNeural",
ru = "ru-RU-SvetlanaNeural",
sk = "sk-SK-ViktoriaNeural",
sl = "sl-SI-PetraNeural",
sq = "sq-AL-AnilaNeural",
sr = "sr-Latn-RS-NicholasNeural",
sv = "sv-SE-SofieNeural",
ta = "ta-IN-PallaviNeural",
te = "te-IN-ShrutiNeural",
th = "th-TH-PremwadeeNeural",
tl = "en-US-AlloyMultilingualNeural",
tr = "tr-TR-EmelNeural",
uk = "uk-UA-PolinaNeural",
ur = "ur-IN-GulNeural",
vi = "vi-VN-HoaiMyNeural",
zh = "zh-CN-YunxiNeural",
}

const openaiModels = [
"gpt-3.5-turbo",
Expand Down
6 changes: 6 additions & 0 deletions app/global.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,9 @@ declare interface Window {
};
};
}

declare module "eld" {
export const eld: {
detect: (text: string) => { language: string };
};
}
1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
"@vercel/analytics": "^0.1.11",
"@vercel/speed-insights": "^1.0.2",
"axios": "^1.7.5",
"eld": "^1.0.0",
"emoji-picker-react": "^4.9.2",
"fuse.js": "^7.0.0",
"heic2any": "^0.0.4",
Expand Down
5 changes: 5 additions & 0 deletions yarn.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3959,6 +3959,11 @@ eastasianwidth@^0.2.0:
resolved "https://registry.yarnpkg.com/eastasianwidth/-/eastasianwidth-0.2.0.tgz#696ce2ec0aa0e6ea93a397ffcf24aa7840c827cb"
integrity sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==

eld@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/eld/-/eld-1.0.0.tgz#507cf2229fe3d809430fdfc3689edafeb53aafc1"
integrity sha512-+wehmSoVbWYIsQ0wDUN6b1QCGNvnfT49KD/W6QavCEq8D5/ANx+B+qy5m2sKOTh3MyBqMEDTUxc1wcajSAo/2Q==

electron-to-chromium@^1.4.284:
version "1.4.345"
resolved "https://registry.yarnpkg.com/electron-to-chromium/-/electron-to-chromium-1.4.345.tgz#c90b7183b39245cddf0e990337469063bfced6f0"
Expand Down