Trying to run it on a low VRAM system

[mercurial-moon]

Hi,

I was testing your example_browserclient tutorial and it works well on a 8GB VRAM system, memory usage is around 3.5 to 4GB when loading the tiny.en and large-v3 models together as required by your system.

I'm trying to run it on a more resource constrained system with 2GB of VRAM so had some questions.

1. Does your library support turbo model?, from what I've heard its faster that large-v3 with minor reduction in accuracy.
2. Is it possible to run the example_browserclient with only 1 model that is turbo? is this allowed? would this cause accuracy issues or simply not work. I see that in your AudioToTextRecorder you pass in 2 models tiny.en and large-v2.
3. Do you plan to support Nvidia Parakeet, from what I understand that it was designed for streaming use case.. not sure if streaming is the same as real time though.

[KoljaB]

1. Yes, you can run any CTranslate2 ASR model on huggingface like deepdml/faster-whisper-large-v3-turbo-ct2
2. Yes, set the use_main_model_for_realtime parameter to true
3. Maybe in the future. It's Linux/Unix only, so I'm hesitating. Maybe with sherpa onnx there's a possibility to run it on Windows, but I have to evaluate.

[csukuangfj]

For English ASR, i suggest that you use
https://k2-fsa.github.io/sherpa/onnx/pretrained_models/offline-transducer/nemo-transducer-models.html#sherpa-onnx-nemo-parakeet-tdt-0-6b-v2-int8-english
with sherpa-onnx.

It is the state-of-the-art English ASR model, even more accurate than whisper .

You can find pre-built Android APK with the above model for real-time/streaming speech recognition on your phone at
https://k2-fsa.github.io/sherpa/onnx/android/apk-simulate-streaming-asr.html

(Search for parakeet_tdt_0.6b_v2.apk in the above page)

[mercurial-moon]

For English ASR, i suggest that you use https://k2-fsa.github.io/sherpa/onnx/pretrained_models/offline-transducer/nemo-transducer-models.html#sherpa-onnx-nemo-parakeet-tdt-0-6b-v2-int8-english with sherpa-onnx.

What kind of VRAM usage does that model have for realtime microphone?

[csukuangfj]

The model file size is close to 700MB. I think you need 2GB RAM to run it.

If you want an ultra low RAM model, please see
https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/zipformer-transucer-models.html#marcoyang-sherpa-ncnn-streaming-zipformer-20m-2023-02-17-english

512MB RAM is enough to run it.

[mercurial-moon]

@csukuangfj
thanks for the update.. unfortunately i haven't been able to get sherpa onnx running on my system , there are so many exe files in the releases page, that I'm not sure what to use. But i will further discuss on sherpa-onnx forum.

@KoljaB
Interestingly i tried using a single model the deepdml turbo one and set use_main_model_for_realtime to true it threw cuda out of memory error.
But it i use 2 models like
main: small.en and realitime: tiny.en -> this works well
main: medium and realtime: tiny.en -> this works sluggish and with some spoken sentences skipped.
but no out of memory error in 2 model mode.