Skip to content

CER estimation speedup for audio-based text normalization #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ekmb merged 2 commits into from
May 27, 2023
Merged

CER estimation speedup for audio-based text normalization #73

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ecec40d
Replaced jiwer with editdistance to speed up CER estimation
vsl9 May 24, 2023
ab90ca5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions nemo_text_processing/text_normalization/normalize_with_audio.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
from time import perf_counter
from typing import List, Optional, Tuple

import jiwer
import editdistance
import pynini
from nemo_text_processing.text_normalization.data_loader_utils import post_process_punct, pre_process
from nemo_text_processing.text_normalization.normalize import Normalizer
Expand Down Expand Up @@ -406,7 +406,7 @@ def calculate_cer(normalized_texts: List[str], pred_text: str, remove_punct=Fals
for punct in "!?:;,.-()*+-/<=>@^_":
text_clean = text_clean.replace(punct, " ").replace(" ", " ")

cer = jiwer.cer(pred_text, text_clean) * 100
cer = editdistance.eval(pred_text, text_clean) * 100.0 / len(pred_text)
normalized_options.append((text, cer, i))
return normalized_options

Expand Down
2 changes: 1 addition & 1 deletion requirements/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
cdifflib
editdistance
inflect
jiwer>2.3.0
joblib
pandas
pynini==2.1.5
Expand Down