[TTS] Create script for processing TTS training audio

[rlangman]

Signed-off-by: Ryan rlangman@nvidia.com

What does this PR do ?

Add a script for doing audio preprocessing for TTS training.

Collection: [TTS]

Changelog

• Created a "data" directory under nemo.collections.tts following the same convention as the ASR & NLP code.
• Added data_util class for generic operations like processing training manifest. Not sure of volume normalization should be there, but currently have no better place to put it.
• Added silence trimming implementations using energy and VAD based approaches.
• Added audio_preprocessing script for running the data processing logic over a training manifest, with example config files.

Usage

HYDRA_FULL_ERROR=1 python /home/rlangman/Code/NeMo/scripts/dataset_processing/tts/audio_processing/preprocess_audio.py \
    --config-path=/home/rlangman/Code/NeMo/scripts/dataset_processing/tts/audio_processing/config \
	--config-name=preprocessing.yaml \
	data_base_dir="/home/rlangman/Data/nemo_processing" \
	config.num_workers=2 \
	config.max_entries=100 \
	trim=vad \
	trim.device=cuda

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

Additional Information

I manually tested the silence trimming using the openSLR Spanish data. For the 2 modes the dataset size decreases from:

Energy: 35 hours -> 28.8 hours
VAD: 35 hours -> 22.6 hours

This is because lots of the audio has noise in the long silences at the start and end of files. The VAD can differentiate the noise from speech while energy cannot.

Doing trimming on my workstation with the ~24k utterance dataset takes about 5 minutes with energy and 30 minutes with VAD.

More automated testing might be possible if we add CI tests that use actual audio, but I would leave that for a future PR as we do not yet have guidelines for TTS CI tests.

[redoctopus]

Thanks for adding this!

[XuesongYang]

LGTM. Added some nitpicks and discussion.

It seems that this PR support either energy-based or vad-based trimming solution. For entergy-based method, the classic way is using two measures of energy and zero-crossing rate (see ref https://web.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints/078_endpoint%20detection.pdf). We are good to ship energy only method, but we can add ZCR in future.

another thought, is it possible to make a hybrid solution with both energy and VAD?

[rlangman]

It seems that this PR support either energy-based or vad-based trimming solution. For entergy-based method, the classic way > is using two measures of energy and zero-crossing rate (see ref https://web.ece.ucsb.edu/Faculty/Rabiner/ece259/Reprints /078_endpoint%20detection.pdf). We are good to ship energy only method, but we can add ZCR in future.

another thought, is it possible to make a hybrid solution with both energy and VAD?

The intention is to merge a prototype/skeleton implementation here and then improve on the algorithm(s) over time. ZCR could definitely be useful.

A hybrid solution with both energy and VAD also sounds like a good idea. I will need to think about how to refactor the code to support such a mode. For those 2 you can simply take the boolean speech frame array and do an AND with the two outputs, but that might not be as applicable for other approaches.

[redoctopus]

@rlangman what's the status of this PR?

[rlangman]

@rlangman what's the status of this PR?

I will update the PR next week. In addition to addressing the PR comments, I talked with Fei and am seeing if I can make the VAD config more consistent with what it was trained on and potentially reuse some of the code from the VAD post-processor.

[lgtm-com]

This pull request introduces 1 alert when merging a895e9f into 9838b3b - view on LGTM.com

new alerts:

• 1 for Unused import

[redoctopus]

LGTM. @XuesongYang to check over his comments before approval

[XuesongYang]

LGTM. thanks.!

[XuesongYang]

no need to bother. Let's keep as-is.

[XuesongYang]

sounds good!