Skip to content

Added ConvertToTarredAudioDataset#110

Closed
ssh-meister wants to merge 2 commits intomainfrom
tarred
Closed

Added ConvertToTarredAudioDataset#110
ssh-meister wants to merge 2 commits intomainfrom
tarred

Conversation

@ssh-meister
Copy link
Collaborator

Processor converts an ASR dataset into a tarred format compatible with TarredAudioToTextDataLayer

Signed-off-by: Sasha Meister <sasha.meister.work@gmail.com>
Signed-off-by: Sasha Meister <sasha.meister.work@gmail.com>
Copy link
Collaborator

@lilithgrigoryan lilithgrigoryan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Overall looks good to me. Let's have a unit test covering 2 values of num_workers and num_shards. And minor comments above. Also, please update with main

entries, total_duration, filtered_entries, filtered_duration = self._read_manifest(manifest_path, config)

if len(filtered_entries) > 0:
print(f"Filtered {len(filtered_entries)} files which amounts to {filtered_duration} seconds of audio.")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please avoid prints and use logger instead


cuts = CutSet(LazyNeMoIterator(manifest_path, metadata_only=True))
bins = estimate_duration_buckets(cuts, num_buckets=num_buckets)
print(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, replace prints with logging everywhere

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments