Sortformer Diarizer 4spk v1 model PR Part 1: models, modules and dataloaders by tango4j · Pull Request #11282 · NVIDIA-NeMo/NeMo

tango4j · 2024-11-14T08:51:55Z

What does this PR do ?

Sortformer Diarizer Model, 4 speaker limit, v1

Sortformer Paper Link

In this PR, we are adding: model files, module files and corresponding dataloader and evalutations.

Collection: ASR/speaker_tasks

Changelog

model files
nemo/collections/asr/models/sortformer_diar_models.py
module files
nemo/collections/asr/modules/sortformer_modules.py
evaluation files
nemo/collections/asr/metrics/der.py
nemo/collections/asr/metrics/multi_binary_acc.py
dataloader files
NeMo/nemo/collections/asr/data/audio_to_diar_label.py
NeMo/nemo/collections/asr/data/audio_to_diar_label_lhotse.py
training yaml
examples/speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml
post-processing yaml files
NeMo/examples/speaker_tasks/diarization/conf/post_processing/sortformer_diar_4spk-v1_callhome-part1.yaml
NeMo/examples/speaker_tasks/diarization/conf/post_processing/sortformer_diar_4spk-v1_dihard-dev.yaml
NeMo/nemo/collections/asr/data/audio_to_diar_label.py
NeMo/nemo/collections/asr/data/audio_to_diar_label_lhotse.py
util files
NeMo/nemo/collections/asr/parts/utils/speaker_utils.py
NeMo/nemo/collections/asr/parts/utils/vad_utils.py
Changed the file names of these yaml files
examples/speaker_tasks/diarization/neural_diarizer/sortformer_diar_train.py
nemo/collections/asr/data/audio_to_diar_label.py
nemo/collections/asr/models/init.py
nemo/collections/asr/modules/sortformer_modules.py
nemo/collections/asr/parts/utils/asr_multispeaker_utils.py
nemo/collections/asr/parts/utils/speaker_utils.py
nemo/collections/asr/parts/utils/vad_utils.py
nemo/collections/common/parts/preprocessing/collections.py

Usage

You can potentially add a usage example below

python ${NEMO_ROOT}/examples/speaker_tasks/diarization/neural_diarizer/e2e_diarize_speech.py \
     model_path=/path/to/diar_sortformer_4spk-v1.nemo \
     dataset_manifest=/path/to/eval_dataset.json

GitHub Actions CI

CI tests will be added in the second PR.
Third PR will include documents and tutorials.

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the ASR and speaker_tasks

Signed-off-by: taejinp <tango4j@gmail.com>

github-advanced-security

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

.../speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml

nemo/collections/asr/data/audio_to_diar_label.py

Signed-off-by: taejinp <tango4j@gmail.com>

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

examples/speaker_tasks/diarization/neural_diarizer/e2e_diarize_speech.py

nemo/collections/asr/data/audio_to_diar_label.py

nemo/collections/asr/models/sortformer_diar_models.py

nemo/collections/asr/parts/utils/asr_multispeaker_utils.py

nemo/collections/asr/parts/utils/speaker_utils.py

nemo/collections/asr/parts/utils/vad_utils.py

nemo/collections/common/parts/preprocessing/collections.py

Signed-off-by: taejinp <tango4j@gmail.com>

…to sortformer/pr_01

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

…to sortformer/pr_01

.../speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml

examples/speaker_tasks/diarization/neural_diarizer/sortformer_diar_train.py

nemo/collections/common/parts/preprocessing/collections.py

Signed-off-by: taejinp <tango4j@gmail.com>

…to sortformer/pr_01

Signed-off-by: taejinp <tango4j@gmail.com>

…to sortformer/pr_01

Signed-off-by: taejinp <tango4j@gmail.com>

…to sortformer/pr_01

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

github-actions · 2024-11-27T01:35:33Z

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

Your code was analyzed with PyLint. The following annotations have been identified:


------------------------------------
Your code has been rated at 10.00/10

Thank you for improving NeMo's documentation!

github-actions · 2024-11-27T01:35:52Z

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

Your code was analyzed with PyLint. The following annotations have been identified:


------------------------------------
Your code has been rated at 10.00/10

Thank you for improving NeMo's documentation!

github-actions · 2024-11-27T01:36:18Z

beep boop 🤖: 🚨 The following files must be fixed before merge!

Your code was analyzed with PyLint. The following annotations have been identified:


------------------------------------
Your code has been rated at 10.00/10

Thank you for improving NeMo's documentation!

nithinraok

Merge it, once it passes CI

…loaders (NVIDIA-NeMo#11282) * Adding the first pr files models and dataset Signed-off-by: taejinp <tango4j@gmail.com> * Tested all unit-test files Signed-off-by: taejinp <tango4j@gmail.com> * Name changes on yaml files and train example Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Reflecting comments and removing unnecessary parts for this PR Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Adding docstrings to reflect the PR comments Signed-off-by: taejinp <tango4j@gmail.com> * removed the unused find_first_nonzero Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Fixed all pylint issues Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Resolving pylint issues Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Removing unused varialbe in audio_to_diar_label.py Signed-off-by: taejinp <tango4j@gmail.com> * Fixed docstrings in training script Signed-off-by: taejinp <tango4j@gmail.com> * Line-too-long issue from Pylint fixed Signed-off-by: taejinp <tango4j@gmail.com> * Adding get_subsegments_scriptable to prevent jit.script error Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Addressed Code-QL issues Signed-off-by: taejinp <tango4j@gmail.com> * Resolved conflicts on bce_loss.py Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Fixed uninit variable issue in bce_loss.py spotted by codeQL Signed-off-by: taejinp <tango4j@gmail.com> * Reflecting PR comments from weiqingw Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Line too long pylint issue resolved in e2e_diarize_speech.py Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Reflecting the comment on Nov 21st 2024. Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Unused variable import time Signed-off-by: taejinp <tango4j@gmail.com> * Adding docstrings to score_labels() function in der.py Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Reflecting comments on YAML files and model file variable changes. Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Added get_subsegments_scriptable for legacy get_subsegment functions Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Resolved line too long pylint issues Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> --------- Signed-off-by: taejinp <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com>

…loaders (NVIDIA-NeMo#11282) * Adding the first pr files models and dataset Signed-off-by: taejinp <tango4j@gmail.com> * Tested all unit-test files Signed-off-by: taejinp <tango4j@gmail.com> * Name changes on yaml files and train example Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Reflecting comments and removing unnecessary parts for this PR Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Adding docstrings to reflect the PR comments Signed-off-by: taejinp <tango4j@gmail.com> * removed the unused find_first_nonzero Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Fixed all pylint issues Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Resolving pylint issues Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Removing unused varialbe in audio_to_diar_label.py Signed-off-by: taejinp <tango4j@gmail.com> * Fixed docstrings in training script Signed-off-by: taejinp <tango4j@gmail.com> * Line-too-long issue from Pylint fixed Signed-off-by: taejinp <tango4j@gmail.com> * Adding get_subsegments_scriptable to prevent jit.script error Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Addressed Code-QL issues Signed-off-by: taejinp <tango4j@gmail.com> * Resolved conflicts on bce_loss.py Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Fixed uninit variable issue in bce_loss.py spotted by codeQL Signed-off-by: taejinp <tango4j@gmail.com> * Reflecting PR comments from weiqingw Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Line too long pylint issue resolved in e2e_diarize_speech.py Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Reflecting the comment on Nov 21st 2024. Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Unused variable import time Signed-off-by: taejinp <tango4j@gmail.com> * Adding docstrings to score_labels() function in der.py Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Reflecting comments on YAML files and model file variable changes. Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Added get_subsegments_scriptable for legacy get_subsegment functions Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> * Resolved line too long pylint issues Signed-off-by: taejinp <tango4j@gmail.com> * Apply isort and black reformatting Signed-off-by: tango4j <tango4j@users.noreply.github.com> --------- Signed-off-by: taejinp <tango4j@gmail.com> Signed-off-by: tango4j <tango4j@users.noreply.github.com> Co-authored-by: tango4j <tango4j@users.noreply.github.com> Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>

tango4j added 3 commits November 13, 2024 18:46

Adding the first pr files models and dataset

e69ec8e

Signed-off-by: taejinp <tango4j@gmail.com>

Tested all unit-test files

2914325

Signed-off-by: taejinp <tango4j@gmail.com>

Name changes on yaml files and train example

9a468ac

Signed-off-by: taejinp <tango4j@gmail.com>

tango4j requested review from nithinraok, stevehuang52 and weiqingw4ng November 14, 2024 08:51

github-actions bot added ASR Speaker Tasks common labels Nov 14, 2024

github-advanced-security bot found potential problems Nov 14, 2024

View reviewed changes

Merge branch 'main' into sortformer/pr_01

a910d30

tango4j marked this pull request as ready for review November 14, 2024 09:07

Apply isort and black reformatting

2f44fe1

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

stevehuang52 reviewed Nov 14, 2024

View reviewed changes

.../speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml Outdated Show resolved Hide resolved

stevehuang52 reviewed Nov 14, 2024

View reviewed changes

.../speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml Show resolved Hide resolved

stevehuang52 reviewed Nov 14, 2024

View reviewed changes

nemo/collections/asr/data/audio_to_diar_label.py Outdated Show resolved Hide resolved

tango4j and others added 3 commits November 14, 2024 16:56

Reflecting comments and removing unnecessary parts for this PR

4ddc59b

Signed-off-by: taejinp <tango4j@gmail.com>

Resolved conflicts

43d95f0

Signed-off-by: taejinp <tango4j@gmail.com>

Apply isort and black reformatting

40e9f95

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

github-advanced-security bot found potential problems Nov 15, 2024

View reviewed changes

tango4j and others added 7 commits November 14, 2024 17:53

Adding docstrings to reflect the PR comments

f7f84bb

Signed-off-by: taejinp <tango4j@gmail.com>

Resolved the new conflict

95acd79

Signed-off-by: taejinp <tango4j@gmail.com>

Merge branch 'main' into sortformer/pr_01

919f4da

removed the unused find_first_nonzero

4134e25

Signed-off-by: taejinp <tango4j@gmail.com>

Merge branch 'sortformer/pr_01' of https://github.com/tango4j/NeMo in…

d3432e5

…to sortformer/pr_01

Apply isort and black reformatting

5dd4d4c

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

Merge branch 'sortformer/pr_01' of https://github.com/tango4j/NeMo in…

ca5eea3

…to sortformer/pr_01

tango4j commented Nov 15, 2024

View reviewed changes

.../speaker_tasks/diarization/conf/neural_diarizer/sortformer_diarizer_hybrid_loss_4spk-v1.yaml Outdated Show resolved Hide resolved

tango4j commented Nov 15, 2024

View reviewed changes

examples/speaker_tasks/diarization/neural_diarizer/sortformer_diar_train.py Show resolved Hide resolved

tango4j commented Nov 15, 2024

View reviewed changes

nemo/collections/common/parts/preprocessing/collections.py Outdated Show resolved Hide resolved

tango4j added 2 commits November 26, 2024 09:09

Added the missing parse_func in preprocessing/collections.py

0eb260e

Signed-off-by: taejinp <tango4j@gmail.com>

Merge branch 'sortformer/pr_01' of https://github.com/tango4j/NeMo in…

895b4ed

…to sortformer/pr_01

tango4j dismissed stevehuang52’s stale review via 895b4ed November 26, 2024 17:10

Adding the missing parse_func in preprocessing/collections.py

37d4240

Signed-off-by: taejinp <tango4j@gmail.com>

tango4j added Run CICD and removed Run CICD labels Nov 26, 2024

tango4j added 2 commits November 26, 2024 09:22

Merge branch 'main' into sortformer/pr_01

01085ab

Merge branch 'main' into sortformer/pr_01

470579d

tango4j added Run CICD and removed Run CICD labels Nov 26, 2024

tango4j added 2 commits November 26, 2024 16:48

Resolved multi_bin_acc and bce_loss issues

73944e3

Signed-off-by: taejinp <tango4j@gmail.com>

Merge branch 'sortformer/pr_01' of https://github.com/tango4j/NeMo in…

ea4c2a7

…to sortformer/pr_01

tango4j added Run CICD and removed Run CICD labels Nov 27, 2024

Merge branch 'main' into sortformer/pr_01

586c64c

tango4j removed the Run CICD label Nov 27, 2024

tango4j and others added 4 commits November 26, 2024 17:33

Resolved line-too-long for msdd_models.py

567f927

Signed-off-by: taejinp <tango4j@gmail.com>

Merge branch 'sortformer/pr_01' of https://github.com/tango4j/NeMo in…

4c4eb1e

…to sortformer/pr_01

Apply isort and black reformatting

b503d3e

Signed-off-by: tango4j <tango4j@users.noreply.github.com>

Merge branch 'main' into sortformer/pr_01

fa2a663

tango4j added the Run CICD label Nov 27, 2024

nithinraok approved these changes Nov 27, 2024

View reviewed changes

tango4j merged commit 505acac into NVIDIA-NeMo:main Nov 27, 2024

stevehuang52 approved these changes Nov 27, 2024

View reviewed changes

tango4j mentioned this pull request Dec 9, 2024

Sortformer Diarizer 4spk v1 model PR Part 3: Speaker Diarization Mixin #11511

Merged

8 tasks

Comments

Conversation

tango4j commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Uh oh!

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 27, 2024

Uh oh!

github-actions bot commented Nov 27, 2024

Uh oh!

github-actions bot commented Nov 27, 2024

Uh oh!

nithinraok left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tango4j commented Nov 14, 2024 •

edited

Loading