Skip to content

[Audio] Metric with Squim objective and MOS#9751

Merged
anteju merged 2 commits intoNVIDIA-NeMo:mainfrom
anteju:pr/audio-metric-squim
Jul 23, 2024
Merged

[Audio] Metric with Squim objective and MOS#9751
anteju merged 2 commits intoNVIDIA-NeMo:mainfrom
anteju:pr/audio-metric-squim

Conversation

@anteju
Copy link
Collaborator

@anteju anteju commented Jul 16, 2024

What does this PR do ?

Add a Metric object for calculating Squim MOS.

Collection: audio

Changelog

  • Added a metric wrapper for Squim objective and MOS models
  • Added unit tests
  • Fixed a bug in transcribe_utils

Usage

For example, in model config:

...
model:
...
  metrics:
    val:
      squim_mos: # Squim MOS
        _target_: nemo.collections.audio.metrics.SquimMOSMetric
        fs: ${model.sample_rate}
      squim_stoi: # Squim STOI
        _target_: nemo.collections.audio.metrics.SquimObjectiveMetric
        metric: stoi
        fs: ${model.sample_rate}
...

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you add or update any necessary documentation?
  • Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
    • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

  • New Feature
  • Bugfix
  • Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

  • Related to # (issue)

@anteju anteju force-pushed the pr/audio-metric-squim branch 2 times, most recently from daf237a to 4166145 Compare July 16, 2024 23:41
@anteju anteju changed the title [Draft] Metric for Squim MOS [Draft] Metric with Squim objective and MOS Jul 16, 2024
@anteju anteju force-pushed the pr/audio-metric-squim branch from 4166145 to b258ba4 Compare July 17, 2024 00:02
@anteju anteju marked this pull request as ready for review July 17, 2024 00:06
@anteju anteju added Run CICD and removed Run CICD labels Jul 17, 2024
@anteju anteju changed the title [Draft] Metric with Squim objective and MOS [Audio] Metric with Squim objective and MOS Jul 17, 2024
@anteju anteju force-pushed the pr/audio-metric-squim branch 2 times, most recently from f09334f to d500fa6 Compare July 17, 2024 20:24
@anteju anteju force-pushed the pr/audio-metric-squim branch from 12db5f2 to bbc24a2 Compare July 17, 2024 22:15
@anteju anteju added Run CICD and removed Run CICD labels Jul 17, 2024
@anteju anteju force-pushed the pr/audio-metric-squim branch from bbc24a2 to 89a9c5b Compare July 17, 2024 22:15
@anteju anteju added Run CICD and removed Run CICD labels Jul 17, 2024
@anteju anteju requested a review from galv July 17, 2024 22:16
@anteju anteju force-pushed the pr/audio-metric-squim branch 2 times, most recently from d467328 to b564001 Compare July 18, 2024 00:48
@anteju anteju added Run CICD and removed Run CICD labels Jul 18, 2024
@anteju anteju requested a review from racoiaws July 18, 2024 15:21
@anteju anteju force-pushed the pr/audio-metric-squim branch from b564001 to ea4997f Compare July 19, 2024 05:32
@anteju anteju requested a review from racoiaws July 19, 2024 05:32
@anteju anteju added Run CICD and removed Run CICD labels Jul 19, 2024
@anteju anteju force-pushed the pr/audio-metric-squim branch from 2798b8d to f8a227b Compare July 22, 2024 23:21
@anteju anteju added Run CICD and removed Run CICD labels Jul 22, 2024
Copy link
Collaborator

@racoiaws racoiaws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

anteju added 2 commits July 23, 2024 09:06
Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: Ante Jukić <ajukic@nvidia.com>
@anteju anteju force-pushed the pr/audio-metric-squim branch from 35a3aad to 6072ac7 Compare July 23, 2024 16:07
@anteju anteju added Run CICD and removed Run CICD labels Jul 23, 2024
@anteju anteju merged commit 1711334 into NVIDIA-NeMo:main Jul 23, 2024
tonyjie pushed a commit to tonyjie/NeMo that referenced this pull request Jul 24, 2024
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
akoumpa pushed a commit that referenced this pull request Jul 25, 2024
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>
BoxiangW pushed a commit to BoxiangW/NeMo that referenced this pull request Jul 30, 2024
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: Boxiang Wang <boxiangw@nvidia.com>
xuanzic pushed a commit to xuanzic/NeMo that referenced this pull request Aug 1, 2024
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: Vivian Chen <xuanzic@example.com>
kchike pushed a commit to kchike/NeMo that referenced this pull request Aug 8, 2024
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: kchike <kohei.chike@jp.ricoh.com>
monica-sekoyan pushed a commit that referenced this pull request Oct 14, 2024
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
hainan-xv pushed a commit to hainan-xv/NeMo that referenced this pull request Nov 5, 2024
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Signed-off-by: Hainan Xu <hainanx@nvidia.com>
XuesongYang pushed a commit to paarthneekhara/NeMo that referenced this pull request Jan 18, 2025
* Metric with Squim Objective and MOS

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

* Removed utility functions

Signed-off-by: Ante Jukić <ajukic@nvidia.com>

---------

Signed-off-by: Ante Jukić <ajukic@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments