[benchmarking] Adds audio curation benchmark to nightly

[rlratzel]

Adds an audio benchmark to the nightly benchmark suite.

This benchmark is based on the current audio example and adds code to save metadata and results used by the benchmarking framework.

Note to reviewers: this PR depends on features in #1341, so it has been merged with this branch. When #1341 is merged, the diff should only include changes needed for adding the audio benchmark.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This PR adds an audio curation benchmark based on the FLEURS dataset to the nightly benchmark suite. The implementation follows established patterns from existing benchmarks and adds a reusable write_benchmark_results utility function.

Key Changes:

• Added audio_fleurs_benchmark.py script that runs ASR inference on FLEURS dataset and filters by WER threshold
• Added write_benchmark_results utility function in runner/utils.py to standardize result file writing across benchmarks
• Added audio_fleurs configuration entry to nightly-benchmark.yaml with appropriate parameters

Issues Found:

• Copyright year in new script is 2026 instead of 2025 (already noted in previous reviews)

Confidence Score: 4/5

• This PR is safe to merge after fixing the copyright year
• The implementation follows established benchmark patterns and the core logic is sound. The only blocking issue is the incorrect copyright year (2026 vs 2025), which is a trivial fix. The code reuses well-tested components from the tutorial and adds appropriate error handling and result tracking.
• Pay attention to benchmarking/scripts/audio_fleurs_benchmark.py to fix the copyright year

Important Files Changed

Filename	Overview
benchmarking/scripts/audio_fleurs_benchmark.py	New audio benchmark script with copyright year issue (2026 vs 2025) and minor inconsistencies with tutorial
benchmarking/runner/utils.py	Adds write_benchmark_results utility function, clean implementation
benchmarking/nightly-benchmark.yaml	Configuration entry for audio benchmark with appropriate parameters

Sequence Diagram

sequenceDiagram
    participant Main as main()
    participant Run as run_audio_fleurs_benchmark()
    participant Pipeline as Audio Pipeline
    participant Executor as XennaExecutor
    participant Utils as write_benchmark_results()
    participant FS as File System

    Main->>Main: Parse arguments
    Main->>Main: Initialize result_dict
    Main->>Main: Convert paths to Path objects
    Main->>Run: run_audio_fleurs_benchmark(args)
    Run->>FS: Check results_dir exists
    Run->>Executor: Create XennaExecutor
    Run->>Pipeline: Create Pipeline
    Run->>Pipeline: Add CreateInitialManifestFleursStage
    Run->>Pipeline: Add InferenceAsrNemoStage
    Run->>Pipeline: Add GetPairwiseWerStage
    Run->>Pipeline: Add GetAudioDurationStage
    Run->>Pipeline: Add PreserveByValueStage
    Run->>Pipeline: Add AudioToDocumentStage
    Run->>Pipeline: Add JsonlWriter
    Pipeline->>Executor: pipeline.run(executor)
    Executor-->>Pipeline: results (tasks)
    Pipeline-->>Run: tasks
    Run-->>Main: result dict with metrics and tasks
    Main->>Main: Update result_dict
    Main->>Main: Set success_code based on is_success
    Main->>Utils: write_benchmark_results(result_dict, path)
    Utils->>FS: Create output directory
    Utils->>FS: Write params.json
    Utils->>FS: Write metrics.json
    Utils->>FS: Write tasks.pkl
    Utils-->>Main: Done
    Main-->>Main: Return success_code

Loading

[greptile-apps]

_{4 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

The copyright year is set to 2026, which is in the future. This should be 2025 to match the other files in this PR (entry.py, session.py, utils.py all use 2025).

Suggested change

	# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
	# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.

[greptile-apps]

The --gpus argument is defined as type=int, but the original tutorial code in tutorials/audio/fleurs/pipeline.py line 110 uses type=float. This creates an inconsistency.

The Resources class used in line 104 likely expects a float (GPUs can be fractional for resource allocation). Consider using type=float to match the original tutorial:

Suggested change

	parser.add_argument("--gpus", type=int, default=1, help="Number of GPUs to use")
	parser.add_argument("--gpus", type=float, default=1, help="Number of GPUs to use")

[greptile-apps]

The hardcoded _expected_num_results = 50 makes the benchmark fragile. If dataset parameters change (different split, language, or dataset size), the assertions on lines 137 and 139 will fail.

Consider making this configurable or removing the strict assertions in favor of logging the actual number of results, or at least document why exactly 50 results are expected for this specific configuration (dev split, hy_am language).

[greptile-apps]

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{No files reviewed, no comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

syntax: Copyright year should be 2025, not 2026

Suggested change

	# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
	# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.

[greptile-apps]

_{5 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

style: type=int differs from the tutorial's type=float (tutorials/audio/fleurs/pipeline.py:110), though both work with Resources.gpus

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[praateekmahajan]

LGTM!

[greptile-apps]

_{5 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

syntax: Copyright year should be 2025 to match other files in this PR (entry.py, utils.py, session.py all use 2025)

Suggested change

	# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
	# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{3 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

syntax: copyright year should be 2025 to match other files in the repository

Suggested change

	# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
	# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.