fix: add trtllm build dependency for fault tolerance tests

[nv-tusharma]

Summary

This PR fixes the fault tolerance tests in the nightly CI pipeline by adding the missing build dependency for TRT-LLM.

Changes

• Added build-cuda13-amd64 to fault-tolerance-tests needs array
• Updated BUILD_JOB_PATTERN to include CUDA13 conditional for trtllm
• Aligns with the pattern used by other test jobs (unit-tests, integration-tests, etc.)

Problem

The fault-tolerance tests were failing because:

1. TRT-LLM is only built in build-cuda13-amd64, not build-amd64
2. The build check pattern didn't account for the "CUDA13" suffix in TRT-LLM build job names

Testing

This can be tested by triggering the nightly CI workflow on this branch:

gh workflow run nightly-ci.yml --ref tusharma/fix-nightly-ci-fault-tolerance-tests

Closes OPS-3217

Summary by CodeRabbit

• Chores
  • Expanded continuous integration pipeline to support CUDA 13 testing, enabling validation across additional GPU configurations for enhanced platform compatibility.
  • Improved test workflow dependencies and execution sequencing to ensure test suites run with correct build artifacts in proper order.

[coderabbitai]

Walkthrough

The CI test suite workflow configuration is updated to add CUDA 13 build dependency support. The fault-tolerance-tests job now depends on the CUDA 13 build job, and the build job pattern matching logic is adjusted to handle the CUDA 13 suffix when identifying corresponding build jobs.

Changes

Cohort / File(s)	Summary
CI Workflow Configuration .github/workflows/ci-test-suite.yml	Added build-cuda13-amd64 to fault-tolerance-tests job dependencies and updated BUILD_JOB_PATTERN to include CUDA 13 suffix matching for build job identification.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

A rabbit hops through workflows neat,
With CUDA13 now complete! 🐰✨
Build jobs linked in perfect dance,
Tests flow forth with steady glance,
CI pipelines skip, hop, and prance!

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding a build dependency for TRT-LLM's CUDA13 build to the fault tolerance tests.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description check	✅ Passed	The pull request description follows the required template structure with all key sections present: Overview (Summary), Details (Changes and Problem), and Related Issues (Closes OPS-3217). Content is clear and complete.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[nv-tusharma]

Update: Additional Fix for Operator Image

After the first run, I discovered a second issue - the fault-tolerance tests were trying to use an operator image that doesn't exist.

Root Cause

The nightly CI doesn't build the operator image, but the fault-tolerance tests were trying to use ${github.sha}-operator-amd64 which doesn't exist.

Solution (commit 5cf0867)

Added operator image fallback logic (similar to pr.yaml):

• Try to use nightly-operator-amd64 if available
• Fall back to main-operator (stable image) if not

Testing

Triggered new nightly CI run to validate both fixes:

• Build dependency fix (first commit)
• Operator image fallback (this commit)

Run: https://github.com/ai-dynamo/dynamo/actions/runs/21962861746

[nv-tusharma]

Update: Fixed Registry Mismatch

Issue #3: Registry Mismatch

The operator image check was querying ECR (AWS) but helm was pulling from ACR (Azure), causing deployment failures.

Fix (commit 547ebbb)

• Updated operator image check to query Azure ACR instead of ECR
• Now checking the same registry where helm pulls images from

Testing

Triggered new nightly CI run: https://github.com/ai-dynamo/dynamo/actions/runs/$(gh run list --workflow=nightly-ci.yml --branch tusharma/fix-nightly-ci-fault-tolerance-tests --limit 1 --json databaseId -q '.[0].databaseId')

Follow-up

Created task #6 to refactor operator deployment into reusable composite action (GitHub Actions best practice) after tests are working.

[github-actions]

This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This PR has been closed due to inactivity. If you believe this PR is still relevant, please feel free to reopen it with additional context or information.