ci: cherry-pick #8324 (switch sccache auth to IRSA) to release/1.1.0

[nv-yna]

Summary

Cherry-pick of #8324 (refactor(ci): switch sccache auth to IRSA web identity) from main to release/1.1.0. Single-commit, CI-only change — no runtime behavior change.

Why

PR #8324 merged to main at 2026-04-17T20:54Z and, as part of the switch from static AWS keys to IRSA web-identity auth for sccache S3, effectively revoked the old static keys. Since then, every PR targeting release/1.1.0 that triggers a container or Rust build fails with:

sccache: error: Server startup failed: cache storage failed to read:
  PermissionDenied (permanent) at read => S3Error {
    code: "InvalidAccessKeyId",
    message: "The AWS Access Key Id you provided does not exist in our records."
  }

release/1.0.2 already received this cherry-pick and is unblocked. release/1.1.0 is stuck without it.

Evidence that this cherry-pick fixes the issue

PR #8338 (a combined experimental PR with this cherry-pick + the DYN-2715 trtllm runtime fix from #8297) runs green on release/1.1.0 CI:

• ✅ Build, Rust Checks, Pre Merge — previously failed on fix(trtllm): install pip into runtime venv for NVRTC JIT include discovery #8297 with InvalidAccessKeyId
• ✅ trtllm-runtime / Build multi-arch cuda13.1, trtllm-dev / Build multi-arch cuda13.1
• ✅ vllm-runtime, vllm-dev, sglang-runtime, sglang-dev, planner multi-arch builds — all green

Only remaining failures on #8338 are a pre-existing broken markdown link (lychee 404 on lib/bench/src/bin/README.md, shared across other release/1.1.0 PRs, not introduced by any of these commits) and the PR-title lint on an earlier rename — neither is related to the sccache fix.

Diff: git diff main...origin/release/1.1.0 -- <affected files> produces only the two trivial comment/whitespace-style conflicts that were resolved in favor of the #8324 side:

• .github/actions/docker-remote-build/action.yml: comment block describing the auth method (old: "AWS credentials"; new: "IRSA web identity token").
• container/templates/wheel_builder.Dockerfile: one additional line that deletes config.log / config.status after ffmpeg build (find /tmp/ffmpeg-${FFMPEG_VERSION} \( -name config.log -o -name config.status \) -delete).

Test plan

• Clean cherry-pick (two comment-style conflicts, resolved toward refactor(ci): switch sccache auth to IRSA web identity #8324).
• DCO sign-off.
• End-to-end CI validation on an experimental branch that includes this commit (PR [test] fix(trtllm): DYN-2715 on release/1.1.0 with #8324 IRSA cherry-pick #8338): all container and Rust builds pass.
• CI re-run on this branch to double-verify in isolation (will attach once complete).

Links

• Original PR to main: refactor(ci): switch sccache auth to IRSA web identity #8324
• Combined experimental PR that proved the fix: [test] fix(trtllm): DYN-2715 on release/1.1.0 with #8324 IRSA cherry-pick #8338
• Release branch PRs blocked by this issue (will unblock once this merges): fix(trtllm): install pip into runtime venv for NVRTC JIT include discovery #8297, fix(tests): keep shadows cold before failover quiesce #8325 (Schwinn's shadow-failover backport), plus any future release/1.1.0 PRs that touch container code.

---

[github-actions]

👋 Hi nv-yna! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[devin-ai-integration]

Devin Review found 2 potential issues.

View 3 additional findings in Devin Review.

[devin-ai-integration]

🚩 Rust-checks step still uses static AWS credentials at runtime

The rust-checks job in container-validation-dynamo.yml:182-183 still passes AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables to docker run for sccache inside the running container. This is a runtime (not build-time) usage and is intentionally outside the scope of this PR. However, if the organization fully deprecates these static keys, this step will lose sccache caching silently (the use-sccache.sh script handles this gracefully via its if sccache --start-server guard). Worth tracking as a follow-up migration item.

(Refers to lines 182-183)

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🚩 IRSA token expiration during long builds

The old approach used static IAM keys which never expire. IRSA web identity tokens have a limited lifetime (typically 1 hour, configurable up to 12 hours). The token content is captured at docker buildx build invocation time via --secret id=aws-web-identity-token,src=${TOKEN_FILE} and remains fixed throughout the build. For very long builds (some framework builds take 60+ minutes per build_timeout_minutes), the token could expire mid-build, causing sccache S3 writes to fail. The use-sccache.sh setup-env logic would have already started the server successfully, so failures would occur on individual cache operations rather than at startup. This is an inherent tradeoff of moving to short-lived credentials, and sccache should degrade gracefully (cache misses, not build failures).

---

Was this helpful? React with 👍 or 👎 to provide feedback.