chore: Pin YAML pip installs to reduce supply chain risk#8306
Conversation
Unpinned `pip install` commands in `recipes/` and `examples/` allow those jobs to resolve whatever package version happens to be current at execution time. That creates avoidable supply chain exposure: a compromised upstream release, malicious dependency swap, or unexpected breaking update can change runtime behavior without any repo change. This change pins the previously floating package installs in YAML jobs so the resolved artifacts are stable and reviewable. Version selection: - `huggingface_hub==1.11.0`, `huggingface-hub==1.11.0`, `hf_transfer==0.1.9`, and `tokenizers==0.22.2` were pinned to the latest versions available on PyPI at the time of this change. - `aiperf==0.6.0` was pinned to the repo’s existing benchmark baseline already used in `container/deps/requirements.benchmark.txt` and `benchmarks/pyproject.toml`. - `transformers==4.57.3` was pinned in the SGLang perf job to match the repo’s documented compatibility guidance for that stack. - `awscli==1.44.80` was pinned to the current PyPI release used by the example sync jobs. - Existing `git+...@<commit>` installs were left unchanged because they are already pinned to immutable commits. This makes the YAML-based install surface deterministic and reduces the chance of silently pulling unreviewed upstream changes at deploy or benchmark time. Assisted-By: Codex:gpt-5.4
WalkthroughMultiple Kubernetes job and pod manifests were updated to pin Python package versions and adjust Hugging Face transfer environment variables; several YAML files also had trailing-newline/formatting fixes applied. No exported/public code entities were changed. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
recipes/deepseek-v32-fp4/model-cache/model-download.yaml (1)
35-41:⚠️ Potential issue | 🟠 MajorRemove unused
hf_transferdependency andHF_HUB_ENABLE_HF_TRANSFERenvironment variable, or pin tohuggingface_hub0.x.With
huggingface_hub1.x (released October 2025, current v1.11.0), hf_transfer support has been completely removed and hf_xet is now the only high-performance upload/download method. TheHF_HUB_ENABLE_HF_TRANSFER=1environment variable on Line 36 is ignored, and thehf_transfer==0.1.9dependency is unused (adding unnecessary supply chain risk).Choose one approach:
- Drop both
hf_transferand the env var; hf_xet is included by default in huggingface_hub 1.x. Optionally useHF_XET_HIGH_PERFORMANCE=1if performance optimization is needed.- Pin to the last
huggingface_hub0.x version if hf_transfer acceleration is specifically required.Note: huggingface_hub 1.11.0 was released April 16, 2026 (one day before this PR). Confirm it has been smoke-tested in your jobs before fleet-wide deployment.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/deepseek-v32-fp4/model-cache/model-download.yaml` around lines 35 - 41, The YAML currently installs huggingface_hub==1.11.0 and hf_transfer==0.1.9 and sets HF_HUB_ENABLE_HF_TRANSFER=1, but hf_transfer is unused with huggingface_hub 1.x and the env var is ignored; either remove the HF_HUB_ENABLE_HF_TRANSFER env var and drop hf_transfer==0.1.9 from the pip install (optionally add HF_XET_HIGH_PERFORMANCE=1 if high-performance transfer is desired), or instead pin huggingface_hub to the last 0.x release and keep hf_transfer; update the args block where pip install and HF_HUB_ENABLE_HF_TRANSFER are referenced and the pip dependency list (hf_transfer==0.1.9 and huggingface_hub==1.11.0) to implement your chosen approach.recipes/qwen3-vl-30b/model-cache/model-download.yaml (1)
29-37:⚠️ Potential issue | 🟠 Major
hf_transfersupport was removed inhuggingface_hub1.0 — pinning it alongsidehuggingface_hub==1.11.0silently no-ops theHF_HUB_ENABLE_HF_TRANSFER=1acceleration.huggingface_hub 1.0 removed
hf_transferentirely, andhf_xetis now the built-in default for downloads/uploads. Withhuggingface_hub==1.11.0, theHF_HUB_ENABLE_HF_TRANSFER=1env var no longer has any effect, and installinghf_transfer==0.1.9has no runtime impact. Downloads still work via the defaulthf_xetbackend, but the intended acceleration path is lost.This same mismatch exists in all 7 manifests touched by this PR:
recipes/kimi-k2.5/model-cache/baseten/model-download.yaml,recipes/kimi-k2.5/model-cache/nvidia/model-download.yaml,recipes/glm-5-nvfp4/model-cache/model-download.yaml,recipes/llama-3-70b/model-cache/model-download.yaml,recipes/deepseek-r1/model-cache/model-download.yaml,recipes/deepseek-r1/model-cache/model-download-sglang.yaml.Consider one of:
- Remove the
HF_HUB_ENABLE_HF_TRANSFERenv var andhf_transferdependency entirely (lethf_xethandle acceleration automatically, or explicitly pinhuggingface_hub[hf-xet]orhf_xetfor determinism), or- Pin
huggingface_hubto the latest0.xrelease if you needhf_transferspecifically.Option A: remove hf_transfer and env var
- - name: HF_HUB_ENABLE_HF_TRANSFER - value: "1" - name: MODEL_REVISION value: "main" args: - | set -eux - pip install --no-cache-dir huggingface_hub==1.11.0 hf_transfer==0.1.9 + pip install --no-cache-dir huggingface_hub==1.11.0 hf download "$MODEL_NAME" --revision "$MODEL_REVISION"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/qwen3-vl-30b/model-cache/model-download.yaml` around lines 29 - 37, The HF_HUB_ENABLE_HF_TRANSFER env var and installing hf_transfer alongside huggingface_hub==1.11.0 is ineffective because hf_transfer was removed in huggingface_hub 1.0; update the model-download manifest by removing the HF_HUB_ENABLE_HF_TRANSFER entry and drop the pip install hf_transfer==0.1.9 from the args, and either rely on the default hf_xet backend or explicitly pin huggingface_hub[hf-xet] (or hf_xet) in the pip install line so the hf download "$MODEL_NAME" --revision "$MODEL_REVISION" command uses the expected accelerated backend.
♻️ Duplicate comments (5)
recipes/qwen3-32b/model-cache/model-download.yaml (1)
29-37:⚠️ Potential issue | 🟠 MajorSame
hf_transfer/ hub 1.x mismatch.See the comment on
recipes/deepseek-v32-fp4/model-cache/model-download.yaml—hf_transferis no longer wired up inhuggingface_hub1.x, so the pin and env var are dead weight.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/qwen3-32b/model-cache/model-download.yaml` around lines 29 - 37, The HF_HUB_ENABLE_HF_TRANSFER env var and the hf_transfer package pin are obsolete with huggingface_hub 1.x; remove the HF_HUB_ENABLE_HF_TRANSFER environment variable and stop installing hf_transfer in the setup block (replace "pip install ... huggingface_hub==1.11.0 hf_transfer==0.1.9" with only the supported hub package or a compatible hub-only pin), and ensure the download command still uses hf download $MODEL_NAME --revision $MODEL_REVISION (keep MODEL_REVISION) so behavior remains the same without the dead-weight hf_transfer wiring.recipes/nemotron-3-super-fp8/model-cache/model-download.yaml (1)
29-35:⚠️ Potential issue | 🟠 MajorSame
hf_transfer/ hub 1.x mismatch.Same issue and remediation options as in
recipes/deepseek-v32-fp4/model-cache/model-download.yaml.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/nemotron-3-super-fp8/model-cache/model-download.yaml` around lines 29 - 35, The pip install line pins incompatible versions (huggingface_hub==1.11.0 with hf_transfer==0.1.9), causing runtime mismatch; update the pip install invocation to use matching versions (either bump hf_transfer to a release compatible with huggingface_hub==1.11.0 or pin huggingface_hub to the version required by hf_transfer==0.1.9), i.e., edit the pip install in the block containing HF_HUB_ENABLE_HF_TRANSFER and the hf download command so the two packages are version-compatible.recipes/gpt-oss-120b/model-cache/model-download.yaml (1)
29-37:⚠️ Potential issue | 🟠 MajorSame
hf_transfer/ hub 1.x mismatch.Drop
hf_transfer==0.1.9and theHF_HUB_ENABLE_HF_TRANSFERenv var (or pin hub to the last 0.x line). See the comment onrecipes/deepseek-v32-fp4/model-cache/model-download.yaml.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/gpt-oss-120b/model-cache/model-download.yaml` around lines 29 - 37, The deployment mixes hf_transfer==0.1.9 with a 1.x huggingface_hub; either remove hf_transfer and the HF_HUB_ENABLE_HF_TRANSFER env var or instead pin huggingface_hub to the compatible 0.x series. Edit the env block to delete the HF_HUB_ENABLE_HF_TRANSFER entry and modify the pip install line (remove hf_transfer==0.1.9) OR change huggingface_hub==1.11.0 to the last 0.x-compatible version so hf_transfer remains compatible; locate the changes around the HF_HUB_ENABLE_HF_TRANSFER env var and the pip install line that references huggingface_hub and hf_transfer.recipes/qwen3-235b-a22b-fp8/model-cache/model-download.yaml (1)
29-37:⚠️ Potential issue | 🟠 MajorSame
hf_transfer/ hub 1.x mismatch.See the comment on
recipes/deepseek-v32-fp4/model-cache/model-download.yaml.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/qwen3-235b-a22b-fp8/model-cache/model-download.yaml` around lines 29 - 37, The pip install pins are mismatched with hf_transfer; update the install so huggingface_hub and hf_transfer are compatible: change the pip install line that currently pins "huggingface_hub==1.11.0 hf_transfer==0.1.9" to a matching pair (or remove hf_transfer if you intend to use hub 1.11.0 alone), keeping the surrounding env var HF_HUB_ENABLE_HF_TRANSFER and the MODEL_REVISION/hf download steps intact; ensure the updated versions are used in the command that runs "hf download $MODEL_NAME --revision $MODEL_REVISION".recipes/qwen3-32b-fp8/model-cache/model-download.yaml (1)
29-37:⚠️ Potential issue | 🟠 MajorSame
hf_transfer/ hub 1.x mismatch as the deepseek-v32-fp4 download job.
HF_HUB_ENABLE_HF_TRANSFER=1plushf_transfer==0.1.9are effectively dead alongsidehuggingface_hub==1.11.0(hf_xetis the v1.x default). See the comment onrecipes/deepseek-v32-fp4/model-cache/model-download.yamlfor the full rationale and suggested fixes.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/qwen3-32b-fp8/model-cache/model-download.yaml` around lines 29 - 37, The job sets HF_HUB_ENABLE_HF_TRANSFER=1 but installs huggingface_hub==1.11.0 which doesn't enable hf_transfer; fix by aligning the pip install and env var: either remove HF_HUB_ENABLE_HF_TRANSFER and drop hf_transfer from the pip install, or upgrade huggingface_hub to a v1.x that supports hf_transfer (change the pip install in the args line to a matching huggingface_hub version that supports hf_transfer and keep hf_transfer==0.1.9), making sure the hf download $MODEL_NAME --revision $MODEL_REVISION command runs with consistent tooling.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@recipes/deepseek-v32-fp4/model-cache/model-download.yaml`:
- Around line 35-41: The YAML currently installs huggingface_hub==1.11.0 and
hf_transfer==0.1.9 and sets HF_HUB_ENABLE_HF_TRANSFER=1, but hf_transfer is
unused with huggingface_hub 1.x and the env var is ignored; either remove the
HF_HUB_ENABLE_HF_TRANSFER env var and drop hf_transfer==0.1.9 from the pip
install (optionally add HF_XET_HIGH_PERFORMANCE=1 if high-performance transfer
is desired), or instead pin huggingface_hub to the last 0.x release and keep
hf_transfer; update the args block where pip install and
HF_HUB_ENABLE_HF_TRANSFER are referenced and the pip dependency list
(hf_transfer==0.1.9 and huggingface_hub==1.11.0) to implement your chosen
approach.
In `@recipes/qwen3-vl-30b/model-cache/model-download.yaml`:
- Around line 29-37: The HF_HUB_ENABLE_HF_TRANSFER env var and installing
hf_transfer alongside huggingface_hub==1.11.0 is ineffective because hf_transfer
was removed in huggingface_hub 1.0; update the model-download manifest by
removing the HF_HUB_ENABLE_HF_TRANSFER entry and drop the pip install
hf_transfer==0.1.9 from the args, and either rely on the default hf_xet backend
or explicitly pin huggingface_hub[hf-xet] (or hf_xet) in the pip install line so
the hf download "$MODEL_NAME" --revision "$MODEL_REVISION" command uses the
expected accelerated backend.
---
Duplicate comments:
In `@recipes/gpt-oss-120b/model-cache/model-download.yaml`:
- Around line 29-37: The deployment mixes hf_transfer==0.1.9 with a 1.x
huggingface_hub; either remove hf_transfer and the HF_HUB_ENABLE_HF_TRANSFER env
var or instead pin huggingface_hub to the compatible 0.x series. Edit the env
block to delete the HF_HUB_ENABLE_HF_TRANSFER entry and modify the pip install
line (remove hf_transfer==0.1.9) OR change huggingface_hub==1.11.0 to the last
0.x-compatible version so hf_transfer remains compatible; locate the changes
around the HF_HUB_ENABLE_HF_TRANSFER env var and the pip install line that
references huggingface_hub and hf_transfer.
In `@recipes/nemotron-3-super-fp8/model-cache/model-download.yaml`:
- Around line 29-35: The pip install line pins incompatible versions
(huggingface_hub==1.11.0 with hf_transfer==0.1.9), causing runtime mismatch;
update the pip install invocation to use matching versions (either bump
hf_transfer to a release compatible with huggingface_hub==1.11.0 or pin
huggingface_hub to the version required by hf_transfer==0.1.9), i.e., edit the
pip install in the block containing HF_HUB_ENABLE_HF_TRANSFER and the hf
download command so the two packages are version-compatible.
In `@recipes/qwen3-235b-a22b-fp8/model-cache/model-download.yaml`:
- Around line 29-37: The pip install pins are mismatched with hf_transfer;
update the install so huggingface_hub and hf_transfer are compatible: change the
pip install line that currently pins "huggingface_hub==1.11.0
hf_transfer==0.1.9" to a matching pair (or remove hf_transfer if you intend to
use hub 1.11.0 alone), keeping the surrounding env var HF_HUB_ENABLE_HF_TRANSFER
and the MODEL_REVISION/hf download steps intact; ensure the updated versions are
used in the command that runs "hf download $MODEL_NAME --revision
$MODEL_REVISION".
In `@recipes/qwen3-32b-fp8/model-cache/model-download.yaml`:
- Around line 29-37: The job sets HF_HUB_ENABLE_HF_TRANSFER=1 but installs
huggingface_hub==1.11.0 which doesn't enable hf_transfer; fix by aligning the
pip install and env var: either remove HF_HUB_ENABLE_HF_TRANSFER and drop
hf_transfer from the pip install, or upgrade huggingface_hub to a v1.x that
supports hf_transfer (change the pip install in the args line to a matching
huggingface_hub version that supports hf_transfer and keep hf_transfer==0.1.9),
making sure the hf download $MODEL_NAME --revision $MODEL_REVISION command runs
with consistent tooling.
In `@recipes/qwen3-32b/model-cache/model-download.yaml`:
- Around line 29-37: The HF_HUB_ENABLE_HF_TRANSFER env var and the hf_transfer
package pin are obsolete with huggingface_hub 1.x; remove the
HF_HUB_ENABLE_HF_TRANSFER environment variable and stop installing hf_transfer
in the setup block (replace "pip install ... huggingface_hub==1.11.0
hf_transfer==0.1.9" with only the supported hub package or a compatible hub-only
pin), and ensure the download command still uses hf download $MODEL_NAME
--revision $MODEL_REVISION (keep MODEL_REVISION) so behavior remains the same
without the dead-weight hf_transfer wiring.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: dc9a18ee-9103-404d-9b9b-6f955c106eb5
📒 Files selected for processing (21)
examples/backends/vllm/deploy/lora/multimodal/sync-lora-job.yamlexamples/backends/vllm/deploy/lora/sync-lora-job.yamlrecipes/deepseek-r1/model-cache/model-download-sglang.yamlrecipes/deepseek-r1/model-cache/model-download.yamlrecipes/deepseek-r1/trtllm/disagg/wide_ep/gb200/perf.yamlrecipes/deepseek-v32-fp4/model-cache/model-download.yamlrecipes/glm-5-nvfp4/model-cache/model-download.yamlrecipes/glm-5-nvfp4/sglang/disagg/perf.yamlrecipes/gpt-oss-120b/model-cache/model-download.yamlrecipes/kimi-k2.5/model-cache/baseten/model-download.yamlrecipes/kimi-k2.5/model-cache/nvidia/eagle-download.yamlrecipes/kimi-k2.5/model-cache/nvidia/model-download.yamlrecipes/llama-3-70b/model-cache/model-download.yamlrecipes/nemotron-3-super-fp8/model-cache/model-download.yamlrecipes/qwen3-235b-a22b-fp8/model-cache/model-download.yamlrecipes/qwen3-32b-fp8/model-cache/model-download.yamlrecipes/qwen3-32b/model-cache/model-download.yamlrecipes/qwen3-32b/vllm/agg-round-robin/perf.yamlrecipes/qwen3-32b/vllm/disagg-kv-router/perf.yamlrecipes/qwen3-vl-30b/model-cache/model-download.yamlrecipes/qwen3-vl-30b/vllm/agg-embedding-cache/perf.yaml
Signed-off-by: Graham King <grahamk@nvidia.com>
|
@CodeRabbit review |
✅ Actions performedReview triggered.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@recipes/gpt-oss-120b/model-cache/model-download.yaml`:
- Around line 29-36: Summary: Pin huggingface_hub to a safer, older release and
document the env var/backend migration. Change the pip install from
huggingface_hub==1.11.0 to huggingface_hub==1.10.1 to reduce supply-chain risk,
and update the PR/commit message to explicitly mention the functional migration:
HF_HUB_ENABLE_HF_TRANSFER was replaced by HF_XET_HIGH_PERFORMANCE and the
hf_transfer extra was removed in favor of hf_xet; call out that this is a
behavioral change so reviewers and future bisects see it. Ensure references to
HF_XET_HIGH_PERFORMANCE, HF_HUB_ENABLE_HF_TRANSFER, hf_transfer, and the
huggingface_hub version (1.10.1 → 1.11.x when stabilized) are included in the
commit message or PR description.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 10228d65-8a38-4db7-bd6c-2d4e99ed50c9
📒 Files selected for processing (14)
recipes/deepseek-r1/model-cache/model-download-sglang.yamlrecipes/deepseek-r1/model-cache/model-download.yamlrecipes/deepseek-v32-fp4/model-cache/model-download.yamlrecipes/glm-5-nvfp4/model-cache/model-download.yamlrecipes/gpt-oss-120b/model-cache/model-download.yamlrecipes/kimi-k2.5/model-cache/baseten/model-download.yamlrecipes/kimi-k2.5/model-cache/nvidia/eagle-download.yamlrecipes/kimi-k2.5/model-cache/nvidia/model-download.yamlrecipes/llama-3-70b/model-cache/model-download.yamlrecipes/nemotron-3-super-fp8/model-cache/model-download.yamlrecipes/qwen3-235b-a22b-fp8/model-cache/model-download.yamlrecipes/qwen3-32b-fp8/model-cache/model-download.yamlrecipes/qwen3-32b/model-cache/model-download.yamlrecipes/qwen3-vl-30b/model-cache/model-download.yaml
🚧 Files skipped from review as they are similar to previous changes (8)
- recipes/qwen3-32b/model-cache/model-download.yaml
- recipes/qwen3-235b-a22b-fp8/model-cache/model-download.yaml
- recipes/deepseek-v32-fp4/model-cache/model-download.yaml
- recipes/nemotron-3-super-fp8/model-cache/model-download.yaml
- recipes/qwen3-vl-30b/model-cache/model-download.yaml
- recipes/glm-5-nvfp4/model-cache/model-download.yaml
- recipes/kimi-k2.5/model-cache/nvidia/eagle-download.yaml
- recipes/qwen3-32b-fp8/model-cache/model-download.yaml
nnshah1
left a comment
There was a problem hiding this comment.
LGTM incrementally
@nealvaidya what's the plan to maintain this moving forward - is there a way we can centralize this in one spot or a way to reuse a pyproject - maybe a special build of the container .... just concerned with these getting out of date quickly
Signed-off-by: Graham King <grahamk@nvidia.com> Signed-off-by: Indrajit Bhosale <iamindrajitb@gmail.com>
Unpinned
pip installcommands inrecipes/andexamples/allow those jobs to resolve whatever package version happens to be current at execution time. That creates avoidable supply chain exposure: a compromised upstream release, malicious dependency swap, or unexpected breaking update can change runtime behavior without any repo change.This change pins the previously floating package installs in YAML jobs so the resolved artifacts are stable and reviewable.
Version selection:
huggingface_hub==1.11.0,huggingface-hub==1.11.0,hf_transfer==0.1.9, andtokenizers==0.22.2were pinned to the latest versions available on PyPI at the time of this change.aiperf==0.6.0was pinned to the repo’s existing benchmark baseline already used incontainer/deps/requirements.benchmark.txtandbenchmarks/pyproject.toml.transformers==4.57.3was pinned in the SGLang perf job to match the repo’s documented compatibility guidance for that stack.awscli==1.44.80was pinned to the current PyPI release used by the example sync jobs.git+...@<commit>installs were left unchanged because they are already pinned to immutable commits.This makes the YAML-based install surface deterministic and reduces the chance of silently pulling unreviewed upstream changes at deploy or benchmark time.
Assisted-By: Codex:gpt-5.4
Also fixes a performance regression by replacing old go-faster env var
HF_HUB_ENABLE_HF_TRANSFERwith the new one for the new backend:HF_XET_HIGH_PERFORMANCE.Summary by CodeRabbit