Cumulative feature and defect updates from recent Transformers PRs

.ai/AGENTS.md

@@ -59,6 +59,22 @@ Do not raise PRs without human validation.
 - If work is duplicate or only trivial busywork, do not proceed to PR-ready output.
 - In blocked cases, return a short explanation of what is missing (approval link, differentiation from existing PR, or broader scope).
 
+## Learning transformers primitives by example
+
+The `src/transformers/cli/agentic/` directory contains concise, self-contained
+examples of how to use the core transformers primitives (`AutoModel`,
+`AutoTokenizer`, `AutoProcessor`, `AutoImageProcessor`, etc.) for a wide
+range of tasks — text classification, NER, QA, summarization, translation,
+image classification, object detection, segmentation, depth estimation,
+speech recognition, audio classification, text-to-speech, video
+classification, visual QA, captioning, OCR, and more.
+
+Each file (`text.py`, `vision.py`, `audio.py`, `multimodal.py`) follows the
+same pattern: load a model and processor with `from_pretrained`, preprocess
+inputs, run a forward pass or `generate`, and post-process the outputs. If
+you need to write code that uses transformers and are unsure how to get
+started, read the relevant command in that folder first.
+
 ## Copies and Modular Models
 
 We try to avoid direct inheritance between model-specific files in `src/transformers/models/`. We have two mechanisms to manage the resulting code duplication:

.circleci/create_circleci_config.py

@@ -398,6 +398,15 @@ def job_name(self):
     parallelism=6,
 )
 
+training_distributed_ci_job = CircleCIJob(
+    "training_distributed_ci",
+    additional_env={"RUN_TRAINING_TESTS": True},
+    docker_image=[{"image": "huggingface/transformers-torch-light"}],
+    install_steps=["uv pip install ."],
+    marker="is_training_distributed_test",
+    parallelism=6,
+)
+
 # We also include a `dummy.py` file in the files to be doc-tested to prevent edge case failure. Otherwise, the pytest
 # hangs forever during test collection while showing `collecting 0 items / 21 errors`. (To see this, we have to remove
 # the bash output redirection.)
@@ -427,7 +436,7 @@ def job_name(self):
 PIPELINE_TESTS = [pipelines_torch_job]
 REPO_UTIL_TESTS = [repo_utils_job]
 DOC_TESTS = [doc_test_job]
-TRAINING_CI_TESTS = [training_ci_job]
+TRAINING_CI_TESTS = [training_ci_job, training_distributed_ci_job]
 TENSOR_PARALLEL_CI_TESTS = [tensor_parallel_ci_job]
 ALL_TESTS = REGULAR_TESTS + EXAMPLES_TESTS + PIPELINE_TESTS + REPO_UTIL_TESTS + DOC_TESTS + [custom_tokenizers_job] + [exotic_models_job] + TRAINING_CI_TESTS + TENSOR_PARALLEL_CI_TESTS  # fmt: skip
 

.github/scripts/assign_reviewers.py

@@ -1,4 +1,3 @@
-# coding=utf-8
 # Copyright 2025 the HuggingFace Inc. team. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");

.github/workflows/model_jobs.yml

@@ -186,7 +186,18 @@ jobs:
         env:
           report_name_prefix: ${{ inputs.report_name_prefix }}
         run: |
-          cat "/transformers/reports/${machine_type}_${report_name_prefix}_${matrix_folders}_test_reports/captured_info.txt"
+          shopt -s nullglob
+          captured_info_files=("/transformers/reports/${machine_type}_${report_name_prefix}_${matrix_folders}_test_reports"/captured_info*.txt)
+
+          if [ ${#captured_info_files[@]} -eq 0 ]; then
+            echo "No captured information files found."
+            exit 0
+          fi
+
+          for captured_info_file in "${captured_info_files[@]}"; do
+            echo "===== ${captured_info_file##*/} ====="
+            cat "$captured_info_file"
+          done
 
       - name: Copy test_outputs.txt
         if: ${{ always() }}

README.md

@@ -134,8 +134,9 @@ pipeline("the secret to baking a really good cake is ")
 To chat with a model, the usage pattern is the same. The only difference is you need to construct a chat history (the input to `Pipeline`) between you and the system.
 
 > [!TIP]
-> You can also chat with a model directly from the command line, as long as [`transformers serve` is running](https://huggingface.co/docs/transformers/main/en/serving).
+> You can also chat with a model directly from the command line, as long as the `chat` extra is installed and [`transformers serve` is running](https://huggingface.co/docs/transformers/main/en/serving).
 > ```shell
+> pip install .[chat]  # or pip install transformers[chat]
 > transformers chat Qwen/Qwen2.5-0.5B-Instruct
 > ```
 

all_requirements.txt

@@ -0,0 +1,98 @@
+gpustat==1.1.1
+psutil==6.0.0
+psycopg2==2.9.9
+pandas>=1.5.0
+numpy>=1.21.0
+psutil>=5.8.0
+nvidia-ml-py>=12.0.0
+torch>=2.0.0
+datasets>=2.10.0
+huggingface_hub>=0.16.0
+amdsmi>=7.0.2
+git+https://github.com/huggingface/transformers.git@main # install main or adjust it with vX.X.X for installing version specific transforms
+datasets==1.8.0accelerate >= 0.12.0
+datasets >= 1.8.0
+torch >= 1.3.0
+evaluateaccelerate >= 0.21.0
+sentencepiece != 0.1.92
+protobuf
+torch >= 1.3
+datasets[audio]>=1.14.0
+evaluate
+librosa
+torchaudio
+torch>=1.6
+accelerate >= 0.12.0
+datasets >= 1.8.0
+sentencepiece != 0.1.92
+protobuf
+sacrebleu >= 1.4.12
+py7zr
+torch >= 1.3
+evaluatedatasets >= 2.0.0
+torch >= 1.3
+accelerate
+evaluate
+Pillow
+albumentations >= 1.4.16
+accelerate >= 0.12.0
+datasets >= 1.8.0
+sentencepiece != 0.1.92
+protobuf
+rouge-score
+nltk
+py7zr
+torch >= 1.3
+evaluate
+torch>=1.5.0
+torchvision>=0.6.0
+datasets>=1.8.0accelerate >= 0.12.0
+datasets >= 1.8.0
+sentencepiece != 0.1.92
+scipy
+scikit-learn
+protobuf
+torch >= 1.3
+evaluateaccelerate>=0.12.0
+torch>=1.5.0
+torchvision>=0.6.0
+datasets>=2.14.0
+evaluate
+scikit-learnaccelerate >= 0.12.0
+torch >= 1.3
+datasets >= 2.14.0
+sentencepiece != 0.1.92
+protobuf
+evaluate
+scikit-learn
+accelerate >= 0.12.0
+seqeval
+datasets >= 1.8.0
+torch >= 1.3
+evaluatealbumentations >= 1.4.16
+timm
+datasets>=4.0
+torchmetrics
+pycocotools
+datasets[audio] >= 1.18.0
+torch >= 1.5
+torchaudio
+librosa
+jiwer
+evaluate
+datasets[audio] >= 1.12.0
+torch >= 1.5
+torchaudio
+accelerate >= 0.12.0
+librosatorch>=1.5.0
+torchvision>=0.6.0
+datasets>=1.8.0albumentations >= 1.4.16
+timm
+datasets
+torchmetrics
+pycocotools
+accelerate >= 0.12.0
+sentencepiece != 0.1.92
+protobuf
+torch >= 1.3
+evaluate