Feature/parallelize model download

[RituAddepalli]

This PR modifies the cached_files() function in hub.py to download multiple files (like sharded model weights) in parallel instead of sequentially.This change safely parallelizes multi-file downloads without altering any existing logic. Single-file downloads remain unchanged, and all files are still resolved to the same cache locations. This improves performance without introducing functional risks.

Details:

Before:
Single file → uses hf_hub_download().
Multiple files → uses snapshot_download(), downloading files sequentially.

After (this PR):
Single file → still uses hf_hub_download().
Multiple files → each file is downloaded in parallel using concurrent.futures.ThreadPoolExecutor with hf_hub_download().
This improves download speed for multiple files without changing any other functionality.

Example snippet of the change:
from concurrent.futures import ThreadPoolExecutor
def cached_files(...):
...
if len(full_filenames) == 1:
return [hf_hub_download(...)]
else:
resolved_files = []

def download_file(file):
    return hf_hub_download(
        path_or_repo_id,
        file,
        subfolder=None if len(subfolder) == 0 else subfolder,
        repo_type=repo_type,
        revision=revision,
        cache_dir=cache_dir,
        user_agent=user_agent,
        force_download=force_download,
        proxies=proxies,
        token=token,
        local_files_only=local_files_only,
    )

with ThreadPoolExecutor(max_workers=min(8, len(full_filenames))) as executor:
    for f in executor.map(download_file, full_filenames):
        resolved_files.append(f)

return resolved_files

Impact:

No functional changes for single-file downloads.
Multi-file downloads are faster due to parallelization.

Testing Done

Test 1: test_parallel_download.py
3 files (config, vocab, model) start downloading at the same time
Verified parallel start timestamps
output:
Directory: C:\gitcontributions\transformers

Mode LastWriteTime Length Name

d----- 04-12-2025 22:00 hf_cache_parallel_test2
Starting download: bert-base-uncased/config.json at 22:00:13
Starting download: bert-base-uncased/vocab.txt at 22:00:13
Starting download: bert-base-uncased/pytorch_model.bin at 22:00:13

Parallel download completed in 0.01 seconds

Test 2: test_parallel_multi_models.py

Downloaded 3 models in parallel:
bert-base-uncased
distilbert-base-uncased
roberta-base
All resolved files appeared in correct HF cache structure.
output:

Conclusion:

In this PR, we partially addressed the issue:

Implemented parallel downloads for multiple files in cached_files() using ThreadPoolExecutor with hf_hub_download().
Single-file downloads and actual weight-loading integration remain unchanged.
While this does not fully implement parallel weight loading, it demonstrates the approach and improves download speed for multiple files.
Value of this partial work:

Acts as a proof of concept for parallel downloads.
Shows performance gains for multi-file downloads, which is relevant to sharded weights.
Provides a foundation for integrating full parallel weight loading in future updates.
This PR is therefore helpful for demonstrating the technique, even if it does not completely resolve the original issue.
Request for Review
Tagging:
@huggingface/transformers-maintainers @Cyrilvallez

Please review when convenient — happy to improve or modify any part!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto