Fix trust_remote_code local cache collisions for local models (#45632)

[Jeevang1-epic]

Fixes #45632

Summary

This PR fixes local trust_remote_code cache collisions when different local model paths share the same leaf directory name.

What changed

• Updated get_cached_module_file in dynamic_module_utils.py:
  • local cache subdirectory is now keyed by a stable SHA-256 hash of local source file bytes
  • hash includes main module file + direct relative-import source files
  • local/remote branch handling now uses explicit is_local logic (not basename equality)
• Added regression tests in tests/utils/test_dynamic_module_utils.py for:
  • same leaf dir + different source => different cache dirs
  • different paths + identical source => same cache dir
  • same main file + different relative-import source => different cache dirs

Validation

• python -m compileall src/transformers/dynamic_module_utils.py tests/utils/test_dynamic_module_utils.py
• Added regression tests above.
• Manually validated the 3 collision/dedup scenarios via direct get_cached_module_file checks.

Coordination / duplicate-work check

• Coordinated on issue: trust_remote_code cache path collides for local models sharing a leaf directory name #45632
• No overlapping open PR found for this issue at submission time.

Notes

• Documentation update: not needed (bug fix only).
• AI was used to assist drafting; I reviewed the patch and validation results before submission.

[nurpax]

Tried locally and it worked as expected. Added a comment about hexdigest which may be worth considering. Lgtm anyway.

[nurpax]

The full digest leads to pretty long path names now:

find HF_MODULES_CACHE
HF_MODULES_CACHE
HF_MODULES_CACHE/transformers_modules
HF_MODULES_CACHE/transformers_modules/c188095df8cba9168884f43b3c5821a3b820169c90954a33d3eea494952bd409
HF_MODULES_CACHE/transformers_modules/c188095df8cba9168884f43b3c5821a3b820169c90954a33d3eea494952bd409/custom_model.py
HF_MODULES_CACHE/transformers_modules/c188095df8cba9168884f43b3c5821a3b820169c90954a33d3eea494952bd409/__init__.py
HF_MODULES_CACHE/transformers_modules/bf712f2bc7deed84f526285911e4816296f3d92ccf15004b2a9800164f5e8831
HF_MODULES_CACHE/transformers_modules/bf712f2bc7deed84f526285911e4816296f3d92ccf15004b2a9800164f5e8831/custom_model.py
HF_MODULES_CACHE/transformers_modules/bf712f2bc7deed84f526285911e4816296f3d92ccf15004b2a9800164f5e8831/__init__.py
HF_MODULES_CACHE/transformers_modules/__init__.py
HF_MODULES_CACHE/__init__.py

Worth truncating to 16 chars like below?

return source_files_hash.hexdigest()[:16]

Some systems such as Windows are known to have issues with long path names.

[Jeevang1-epic]

Hey @nurpax , glad to hear it's working smoothly on your end! That is a great catch regarding the Windows path length limits—definitely don't want to trigger any OS-level crashes. I just pushed a quick update to truncate the hex digest to 16 characters as suggested. Thanks for testing it out!

[nurpax]

I reviewed the code and tried it locally. Lgtm.

Verified the fix on this branch against the repro in #45632. Three cases:

1. Two local paths with the same leaf name but different source -> separate cache dirs (the original bug, no longer triggers).
2. Two local paths with identical source -> one shared cache dir.
3. Same custom_model.py at two paths but a differing helpers.py pulled in via from .helpers import ... -> separate cache dirs, and each cache dir contains both files.

All three behave as expected. Thanks for the fix.

[Rocketknight1]

Hey! The issue is real, and I agree we should fix it, but I also think the very long hash pathnames clutter the cache directory a bit. How about if we just use os.path.basename(pretrained_model_name_or_path) / hash? Then we could shorten the hash and there should still be no collisions, and the cache directory still has human-readable names.

[nurpax]

@Rocketknight1 Hi! If the original author @Jeevang1-epic is not available to update the PR, I can take a crack at amending the changes re: your feedback.

[Jeevang1-epic]

sorry for delay, I will be updating the pr

[Jeevang1-epic]

Thanks for the review @nurpax and @Rocketknight1 and great suggestion. I updated the PR to use basename/hash for local cache paths, so the directory names stay readable while still keeping the collision-safe hash behavior. I only made the requested incremental change on top of the existing fix and added/updated tests to cover it and is there any other required changes from my side or its fine?

[Rocketknight1]

Yep, LGTM now!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.