Reuse the cache created for latest main on PRs/branches if setup.py is not modified

[ydshieh]

What does this PR do?

For a PR or a branch, if setup.py is not modified (compare to the common ancestor with the main), let's use cache that is created for the setup.py from the latest commit on the main branch.

latest means the latest one on main at the moment where a run is triggered.

Motivation:

• avoid having cache for main and pull (for most cases), which is introduced in Separate CircleCI cache between main and pull or other branches #24886 to avoid unexpected/undesired edge cases.
• avoid the cache creation and store for PRs/branches that still have (very) old setup.py as they don't rebase on more recent main

With this PR, we expect the storage usage for cache could be reduced dramatically 🤞 .

The artifact in this job run shows generated_config.txt has explicit checksum used v0.7-pipelines_torch-main-pip-9RXs1YQ8L2beP4cdAfRDkWX0VRTtWaQodDVKzvyJwPI= rather than {{ checksum "setup.py" }}

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

I will remove this before merge

[sgugger]

Thanks for working on this! Let's try it for a bit and see if it impacts any workflow negatively.

[ydshieh]

need this in create_circleci_config.py to check if tests is in the list

[ydshieh]

We need to check the summary file instead of filtered_test_list as it is already changed to a list in filter_tests method

    if test_files == ["tests"]:
        test_files = [os.path.join("tests", f) for f in os.listdir("tests") if f not in ["__init__.py"] + filters]

[ydshieh]

@sgugger FYI: I need to change 2 more places to make it work correctly for all cases. See my last 2 review comments.

[sgugger]

Works for me!

[ydshieh]

Hi @sgugger

I am afraid I have to revert this PR until we do something to enabling sharing cache (see below at the end). From this section

PRs from the same fork repo share a cache (this includes, as previously stated, that PRs in the main repo share a cache with main).
Two PRs in different fork repos have different caches.

The cache is never shared between PRs/branches from different forks. This might explains the question I posted about why the same cache key could be found sometimes but not other times.

The PR description of #24886 is partially valid as lhoestq created a branch in transformers rather than a PR from his own forked repo.

The way to share cache is

Enabling the sharing of environment variables allows cache sharing between the original repo and all forked builds.

But we need to be careful if we have sensitive env. variables or not.