chore(tooling,ci): throttle py3 xdist workers locally, use all cores in CI

[danceratopz]

🗒️ Description

This PR changes the xdist behavior of the py3 and tests_pytest_py3 tox environments such that:

• All physical available cores are used in CI on self-hosted runners (-n auto) instead of being capped via --maxprocesses.
• Only 6 cores are used for local development, by default; overridable via PYTEST_XDIST_AUTO_NUM_WORKERS. This lower default intends to avoid xdist utilizing all cores, which leaves the local workstation unresponsive until the end of the pytest run.

It additionally adds psutil to the dev dependency group in order to accurately detect the number of physical cores, cf https://pytest-xdist.readthedocs.io/en/stable/distribution.html#running-tests-across-multiple-cpus

Results

py3 runs:

• 16 workers, but before chore(tests): use representative blob combos in high blob count forks #2127 (removes 23k tests) and other opts, e.g feat(test-fill): speed up filling #2079, feat(test-fill): performance improvements for release processes and fill #2140:

  ======== 110675 passed, 1603 skipped, 17 warnings in 1917.08s (0:31:57) ========
  

• 16 workers, but after the above optimizations:

  ======== 87957 passed, 1603 skipped, 17 warnings in 1411.02s (0:23:31) =========
  

• 10 workers (before this PR), with the above optimizations:

  ======== 87957 passed, 1603 skipped, 11 warnings in 1853.90s (0:30:53) =========
  

Approach

Replace -n auto (and -n auto --maxprocesses N) with:

-n {env:PYTEST_XDIST_AUTO_NUM_WORKERS:6}

Tox resolves {env:VAR:default} at config-parse time from the host shell. CI workflows set the env var to auto; locally it falls back to 6.

Scenario	Host env	Resolves to
Local (default)	unset	-n 6
Local (override)	PYTEST_XDIST_AUTO_NUM_WORKERS=3	-n 3
CI	PYTEST_XDIST_AUTO_NUM_WORKERS=auto	-n auto

Why not passenv + PYTEST_XDIST_AUTO_NUM_WORKERS directly?

xdist reads PYTEST_XDIST_AUTO_NUM_WORKERS from the child process environment to resolve -n auto to a worker count and requires an integer. Passing auto through passenv would cause xdist to crash on int("auto"). By resolving the env var in tox's own substitution layer, the child process only ever sees -n <int> or -n auto (with the env var absent), so xdist never encounters a non-integer value.

🔗 Related Issues or PRs

N/A.

✅ Checklist

• All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  uvx tox -e static

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Set appropriate labels for the changes (only maintainers can apply labels).

Cute Animal Picture

🦖

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.07%. Comparing base (342c7bc) to head (8cc9f9a).
⚠️ Report is 20 commits behind head on forks/amsterdam.

Additional details and impacted files

@@               Coverage Diff                @@
##           forks/amsterdam    #2120   +/-   ##
================================================
  Coverage            86.07%   86.07%           
================================================
  Files                  599      599           
  Lines                39472    39472           
  Branches              3780     3780           
================================================
  Hits                 33977    33977           
  Misses                4862     4862           
  Partials               633      633           

Flag	Coverage Δ
unittests	86.07% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[SamWilsn]

avoid deadlocks when xdist spawns a worker per core

How does limiting the number of workers avoid a deadlock?

[SamWilsn]

I'm thinking we maybe combine this with #1393? It should be possible with pytest_xdist_auto_num_workers to:

• Read numbers of cores
• Read total system memory
• Determine if CPython or PyPy

And come up with a reasonable number of cores.

[danceratopz]

avoid deadlocks when xdist spawns a worker per core

How does limiting the number of workers avoid a deadlock?

Deadlock was a poor choice of wording! What I mean is: Running -n auto leads to full load, which makes your (local) machine unresponsive until pytest finishes.

Updated the description! :)

[danceratopz]

I'm thinking we maybe combine this with #1393?

I took a look #1393 and commented there: #1393 (comment)

According to this, I've just updated this PR to:

• Add psutil as a dev dep,

It should be possible with pytest_xdist_auto_num_workers to:

* Read numbers of cores

* Read total system memory

* Determine if CPython or PyPy

And come up with a reasonable number of cores.

This is a great idea! This would def help future-proof our setup for CI runner changes. I'd prefer to defer this to a follow-up issue though (tbh, I need to move on in terms of priorities and patience), but happy if someone else picks it up!

I'd just add for the follow-up issue, that this calculation probably won't be set in stone, as new optimizations will impact the memory footprint of generating tests.

For now, this PR means:

• In CI:
  • We accurately detect the number of physical cores and don't increase the count unintentionally to match the number of logical cores.
  • We use the maximum number of physical cores with -n auto; this might be too memory hungry in the future.
• For local development:
  • Defaults to 6.
  • Allows devs to override to their preferred value via the PYTEST_XDIST_AUTO_NUM_WORKERS env var.

[spencer-tb]

LGTM for now! I noticed tests_pytest_py3 has --dist=loadgroup but py3 doesn't

[danceratopz]

Rebased on forks/amsterdam to get the changes from #2116.

LGTM for now! I noticed tests_pytest_py3 has --dist=loadgroup but py3 doesn't

Thanks for flagging! It's the other way round (py3 uses loadgroup). I added an inline comment explaining why for now:
https://github.com/danceratopz/execution-specs/blob/8cc9f9a1f5765f45ccab81accdfaf35cb0c20cee/tox.ini#L90-L95

We'll likely revisit this flag very soon, as we'll likely somehow also enable --dist=worksteal for better balance/distribution across workers.

[danceratopz]

@SamWilsn I made a follow-up issue to not forget your idea [above](https://github.com//pull/2120#issuecomment-3843476312): - https://github.com//issues/2163

---

Ah, #1393 is already exactly this issue. I hadn't understood that from the description! I added your comment to #1393's description to make it clearer.

[fselmo]

It seems like this helps those of us who run these locally without an additional -n flag. I think this opens up some flexibility which is nice. Approving based on the other support here for this feature 👍🏼