Support outdir created at runtime

[atravitz]

Resolves #986 (comment)

Checklist

• Added a news entry

Developers certificate of origin

• I certify that this contribution is covered by the MIT License here and the Developer Certificate of Origin at https://developercertificate.org/.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.74%. Comparing base (2eb0ec8) to head (29bb619).
Report is 143 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1072      +/-   ##
==========================================
- Coverage   94.47%   92.74%   -1.73%     
==========================================
  Files         135      135              
  Lines       10092    10090       -2     
==========================================
- Hits         9534     9358     -176     
- Misses        558      732     +174     

Flag	Coverage Δ
fast-tests	92.74% <100.00%> (?)
slow-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mikemhenry]

LGTM!

Edit: sorry changed my mind!

[mikemhenry]

Since this does fix a bug, can you add a change log entry saying that the bug is fixed?
And delete the changelog entry that contradicts the behavior implemented in this PR?

[github-actions]

No API break detected ✅

[mikemhenry]

LGTM -- Thanks!