Too many tests per release

[emirkmo]

I want to move to making more frequent releases, but there are 5 rounds of the same CI tests being run per release with the current workflow. Need to cut down on that using GitHub actions.

[emirkmo]

I think step 1 here is to remove the redundant py 3.9 tests. As this is a pipeline and not a library, we are enforcing dependency locking (somewhat), so it's not unreasonable to also lock python to 3.10+