ci: create a new library compatibility test suite

[wjones127]

This adds a new compat test suite that replaces the forwards_compat suite. Key changes:

1. It tests both forwards and backwards compatibility.
2. It adds the most recent stable and beta release as comparison targets. Individual tests can also choose which versions to test against.
3. It can be run easily locally via a single command make compattest

Closes #4416

How it works

Tests are written like:

@compat_test()
class MyTest(UpgradeDowngradeTest):
    def __init__(self, path: Path):
        self.path = path

    def create(self):
        # write initial data

    def check_read(self):
        # test reading data

    def check_write(self):
        # test writing data

Then two tests will be generated:

1. downgrade: We call create in current version, then check_read and check_write in old version.
2. upgrade_downgrade: We call create in old version, then check_read and check_write in current version, then go back and call check_read and check_write in old version again.

The way we execute on old versions of the library is using a persistent executor process within special virtual environment. We create one virtual env per old version, and one executor process per virtual env. The executor process receives a tuple (TestObject, method_name) (for example, (MyTest, 'create')), which is serialized via pickle. The executor process runs the method, and returns back a status indicating if there were any errors.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[westonpace]

A few minor nits. I suppose one question would be why is this approach better than the old CI-based one?

[westonpace]

Why wrap the method in an object? Why not just pickle the method itself? Then you could use __name__ to get the method name for logging.

[wjones127]

I suppose there's two ways you could go here:

1. (What I did): Put all the parameters (such as the temp directory) in the class attributes. Then you can just pickle the object and call it's method without having to pass them.
2. Pickle the function and the parameters separately, and pass the parameters in on the other side.

It doesn't seem like pickle works with closures. So I couldn't just create a closure with the parameters passed in, pickle it, and run it on the other side.

I felt like option (1) was a simpler way to implement this. Plus I kind of liked using the class to organize the methods into a logical grouping.

[westonpace]

Makes sense to me

[westonpace]

Looks like we still have a hard-coded list of versions. At one point we had discussed using "the latest X major releases" Is that still a follow-up or do we want this approach for some reason?

[wjones127]

We do the last stable and last beta release. We could expand that further.

Separately, there's the issue of whether we want to hard code versions. Some of the hardcoded ones here are just representing the first time we have compatibility working for that feature. I think at least that version is worth hard-coding in.

We might be able to take some of these versions out in the middle. I think some are there for no particular reason. But it's possible we might want to record some specific version in there if there was a risky compatibility change that happened in that version.

[wjones127]

Thinking about it, I think what I'd like to do is for all test, have it test against:

• Last stable release
• Last preview release
• Last 2 major releases

I can make that change.

[wjones127]

I suppose one question would be why is this approach better than the old CI-based one?

Main advantage is that anyone can run this locally with one command: make compattest. Before they had to run a sequence of steps.