Add a way to add tests without breaking solvers

[odow]

I think we need a more structured system for adding new tests to MOI.Test.

• Never modify an existing test
  • Unless it is incorrect => bugfix
• Process for adding a new test
  • The current release is v1.1.1.
  • A new test test_new_feature is added
    • Next release will be v1.1.2
    • By default, MOI.Test.runtests does not run this test.
  • v1.2.0 is released.
  • By default, MOI.Test.runtests runs the test warns if it fails, but does not error
  • v1.3.0 is released
  • By default, MOI.Test.runtests runs the test and errors if it fails

We also need a mechanism for solver authors to always run the latest tests (opt-in):

MOI.Test.runtests(model, config; include_new_tests::Bool = false)

This would always run all tests and error on failure.

The alternative is that every new test is a minor release, and tests might break between releases.

Thoughts?

[joaquimg]

Why do you need 1.2.0 to warn and not error?
We would be breaking between minors anyway...
In the proposed framework, the suggestions would be that solvers use tilde specifiers: MathOptIntercace = "~1.2.0"

Another (crazy?) idea would be to explicitly annotate the version of MOI that each test got in... then, when testing, check the project.toml (is it doable?) and only error on tests that got in before the selected version, warn and suggest update otherwise.

[blegat]

Would we be breaking between minors ? Making tests break is not breaking anyway but I agree that it's annoying when you have the tests that suddenly fail after a patch release of MOI. Having the ; include_new_tests::Bool = false new test is a good idea. Needing two minor release to have the test be enabled is maybe a bit too much. Maybe something like new_tests=IGNORE|WARN|ERROR where new tests are those that have been added in patch release of the current minor release.

[odow]

My worry was that if we release 1.1 and then 1.2 in short order, the solvers won't have updated and we'll break things anyway.

Adding new tests is a patch release, and new tests are automatically skipped in patch releases, seems okay.

[odow]

What about a function like

version_added(::typeof(test_function_foo)) = v"0.10.5"

Then solvers could run:

MOI.Test.runtests(model, config; include_tests_before = v"0.10.4")

and explicitly limit their tests to a fixed version. When you update the wrapper to include new features, you also have to update that version.