Test versioning

[robphoenix]

Has there been any previous discussion about test versioning and/or stub files for xlua?

Is there a stance either way for either of these?

[ryanplusplus]

No real discussion thus far regarding test versioning. I got a little bit of experience with test versioning on the Ruby track, but I didn't really understand how it helped. What was in place (which may have changed) was that each class simply had a version property that indicated which version of the exercise it was developed against. The major issues I saw were that 1) people doing the track likely had no idea what the version property was even for (I didn't) and 2) there wasn't really a way to use the information because exercism.io doesn't provide an obvious way to know what version is current or how the exercise changed between versions. Is there a good way to accomplish this?

As far as stubs, do you mean to provide a skeleton for some exercises to make it easier to get started?

[robphoenix]

We use it in the go track, and are in the process of updating all exercises to have a test version. From the README:

The benefit of all this is that reviewers can see which test version a posted solution was written for and be spared confusion over why an old posted solution might not pass current tests.

And in the hello-world stub file it's explained:

// testVersion identifies the version of the test program that you are
// writing your code to. If the test program changes in the future --
// after you have posted this code to the Exercism site -- reviewers
// will see that your code can't necessarily be expected to pass the
// current test suite because it was written to an earlier test version.

with the expectation that it then just becomes part of the boilerplate.

I ask only because with my recent PR (#191) I would have bumped the test version as it changed the expected solution. And I just wondered if test versioning was something you felt was worth implementing. I don't feel strongly either way to be honest, but would look into how we might achieve it if you felt it would improve things.

And as for stub files, yeah a skeleton file of sorts. It just occurred to me thinking about test versioning. Maybe it's worth it anyway for the hello-world exercise?

[ryanplusplus]

@robphoenix I think a stub for hello-world makes sense. There's a change coming down from x-common that simplifies the hello-world exercise so we should probably add the stub when we make changes for that.

For the test version in Go, do you find that it's easy for people to tell whether the submitted exercise's test version is current? How do you make sure it is easy to close the loop?

[robphoenix]

No, I don't think there is an easy way to see the current test version, as compared to the submitted exercise version, apart from noticing that it is different to one's own test version, if that is the case, or viewing it in the github repo.

I think it's as much a sanity check as anything, especially with significant changes to test suites. The original explanation appears to be this

Some exercises require you to define a TestVersion constant in your package.
There are sometimes problems where tests change and solutions which worked for
the old version no longer work. The TestVersion system makes it obvious when
this happens.

To avoid compilation warnings due to TestVersion not being defined
just add const TestVersion = VERSION (where VERSION should be replaced by the
value of testVersion in the test file) to your submission file.

Should the tests change in a way that old submissions will or could no longer
work the test writer should increase the testVersion value which will cause
the tests to fail with a clear message when an old submission is run with them.

The more I read this line There are sometimes problems where tests change and solutions which worked for the old version no longer work. the more I wonder if this is something that can even occur while someone is working on a solution. Surely even if the test suite changes in the language repo, it's not going to change on the users machine, even if they do an exercism fetch, and so is this even going to be a problem?

Sorry, I'm over thinking this I think, and most likely taking up more of your time rather than easing your maintainer load!

If it's not presented a problem so far, then it's perhaps best to leave it. Maybe the best consideration of its merits is the recent significant change to the hello-world exercise exercism/problem-specifications#544 and when updating that exercise here do you feel it warrants a mechanism for acknowledging and checking the change for new users?

[ryanplusplus]

Ha, no problem.

I think that it just ends up being a partial solution to a mostly non-issue that adds a bit of clutter. Test suites don't auto-update on exercism fetch so it's really only something that will pop up for reviewers. Reviewers generally aren't going to be looking at really old submissions, and without an easy way to know what the current version is or see the test suite that the code was written against I'm not sure what advantage this really brings in practical terms.

Another challenge is that a lot of the Lua exercises don't return tables, just functions. I think it's important to show that not everything has to be a table like a class-based language, so we'd either have to compromise on this, use globals, or use the __call metamethod to implement function-based modules:

return setmetatable({
  test_version = 2
}, {
  __call = function()
    -- function implementation
  end
})

[robphoenix]

Yeah, fair assessment. Plus it doesn't look like a pretty thing to implement. Thanks for going into this for me, I wanted to make sure I wasn't missing something and it's really made me think about the test versioning overall.