Add instructions on how to extract canonical data to the Contributing Guide

[petertseng]

the TODO issues like https://github.com/exercism/todo/issues/100 point us to some examples as guidelines for what should go in a test inputs/outputs json file. It is probably useful to assemble some guide somewhere about that using what we know of our existing json files.

This is prompted by exercism/discussions#10 (comment) where I muse that it could have been good if some non-obvious corner cases had been commented with their purpose. Sure, in many situations the test name is enough, but when it isn't, I'm always for more documentation.

[kotp]

I had noted in this comment how(what appears to be) a comment made it to the test title. That should not happen. If we do need comments, they should be in the json file as "notes" or something similar, rather than the title.

[kytrinyx]

Agreed, it would be really great to have a comprehensive guide (that we could add to). Probably somewhere in the contributing guide.

We haven't quite worked out the format for comments--if you have suggestions for how to improve them, that would be great.

Maybe an optional key "notes" for any test case? That way we can put notes as comments in the test suite if they exist.

[kotp]

Hey! "notes" is a good idea. 😆 I was thinking that messages for the tests may be helpful. The generated tests simply fail at the moment with no clarification of what may be the problem, other than the test title. So we have two kinds of "comments" to consider.

I am not familiar enough with other testing frameworks to know how error messaging is handled, but "test-message" as a key for the test messages when deemed appropriate, and "notes" for commentary notes, such as "explanation for a magic number" may be nice.

[kytrinyx]

We did have messages for some tests, but I found that it was incredibly redundant. I get the benefit of working with both Go and Ruby which have very different approaches to testing. I think we can use the test data to generate messages where necessary (e.g. leap year test suite could have a message that says "case: 1998" or something like that).

I really like the idea of having notes that are meant for the end user, though, for places where it makes sense.

[kotp]

I agree, comments like the ones in food_chain (not generated) are meant to be informative to the learner, and I think they are in the right place. As far as messages... since I have (finally) done a generator class, I think it isn't such a big deal to get values in the messages for the tests. If the tests are actually descriptive enough (Perhaps like a test title that translates to this^#231 pattern: "standard and odd year") we can then use the description in the failure message of the test, perhaps with the value of the offending call and the inputs.

---

#231: A little more descriptive Leap Year test titles.

[kytrinyx]

Agreed - if the description is good, then that combined with inputs/outputs should help dramatically.

[petertseng]

Notes: In #289 there is a question on whether there should be any facility for marking tests as primarily testing performance so that tracks that auto-generate can read that and react accordingly (if they want to treat performance tests differently)

Other musings I had were: Should there be a difference between comments that need only be visible to the track maintainers/anyone who want to port the exerciser to a new language) versus those that should directly be added to auto-generated test files so that student can see them?

[kytrinyx]

Quick note: this might be helpful: http://x.exercism.io/v3/problems/data-todos

[IanWhitney]

My general strategy:

1. Look at the list of the tracks that currently implement the problem. I use http://x.exercism.io/problems/:problem-name
2. Open up a bunch of tabs, 1 per language that implements the problem, to http://exercism.io/exercises/:language-name/:problem-name
3. Scan the test suites to find Families and Outliers.
   • Usually a problem has a Family of common set of tests that started off in one language and have been copied into a bunch of other languages, sometimes with minor modifications. If you're lucky, the problem only has one Family.
   • Problems usually have one Outlier language (Looking at you Go, and sometimes Haskell) that have decided to forge their own path and write a brand new test suite from scratch.
4. From that survey, compile a set of tests that:
   • Covers the described problem thoroughly enough (note: not thoroughly, just enough)
   • Generally represents the tests that are currently in Production, because asking every track maintainer to fully rewrite their test suite is mean.
5. Determine test order. General rules for that:
   • Put the simplest test first. Get students to a green test ASAP.
     • Custom Set, for example frequently put one of the hardest tests first.
   • Introduce 1 new wrinkle per test
   • Try not to have tests that pass without any code changes. If the test already passes, why is it there?
   • Try to avoid tests that exist to check performance. Or clearly mark them as optional. These tests are unit tests and should report if the code works or does not work. Performant code is great, but those tests belong in an acceptance suite.

I gots lots of opinions on this, apparently. Take or leave as you see fit.