Create test generators framework

[ErikSchierboom]

Most exercises have a set of canonical data associated with them. This data can change from time to time, so the burden is on us to keep track of it. Wouldn't it be great if we could write a generator to generate the tests automatically based on the canonical data (which is described as JSON document)?

There is also an ongoing effort to update the canonical data to a fixed standard. This should make it possible to write something like a base test generator.

[robkeim]

That would definitely be great @ErikSchierboom, do you have an idea on how you're going to tackle this?

[ErikSchierboom]

I do. There will be a generic parser to parse the canonical data JSON files. Then there will be a generic test file generator, which can setup the structure and such to render the tests. Finally, each exercise will have its own test generator class deriving from the base one. At least, that's how I plan to structure it. The generators themselves will just be part of one .NET Core Console Application.

You can follow my progress here. I aim to use RazorLight to render the test file templates, which will allow using Razor-like syntax in the templates. To verify the correctness, I aim to use the C# Scripting library, which uses Roslyn to parse and verify C# source code.

It is an ambitious project, but part of the reason why I want to do this, is to have to do #195 only once.

[robkeim]

That sounds like a good approach @ErikSchierboom! The reason that I haven't jumped on #195 is because there's currently no way to get updates when things fall out of date again. If you do this work then it should be trivial to update all of the cases whenever any of them change :)

[ErikSchierboom]

@robkeim Precisely!

[ErikSchierboom]

@robkeim I've just submitted PR #228, which adds the basic infrastructure for the test generators. In PR #229 (which is on hold until #228 is merged), you can see what an actual test generator would look like. If you think it looks okay, feel free to merge both PR's :)