Running smoke tests in docker

[jiegillet]

I've added a bin/run-tests-in-docker.sh in the CI that runs the analyzer on the exercises in test_data in docker.

I adapted bin/smoke_test.sh and bin/check_files.sh from elixir-test-runner.
They work by running the analyzer in every folder in test_data and comparing the result to expected results (that I added as well).

[angelikatyborska]

Should both the step name and the bin filename mention that those are smoke tests only, not all tests?

[angelikatyborska]

👍 (sanity check: the script fails when either the first or the second stage of the docker build fails, which was the problem Erik wanted to address)

[angelikatyborska]

When I mess up with one of the expected results (e.g. change the emoji in test_data/lasagna/failing_solution/expected_analysis.json) to simulate a failure, running ./bin/run-tests-in-docker.sh still has an exit code 0. I think that means the CI won't really report smoke test failures.

[jiegillet]

Oh, good catch.
I guess using set +e in this way is inappropriate.

[jiegillet]

Sorry for the mess.
It turns out ALL the smoke tests were failing all along, and I didn't notice because

1. The smoke tests were fine on my machine
2. I used set +e which ignored the failures

The reason for failure was that the script didn't have permission to write the output file. I did however have the right to write to tmp so that's what I used.

I'm not too sure what I could have done for reason 1, I'm not too experienced with docker, but I thought the whole point was to have reproducible behavior...
Reason 2 however was a major mistake on my part, lesson learned.

[angelikatyborska]

I think ideally the tests run in Docker would use the same docker run flags as in production, and the same as other tooling in their tests, for example:
https://github.com/exercism/c-test-runner/blob/main/bin/run-tests-in-docker.sh#L21-L24

There's no network, file-read only filesystem, but also mounting a temporary write filesystem to /tmp. If /tmp is the only directory that needs write permissions to make the smoke tests pass, that should work for us.

[jiegillet]

According to Erik, they don't use read-only yet, but according to his next 7 words, they might in the future.
Yes, that config should work, I'll try it tomorrow!

[angelikatyborska]

Perfect! Now we will know in CI if something goes wrong with the Docker container.