File tests #4
Description
Cretonne has unit tests that are run with cargo test. These tests are important, but it is tedious to write Rust code that generates significant IR functions to test certain features.
We need a file-level test mechanism that loads IR from a .cton file and runs a test on the functions contained. This mechanism should be controlled by a cton-util test sub-command. Each test case is a single .cton file containing a test command and a number of functions. The test command determines how the functions should be tested.
This assumes that the
.ctonfile can be parsed correctly. Tests for parser errors should be written in a different way. Probably ascargo testunit tests.
The cton-util test sub-command can be used to run a single test case or a whole directory tree full of test files. When running multiple tests, multi-core machines should be exploited.
Parser roundtrip tests
The test parse-roundtrip command is used to verify the IL parser and serializer. For each function in the file this tests that:
- The function can be parsed without errors.
- Serializing the parsed IR produces the exact same text.
If the file contains two consecutive functions with the same name, the text produced by serializing the first function is compared against the text of the second function. This is used to test certain canonicalizing features of the parser: Comments are removed and whitespace is canonicalized, for example.
This type of test is also used to check that all IR features and attributes are preserved in the serialization roundtrip. The textual compare is rough, but it doesn't miss anything. If we compared the in-memory IR, the comparison function would need constant updating to avoid missing new features.
An IR-text-IR roundtrip test would also be useful. This should focus on preserving things like CFG predecessor list orders and other derived properties not stored directly in the IR. This kind of testing is more difficult because IR just read from a file has less variance than IR coming out of some optimization pass. For example, if a test case runs text-IR-text-IR and compares the CFG predecessor lists in the two IRs, it is unlikely to find any differences sine those CFGs were just computed from the text.
Verifier tests
The test verify-ok command is used to test the IR verifier. For each function in the file this tests that:
- The function can be parsed without errors.
- The function can be verified without errors.
The test verify-fail command is used to test verifier failures. For each function in the file this tests that:
- The function can be parsed without errors.
- Verifying the function fails.
The file may also contain FileCheck-style comments for checking for specific diagnostics.
Compiler pass tests
Syntax TBD, but we want to be able to inject an IR function into a specific pass and run FileCheck-style checks against the output.