Add E2E concrete playback tests

[sanjit-bhat]

See title. This involves adding a new compiletest mode for these kinds of tests.

[sanjit-bhat]

I'm working on this issue right now.

[celinval]

I believe @adpaco-aws added an option to run scripts as part of compiletest: #2102. So we could just write a few e2e tests using this infrastructure. We really need to add tests that cover adding tests in place.

[jaisnan]

#2353 Adds an E2E unit test that also runs the generated unit test to get the output. (The test for generating the unit test is already present in the UI section of the regression tests).