InlineExpectationsTest documentation regarding `.expected` file is misleading

[Marcono1234]

For all languages the InlineExpectationsTest documentation is misleading regarding the .expected file:

Provides a library for writing QL tests whose success or failure is based on expected results embedded in the test source code as comments, rather than a .expected file.

To me that sounds like I can omit the .expected file, but that is not actually the case. When no .expected file is present CodeQL CLI will consider the test to fail because apparently expects some (arbitrary?) result:

0 tests passed; 1 tests failed:
FAILED: path/to/test.ql

failed: path/to/test.ql
Query produced no results.

Therefore it would be good to:

• Improve the InlineExpectationsTest documentation by mentioning that an empty .expected file is needed
• Improve CodeQL CLI to inform the user in the test output that no .expected file was found and therefore the expects the test query to yield some (arbitrary?) results

Or it should be verified, what result CodeQL CLI is actually expecting when no .expected file is present; the CodeQL CLI documentation is also not specific regarding this.

[sauyon]

That absolutely sounds like a useful documentation improvement. Would you like to make a PR for it?

[sauyon]

0 tests passed; 1 tests failed:
FAILED: path/to/test.ql
failed: path/to/test.ql
Query produced no results.

The CodeQL CLI tells me Cannot find ResponseBody.expected file.; where are you getting this result?

[yoff]

An .expected file is always needed, otherwise the test cannot be evaluated and will fail (there is no arbitrary output allowed, but you can see what output you got in the .actual file).

I agree that the CLI documentation is not as clear up-front about the need for an .expected file, as I would have thought. (Formulations such as "You can also define the results you expect to see when you run the query against the example code, by creating a file with the extension .expected.", should perhaps say "need to".)
It does become clear when we get to step 6 of running the first test.

[yoff]

@Marcono1234 does my PR help in reading the InlineExpectationsTest documentation?

[Marcono1234]

The CodeQL CLI tells me Cannot find ResponseBody.expected file.; where are you getting this result?

@sauyon, it appears to be an issue with the VSCode extension (reported it as github/vscode-codeql#774), sorry. When running from command line, the message "Cannot find remote-flow-sink.expected file." does appear indeed.

---

An .expected file is always needed, otherwise the test cannot be evaluated and will fail (there is no arbitrary output allowed, but you can see what output you got in the .actual file).

Thanks for your answer. Though CodeQL CLI behaves, in my opinion, a little bit weird:

1. The JSON output shows that the tests failed at "failureStage" : "RESULT"; shouldn't the missing .expected file already be detected earlier and cause a failure (assuming there are more stages)
   Edit: However, reporting it at this stage has the advantage that the .actual file has already been written, allowing to copy it as .expected file.
2. It is weird that the JSON output has a diff (with "Query produced no results.") when the test could not be executed at all; respectively this message on its own is confusing

[github-actions]

This issue is stale because it has been open 14 days with no activity. Comment or remove the stale label in order to avoid having this issue closed in 7 days.

[smowton]

With #5356 merged and github/vscode-codeql#774 resolved this looks fixed to me

[Marcono1234]

The only remaining question was whether it is intended that CodeQL CLI has "Query produced no results." as diff, but that probably does not matter much anymore since the other issues have been resolved (as mentioned by you).