Reassess how "integration test" is used

[GaryJones]

The "integration test" file seems to be mis-used, or at least not up to date.

We already have individual unit tests, that can test for the presence of violations in known "bad" code, and check there are no violations reported for "good" code.

I recently saw @jrfnl (I think) mention that the purpose of an extra file like this was to check that all the sniff classes in the repo were called against a file, without throwing any errors (I think). I can't remember or find the exact wording of this statement, or find any more background research or examples, but I'm pretty sure that the current VIPCS integration test does NOT contain code that triggers each of our sniffs (i.e. recently added sniffs may not be triggered).

I'd like to get to the bottom of the purpose of this file, and find a way to ensure that it serves its correct purpose, and doesn't just be somewhat of a duplicate of unit tests.

cc @david-binda for any historical reference he might have.

[david-binda]

Yeah, so the purpose of the file was to make sure that any sniff being a part of the standard is still a part of the standard. The file should ideally contain a piece of code for every single sniff which we want to have in the standard.

Eg.: if the standard contains 3 sniffs flagging 3 different things, there should be 3 pieces of code in here, triggering all 3 sniffs.

The file was created for help with transition to next major version of PHPCS / WPCS (upstream repositories), as the VIPCS were extending the WPCS/VIP standard. It was able to catch removal of sniff from upstream repository.

I'm really not sure it's clear :) It was meant to be "integration test" for the standard, living alongside the unit tests.

[jrfnl]

@GaryJones I think you are referring to this issue: WordPress/WordPress-Coding-Standards#1546

[GaryJones]

I guess we'll also need a separate (but similar) integration test file for WordPress-VIP-Go as well then, and each test (.inc + .php) can be moved into {standard}/Tests.

[GaryJones]

Also noting that there are completely different approaches to the integration test functionality, between WPCS and VIPCS.

• WPCS runs phpcs against the "clean" test file and checks for no violations being reported. It does the run four times, once for each Standard, against the exact same test file. This only needs to see that at least one token for each sniff is added.
• VIPCS runs the ruleset_test.php file, which internally runs phpcs, against the "dirty" test file, and looks out for a known number of errors and warnings on each line. It adds a code snippet that will specifically trigger each violation.

[GaryJones]

Between #371 and the solution that comes for #375 we've established what the ruleset tests are, and how they work. I think we can make some notes in a contributors doc to consolidate this knowledge, but afterwards, this Issue can be closed.