[hygiene] Track and enforce checks without documented severity #968
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR furthers #404 by adding programmatic enforcement disallowing new checks to be defined without documented severity, and disallowing documented severity from being removed from existing checks. It also tracks progress toward eliminating checks without documented severity. This was prompted by the new bug fixed in #967.
This is accomplished by tracking all current checks without documented severity in checks_without_severity.json.
make lintensures that all checks without documented severity are listed in this file -- if any checks without documented severity exist that aren't listed in that file, they are treated as new bad checks andmake lintfails. When checks without documented severity are removed (by documenting the severity of one or more checks),make formatautomatically updates checks_without_severity.json.make lintdetects when this is necessary and fails with a message to runmake formatwhen necessary.Tested manually via:
make lint-> verify pass with warning about current number of checks without documented severity (622)make format-> verify no changes, verify message that number of checks without documented severity has not changedmake lint-> verify fail with message to runmake formatmake format-> verify changes to checks_without_severity.json, verify message about updatemake lint-> verify passmake lint-> verify fail with message indicating new check with missing severity documentationmake format-> verify no changes, verify warning message about new checks with undocumented severityOnce there are zero entries in checks_without_severity.json, it should be possible to remove the
severityargument fromrecord_failedand thus complete the task in 404. At that point, much of this PR can be reverted/removed, though detection of check documentation without severity will remain permanently.