-
Notifications
You must be signed in to change notification settings - Fork 29
Description
Is your feature request related to a problem? Please describe.
Currently, an understanding of the expected overall behavior of a test scenario from the test scenario documentation is missing information regarding what will/should happen when a check fails. This is important because failed checks sometimes/often serve as flow control to, for instance, stop the test scenario early or even stop the test run early. Instead, this information is provided only in Python when check.record_failed is called.
Describe the solution you'd like
We already have the ability to specify the severity of a failed check in documentation, and that documentation information is already available programmatically. We should migrate all check severity information into documentation and remove the ability to specify severity at runtime. This will clarify and simplify test behavior.
Describe alternatives you've considered
We could leave maximum flexibility to specify severity in documentation or in code or override documentation in code. While that would provide maximum developer flexibility, I am unaware of any value we would gain/retain under that approach and we would lose the clarity and simplicity that would accompany specification of check severity in documentation only.
Execution plan
The record_failed method has wide usage throughout the codebase (~260 instances) and currently severity is a required parameter. Therefore, we will probably need to phase this change in over many PRs. I suspect a strategy like the following would be effective:
- Make
severityan optional parameter torecord_failedand have it default to the documented value when not specified (throwing ValueError if not specified via either means) - Remove
severityparameter fromrecord_failedcalls where severity is specified in documentation - Add severity to check documentation in some subset of the codebase (probably not all at once)
- Repeat steps 2-3 until all specifications of the
severityparameter ofrecord_failedhave been removed - Remove the
severityparameter fromrecord_failed