`--strict` deprecation was botched - what to do about it

[bluetech]

In pytest 6.2, we deprecated the --strict option. In pytest 8, we purportedly removed it, however I completely botched the removal (1f8b39e), instead of removing --strict I removed its deprecation, effectively un-deprecating it.

I wonder if we should:

1. re-deprecate it and remove in pytest 10

2. remove it, on the idea that it did go through a deprecation period after all

3. go with our original plan which is to make --strict an "open" flag which enables all current and future --strict-* flags (currently --strict-markers and --strict-config). Since --strict currently means --strict-markers this will add --strict-config to --strict and document --strict as a proper feature.

4. other

I'd like to do 3 (and BTW also add ini options for strict and the current strict flags since I see them as project settings not per-run settings...).

(In #13757 I'm thinking it should be another strictness flag but not decided yet)

[Pierre-Sassoulas]

3. sounds like the best plan to me.

[RonnyPfannschmidt]

I like 3 with the option to use a list for it

That way we can eventually replace dozens of strict options with one thats a lists

[bluetech]

I like 3 with the option to use a list for it

Interesting idea but I see a couple of downsides:

• It makes it hard to set strict = true and then turn some specific strict_foo = false. The advantage of strict = true over an explicit list that it automatically enables new strict options when they are added. We can add syntax like strict = ["all", "-strict_foo"] but I think that's undesirable.

• We won't have a :confval: reference for each option.

Therefore I think separate options and a boolean strict, where the specific options override strict, is better.

The model I have is mypy which I think works pretty well. Users who want the "automatically updating strict" behavior use strict, those who don't set the specific options.

[bluetech]

BTW I forgot to mention xfail_strict, IIRC we want it in --strict as well since it's the recommended behavior (@RonnyPfannschmidt can you confirm?)

[RonnyPfannschmidt]

Yes

[nicoddemus]

I agree with option 3, I also like the mypy model: perfect for now projects to use, and easy to override specific options in existing projects. 👍

[The-Compiler]

Agree with option 3 as well (but see below). I think it works well because a) it's a loud failure (rather than a silent one), and b) it should be trivial to adjust (just replace it with --strict-markers in presumably 1-2 easily findable places) if someone still wants the old behavior.

Indeed that should also enable xfail_strict IMHO.

Possible option 4: Add strict = true as an ini option, and make --strict a re-deprecated (or possibly removed) alias for that. Main reasoning being that it doesn't make sense to want strictness for one single pytest run, this strikes me as something you'd typically set once for all runs (and if not, -o strict=true still works). However, I haven't really thought in detail about what would happen to --strict-markers and --strict-config in this scenario, and I suppose it would be good to be consistent between the three.

[bluetech]

Replying to @The-Compiler's comment in #13829 (comment)

If anything, IMHO we should have strict / strict_markers / strict_config / strict_xfail as ini options (see #13823 (comment)) rather than CLI flags, and then it makes sense to make the naming consistent. But if we go there, please let's do it in one step instead of doing the xfail_strict rename/alias in one release and the CLI -> INI move of the rest in another.

My plan is to do the following in one PR:

• Make xfail_strict be an alias to strict_xfail.
• Add the missing strict_* ini flags (strict_config and strict_markers), keep the CLI flags but de-emphasize them in the docs and document them as e.g. "overrides the strict_markers option`.
• Add strict ini option, keeping --strict as CLI version

If we end up with aliasing options, IMHO there's a discussion to be had whether to deprecate the old way. Yes, it causes churn, but so does having two ways to do the same thing (especially for newcomers which might see both in different tutorials, and then they also need to know about which pytest version they use, etc.).

I can implement whatever we decide, my opinion is to leave --strict-config, --strict-markers and --strict alone for backward compatibility and not deprecate them, since I personally give more weight to "don't break/churn things for existing users" over "reduce cognitive overhead for newcomers".

[nicoddemus]

Sounds good to me @bluetech