More powerful whitelisting system

[blueyed]

A lot of surviving mutants in a project are due to changing constants, e.g. FOO = 1, where FOO itself is used in tests then also.
Killing those would need a test where the value itself is used, which does not make much sense, does it?

I think it would be good if those mutations could be skipped (e.g. by not mutating anything that matches [A-Z_]+ maybe?

It would be good if those could be deselected either by name of the mutation (which would need to be something like mutate-constant-X), or by specifying a pattern of identifiers not to mutate.
(it is not possible to deselect mutations currently, is it?)

[boxed]

This seems like mostly the same as what we discussed in #28 but for different reasons obviously. Maybe one should be able to just plug in an arbitrary python function that gets called to ask for exclusion?

[boxed]

Actually, thinking more about this it would be interesting if you could explain more about your situation. In my experience mutating constants is one of the more useful things for finding lacking tests or even outright bugs. I don't understand your description.

[blueyed]

If you have EXIT_OK = 0 and then check if your program exits with EXIT_OK in some case, the test will also pass with EXIT_OK = 1 (because the constant is used in the test also).
You would need a test that EXIT_OK == 0, or use the number itself in the test. But in this case the constant more or less defines the behavior.

[boxed]

Ah, I see. Well yea, in that case the current system to deal with it is # pragma: no mutate. What do you think about more advanced exclusions like having a callback you can specify in the config?

[nklapste]

Just to chime in for a similar issue. I was experiencing false positives due to mutmut modifying the __version__ within a __init__.py

--- graypy/__init__.py
+++ graypy/__init__.py
@@ -20,5 +20,5 @@
     pass  # amqplib is probably not installed


-__version__ = (0, 3, 1)
+__version__ = None

It would likely be good practice to avoid mutating misc python doc/notes variables e.g.

__author__ = "Software Authors Name"
__copyright__ = "Copyright (C) 2004 Author Name"
__license__ = "Public Domain"
__version__ = "1.0"

[boxed]

Yea, maybe a global and built in whitelist would be a good idea. I have another candidate for that:

__import__('pkg_resources').declare_namespace(__name__)

we always have to put a pragma no mutate on those.

[boxed]

@nklapste I've released a new version of mutmut that ignored a bunch of these by default. It's also got an overhauled cache system that is way better. Give it a try!

[lundybernard]

A related issue that I'm dealing with: I want to skip mutations on log messages.
I normally avoid making assertions about log messages, as it feels more like locking down the code than a behavioral test.

ex:

 log.debug(
     'whatever message you like, maybe its long and'
-     'broken up over multiple lines'
+    'XXbroken up over multiple lines'
 )

# pragma: no mutate isn't a good option at a cost of 19 characters per line

[blueyed]

A more flexible approach, that could live outside the actual source would be nice. E.g. having a callback in the config like mentioned already.
Not sure what information it should get, but e.g. matching regular expressions against (original) code lines would be useful there I think.

[boxed]

For the case @lundybernard has, the line obviously won't do because the mutated string is on a different line than log.debug, so maybe we should just pass the entire multiline statement.

If we allow python code in the config file, then we should probably require (and validate!) that the callback provided has **_ in the parameter list so we can add keyword arguments when we find new uses cases down the line.

[djmattyg007]

I can't help but feel that needing to achieve 100% killed mutants will always be an endless game of whack-a-mole that isn't worth playing. I would prefer a different strategy that accepts the fact that false positives will occur.

Take a look at how the PHP mutation testing framework 'Infection' handles this:

https://infection.github.io/guide/using-with-ci.html

The notion of not being able to achieve 100% killed mutants is baked into the design of the tool. I think this is a much better approach than requiring that everything be whitelisted.