Support Django Test Runs with Coverage

[eleanorjboyd]

make sure Coverage works while running Django and that the scenario is supported.

originally raised in #22827 (comment)

[slank]

The problem occurs because coverage is run in-process using coverage.start() (here), but the test runs under a subprocess and is unobservable to coverage.

Here are some approaches to fixing it.

---

Keep coverage in-process. Also run manage.py in-process.

Three variations on this:

1. Use exec() to run manage.py in the same interpreter. This has a bad security smell but it's hard to say if there's really any difference from ...
2. Import manage.py and run main() (here). Because manage.py is hard-coded to read its arguments from sys.argv, this approach would require manipulation of that variable.
3. Do the stuff manage.py does. Something like this:

from django.core.management import execute_from_command_line
execute_from_command_line(["/path/to/manage.py", "test", ...])

Don't run coverage in-process. Call manage.py as designed

In this case, we'd simply replace [sys.executable] (here) with ["coverage", "run"]. The test execution code would need to be cleaned up.

---

I think all of these have their plusses and minuses. This is my first time here and I'm just working on a weekend project so I don't know the backstory on the current implementation. I like the execute_from_command_line() approach for simplicity.

[mcobalchinisoftfocus]

I read the documentation about testing and it doesn't mention that coverage in Django tests is not yet supported, so I was lost in the dark until I found this issue right here, I think it would be nice if this information was added as a disclaimer

[eleanorjboyd]

great point @mcobalchinisoftfocus! @cwebster-99, could we include this in our docs update?

[danila-grobov]

Hi,

@eleanorjboyd Could you take a look at my PR #24927? I think I've fixed the issue.