Allow extension to fail a test immediately after the test is executed

[ruudk]

I'm trying to create an extension for PHPUnit 10 that runs immediately after the test is executed.

phpunit/src/Framework/TestCase.php

Line 623 in ddf1b95

$this->testResult = $this->runTest();

Currently, only PostConditionFinished can be used for this. But this only works when the TestCase has an assertPostConditions method. When that method is not defined, it will never call PostConditionFinished.

I want to be able to do extra assertions for a library that I'm working on. I don't want to require a trait or abstract TestCase that sets up this assertPostConditions method.

When the assertions fail in my extension, they let the test fail, which is exactly what I want.

The benefit of running the extension immediately after runTest() is that any exception thrown by the extension, will be caught by this try/catch block:

phpunit/src/Framework/TestCase.php

Lines 602 to 628 in ddf1b95

	try {
	$this->checkRequirements();
	$hasMetRequirements = true;
	if ($this->inIsolation) {
	$this->invokeBeforeClassHookMethods($hookMethods, $emitter);
	}
	if (method_exists(static::class, $this->name) &&
	MetadataRegistry::parser()->forMethod(static::class, $this->name)->isDoesNotPerformAssertions()->isNotEmpty()) {
	$this->doesNotPerformAssertions = true;
	}
	$this->invokeBeforeTestHookMethods($hookMethods, $emitter);
	$this->invokePreConditionHookMethods($hookMethods, $emitter);
	$emitter->testPrepared(
	$this->valueObjectForEvents()
	);
	$this->wasPrepared = true;
	$this->testResult = $this->runTest();
	$this->verifyMockObjects();
	$this->invokePostConditionHookMethods($hookMethods, $emitter);
	$this->status = TestStatus::success();

Another solution could be to introduce a new Event that would allow an extension to fail the test after the test was passed. This was also discussed here: phpspec/prophecy-phpunit#45 (comment)

References:

• https://localheinz.com/articles/2023/02/14/extending-phpunit-with-its-new-event-system/
• Add support for Phpunit 10 phpspec/prophecy-phpunit#45 (comment)
• https://twitter.com/Ruud_/status/1627294353151479809

[ruudk]

With such an extension point, Mockery could also run independently from the MockeryPHPUnitIntegration trait. And do mock verification in an extension.

[sebastianbergmann]

The event system is intended to be read-only and with #5219 fixed, exceptions thrown in third-party subscribers no longer affect how tests are run or interpreted.

[ruudk]

Alright. So the only way to provide this functionality is always to use hooks inside the TestCase instead (via trait, abstract classes, etc)?

[sebastianbergmann]

When the assertions fail in my extension, they let the test fail, which is exactly what I want.

This is not a use case the event system supports.

The events emitted by PHPUnit are subscribed to by PHPUnit itself to report progress and test results on the command-line as well as to generate logfiles.

The events emitted by PHPUnit can be subscribed to by third-party extensions for PHPUnit's test runner to implement custom logfile formats, for instance. But the event system cannot be used to change how tests are run or how their result should be interpreted.

TL;DR: Yes, for your use case you have to implement a template method such as assertPostConditions(). You can do that in your test case directly or in an abstract test case class your concrete test case class inherits from. You can also do this in a trait that you use in your test case class.

A test double library such as Mockery needs to do two things: it needs its expectations to be verified after the test has finished and it needs to communicate to PHPUnit the result of this verification making sure that it is correctly interpreted (as a failure, not as an error). For the latter, a clean way was implemented in #5201 for PHPUnit 10.1 that will be used by Prophecy.

[Slamdunk]

I need this very feature too: be able to do assertions in an Extension, for the whole test suite, without adding extra steps on each test.

#5201 seems good only for a tool that extends TestCase.

As suggested in phpspec/prophecy-phpunit#45 (comment), the solution is easy, if you accept to use @internal classes 😉

final class MyListenerAssertingSomething implements \PHPUnit\Runner\Extension\Extension
{
    public function bootstrap(Configuration $configuration, Facade $facade, ParameterCollection $parameters): void
    {
        $facade->registerSubscribers(
            new class($this) implements FinishedSubscriber {
                public function __construct(
                    private readonly MyListenerAssertingSomething $listener
                ) {
                }

                public function notify(Finished $event): void
                {
                    $this->listener->endTest($event);
                }
            },
        );
    }

    public function endTest(Finished $event): void
    {
        // [...]

        if ($everythingsOk) {
            return;
        }

        // Both Facade and Emitter are @internal, but YOLO
        \PHPUnit\Event\Facade::emitter()->testFailed(
            $event->test(),
            ThrowableBuilder::from(new ExpectationFailedException('Something bad happened')),
            null
        );
    }
}