JUnit @Rule executes outside of transaction when using the TransactionalTestExecutionListener [SPR-9232]

[spring-projects-issues]

Phil Webb opened SPR-9232 and commented

Status Quo

The order in which JUnit rules are run was changed between Spring 3.0 and 3.1 to match JUnit (see #12361). As a side effect of this, @Rule callbacks (such as the one developed for #11259) are now executed after the callbacks in TestExecutionListeners. This can be problematic if your rule is running within a transaction as the TransactionalTestExecutionListener will perform the rollback before the rule runs. The opposite may also be true: a transaction might not be started before the rule runs.

Current Implementation of SpringJUnit4ClassRunner.methodBlock()

Statement statement = methodInvoker(frameworkMethod, testInstance);
statement = possiblyExpectingExceptions(frameworkMethod, testInstance, statement);
statement = withBefores(frameworkMethod, testInstance, statement);
statement = withAfters(frameworkMethod, testInstance, statement);
statement = withRulesReflectively(frameworkMethod, testInstance, statement);
statement = withPotentialRepeat(frameworkMethod, testInstance, statement);
statement = withPotentialTimeout(frameworkMethod, testInstance, statement);

Goals

Ideally, one could argue that in most circumstances the TransactionalTestExecutionListener should always be the first and last thing to run.

Deliverables

1. Consider changing the SpringJUnit4ClassRunner.methodBlock() method to call TestExecutionListener callbacks outside of JUnit @Before calls, something like:

Statement statement = methodInvoker(frameworkMethod, testInstance);
statement = possiblyExpectingExceptions(frameworkMethod, testInstance, statement);
statement = withTestExecutionListenerBefores(frameworkMethod, testInstance, statement);
statement = withTestExecutionListenerAfters(frameworkMethod, testInstance, statement);
statement = withRulesReflectively(frameworkMethod, testInstance, statement);
statement = withBefores(frameworkMethod, testInstance, statement);
statement = withAfters(frameworkMethod, testInstance, statement);
statement = withPotentialRepeat(frameworkMethod, testInstance, statement);
statement = withPotentialTimeout(frameworkMethod, testInstance, statement);

This would ensure that the before and after callbacks of TransactionalTestExecutionListener (and any TestExecutionListener) get called around the @Rule.

Alternatives

Another option would be to introduce a RuleAwareTestExecutionListener interface that adds beforeRules() and afterRules() methods and have TransactionalTestExecutionListener implement this interface as well (or possibly instead of) TestExecutionListener.

---

Affects: 3.1.1

Issue Links:

• Provide @Rule alternative to SpringJUnit4ClassRunner [SPR-7731] #12387 Provide @Rule alternative to SpringJUnit4ClassRunner ("depends on")
• jUnit 4 and @Rule execution order [SPR-7705] #12361 jUnit 4 and @Rule execution order

4 votes, 6 watchers

[spring-projects-issues]

Dave Syer commented

I think there are probably just as many use cases where a @Rule should run before the transaction starts as after, so it won't help the general case to simply flip the order. So RuleAwareTestExecutionListener seems like the only real option.

I think JUnit changed in 4.9 as well so the order of @Before and custom @Rules is flipped (or was that 4.8)? That it's a mess in JUnit internally is a given The best way round it for us might include a way to specify which rules are before the transaction (e.g. accepting @BeforeTransaction on a @Rule declaration).

[spring-projects-issues]

Phil Webb commented

I see your point, really the problem is not to do with transactions at all but with the order that @Rules run within relation to the TransactionTestExecutionLister. That means that the problem cannot be solved at the TestExecutionListener level but must be pushed back to the rule.

@BeforeTransaction seems like an option but I wonder if might end up spreading complexity around. If you are suggesting:

@Rule
@BeforeTransaction
public SomeRule = new SomeRule()

Then the SpringJUnit4ClassRunner needs to know what @BeforeTransaction rules to run early and to filter them out of the regular @Rules list so that they don't run twice.

Following on from your thought process perhaps @Order might be an option? It could solve a couple of issues in that it would let you specify the general order that @Rules run (not that easy at the moment) and perhaps any order of SpringJUnit4ClassRunner.ORDER_EARLY could run before the TestExecutionListeners

@Rule
@Order(SpringJUnit4ClassRunner.ORDER_EARLY)
public MyTransactionRule rule1 = new MyTransactionRule();

@Rule
@Order(1)
public AnotherRule rule2 = new AnotherRule();

@Rule
@Order(2)
public YetAnotherRule ruleMustRunAfter2 = new YetAnotherRule();

There is a protected method in BlockJUnit4ClassRunner called getTestRules() that might provide a suitable place to hook in the ordering.

[spring-projects-issues]

Sam Brannen commented

Hi guys,

Please keep in mind that the JUnit team designed the original method-level callbacks (e.g., @Before, @After, etc.) completely independent of @Rule. In other words, there is a 100% disconnect -- in terms of execution order, nesting, and wrapping -- between annotated method callbacks and the seemingly analogous callbacks in rules. Furthermore, the method-level callbacks supported by the Spring TestContext Framework (TCF) (e.g., @BeforeTransaction and @AfterTransaction) are indirectly tied to the implementation of SpringJUnit4ClassRunner in terms of life cycle (i.e., when those callbacks are actually executed).

Keeping those points in mind, it follows that:

1. As it stands now (Spring 2.5 - 3.1), @Before, @After, @BeforeTransaction, and @AfterTransaction should be used with SpringJUnit4ClassRunner but should generally not be mixed with @Rule (if execution order, nesting, and wrapping are important).
2. Conversely, if Provide @Rule alternative to SpringJUnit4ClassRunner [SPR-7731] #12387 is implemented there would no longer be a need to use SpringJUnit4ClassRunner, and the restriction not to mix rules and annotated method callbacks would be removed.
3. In other words, if we have 100% @Rule-based integration of the TCF in JUnit, there is no longer a strong requirement to use @Before and @After methods, and the semantics for @Transactional, @BeforeTransaction, and @AfterTransaction should then work consistently with the method-level callbacks supported by @Rule.

Regards,

Sam

[spring-projects-issues]

Sam Brannen commented

This will be looked at in conjunction with #12387.

[spring-projects-issues]

Sam Brannen commented

In light of the fact that #12387 has been resolved, I am closing this issue as "Works as Designed" with the recommendation that developers either use the SpringJUnit4ClassRunner without rules or use the new SpringClassRule and SpringMethodRule with rules.