Introduce support for ordering test classes

[ledoyen]

Issue: #1948

Overview

Introduce support for class ordering through the addition of a new configuration parameter:

• junit.jupiter.testclass.order.default accepting a fully-qualified class-name implementing the newly introduced org.junit.jupiter.api.ClassOrderer interface

---

I hereby agree to the terms of the JUnit Contributor License Agreement.

---

Definition of Done

• There are no TODOs left in the code
• Method preconditions are checked and documented in the method's Javadoc
• Coding conventions (e.g. for logging) have been followed
• Change is covered by automated tests including corner cases, errors, and exception handling
• Public API has Javadoc and @API annotations
• Change is documented in the User Guide and Release Notes
• All continuous integration builds pass

[codecov]

Codecov Report

Merging #2488 (82cbc6d) into main (e08ab9e) will decrease coverage by 0.02%.
The diff coverage is 87.70%.

@@             Coverage Diff              @@
##               main    #2488      +/-   ##
============================================
- Coverage     90.56%   90.53%   -0.03%     
- Complexity     4683     4708      +25     
============================================
  Files           407      413       +6     
  Lines         11568    11646      +78     
  Branches        923      924       +1     
============================================
+ Hits          10476    10544      +68     
- Misses          814      824      +10     
  Partials        278      278              

Impacted Files	Coverage Δ	Complexity Δ
...main/java/org/junit/jupiter/api/MethodOrderer.java	100.00% <ø> (ø)	1.00 <0.00> (ø)
.../discovery/AbstractAnnotatedElementDescriptor.java	69.23% <69.23%> (ø)	5.00 <5.00> (?)
...jupiter/engine/discovery/ClassOrderingVisitor.java	70.58% <70.58%> (ø)	6.00 <6.00> (?)
...piter/engine/discovery/DefaultClassDescriptor.java	75.00% <75.00%> (ø)	2.00 <2.00> (?)
.../main/java/org/junit/jupiter/api/ClassOrderer.java	84.37% <84.37%> (ø)	0.00 <0.00> (?)
...ter/engine/config/CachingJupiterConfiguration.java	100.00% <100.00%> (ø)	21.00 <2.00> (+2.00)
...ter/engine/config/DefaultJupiterConfiguration.java	100.00% <100.00%> (ø)	14.00 <1.00> (+1.00)
...iter/engine/discovery/AbstractOrderingVisitor.java	100.00% <100.00%> (ø)	12.00 <12.00> (?)
...r/engine/discovery/DefaultClassOrdererContext.java	100.00% <100.00%> (ø)	3.00 <3.00> (?)
...iter/engine/discovery/DefaultMethodDescriptor.java	75.00% <100.00%> (+13.46%)	2.00 <2.00> (-3.00)	⬆️
... and 9 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e08ab9e...82cbc6d. Read the comment docs.

[marcphilipp]

Looks promising! 👍

[ledoyen]

Thanks for reviewing this so quickly.

Let me know if that is ok so I can squash commits.

Build on JDK16 & 17 was failing, so I rebased the branch on main, but the issue is still there.
It can be a bug in byte-buddy given similar symptoms with

• IllegalArgumentException: Cannot attach undefined variable: T raphw/byte-buddy#117
• https://stackoverflow.com/questions/38622789/bytebuddy-java-lang-illegalargumentexception-cannot-attach-undefined-variable

FYI: Upgrading to mockito 3.6.28 did not fix the issue, but maybe some other library is getting byte-buddy and I am not confortable enough with Gradle to display the test dependency tree, maybe you can see something with the build scan ?

[marcphilipp]

The breakage is most likely unrelated to the changes in this branch. I've upgrade the versions of JDK 16 and 17 in junit-team/build-env@2a4a8a5.

[marcphilipp]

Mockito 3.6.48 also doesn't solve the issue: https://ge.junit.org/s/7nva27dj6tilo/dependencies?dependencies=mockito&expandAll

[marcphilipp]

I reported an issue for Mockito: mockito/mockito#2156

[marcphilipp]

I've reverted back to the working versions of JDK 16 and 17 for now.

[ledoyen]

Just for my knowledge, do you use some extra tool that is not (yet ?) scripted in CI builds to check on line lengths or is it a manual verification ?

[sbrannen]

Just for my knowledge, do you use some extra tool that is not (yet ?) scripted in CI builds to check on line lengths or is it a manual verification ?

We check line lengths manually in documentation (i.e., Asciidoc and Javadoc) and build scripts.

For Java code, we use Spotless.

Changes already made

[sbrannen]

This has been merged into main in b3ce7ae and further refined in 49565e9 and 76d4ca3.

Thanks for all of the hard work you put into this new feature.

We really appreciate it! 👍

[sbrannen]

@ledoyen, if you have a Twitter account, I'd be grateful if you could share it with me (either here or on Twitter: @sam_brannen).

[ledoyen]

@sbrannen you can find me on Twitter @ledoyen_loic but I am not very active on this platform 😉

[marcphilipp]

Thanks, both of you! 🎉

[sbrannen]

@sbrannen you can find me on Twitter @ledoyen_loic but I am not very active on this platform 😉

Here's why I asked: https://twitter.com/sam_brannen/status/1355131508101705734

[mpkorstanje]

@sbrannen this MR is causing the Cross-Version / OpenJDK 17 step to fail for #2416 I think the CI might be leaky.

2021-01-30T14:43:03.5462872Z /__w/junit5/junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/ClassOrderer.java:78: warning: [missing-explicit-ctor] class ClassName in exported package org.junit.jupiter.api declares no explicit constructors, thereby exposing a default constructor to clients of module org.junit.jupiter.api
2021-01-30T14:43:03.5464651Z 	class ClassName implements ClassOrderer {
2021-01-30T14:43:03.5465108Z 	^
2021-01-30T14:43:03.5467020Z /__w/junit5/junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/ClassOrderer.java:98: warning: [missing-explicit-ctor] class DisplayName in exported package org.junit.jupiter.api declares no explicit constructors, thereby exposing a default constructor to clients of module org.junit.jupiter.api
2021-01-30T14:43:03.5468749Z 	class DisplayName implements ClassOrderer {
2021-01-30T14:43:03.5469217Z 	^
2021-01-30T14:43:03.5471373Z /__w/junit5/junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/ClassOrderer.java:128: warning: [missing-explicit-ctor] class OrderAnnotation in exported package org.junit.jupiter.api declares no explicit constructors, thereby exposing a default constructor to clients of module org.junit.jupiter.api
2021-01-30T14:43:03.5473178Z 	class OrderAnnotation implements ClassOrderer {
2021-01-30T14:43:03.5473673Z 	^
2021-01-30T14:43:03.5475541Z /__w/junit5/junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/ClassOrderer.java:165: warning: [missing-explicit-ctor] class Random in exported package org.junit.jupiter.api declares no explicit constructors, thereby exposing a default constructor to clients of module org.junit.jupiter.api
2021-01-30T14:43:03.5477176Z 	class Random implements ClassOrderer {
2021-01-30T14:43:03.5477605Z 	^
2021-01-30T14:43:03.5478158Z error: warnings found and -Werror specified
2021-01-30T14:43:03.5478589Z 1 error
2021-01-30T14:43:03.5478911Z 4 warnings
2021-01-30T14:43:03.5479135Z 
2021-01-30T14:43:03.5479533Z FAILURE: Build failed with an exception.
2021-01-30T14:43:03.5479877Z 
2021-01-30T14:43:03.5480250Z * What went wrong:
2021-01-30T14:43:03.5480942Z Execution failed for task ':junit-jupiter:compileModule'.

[marcphilipp]

Thanks for reporting, I'm on it.

[marcphilipp]

As it turns out, the compileModule tasks were not tracking their inputs correctly. This led to false cache hits. I've fixed it in 31aaac8 and cherry-picked your commit (dc7cab3) to fix the warnings. Sorry for the inconvenience!

[mpkorstanje]

Cheers!