MINOR: rat should depend on processMessages task

[dajac]

This fix the following issue that we occasionally see in builds.

[2023-06-14T11:41:50.769Z] * What went wrong:
[2023-06-14T11:41:50.769Z] A problem was found with the configuration of task ':rat' (type 'RatTask').
[2023-06-14T11:41:50.769Z]   - Gradle detected a problem with the following location: '/home/jenkins/jenkins-agent/workspace/Kafka_kafka-pr_PR-13848'.
[2023-06-14T11:41:50.769Z]     
[2023-06-14T11:41:50.769Z]     Reason: Task ':rat' uses this output of task ':clients:processTestMessages' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
[2023-06-14T11:41:50.769Z]     
[2023-06-14T11:41:50.769Z]     Possible solutions:
[2023-06-14T11:41:50.769Z]       1. Declare task ':clients:processTestMessages' as an input of ':rat'.
[2023-06-14T11:41:50.769Z]       2. Declare an explicit dependency on ':clients:processTestMessages' from ':rat' using Task#dependsOn.
[2023-06-14T11:41:50.769Z]       3. Declare an explicit dependency on ':clients:processTestMessages' from ':rat' using Task#mustRunAfter.
[2023-06-14T11:41:50.769Z]     
[2023-06-14T11:41:50.769Z]     Please refer to https://docs.gradle.org/8.1.1/userguide/validation_problems.html#implicit_dependency for more details about this problem.

Validated manually as well:

% ./gradlew rat

> Configure project :
Starting build with version 3.6.0-SNAPSHOT (commit id 874081ca) using Gradle 8.1.1, Java 17 and Scala 2.13.10
Build properties: maxParallelForks=10, maxScalacThreads=8, maxTestRetries=0

> Task :storage:processMessages
MessageGenerator: processed 4 Kafka message JSON files(s).

> Task :raft:processMessages
MessageGenerator: processed 1 Kafka message JSON files(s).

> Task :core:processMessages
MessageGenerator: processed 2 Kafka message JSON files(s).

> Task :group-coordinator:processMessages
MessageGenerator: processed 16 Kafka message JSON files(s).

> Task :streams:processMessages
MessageGenerator: processed 1 Kafka message JSON files(s).

> Task :metadata:processMessages
MessageGenerator: processed 20 Kafka message JSON files(s).

> Task :clients:processMessages
MessageGenerator: processed 146 Kafka message JSON files(s).

> Task :clients:processTestMessages
MessageGenerator: processed 4 Kafka message JSON files(s).

BUILD SUCCESSFUL in 8s

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[dajac]

Similar to #13316.

[divijvaidya]

This problem is annoying and real. Thank you for starting a PR to fix this. Another example build that fails with similar problems: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13676/5/pipeline

However, I am wondering if an alternative way to do it is:

Could we create a new root level task called processMessages which will dependOn the following tasks:

:clients:processMessages
:core:processMessages
:group-coordinator:processMessages
:metadata:processMessages
:raft:processMessages
:storage:processMessages
:streams:processMessages

and then, check depends on processMessages
and then, rat already has a dependency on check

This will ensure that rat will depend on any future new subtasks for processMessages. Note that the problem being solved in this PR is slightly different from what was solved in #13316 because over there we wanted each subtask.srcJar to depend on processMessages but now root is a root level task, hence, it can have a root level dependency on each subtasks's :processMessages separately.

Thoughts?

[divijvaidya]

I just encountered another idea.

Just put rat.dependsOn compileJava at

kafka/build.gradle

Line 211 in e1d5992

}

You can verify the dependency using:

$ ./gradlew rat --dry-run 

> Configure project :
Starting build with version 3.6.0-SNAPSHOT (commit id cabd6d2e) using Gradle 8.1.1, Java 17 and Scala 2.13.10
Build properties: maxParallelForks=12, maxScalacThreads=8, maxTestRetries=0
:compileJava SKIPPED
:rat SKIPPED

and compileJava depends on processMessages using

$ ./gradlew compileJava --dry-run | grep processMessages
:clients:processMessages SKIPPED
:core:processMessages SKIPPED
:group-coordinator:processMessages SKIPPED
:metadata:processMessages SKIPPED
:raft:processMessages SKIPPED
:storage:processMessages SKIPPED
:streams:processMessages SKIPPED

[dajac]

@divijvaidya I just push another approach. It avoids the compilation and the manual dependencies. Let me know what you think.

[divijvaidya]

This is perfect. I was trying a similar approach but somehow my it.tasks wasn't returning the processMessages?!

Can you please verify that it is working fine by running ./gradlew rat --dry-run? It should print the processMessages tasks.

[dajac]

% ./gradlew rat --dry-run

> Configure project :
Starting build with version 3.6.0-SNAPSHOT (commit id 55373f1f) using Gradle 8.1.1, Java 17 and Scala 2.13.10
Build properties: maxParallelForks=10, maxScalacThreads=8, maxTestRetries=0
:generator:compileJava SKIPPED
:generator:processResources SKIPPED
:generator:classes SKIPPED
:generator:jar SKIPPED
:clients:processMessages SKIPPED
:clients:processTestMessages SKIPPED
:core:processMessages SKIPPED
:group-coordinator:processMessages SKIPPED
:metadata:processMessages SKIPPED
:raft:processMessages SKIPPED
:storage:processMessages SKIPPED
:streams:processMessages SKIPPED
:rat SKIPPED

[divijvaidya]

Perfect! Thank you for making this change. I looked into our gradle file today and it really needs refactoring since quite a lot of logic can be combined as we did today. Perhaps, for another day.

[dajac]

The number if failed tests is scary but none of them are related to this change.