Build, CI: Add dependency checking tasks#15857
Conversation
rdblue
force-pushed
the
check-dependencies
branch
from
d05f881 to
02233c4
Compare
|
Note that this does not currently check any modules because there are no dependency lists in this PR. I'll follow up in a new PR to add dependency lists for each runtime module. |
| } | ||
|
|
||
| // Resolve actual dependencies | ||
| def actualDependencies = resolveDependencies(configuration) |
There was a problem hiding this comment.
One thing I found that broke very quickly here is we may be using a Snapshot for Iceberg dependencies which breaks if you have an actual commit hash.
I think we should exclude org.apache.iceberg artifacts
There was a problem hiding this comment.
I agree with removing the Iceberg artifacts. There's no need to update the files for every release.
| // Usage: ./gradlew :iceberg-core:checkDependencyList | ||
| // Usage with custom config: ./gradlew :iceberg-core:checkDependencyList -Pconfiguration=compileClasspath | ||
| // Fails if dependencies-<configuration>.txt exists but doesn't match actual dependencies | ||
| tasks.register('checkDependencyList') { |
There was a problem hiding this comment.
Do we want to run this for every config? Just wondering if we care about changes in dependencies for jars we aren't shipping
There was a problem hiding this comment.
This only runs for -Pconfiguration=<someConfig>. The default is runtimeClasspath, but we could have other configurations. I wanted to be more generic in case we want to be strict in other modules (like iceberg-api) in the future.
| // Task to generate a dependency list for this module | ||
| // Usage: ./gradlew :iceberg-core:generateDependencyList | ||
| // Usage with custom config: ./gradlew :iceberg-core:generateDependencyList -Pconfiguration=compileClasspath | ||
| // Generates a file at <module-root>/dependencies-<configuration>.txt |
There was a problem hiding this comment.
We'll need a rat exclusion for these
| java-version: ${{ matrix.jvm }} | ||
| - run: ./gradlew -Pquick=true javadoc | ||
|
|
||
| check-dependency-lists: |
There was a problem hiding this comment.
good to see zizmor ci check triggered. https://github.com/apache/iceberg/actions/runs/23870316620/job/69600308344?pr=15857
could you rebase off main?
we now enforce all github actions to be pinned to hash commit.
heres a good example to follow
iceberg/.github/workflows/java-ci.yml
Lines 113 to 128 in 05d7ece
| def version = id.version | ||
|
|
||
| // Truncate version to major.minor (remove patch and beyond) | ||
| def truncatedVersion = truncateVersion(version) |
There was a problem hiding this comment.
We are doing this on write here, it may be better to do this on read so the files are contain the full dependency list and we just ignore the patches when we compare.
Add a checkAllRuntimeDeps aggregation task in build.gradle that collects checkRuntimeDeps from all subprojects, and a dedicated check-runtime-deps CI job in java-ci.yml that runs it on every PR. Incorporates ideas from rdblue's apache#15857 (top-level aggregation task and dedicated CI job) into the runtime dependency guard approach.
|
Closing this in favor of #15855. |
3 participants
This adds tasks to validate dependency lists that are committed in git.
The new tasks are:
generateDependencyList: generates a sorted file containing a module's dependencies for a given configuration (runtimeClasspathby default). This keeps only the first two parts of a dependency's version to avoid failing patch release updates from dependabot.checkDependencyList: validates that a module's current set of dependencies match the dependency file. If there is no dependency file for the given configuration, this does not fail.checkAllDependencyLists: runs dependency list validation for all modulesThis also adds a new CI validation,
check-dependency-liststhat callscheckAllDependencyLists.These changes were generated with Claude code.