Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions aws-bundle/runtime-deps.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
com.github.ben-manes.caffeine:caffeine:2.9.3
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to other reviewers: The aws-bundle/LICENSE file and azure-bundle/LICENSE file both include "JCTools (via Netty)". This is correct: Netty shades org.jctools under io/netty/util/shaded, which is then shaded in org/apache/iceberg/aws/shaded.

com.google.errorprone:error_prone_annotations:2.10.0
commons-codec:commons-codec:1.17.1
commons-logging:commons-logging:1.2
io.netty:netty-buffer:4.1.132.Final
io.netty:netty-codec-http2:4.1.132.Final
io.netty:netty-codec-http:4.1.132.Final
io.netty:netty-codec:4.1.132.Final
io.netty:netty-common:4.1.132.Final
io.netty:netty-handler:4.1.132.Final
io.netty:netty-resolver:4.1.132.Final
io.netty:netty-transport-classes-epoll:4.1.132.Final
io.netty:netty-transport-native-unix-common:4.1.132.Final
io.netty:netty-transport:4.1.132.Final
org.apache.httpcomponents:httpclient:4.5.13
org.apache.httpcomponents:httpcore:4.4.16
org.apache.logging.log4j:log4j-api:2.20.0
org.apache.logging.log4j:log4j-core:2.20.0
org.apache.logging.log4j:log4j-slf4j-impl:2.20.0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log4J is not included in the LICENSE file.

org.checkerframework:checker-qual:3.19.0
org.reactivestreams:reactive-streams:1.0.4
org.slf4j:slf4j-api:2.0.17
software.amazon.awssdk.crt:aws-crt:0.43.9
software.amazon.awssdk:annotations:2.42.33
software.amazon.awssdk:apache-client:2.42.33
software.amazon.awssdk:arns:2.42.33
software.amazon.awssdk:auth:2.42.33
software.amazon.awssdk:aws-core:2.42.33
software.amazon.awssdk:aws-json-protocol:2.42.33
software.amazon.awssdk:aws-query-protocol:2.42.33
software.amazon.awssdk:aws-xml-protocol:2.42.33
software.amazon.awssdk:checksums-spi:2.42.33
software.amazon.awssdk:checksums:2.42.33
software.amazon.awssdk:cloudwatch-metric-publisher:2.42.33
software.amazon.awssdk:cloudwatch:2.42.33
software.amazon.awssdk:crt-core:2.42.33
software.amazon.awssdk:dynamodb:2.42.33
software.amazon.awssdk:endpoints-spi:2.42.33
software.amazon.awssdk:glue:2.42.33
software.amazon.awssdk:http-auth-aws-crt:2.42.33
software.amazon.awssdk:http-auth-aws-eventstream:2.42.33
software.amazon.awssdk:http-auth-aws:2.42.33
software.amazon.awssdk:http-auth-spi:2.42.33
software.amazon.awssdk:http-auth:2.42.33
software.amazon.awssdk:http-client-spi:2.42.33
software.amazon.awssdk:iam:2.42.33
software.amazon.awssdk:identity-spi:2.42.33
software.amazon.awssdk:json-utils:2.42.33
software.amazon.awssdk:kms:2.42.33
software.amazon.awssdk:lakeformation:2.42.33
software.amazon.awssdk:metrics-spi:2.42.33
software.amazon.awssdk:netty-nio-client:2.42.33
software.amazon.awssdk:profiles:2.42.33
software.amazon.awssdk:protocol-core:2.42.33
software.amazon.awssdk:regions:2.42.33
software.amazon.awssdk:retries-spi:2.42.33
software.amazon.awssdk:retries:2.42.33
software.amazon.awssdk:s3:2.42.33
software.amazon.awssdk:s3control:2.42.33
software.amazon.awssdk:sdk-core:2.42.33
software.amazon.awssdk:smithy-rpcv2-protocol:2.42.33
software.amazon.awssdk:sso:2.42.33
software.amazon.awssdk:sts:2.42.33
software.amazon.awssdk:third-party-jackson-core:2.42.33
software.amazon.awssdk:third-party-jackson-dataformat-cbor:2.42.33
software.amazon.awssdk:utils-lite:2.42.33
software.amazon.awssdk:utils:2.42.33
software.amazon.eventstream:eventstream:1.0.1
software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin:2.4.1
software.amazon.s3.analyticsaccelerator:analyticsaccelerator-s3:1.3.1
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that these are fine. I didn't check every one against the latest update (fb2c8ac3faf) because they are now grouped into high level SDK modules, but they are all ALv2 and should be okay.

44 changes: 44 additions & 0 deletions azure-bundle/runtime-deps.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
com.azure:azure-core-http-netty:1.16.3
com.azure:azure-core:1.57.1
com.azure:azure-identity:1.18.2
com.azure:azure-json:1.5.1
com.azure:azure-security-keyvault-keys:4.10.6
com.azure:azure-storage-blob:12.33.3
com.azure:azure-storage-common:12.32.2
com.azure:azure-storage-file-datalake:12.26.3
com.azure:azure-storage-internal-avro:12.18.2
com.azure:azure-xml:1.2.1
com.fasterxml.jackson.core:jackson-annotations:2.18.4
com.fasterxml.jackson.core:jackson-core:2.18.4.1
com.fasterxml.jackson.core:jackson-databind:2.18.4
com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.18.4
com.microsoft.azure:msal4j-persistence-extension:1.3.0
com.microsoft.azure:msal4j:1.23.1
io.netty:netty-buffer:4.1.130.Final
io.netty:netty-codec-dns:4.1.128.Final
io.netty:netty-codec-http2:4.1.130.Final
io.netty:netty-codec-http:4.1.130.Final
io.netty:netty-codec-socks:4.1.130.Final
io.netty:netty-codec:4.1.130.Final
io.netty:netty-common:4.1.130.Final
io.netty:netty-handler-proxy:4.1.130.Final
io.netty:netty-handler:4.1.130.Final
io.netty:netty-resolver-dns-classes-macos:4.1.128.Final
io.netty:netty-resolver-dns-native-macos:4.1.128.Final
io.netty:netty-resolver-dns:4.1.128.Final
io.netty:netty-resolver:4.1.130.Final
io.netty:netty-tcnative-boringssl-static:2.0.74.Final
io.netty:netty-tcnative-classes:2.0.74.Final
io.netty:netty-transport-classes-epoll:4.1.130.Final
io.netty:netty-transport-classes-kqueue:4.1.130.Final
io.netty:netty-transport-native-epoll:4.1.130.Final
io.netty:netty-transport-native-kqueue:4.1.130.Final
io.netty:netty-transport-native-unix-common:4.1.130.Final
io.netty:netty-transport:4.1.130.Final
io.projectreactor.netty:reactor-netty-core:1.2.13
io.projectreactor.netty:reactor-netty-http:1.2.13
io.projectreactor:reactor-core:3.7.14
net.java.dev.jna:jna-platform:5.17.0
net.java.dev.jna:jna:5.17.0
org.reactivestreams:reactive-streams:1.0.4
org.slf4j:slf4j-api:2.0.17
33 changes: 33 additions & 0 deletions flink/v1.20/flink-runtime/runtime-deps.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
com.fasterxml.jackson.core:jackson-annotations:2.21
com.fasterxml.jackson.core:jackson-core:2.21.2
com.fasterxml.jackson.core:jackson-databind:2.21.2
com.github.ben-manes.caffeine:caffeine:2.9.3
com.github.luben:zstd-jni:1.5.7-3
com.google.errorprone:error_prone_annotations:2.10.0
dev.failsafe:failsafe:3.3.2
io.airlift:aircompressor:2.0.3
io.dropwizard.metrics:metrics-core:3.2.6
org.apache.avro:avro:1.12.1
org.apache.datasketches:datasketches-java:6.2.0
org.apache.datasketches:datasketches-memory:3.0.2
org.apache.flink:flink-metrics-dropwizard:1.20.1
org.apache.httpcomponents.client5:httpclient5:5.6
org.apache.httpcomponents.core5:httpcore5-h2:5.4
org.apache.httpcomponents.core5:httpcore5:5.4
org.apache.orc:orc-core:1.9.8
org.apache.orc:orc-shims:1.9.8
org.apache.parquet:parquet-avro:1.17.0
org.apache.parquet:parquet-column:1.17.0
org.apache.parquet:parquet-common:1.17.0
org.apache.parquet:parquet-encoding:1.17.0
org.apache.parquet:parquet-format-structures:1.17.0
org.apache.parquet:parquet-hadoop:1.17.0
org.apache.parquet:parquet-jackson:1.17.0
org.apache.parquet:parquet-variant:1.17.0
org.checkerframework:checker-qual:3.19.0
org.eclipse.microprofile.openapi:microprofile-openapi-api:4.1.1
org.locationtech.jts:jts-core:1.20.0
org.projectnessie.nessie:nessie-client:0.107.4
org.projectnessie.nessie:nessie-model:0.107.4
org.roaringbitmap:RoaringBitmap:1.6.14
org.threeten:threeten-extra:1.7.1
33 changes: 33 additions & 0 deletions flink/v2.0/flink-runtime/runtime-deps.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
com.fasterxml.jackson.core:jackson-annotations:2.21
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not reviewing older versions of Flink yet. I think we should determine what needs to change for the current version and then verify the same changes on the older ones.

com.fasterxml.jackson.core:jackson-core:2.21.2
com.fasterxml.jackson.core:jackson-databind:2.21.2
com.github.ben-manes.caffeine:caffeine:2.9.3
com.github.luben:zstd-jni:1.5.7-3
com.google.errorprone:error_prone_annotations:2.10.0
dev.failsafe:failsafe:3.3.2
io.airlift:aircompressor:2.0.3
io.dropwizard.metrics:metrics-core:3.2.6
org.apache.avro:avro:1.12.1
org.apache.datasketches:datasketches-java:6.2.0
org.apache.datasketches:datasketches-memory:3.0.2
org.apache.flink:flink-metrics-dropwizard:2.0.0
org.apache.httpcomponents.client5:httpclient5:5.6
org.apache.httpcomponents.core5:httpcore5-h2:5.4
org.apache.httpcomponents.core5:httpcore5:5.4
org.apache.orc:orc-core:1.9.8
org.apache.orc:orc-shims:1.9.8
org.apache.parquet:parquet-avro:1.17.0
org.apache.parquet:parquet-column:1.17.0
org.apache.parquet:parquet-common:1.17.0
org.apache.parquet:parquet-encoding:1.17.0
org.apache.parquet:parquet-format-structures:1.17.0
org.apache.parquet:parquet-hadoop:1.17.0
org.apache.parquet:parquet-jackson:1.17.0
org.apache.parquet:parquet-variant:1.17.0
org.checkerframework:checker-qual:3.19.0
org.eclipse.microprofile.openapi:microprofile-openapi-api:4.1.1
org.locationtech.jts:jts-core:1.20.0
org.projectnessie.nessie:nessie-client:0.107.4
org.projectnessie.nessie:nessie-model:0.107.4
org.roaringbitmap:RoaringBitmap:1.6.14
org.threeten:threeten-extra:1.7.1
33 changes: 33 additions & 0 deletions flink/v2.1/flink-runtime/runtime-deps.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
com.fasterxml.jackson.core:jackson-annotations:2.21
Copy link
Copy Markdown
Contributor

@rdblue rdblue Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LICENSE file stats that this contains many missing libraries:

The reason for most of the extras is that there were no LICENSE updates after a few recent PRs:

Also, LICENSE contains Google Guava, which is present because this shades iceberg-bundled-guava. But shading in that module means we don't have it listed here (FYI).

Action items:

  • Find out why some libraries were there but are no longer:
    • Arrow, Netty, Apache Commons, OpenTelemetry, javax.annotations
  • Fix the Hive entry in LICENSE. Before chore: several fixes on the LICENSE/NOTICE #15449 it was clear that this was shaded by ORC. Now the only Hive reference I see is META-INF files so I think this is probably incorrect.
  • Remove all of the fixed dependency leaks from LICENSE and NOTICE

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apache Commons is from this commit: 760a20b

#2102 copied array methods into ArrayUtil. This isn't a big problem, but it doesn't seem worth the hassle of tracking it down in LICENSE to have array copy methods. The implementations don't match project style or provide value. A good first issue is to remove them.

com.fasterxml.jackson.core:jackson-core:2.21.2
com.fasterxml.jackson.core:jackson-databind:2.21.2
com.github.ben-manes.caffeine:caffeine:2.9.3
com.github.luben:zstd-jni:1.5.7-3
com.google.errorprone:error_prone_annotations:2.10.0
dev.failsafe:failsafe:3.3.2
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is leaked by iceberg-aws and should not be bundled.

Copy link
Copy Markdown
Contributor

@rdblue rdblue Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is used directly by S3InputStream, which means it needs to be included when iceberg-aws is included becuase it is not provided by the AWS dependencies. I don't think this is a good reason to keep using it and that we should replace it with Tasks, unless it is doing something special.

Since this is in the license docs, I think this isn't a blocker for 1.11.0 or 1.10.2, but we should remove it to keep dependencies to a minimum.

io.airlift:aircompressor:2.0.3
io.dropwizard.metrics:metrics-core:3.2.6
org.apache.avro:avro:1.12.1
org.apache.datasketches:datasketches-java:6.2.0
org.apache.datasketches:datasketches-memory:3.0.2
org.apache.flink:flink-metrics-dropwizard:2.1.0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not listed in the LICENSE file.

I also wonder if this should be excluded and not added to LICENSE because it seems like something that should be included in the Flink runtime. I suspect that we need to add it as a compileOnly dependency or suppress it in the runtime config.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened a PR for this fix: #16093

org.apache.httpcomponents.client5:httpclient5:5.6
org.apache.httpcomponents.core5:httpcore5-h2:5.4
org.apache.httpcomponents.core5:httpcore5:5.4
org.apache.orc:orc-core:1.9.8
org.apache.orc:orc-shims:1.9.8
org.apache.parquet:parquet-avro:1.17.0
org.apache.parquet:parquet-column:1.17.0
org.apache.parquet:parquet-common:1.17.0
org.apache.parquet:parquet-encoding:1.17.0
org.apache.parquet:parquet-format-structures:1.17.0
org.apache.parquet:parquet-hadoop:1.17.0
org.apache.parquet:parquet-jackson:1.17.0
org.apache.parquet:parquet-variant:1.17.0
org.checkerframework:checker-qual:3.19.0
org.eclipse.microprofile.openapi:microprofile-openapi-api:4.1.1
org.locationtech.jts:jts-core:1.20.0
org.projectnessie.nessie:nessie-client:0.107.4
org.projectnessie.nessie:nessie-model:0.107.4
org.roaringbitmap:RoaringBitmap:1.6.14
org.threeten:threeten-extra:1.7.1
114 changes: 114 additions & 0 deletions gcp-bundle/runtime-deps.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
com.fasterxml.jackson.core:jackson-annotations:2.18.2
com.fasterxml.jackson.core:jackson-core:2.18.2
com.fasterxml.jackson.core:jackson-databind:2.18.2
com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.18.2
com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.18.2
com.fasterxml.woodstox:woodstox-core:7.0.0
com.google.android:annotations:4.1.1.4
com.google.api-client:google-api-client:2.7.2
com.google.api.grpc:gapic-google-cloud-storage-v2:2.64.1
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:3.24.0
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:0.196.0
com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:0.196.0
com.google.api.grpc:grpc-google-cloud-storage-v2:2.64.1
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:3.24.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1alpha:3.24.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.196.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:0.196.0
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta:3.24.0
com.google.api.grpc:proto-google-cloud-kms-v1:0.182.0
com.google.api.grpc:proto-google-cloud-monitoring-v3:3.89.0
com.google.api.grpc:proto-google-cloud-storage-v2:2.64.1
com.google.api.grpc:proto-google-common-protos:2.67.0
com.google.api.grpc:proto-google-iam-v1:1.62.0
com.google.api:api-common:2.59.0
com.google.api:gax-grpc:2.76.0
com.google.api:gax-httpjson:2.76.0
com.google.api:gax:2.76.0
com.google.apis:google-api-services-bigquery:v2-rev20251012-2.0.0
com.google.apis:google-api-services-storage:v1-rev20260204-2.0.0
com.google.auth:google-auth-library-credentials:1.43.0
com.google.auth:google-auth-library-oauth2-http:1.43.0
com.google.auto.value:auto-value-annotations:1.11.1
com.google.cloud.gcs.analytics:client:1.2.3
com.google.cloud.gcs.analytics:gcs-analytics-core:1.2.3
com.google.cloud.opentelemetry:detector-resources-support:0.33.0
com.google.cloud.opentelemetry:exporter-metrics:0.33.0
com.google.cloud.opentelemetry:shared-resourcemapping:0.33.0
com.google.cloud:google-cloud-bigquery:2.62.0
com.google.cloud:google-cloud-bigquerystorage:3.24.0
com.google.cloud:google-cloud-core-grpc:2.66.0
com.google.cloud:google-cloud-core-http:2.66.0
com.google.cloud:google-cloud-core:2.66.0
com.google.cloud:google-cloud-kms:2.91.0
com.google.cloud:google-cloud-monitoring:3.89.0
com.google.cloud:google-cloud-storage:2.64.1
com.google.code.findbugs:jsr305:3.0.2
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findbugs is excluded throughout the codebase because it was originally LGPL and cannot be bundled. The license issues weren't clarified, and a clean implementation was created: https://github.com/stephenc/findbugs-annotations

Although the maven metadata reports ALv2, we need to exclude it. If we need the annotations (which are not required to function), then we should use the stephenc verison.

com.google.code.gson:gson:2.12.1
com.google.errorprone:error_prone_annotations:2.42.0
com.google.flatbuffers:flatbuffers-java:24.3.25
com.google.guava:failureaccess:1.0.3
com.google.guava:guava:33.5.0-jre
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
com.google.http-client:google-http-client-apache-v2:2.1.0
com.google.http-client:google-http-client-appengine:2.1.0
com.google.http-client:google-http-client-gson:2.1.0
com.google.http-client:google-http-client-jackson2:2.1.0
com.google.http-client:google-http-client:2.1.0
com.google.j2objc:j2objc-annotations:3.1
com.google.oauth-client:google-oauth-client:1.39.0
com.google.protobuf:protobuf-java-util:4.33.2
com.google.protobuf:protobuf-java:4.33.2
com.google.re2j:re2j:1.8
commons-codec:commons-codec:1.18.0
io.grpc:grpc-alts:1.76.3
io.grpc:grpc-api:1.76.3
io.grpc:grpc-auth:1.76.3
io.grpc:grpc-context:1.76.3
io.grpc:grpc-core:1.76.3
io.grpc:grpc-googleapis:1.76.3
io.grpc:grpc-grpclb:1.76.3
io.grpc:grpc-inprocess:1.76.3
io.grpc:grpc-netty-shaded:1.76.3
io.grpc:grpc-opentelemetry:1.76.3
io.grpc:grpc-protobuf-lite:1.76.3
io.grpc:grpc-protobuf:1.76.3
io.grpc:grpc-rls:1.76.3
io.grpc:grpc-services:1.76.3
io.grpc:grpc-stub:1.76.3
io.grpc:grpc-util:1.76.3
io.grpc:grpc-xds:1.76.3
io.netty:netty-buffer:4.1.110.Final
io.netty:netty-common:4.1.110.Final
io.opencensus:opencensus-api:0.31.1
io.opencensus:opencensus-contrib-http-util:0.31.1
io.opentelemetry.contrib:opentelemetry-gcp-resources:1.37.0-alpha
io.opentelemetry.semconv:opentelemetry-semconv:1.29.0-alpha
io.opentelemetry:opentelemetry-api:1.51.0
io.opentelemetry:opentelemetry-context:1.51.0
io.opentelemetry:opentelemetry-sdk-common:1.51.0
io.opentelemetry:opentelemetry-sdk-extension-autoconfigure-spi:1.51.0
io.opentelemetry:opentelemetry-sdk-logs:1.51.0
io.opentelemetry:opentelemetry-sdk-metrics:1.51.0
io.opentelemetry:opentelemetry-sdk-trace:1.51.0
io.opentelemetry:opentelemetry-sdk:1.51.0
io.perfmark:perfmark-api:0.27.0
javax.annotation:javax.annotation-api:1.3.2
org.apache.arrow:arrow-format:17.0.0
org.apache.arrow:arrow-memory-core:17.0.0
org.apache.arrow:arrow-memory-netty-buffer-patch:17.0.0
org.apache.arrow:arrow-memory-netty:17.0.0
org.apache.arrow:arrow-vector:17.0.0
org.apache.commons:commons-lang3:3.20.0
org.apache.httpcomponents:httpclient:4.5.14
org.apache.httpcomponents:httpcore:4.4.16
org.checkerframework:checker-compat-qual:2.5.6
org.checkerframework:checker-qual:3.49.0
org.codehaus.mojo:animal-sniffer-annotations:1.24
org.codehaus.woodstox:stax2-api:4.2.2
org.conscrypt:conscrypt-openjdk-uber:2.5.2
org.json:json:20250517
org.jspecify:jspecify:1.0.0
org.slf4j:slf4j-api:2.0.17
org.threeten:threeten-extra:1.8.0
org.threeten:threetenbp:1.7.0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rest of these appear to be real dependencies from GCP and correctly included in the LICENSE file.

Loading
Loading