Investigate Avro linkage errors in java-bigquery. If they are safe, add them to suppress list

[suztomo]

googleapis/java-bigquery#1374 (comment)

Jul 20, 2021 6:34:47 PM com.google.cloud.tools.dependencies.linkagemonitor.LinkageMonitor run
SEVERE: Newly introduced problems:
Class org.joda.time.LocalDate is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.Days is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.LocalTime$Property is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.LocalTime is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
Class org.joda.time.DateTime is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateTimeZone is not found
  referenced from org.apache.avro.data.TimeConversions (org.apache.avro:avro:1.8.2)
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateMidnight is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.format.DateTimeFormatter is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.format.ISODateTimeFormat is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDateTime is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadablePeriod is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.Period is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadableDateTime is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.ReadableInstant is not found
  referenced from org.codehaus.jackson.map.ext.JodaDeserializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.DateMidnight$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDate$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)
Class org.joda.time.LocalDateTime$Property is not found
  referenced from org.codehaus.jackson.map.ext.JodaSerializers (org.codehaus.jackson:jackson-mapper-asl:1.9.13)

org.apache.avro:avro:1.8.2 is at:
  com.google.cloud:google-cloud-bigquery:1.137.1-SNAPSHOT (compile) / org.apache.avro:avro:1.8.2 (compile)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 is at:
  com.google.cloud:google-cloud-bigquery:1.137.1-SNAPSHOT (compile) / org.codehaus.jackson:jackson-mapper-asl:1.9.13 (compile)

https://github.com/googleapis/java-bigquery/pull/1374/checks?check_run_id=3117079174

[suztomo]

This gives the same errors.

  @Test
  public void testAvroDependency() throws Exception {
    Artifact avro =
        new DefaultArtifact("org.apache.avro:avro:1.8.2");
    ClassPathResult classPathResult =
        new ClassPathBuilder()
            .resolve(ImmutableList.of(avro), false, DependencyMediation.MAVEN);

    System.out.println(classPathResult);

    LinkageChecker linkageChecker = LinkageChecker.create(classPathResult.getClassPath());

    ImmutableSet<LinkageProblem> linkageProblems = linkageChecker.findLinkageProblems();
    Truth.assertThat(linkageProblems).isEmpty();
  }

Doesn't avro declare joda-time dependency? It's optional.

    <dependency>
      <groupId>joda-time</groupId>
      <artifactId>joda-time</artifactId>
      <version>2.7</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Linkage Monitor sets full=true to traverse optional dependencies. Why joda-time not included in the list?

    ClassPathResult classPathResult =
        (new ClassPathBuilder())
            .resolve(snapshotManagedDependencies, true, DependencyMediation.MAVEN);

In fact, flipping the flag from false to true in the unit test fixes the error.

Investigate why Linkage Monitor doesn't add joda-time there.

[suztomo]

@stephaniewang526 It turned out joda-time is explicitly excluded in the pom.xml (after the flatten plugin; see below). Is this expected?

suztomo-macbookpro44% vim ~/.m2/repository/com/google/cloud/google-cloud-bigquery/1.135.4-SNAPSHOT/google-cloud-bigquery-1.135.4-SNAPSHOT.pom
...
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro</artifactId>
      <version>1.8.2</version>
      <scope>compile</scope>
      <exclusions>
        <exclusion>
          <artifactId>jackson-core-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>jackson-mapper-asl</artifactId>
          <groupId>org.codehaus.jackson</groupId>
        </exclusion>
        <exclusion>
          <artifactId>paranamer</artifactId>
          <groupId>com.thoughtworks.paranamer</groupId>
        </exclusion>
        <exclusion>
          <artifactId>snappy-java</artifactId>
          <groupId>org.xerial.snappy</groupId>
        </exclusion>
        <exclusion>
          <artifactId>commons-compress</artifactId>
          <groupId>org.apache.commons</groupId>
        </exclusion>
        <exclusion>
          <artifactId>xz</artifactId>
          <groupId>org.tukaani</groupId>
        </exclusion>
        <exclusion>
          <artifactId>joda-time</artifactId>
          <groupId>joda-time</groupId>
        </exclusion>
        <exclusion>

[suztomo]

Dependency tree of google-cloud-bigquery (commit 02cfa971b805a49a17c497e96d624283b8775ccc):

suztomo-macbookpro44% mvn dependency:tree
[INFO] Scanning for projects...
[INFO] Inspecting build with total of 1 modules...
[INFO] Installing Nexus Staging features:
[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin
[INFO] 
[INFO] ---------------< com.google.cloud:google-cloud-bigquery >---------------
[INFO] Building BigQuery 1.135.4-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ google-cloud-bigquery ---
[INFO] com.google.cloud:google-cloud-bigquery:jar:1.135.4-SNAPSHOT
[INFO] +- com.google.cloud:google-cloud-core:jar:1.95.4:compile
[INFO] |  +- com.google.protobuf:protobuf-java-util:jar:3.17.3:compile
[INFO] |  +- com.google.api.grpc:proto-google-common-protos:jar:2.3.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.0.14:compile
[INFO] |  +- com.google.auth:google-auth-library-credentials:jar:0.26.0:compile
[INFO] |  +- com.google.http-client:google-http-client-gson:jar:1.39.2:compile
[INFO] |  \- com.google.protobuf:protobuf-java:jar:3.17.3:compile
[INFO] +- com.google.cloud:google-cloud-core-http:jar:1.95.4:compile
[INFO] |  +- com.google.api-client:google-api-client:jar:1.32.1:compile
[INFO] |  +- com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
[INFO] |  +- com.google.api:gax-httpjson:jar:0.83.0:compile
[INFO] |  +- io.opencensus:opencensus-api:jar:0.28.0:compile
[INFO] |  \- io.opencensus:opencensus-contrib-http-util:jar:0.28.0:compile
[INFO] +- com.google.http-client:google-http-client-jackson2:jar:1.39.2:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-core:jar:2.12.3:compile
[INFO] +- com.google.cloud:google-cloud-datacatalog:jar:1.4.2:test
[INFO] |  +- io.grpc:grpc-api:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-context:jar:1.39.0:compile
[INFO] |  +- io.grpc:grpc-stub:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-protobuf:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-protobuf-lite:jar:1.39.0:test
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-datacatalog-v1beta1:jar:0.41.2:test
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.2:test
[INFO] |  +- commons-codec:commons-codec:jar:1.15:test
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.8.7:compile
[INFO] |  +- com.google.api:gax-grpc:jar:1.66.0:test
[INFO] |  +- io.grpc:grpc-auth:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-netty-shaded:jar:1.39.0:test
[INFO] |  +- io.perfmark:perfmark-api:jar:0.23.0:test
[INFO] |  +- io.grpc:grpc-core:jar:1.39.0:test
[INFO] |  +- com.google.android:annotations:jar:4.1.1.4:test
[INFO] |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.20:test
[INFO] |  +- io.grpc:grpc-alts:jar:1.39.0:test
[INFO] |  +- io.grpc:grpc-grpclb:jar:1.39.0:test
[INFO] |  \- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.1:test
[INFO] +- com.google.api.grpc:proto-google-cloud-datacatalog-v1:jar:1.4.2:test
[INFO] +- com.google.cloud:google-cloud-storage:jar:1.117.1:test
[INFO] |  +- com.google.oauth-client:google-oauth-client:jar:1.31.5:compile
[INFO] |  +- com.google.http-client:google-http-client-apache-v2:jar:1.39.2:compile
[INFO] |  \- com.google.apis:google-api-services-storage:jar:v1-rev20210127-1.31.5:test
[INFO] +- com.google.auto.value:auto-value-annotations:jar:1.8.1:provided
[INFO] +- com.google.http-client:google-http-client:jar:1.39.2:compile
[INFO] +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
[INFO] +- com.google.auth:google-auth-library-oauth2-http:jar:0.26.0:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- com.google.apis:google-api-services-bigquery:jar:v2-rev20210617-1.32.1:compile
[INFO] +- com.google.cloud:google-cloud-core:jar:tests:1.95.4:test
[INFO] +- com.google.api:api-common:jar:1.10.4:compile
[INFO] +- com.google.guava:guava:jar:30.1.1-android:compile
[INFO] +- com.google.api:gax:jar:1.66.0:compile
[INFO] +- org.threeten:threetenbp:jar:1.5.1:compile
[INFO] +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.7:compile
[INFO] +- com.google.errorprone:error_prone_annotations:jar:2.7.1:compile
[INFO] +- junit:junit:jar:4.13.2:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- com.google.truth:truth:jar:1.1.3:test
[INFO] |  +- org.checkerframework:checker-qual:jar:3.13.0:test
[INFO] |  \- org.ow2.asm:asm:jar:9.1:test
[INFO] +- org.mockito:mockito-core:jar:2.28.2:test
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.9.10:test
[INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.9.10:test
[INFO] |  \- org.objenesis:objenesis:jar:2.6:test
[INFO] \- org.assertj:assertj-core:jar:2.9.1:test

mvn dependency:tree shows joda-time is not there.

[INFO] +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.1.3:compile
[INFO] |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  +- org.tukaani:xz:jar:1.5:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.7:compile

https://search.maven.org/artifact/org.apache.avro/avro/1.8.2/bundle shows it's optional.

    <dependency>
      <groupId>joda-time</groupId>
      <artifactId>joda-time</artifactId>
      <version>2.7</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Maybe the flutten plugin somehow excluded joda-time because it's optional

[suztomo]

Stephanie was thinking whether Apache Avro or Apache Arrow is better choice. Not taking action for now.

[stephaniewang526]

We are no longer using Avro. We are using Arrow instead. Hence, we can close this ticket.

[suztomo]

Thank you for update.