[BEAM-10961] Strict dependency checking for sdks/java/io/gcp #13791

shehzaadn-vd · 2021-01-22T11:02:00Z

These changes were originally in #13688. They've been moved from that PR to here per
@ibzib's request. Please see the original PR for conversation history around these files.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

shehzaadn-vd · 2021-01-22T11:03:17Z

@ibzib please add additional reviewers. thanks!

shehzaadn-vd · 2021-01-22T17:29:55Z

retest this please

shehzaadn-vd · 2021-01-22T17:45:27Z

@ibzib any idea how I can this particular failing check to be retried? It seems to have been run on creation of the PR, but there seems to be no way to get it to run again e.g. with a "Run ...." comment.

I'm referring to "Java Tests / Java Unit Tests (macos-latest) (pull_request) - third from the top - with Details link: https://github.com/apache/beam/pull/13791/checks?check_run_id=1748199531)

shehzaadn-vd · 2021-01-22T17:45:38Z

Run Java PreCommit

ibzib · 2021-01-22T17:56:51Z

@ibzib any idea how I can this particular failing check to be retried? It seems to have been run on creation of the PR, but there seems to be no way to get it to run again e.g. with a "Run ...." comment.

I'm referring to "Java Tests / Java Unit Tests (macos-latest) (pull_request) - third from the top - with Details link: https://github.com/apache/beam/pull/13791/checks?check_run_id=1748199531)

These are Github actions and work totally differently from our Jenkins tests. As far as I know there is no way to retry an individual action, but you can retry all actions by clicking Details -> Re-run jobs (on the right) -> Re-run all jobs.

ibzib

For reference, here are the failures prior to this PR:

> Dependency analysis found issues.                                                                                           │|    |    +--- com.google.code.findbugs:jsr305:3.0.2
  usedUndeclaredArtifacts:                                                                                                    │|    |    +--- org.codehaus.mojo:animal-sniffer-annotations:1.19
   - beam.model:pipeline:2.28.0-SNAPSHOT@jar                                                                                  │|    |    +--- com.google.errorprone:error_prone_annotations:2.4.0 -> 2.3.1
   - beam.runners:core-java:2.28.0-SNAPSHOT@jar                                                                               │|    |    +--- io.grpc:grpc-protobuf-lite:1.32.1 -> 1.32.2 (*)
   - com.fasterxml.jackson.core:jackson-core:2.10.2@jar                                                                       │|    |    +--- com.google.api:api-common:1.10.0 (*)
   - com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:1.5.5@jar                                                      │|    |    +--- javax.annotation:javax.annotation-api:1.3.2
   - com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:1.16.1@jar                                                      │|    |    +--- com.google.guava:failureaccess:1.0.1
   - com.google.api.grpc:proto-google-cloud-spanner-v1:2.0.2@jar                                                              │|    |    +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava
   - com.google.api:api-common:1.10.0@jar                                                                                     │|    |    +--- org.checkerframework:checker-compat-qual:2.5.5
   - com.google.apis:google-api-services-storage:v1-rev20200611-1.30.10@jar                                                   │|    |    \--- com.google.j2objc:j2objc-annotations:1.3
   - com.google.code.gson:gson:2.8.6@jar                                                                                      │|    +--- com.google.guava:guava:29.0-android -> 25.1-jre (*)
   - com.google.errorprone:error_prone_annotations:2.3.1@jar                                                                  │|    +--- com.google.api:gax:1.58.2 -> 1.60.0 (*)
   - com.google.protobuf:protobuf-java-util:3.12.0@jar                                                                        │|    +--- com.google.api:gax-grpc:1.58.2 -> 1.60.0 (*)
   - io.grpc:grpc-api:1.32.2@jar                                                                                              │|    +--- io.grpc:grpc-alts:1.32.1 -> 1.32.2 (*)
   - org.apache.beam:beam-vendor-grpc-1_26_0:0.3@jar                                                                          │|    +--- io.grpc:grpc-grpclb:1.32.1 -> 1.32.2 (*)
   - org.apache.beam:beam-vendor-guava-26_0-jre:0.1@jar                                                                       │|    +--- org.conscrypt:conscrypt-openjdk-uber:2.2.1 -> 2.5.1
   - org.apache.httpcomponents:httpclient:4.5.13@jar                                                                          │|    +--- org.threeten:threetenbp:1.4.4 -> 1.5.0
   - org.apache.httpcomponents:httpcore:4.4.13@jar                                                                            │|    +--- com.google.code.findbugs:jsr305:3.0.2
   - org.hamcrest:hamcrest:2.1@jar                                                                                            │|    +--- com.google.code.gson:gson:2.8.6
   - org.threeten:threetenbp:1.5.0@jar                                                                                        │|    +--- com.google.auth:google-auth-library-credentials:0.21.1
  unusedDeclaredArtifacts:                                                                                                    │|    \--- org.apache.commons:commons-lang3:3.5 -> 3.9
   - beam.sdks.java:expansion-service:2.28.0-SNAPSHOT@jar                                                                     │+--- com.google.http-client:google-http-client -> 1.36.0 (*)
   - com.google.api.grpc:grpc-google-cloud-pubsublite-v1:0.7.0@jar                                                            │+--- com.google.http-client:google-http-client-jackson2 -> 1.36.0 (*)
   - com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.105.5@jar                                               │+--- io.grpc:grpc-alts -> 1.32.2 (*)
   - com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:2.0.2@jar                                               │+--- io.grpc:grpc-auth -> 1.32.2 (*)
   - com.google.cloud:google-cloud-core-grpc:1.93.9@jar                                                                       │+--- io.grpc:grpc-core -> 1.32.2 (*)
   - io.grpc:grpc-alts:1.32.2@jar                                                                                             │+--- io.grpc:grpc-context -> 1.32.2
   - io.grpc:grpc-context:1.32.2@jar                                                                                          │+--- io.grpc:grpc-grpclb -> 1.32.2 (*)
   - io.grpc:grpc-core:1.32.2@jar                                                                                             │+--- io.grpc:grpc-netty -> 1.32.2
   - io.grpc:grpc-grpclb:1.32.2@jar                                                                                           │|    +--- io.grpc:grpc-core:1.32.2 (*)
   - io.grpc:grpc-netty-shaded:1.32.2@jar                                                                                     │|    +--- io.netty:netty-codec-http2:4.1.51.Final
   - io.grpc:grpc-stub:1.32.2@jar                                                                                             │|    |    +--- io.netty:netty-common:4.1.51.Final
   - io.netty:netty-tcnative-boringssl-static:2.0.33.Final@jar                                                                │|    |    +--- io.netty:netty-buffer:4.1.51.Final
   - org.hamcrest:hamcrest-core:2.1@jar                                                                                       │|    |    |    \--- io.netty:netty-common:4.1.51.Final

cc: @kileys

sdks/java/io/google-cloud-platform/expansion-service/build.gradle

ibzib · 2021-01-22T23:00:41Z

sdks/java/io/google-cloud-platform/build.gradle

Suggested change

compile "org.checkerframework:checker-qual:2.0.0"

ibzib · 2021-01-22T23:05:43Z

sdks/java/io/google-cloud-platform/build.gradle

Why is this necessary?

ibzib · 2021-01-22T23:05:57Z

sdks/java/io/google-cloud-platform/build.gradle

Suggested change

//enableStrictDependencies:true,

enableStrictDependencies:true,

@ibzib to your question from the parent PR:

I ran into this as well. I was able to avoid it temporarily by downgrading from Gradle 6.8 to 6.7, however we will need to figure out a better solution. The first step is to figure out where the :runners:core-java dependency is coming from, since it isn't explicitly declared here.

running ../../../../gradlew -q dependencies shows that :runners:java-core comes from :sdks:java:extensions:google-cloud-platform-core

I figured out that this was a bug in the plugin: https://github.com/wfhartford/gradle-dependency-analyze/issues/125

This bug only occurs when there is a strict dependency violation.

You can see the real violation when doing your local testing by reverting 6763c21.

@ibzib thanks for the tip. the only violation I see is that runners:core-java was usedUndeclared. The only fix I see is to add that explicitly.

ibzib · 2021-01-23T00:11:27Z

sdks/java/io/google-cloud-platform/build.gradle

Many of these com.google dependencies should read from BeamModulePlugin, which uses the version from google_cloud_platform_libraries_bom.

For example:

beam/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

Line 522 in 809a0b7

google_cloud_bigquery_storage : "com.google.cloud:google-cloud-bigquerystorage", // google_cloud_platform_libraries_bom sets version

Except we need to add com.google.api.grpc:proto-google-cloud-bigquerystorage-v1 since it is not exactly the same as com.google.api.grpc:proto-google-cloud-bigquerystorage.

@ibzib Comparing the lines added here with what's available in BeamModulePlugin and my proposed plan to use what's available where possible:

com.google.api.grpc:proto-google-cloud-bigquerystorage-v1 is not available - as you pointed out.

com.google.api.grpc:proto-google-cloud-bigtable-admin-v2 is not available

com.google.api.grpc:proto-google-cloud-spanner-v1 is not available

com.google.api:api-common is available - but the version coming in from google_cloud_platform_libraries_bom is unclear to me. Replacing.

com.google.apis:google-api-services-storage is a match. Replacing.

com.google.code.gson:gson:2.8.6 is a match. Replacing.

com.google.errorprone:error_prone_annotations is 2.3.3 here, but 2.3.1 in BeamModulePlugin. Skipping, but putting in a TODO in the code to match version numbers.

com.google.protobuf:protobuf-java-util is 3.11.1 here, but 3.12.0 in BeamModulePlugin. Skipping, but putting in a TODO in the code to match version numbers.

io.grpc:grpc-api is available - but the version coming in from google_cloud_platform_libraries_bom is unclear to me. Replacing.

org.apache.beam:beam-vendor-guava-26_0-jre is available. ** Replacing**.

org.apache.httpcomponents:httpclient is 4.5.12 here, but 4.5.10 in BeamModulePlugin. Skipping but putting in a TODO in the code to match version numbers.

org.apache.httpcomponents:httpcore is 4.4.13 here, but 4.4.12 in BeamModulePlugin. ** Skipping** but putting in a TODO in the code to match version numbers.

org.hamcrest:hamcrest is not available in BeamModulePlugin

org.threethen:threetenbp is not available in BeamModulePlugin

compile "org.apache.beam:beam-vendor-grpc-1_26_0:0.3" is available. Replacing

the version coming in from google_cloud_platform_libraries_bom is unclear to me.

That's totally fine. The whole point of using the bom is that it provides a single source of truth for dependency versions. If the version provided by the bom mismatches some version the build is actually using, it's a problem.

org.hamcrest:hamcrest is not available in BeamModulePlugin

I've seen this one in enough other modules that it probably should be placed in BeamModulePlugin.

Skipping, but putting in a TODO in the code to match version numbers.

Try replacing these anyway, don't worry about matching the versions exactly. In most cases I expect that it shouldn't matter if we use a slightly newer or older version than one of the transitive dependencies, as long as the versions are compatible. (If we don't match versions, Gradle will do that for us anyway when it resolves dependencies.)

OK. Sounds good. I won't bother with the TODOs, and I'll add hamcrest to BeamModulePlugin.

Please add the "not available" dependencies to BeamModulePlugin so that we can use the bom versions.

codecov · 2021-01-23T06:08:42Z

Codecov Report

Merging #13791 (2187c9a) into master (c137e88) will increase coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #13791      +/-   ##
==========================================
+ Coverage   82.97%   83.00%   +0.02%     
==========================================
  Files         469      469              
  Lines       58300    58355      +55     
==========================================
+ Hits        48377    48438      +61     
+ Misses       9923     9917       -6

Impacted Files	Coverage Δ
...s/python/apache_beam/ml/gcp/videointelligenceml.py
...python/apache_beam/examples/wordcount_dataframe.py
.../build/srcs/sdks/python/apache_beam/io/__init__.py
...ive/messaging/interactive_environment_inspector.py
...am/testing/benchmarks/chicago_taxi/process_tfma.py
...am/portability/api/standard_window_fns_pb2_grpc.py
...rcs/sdks/python/apache_beam/runners/job/manager.py
...dks/python/apache_beam/io/gcp/gce_metadata_util.py
...n/apache_beam/runners/dataflow/dataflow_metrics.py
...on/apache_beam/ml/gcp/naturallanguageml_test_it.py
... and 928 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c137e88...2187c9a. Read the comment docs.

sonam-vend · 2021-01-25T08:27:00Z

Run Java PreCommit

…les, remove testImplementation line

sdks/java/io/google-cloud-platform/build.gradle

ibzib · 2021-01-28T01:53:51Z

Run Python 3.8 PostCommit

shehzaadn-vd · 2021-01-28T02:02:47Z

sdks/java/io/google-cloud-platform/build.gradle

  compile enforcedPlatform(library.java.google_cloud_platform_libraries_bom)
  compile project(path: ":sdks:java:core", configuration: "shadow")
-  compile project(":sdks:java:expansion-service")
+  testCompile project(":sdks:java:expansion-service")


@ibzib does your comment about making the :sdks:java:expansion-service dependency runtimeOnly apply here too?

This one is actually needed at compile time, which is why it passes the strict dependency checker:

beam/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java

Line 32 in 0c01636

import org.apache.beam.sdk.expansion.service.ExpansionService;

Run Python 3.8 PostCommit

@ibzib I see the error
Error: Could not find or load main class org.apache.beam.sdk.expansion.service.ExpansionService
Changing to runtimeOnly.

shehzaadn-vd · 2021-01-28T03:04:51Z

Run Java PreCommit

shehzaadn-vd · 2021-01-28T06:42:07Z

Run Java_Examples_Dataflow_Java11 PreCommit

shehzaadn-vd · 2021-01-28T06:57:42Z

Run Python 3.8 PostCommit

shehzaadn-vd · 2021-02-01T02:39:17Z

@ibzib do you think we are good to go on this one?

sdks/java/io/google-cloud-platform/expansion-service/build.gradle

ibzib · 2021-02-01T22:47:57Z

sdks/java/io/google-cloud-platform/build.gradle

Please add the "not available" dependencies to BeamModulePlugin so that we can use the bom versions.

sdks/java/io/google-cloud-platform/build.gradle

…ctDependencies=true

shehzaadn-vd · 2021-02-06T14:21:36Z

@ibzib i've brought back the compile lines and paired them with permitUnusedDeclared. However, some of those permitUnusedDeclared lines seem to cause this error:

Could not determine the dependencies of task ':sdks:java:io:google-cloud-platform:analyzeClassesDependencies'.
> Could not resolve all dependencies for configuration ':sdks:java:io:google-cloud-platform:permitUnusedDeclared'.
   > Could not find io.grpc:grpc-alts:.
     Required by:
         project :sdks:java:io:google-cloud-platform
   > Could not find io.grpc:grpc-core:.
     Required by:
         project :sdks:java:io:google-cloud-platform
   > Could not find io.grpc:grpc-grpclb:.
     Required by:
         project :sdks:java:io:google-cloud-platform
   > Could not find io.grpc:grpc-netty-shaded:.
     Required by:
         project :sdks:java:io:google-cloud-platform
   > Could not find com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:.
     Required by:
         project :sdks:java:io:google-cloud-platform
   > Could not find com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:.
     Required by:
         project :sdks:java:io:google-cloud-platform

My guess is permitUnusedDeclared doesn't play well with a library variable where the version is set by the BOM
e.g. permitUnusedDeclared library.java.proto_google_cloud_bigquery_storage_v1beta1

Any pointers on how to express such dependencies as permitUnusedDeclared? Hardcoding the version seems ugly...

shehzaadn-vd · 2021-02-06T23:02:04Z

retest this please

shehzaadn-vd · 2021-02-07T04:26:50Z

@ibzib The error I see locally on the latest state of this PR is:

Execution failed for task ':sdks:java:io:google-cloud-platform:analyzeClassesDependencies'.
> Dependency analysis found issues.
  unusedDeclaredArtifacts: 
   - io.grpc:grpc-context:1.32.2@jar
   - io.grpc:grpc-stub:1.32.2@jar

i'm not quite sure why, since I've got explicit permitUnusedDeclared lines for both of these.

…lared to support library.java.* variables

…itUnusedDeclared library.java.proto_google_cloud_bigquerybeta2_storage_v1

ibzib

Mostly LGTM. Looks like this got slightly out of sync with head though.

sdks/java/io/google-cloud-platform/build.gradle

… earlier (2) group compile project() declarations

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

ibzib

LGTM

ibzib · 2021-02-26T01:28:20Z

Run Portable_Python PreCommit

…gger * github/master: (123 commits) [BEAM-11899] Bump commons-pool to 2.8.1 and bump commons-dbcp to 2.8.0, Because there is a library dependency Update pillars.yaml (apache#14142) [BEAM-10632] Checkerframework nullness cleanups (apache#14107) [BEAM-11213] Instantiate SparkListenerApplicationStart in a Spark 3 compatible way Fix typos for excluding testMergingCustomWindowsWithoutCustomWindowTypes Specify the time resolution for TestStreamPayload. [BEAM-10961] Enable strict depdency checking for sdks/java/extensions/euphoria (second attempt) [BEAM-11848] Store Docker images in a variable for consistency. Splitting old Go Precommit and new ULR integration test precommit. Moving runner imports out of ptest. Add the TO_STRING capability to Java and Python [BEAM-11848] Fix Docker images list. jdbc python supported Dataflow runner (apache#13960) Adding a warning to use multi-workers on FnApiRunner Fix legend for Python Directrunner microbenchmarks [BEAM-11740] Estimate PCollection byte size [BEAM-10961] enable strict dependency checking for sdks/java/extensions/zetasketch (apache#14093) Map Dataflow JOB_STATE_CANCELLING to Beam RUNNING state [BEAM-11833] Fix reported watermark after restriction split in UnboundedSourceAsSDFRestrictionTracker [BEAM-10761] add reference to BEAM-11761 [BEAM-10961] enable strict dependency checking for flink/job-server Exclude MapState example integration tests from Dataflow runner v2 suite Remove InvalidWindows from Java SDK, instead track "already merged" bit Fix checkstyle in watermark latency benchmark Fix compile breakage in WindmillStateInternals Improve test, error on ALREADY_MERGED. [BEAM-10961] Strict dependency checking for sdks/java/io/gcp (apache#13791) Initial watermark latency benchmark Attempting improvements on DirectRunner Python dash [BEAM-10961] enable strict dependency checking for sdks/java/extensions/google-cloud-platform-core (apache#14084) Merge pull request apache#13802: [BEAM-1474]. Adding MapState and SetState support for the Dataflow runner Remove some false positives Remove nullness warning suppression [BEAM-11861] Add methods to explicitly provide coder for ParquetIO's Parse and ParseFiles (apache#14078) [BEAM-11531] Use pandas 1.2 for python>=3.7 (apache#14099) [BEAM-10961] add reference to BEAM-11761 [BEAM-10961] add explicit compile for auto_value_annotations in sdks/extensions/ml/build.gradle Attempting improvements on DirectRunner Python dash Recognize JOB_STATE_PENDING from Dataflow and map to RUNNING never run checkerframework on tests Puts more expensive BQ empty table check to the right of the 'and' condition (apache#14094) Use the windowing strategy of the input, not output, PCollection of GBK. Do not stage dataflow worker jar when use runner_v2. [BEAM-11870] Re-raise underlying exception for InvocationTargetException (apache#14098) [BEAM-11778] Create a wrapper for ZetaSQL catalog and refactor accordingly. (apache#13934) [BEAM-9378] Add ignored tests which fail in various ways when querying nested structures (apache#14077) Merge Fn API and runner v2 configurations for DataflowRunner Fix up! formatting Add validate runner test for testing custom merging windows fn without custom window types Revert "Revert "[BEAM-2914] Add portable merging window support to Python. (apache#12995)"" [BEAM-10961] fix stray reordering of lines [BEAM-10961] enable strict dependency checking for sdks/java/extensions/sorter [BEAM-10961] enable strict dependency checking for sdks/java/extensions/sketching [BEAM-10961] enable strict dependency checking for sdks/java/extensions/schemaio-expansion-service [BEAM-10961] enable strict dependency checking for sdks/java/extensions/protobuf [BEAM-10961] enable strict dependency checking for sdks/java/extensions/ml [BEAM-10961] enable strict dependency checking for sdks/java/extensions/kyro [BEAM-10961] enable strict dependency checking for sdks/java/extensions/join-library [BEAM-10961] undo line moves (originally intended for alphabeticization) [BEAM-10961] enable strict dependency checking for sdks/java/extensions/jackson [BEAM-10961] Enable strict dependency checking on sdks/java/extensions/sql (apache#13830) [BEAM-10961] enable strict dependency checking for sdks/java/extensions/euphoria [BEAM-10961] enable strict dependency checking for sdks/java/io/parquet (apache#14062) [BEAM-10961] enable strict dependency checking for sdks/java/io/thrift (apache#14066) Refactor ZetaSqlDialectSpecTest and add some passing tests. (apache#14080) [BEAM-11864] Use objects.equals instead of raw comparison [BEAM-11707] Change WindmillStateCache cache invalidation to be based upon reference invalidation instead of expensive set management. Reduce operations of shared cache by caching per-key object sets locally and flushing as groups to shared cache. Remove byte tracking which could be racy based upon background evictions in favor of just iterating for rendering the status page. This also lets us capture more stats. [BEAM-11730] Reduce context switching overhead for appliance reads by issuing reads directly from calling threads in the case that there is no reads being queued. Fix preview Show string from Dataflow service when job terminates in unrecognized state Log a warning when Dataflow returns an unrecognized state Merge pull request apache#14033 from [BEAM-11408] Integrate Python BigQuery sink with GroupIntoBatches Remove SYNCHRONIZED_PROCESSING_TIME from model proto Remove use of model SYNCHRONIZED_PROCESSING_TIME Merge redundant model feature columns in capability matrix Remove MapReduce runner from capability matrix, because it is on a branch and unreleased Remove JStorm runner from capability matrix, because it is on a branch and unreleased Remove retractions from capability matrix, because they do not exist yet Remove metadata-driven triggers from capability matrix, because they do not exist [BEAM-10937] Add Tour of Beam page (apache#13747) [BEAM-11344] Apply "Become a Committer" changes from Website Revamp (apache#14036) Merge pull request apache#14046 from [BEAM-11791] Adding a microbenchmark for TestStream Returning successful writes in FhirIO.Write.Result (apache#14034) Fixup [BEAM-10961] enable strict dependency checking for sdks/java/io/file-based-io-tests (apache#14052) [BEAM-10961] enable strict dependency checking for sdks/java/io/contextualtextio (apache#14049) [BEAM-10961] enable strict dependency checking for sdks/java/io/kinesis (apache#14058) [BEAM-10961] enable strict dependency checking for sdks/java/io/bigquery-io-perf-tests (apache#14048) [BEAM-10961] enable strict dependency checking for sdks/java/io/elasticsearch (apache#14050) [BEAM-10961] enable strict dependency checking for sdks/java/io/expansion-service (apache#14051) [BEAM-10961] enable strict dependency checking for sdks/java/io/jdbc (apache#14055) [BEAM-10961] enable strict dependency checking for sdks/java/io/jms (apache#14056) [BEAM-10961] enable strict dependency checking for sdks/java/io/kafka (apache#14057) [BEAM-10961] enable strict dependency checking for sdks/java/io/hcatalog (apache#14053) [BEAM-11859] Fixed bug in python S3 IO [BEAM-10114] Fix PerSubscriptionPartitionSdf to not rely on the presence of BundleFinalizer [BEAM-10114] Fix PerSubscriptionPartitionSdf to not rely on the presence of BundleFinalizer [BEAM-10961] fix spacing [BEAM-10961] enable strict dependency checking for sdks/java/io/xml [BEAM-10961] enable strict dependency checking for sdks/java/io/tika ...

…13791) * strict dependency checking for sdks/io/gcp * add compile(runners:core-java) since it was usedUndeclared * [BEAM-10961] suggested changes: better use of existing library variables, remove testImplementation line * reintroduce some dependencies but as runtimeOnly * remove stray space * [BEAM-10961] create variables for some grpc libraries * [BEAM-10961] add back compile+permitUnusedDeclared lines * [BEAM-10961] fix whitespace in comment block and uncomment enableStrictDependencies=true * [BEAM-10961] uncomment out permitUnusedDeclared - to show error in PR * remove trailing whitespace * [BEAM-10961] hardcode dependency in permitUnusedDeclared lines * [BEAM-10961] make commenting for BEAM-11761 consistent with other PRs * [BEAM-10961] use enforcedPlatform dependency to allow permitUnusedDeclared to support library.java.* variables * [BEAM-10961] (1) add reference to BEAM-11761 (2) fix spacing (3) permitUnusedDeclared library.java.proto_google_cloud_bigquerybeta2_storage_v1 * [BEAM-10961] (1) remove stray compile/permitUnusedDeclared pair added earlier (2) group compile project() declarations * [BEAM-10961] remove proto_google_cloud_bigquery_storage_v1beta1

shehzaadn-vd mentioned this pull request Jan 22, 2021

[BEAM-10961] Enabled strict dependency over sdks-io #13688

Closed

4 tasks

shehzaadn-vd changed the title ~~[BEAM-10961] Strict dependency checking for sdks/io/gcp~~ [BEAM-10961] Strict dependency checking for sdks/java/io/gcp Jan 22, 2021

ibzib requested changes Jan 23, 2021

View reviewed changes

shehzaadn-vd force-pushed the sdks-io-gcp branch from c691597 to 2aa2a9c Compare January 23, 2021 06:00

shehzaadn-vd added 2 commits January 27, 2021 04:43

strict dependency checking for sdks/io/gcp

8137647

add compile(runners:core-java) since it was usedUndeclared

b3639e3

shehzaadn-vd force-pushed the sdks-io-gcp branch from 2aa2a9c to b3639e3 Compare January 27, 2021 13:08

[BEAM-10961] suggested changes: better use of existing library variab…

89e3567

…les, remove testImplementation line

shehzaadn-vd commented Jan 28, 2021

View reviewed changes

sdks/java/io/google-cloud-platform/build.gradle Show resolved Hide resolved

shehzaadn-vd commented Jan 28, 2021

View reviewed changes

shehzaadn-vd added 2 commits January 27, 2021 20:36

reintroduce some dependencies but as runtimeOnly

a556379

remove stray space

b06b9f1

ibzib reviewed Feb 1, 2021

View reviewed changes

shehzaadn-vd added 4 commits February 6, 2021 05:19

[BEAM-10961] create variables for some grpc libraries

bef61df

[BEAM-10961] add back compile+permitUnusedDeclared lines

8fcd746

[BEAM-10961] fix whitespace in comment block and uncomment enableStri…

6346e90

…ctDependencies=true

[BEAM-10961] uncomment out permitUnusedDeclared - to show error in PR

7604e0c

shehzaadn-vd added 2 commits February 6, 2021 06:23

remove trailing whitespace

226969b

[BEAM-10961] hardcode dependency in permitUnusedDeclared lines

9b13f14

shehzaadn-vd mentioned this pull request Feb 13, 2021

[BEAM-10961] Enable strict dependency checking on sdks/java/extensions/sql #13830

Merged

4 tasks

shehzaadn-vd added 4 commits February 13, 2021 00:40

[BEAM-10961] make commenting for BEAM-11761 consistent with other PRs

3d08996

[BEAM-10961] use enforcedPlatform dependency to allow permitUnusedDec…

09525a1

…lared to support library.java.* variables

merge master

f44cea0

[BEAM-10961] (1) add reference to BEAM-11761 (2) fix spacing (3) perm…

feac628

…itUnusedDeclared library.java.proto_google_cloud_bigquerybeta2_storage_v1

ibzib reviewed Feb 25, 2021

View reviewed changes

sdks/java/io/google-cloud-platform/build.gradle Show resolved Hide resolved

sdks/java/io/google-cloud-platform/build.gradle Outdated Show resolved Hide resolved

[BEAM-10961] (1) remove stray compile/permitUnusedDeclared pair added…

5d28a06

… earlier (2) group compile project() declarations

ibzib reviewed Feb 25, 2021

View reviewed changes

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy Outdated Show resolved Hide resolved

[BEAM-10961] remove proto_google_cloud_bigquery_storage_v1beta1

2187c9a

ibzib approved these changes Feb 26, 2021

View reviewed changes

ibzib merged commit 03b883b into apache:master Feb 26, 2021

	//enableStrictDependencies:true,
	enableStrictDependencies:true,

[BEAM-10961] Strict dependency checking for sdks/java/io/gcp #13791

[BEAM-10961] Strict dependency checking for sdks/java/io/gcp #13791

Uh oh!

Conversation

shehzaadn-vd commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

shehzaadn-vd commented Jan 22, 2021

Uh oh!

shehzaadn-vd commented Jan 22, 2021

Uh oh!

shehzaadn-vd commented Jan 22, 2021

Uh oh!

shehzaadn-vd commented Jan 22, 2021

Uh oh!

ibzib commented Jan 22, 2021

Uh oh!

ibzib left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Jan 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sonam-vend commented Jan 25, 2021

Uh oh!

Uh oh!

ibzib commented Jan 28, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shehzaadn-vd commented Jan 28, 2021

Uh oh!

shehzaadn-vd commented Jan 28, 2021

Uh oh!

shehzaadn-vd commented Jan 28, 2021

Uh oh!

shehzaadn-vd commented Feb 1, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shehzaadn-vd commented Feb 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shehzaadn-vd commented Feb 6, 2021

Uh oh!

shehzaadn-vd commented Feb 7, 2021

Uh oh!

shehzaadn-vd commented Jan 22, 2021 •

edited

Loading

codecov bot commented Jan 23, 2021 •

edited

Loading

shehzaadn-vd commented Feb 6, 2021 •

edited

Loading