Skip to content

Managed BigQueryIO #31486

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ahmedabu98 merged 30 commits into from
Nov 12, 2024
Merged

Managed BigQueryIO #31486

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
ce14b96
managed bigqueryio
ahmedabu98 Jun 3, 2024
550c1b4
spotless
ahmedabu98 Jun 4, 2024
c94de3c
move managed dependency to test only
ahmedabu98 Jun 4, 2024
912dc08
Merge branch 'master' of https://github.com/ahmedabu98/beam into mana…
ahmedabu98 Jun 5, 2024
f436e62
cleanup after merging snake_case PR
ahmedabu98 Jun 5, 2024
fe60904
choose write method based on boundedness and pipeline options
ahmedabu98 Jul 9, 2024
7d405cf
Merge branch 'master' of https://github.com/ahmedabu98/beam into mana…
ahmedabu98 Jul 9, 2024
d45159f
rename bigquery write config class
ahmedabu98 Jul 9, 2024
989ad0f
spotless
ahmedabu98 Jul 9, 2024
b9b49e7
change read output tag to 'output'
ahmedabu98 Jul 9, 2024
a119bbc
spotless
ahmedabu98 Jul 9, 2024
74bc178
revert logic that depends on DataflowServiceOptions. switching BQ met…
ahmedabu98 Jul 16, 2024
528b504
spotless
ahmedabu98 Jul 16, 2024
dcc398a
fix typo
ahmedabu98 Jul 29, 2024
36edc38
separate BQ write config to a new class
ahmedabu98 Aug 6, 2024
f9be86c
fix doc
ahmedabu98 Aug 6, 2024
bd1e534
Merge branch 'master' of https://github.com/ahmedabu98/beam into mana…
ahmedabu98 Oct 25, 2024
a26765e
resolve after syncing to HEAD
ahmedabu98 Oct 25, 2024
725f7bd
spotless
ahmedabu98 Oct 26, 2024
2631104
fork on batch/streaming
ahmedabu98 Nov 5, 2024
770cf50
cleanup
ahmedabu98 Nov 5, 2024
0a70466
spotless
ahmedabu98 Nov 5, 2024
01a01f7
move forking logic to BQ schematransform side
ahmedabu98 Nov 6, 2024
697c0b8
add file loads translation and tests; add test checks that the correc…
ahmedabu98 Nov 7, 2024
105474b
set top-level wrapper to be the underlying managed BQ transform urn; …
ahmedabu98 Nov 8, 2024
d6b9e69
move unit tests to respectvie schematransform test classes
ahmedabu98 Nov 8, 2024
c0767d7
Merge branch 'master' of https://github.com/ahmedabu98/beam into mana…
ahmedabu98 Nov 8, 2024
a600f62
Merge branch 'master' of https://github.com/ahmedabu98/beam into mana…
ahmedabu98 Nov 8, 2024
ad4dcd9
expose to Python SDK as well
ahmedabu98 Nov 11, 2024
6f325ce
Merge branch 'master' of https://github.com/ahmedabu98/beam into mana…
ahmedabu98 Nov 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/trigger_files/beam_PostCommit_Java_DataflowV2.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run"
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 1
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_Python_Xlang_Gcp_Direct.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 2
"modification": 1
}
4 changes: 4 additions & 0 deletions model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,10 @@ message ManagedTransforms {
"beam:schematransform:org.apache.beam:kafka_read:v1"];
KAFKA_WRITE = 3 [(org.apache.beam.model.pipeline.v1.beam_urn) =
"beam:schematransform:org.apache.beam:kafka_write:v1"];
BIGQUERY_READ = 4 [(org.apache.beam.model.pipeline.v1.beam_urn) =
"beam:schematransform:org.apache.beam:bigquery_storage_read:v1"];
BIGQUERY_WRITE = 5 [(org.apache.beam.model.pipeline.v1.beam_urn) =
"beam:schematransform:org.apache.beam:bigquery_write:v1"];
}
}

Expand Down
1 change: 1 addition & 0 deletions sdks/java/io/google-cloud-platform/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@ dependencies {
testImplementation project(path: ":sdks:java:extensions:google-cloud-platform-core", configuration: "testRuntimeMigration")
testImplementation project(path: ":sdks:java:extensions:protobuf", configuration: "testRuntimeMigration")
testImplementation project(path: ":runners:direct-java", configuration: "shadow")
testImplementation project(":sdks:java:managed")
testImplementation project(path: ":sdks:java:io:common")
testImplementation project(path: ":sdks:java:testing:test-utils")
testImplementation library.java.commons_math3
Expand Down
3 changes: 3 additions & 0 deletions sdks/java/io/google-cloud-platform/expansion-service/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ dependencies {
permitUnusedDeclared project(":sdks:java:io:google-cloud-platform") // BEAM-11761
implementation project(":sdks:java:extensions:schemaio-expansion-service")
permitUnusedDeclared project(":sdks:java:extensions:schemaio-expansion-service") // BEAM-11761
implementation project(":sdks:java:managed")
permitUnusedDeclared project(":sdks:java:managed") // BEAM-11761
Copy link
Member

@kennknowles kennknowles Jul 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notes, no action required on this PR:

  • This is a link to Jira, so probably there's a github issue it is migrated to
  • This should be equivalent to runtimeOnly because it is "implementation" but no static references to it. I would guess this works the same, or else the uberjar plugin might not treat it right.
  • Putting these deps into a docker container without making an uber jar would honestly be better in the case where it does end up in a container, so we keep the original jar metadata.


runtimeOnly library.java.slf4j_jdk14
}

Expand Down
72 changes: 0 additions & 72 deletions ...g/apache/beam/sdk/io/gcp/bigquery/BigQueryFileLoadsWriteSchemaTransformConfiguration.java
View file
Open in desktop

This file was deleted.

256 changes: 0 additions & 256 deletions ...va/org/apache/beam/sdk/io/gcp/bigquery/BigQueryFileLoadsWriteSchemaTransformProvider.java
View file
Open in desktop

This file was deleted.

Loading