Add BigTableRead connector and new feature implemented #35696

arnavarora2004 · 2025-07-25T17:12:09Z

@fozzie15 @ahmedabu98 @damccorm @derrickaw

I added the BigTable Read Connector for BeamYaml

added new logic for bigtable yaml with the option to simplify what the read from bigtable returns, a flattened feature makes it more readable than the old logic,

added tests to integration_test.py (I commented out some bugs in integration_test.py from new commits on the main branch, will remove if no more errors)

fixed BigTableReadSchemaTransformIT to reflect new functionality

made bigtableWrite family_name as string instead of bytes

all logic works, please let me know if anything looks off/funny and if anything can be improved

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

… connected and actually look good on user end for mutations

…est debugging

…test wise

…hemaTransformProviderIT, and testing out new mutations etc

…ted new user input, all mutations work correctly, put demo code for it

…am/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com>

…am/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProviderIT.java Co-authored-by: Derrick Williams <myutat@gmail.com>

sdks/python/apache_beam/yaml/integration_tests.py

github-actions · 2025-07-31T20:39:52Z

Assigning reviewers:

R: @jrmccluskey for label python.
R: @chamikaramj for label java.
R: @meeral-k for label bigtable.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

sdks/python/apache_beam/yaml/tests/bigtable.yaml

...m/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableReadSchemaTransformProvider.java

mutianf · 2025-08-01T14:29:53Z

.../src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java

                + "key\": ByteString\n"
                + "\"type\": String\n"
                + "\"value\": ByteString\n"
                + "\"column_qualifier\": ByteString\n"


The type for qualifier is bytearray but here it's bytestring, is this accurate? Same for key and value

thank you for that catch!

Bump on this ^

...src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableReadSchemaTransformProviderIT.java

sdks/python/apache_beam/io/gcp/bigtableio.py

sdks/python/apache_beam/yaml/standard_io.yaml

derrickaw · 2025-08-06T13:08:01Z

Run PythonDocker PreCommit 3.9

ahmedabu98 · 2025-08-06T16:14:10Z

sdks/python/apache_beam/pvalue.py

+    if type(self) == type(other):
+      other_dict = other.__dict__
+    elif type(other) == type(NamedTuple):
+      other_dict = other._asdict()
+    else:
+      return False
    return (
-        type(self) == type(other) and
-        len(self.__dict__) == len(other.__dict__) and all(
-            s == o
-            for s, o in zip(self.__dict__.items(), other.__dict__.items())))
+        len(self.__dict__) == len(other_dict) and
+        all(s == o for s, o in zip(self.__dict__.items(), other_dict.items())))


Ahh I think the problem described in #35790 is because this only handles top-level NamedTuples. Should be a straightforward fix but not a blocker for this PR

Per Derricks advice, I or derrick can work on it after the PR is merged

ahmedabu98

I don't like that we have BigtableReadSchemaTransformConfiguration implements Serializable, but besides that this LGTM

arnavarora2004 · 2025-08-06T17:31:29Z

Run PythonDocker PreCommit 3.9

* Refactored BigTableReadSchemaTransformConfiguration * changed scope, working on buffer class for making BigTable yaml fully connected and actually look good on user end for mutations * Finished up a bit of standard_io.yaml * Finished up a bit of standard_io.yaml * Added bigTable test * changed some tests for BigTable * Added new IT file for simpleWrite and also made changes integration test debugging * Added new IT file for simpleWrite and also made changes integration test debugging * SetCell mutation test works, I want to see if this draft PR works CI test wise * Fixed a slight error * Added way more changes to integrations test.py, BigTableSimpleWriteSchemaTransformProviderIT, and testing out new mutations etc * BigTableSimpleWriteSchemaTransformProviderIT finished changes to mutated new user input, all mutations work correctly, put demo code for it * Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com> * Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com> * Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com> * Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com> * Update sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProviderIT.java Co-authored-by: Derrick Williams <myutat@gmail.com> * changed comments * Added changes from derrick comments * Added default schema maybe fixes the issues * Added schema to every test specificly, will run tests to see if it works * Added default schema maybe fixes the issues * Following formatting tests * Following formatting tests * Following checkstyle tests * Made schema and test changes * Made schema and test changes * Made schema and test changes * Made schema and test changes * Made schema and test changes * Added final test * changed timestamp values * added all mutations test * added all mutations test * pushed changes to format errors * pushed changes to format errors * Delete 4 * pushed changes to format errors * pushed changes to format errors * pushed changes to format errors * pushed changes to debugging errors * pushed changes to debugging errors * to see internal error added print(will remove) * to see internal error added print(will remove) * to see internal error added print(will remove) * import fixes * import fixes * import fixes * import fixes * import fixes * import fixes * pushed changes to debugging errors * pushed changes to debugging errors * pushed changes to debugging errors, added pulls from other beam * made changes to allMutations test * made changes to allMutations test * pushed changes to debugging errors, added pulls from other beam * pushed changes to debugging errors, added pulls from other beam * pushed changes to debugging errors, added pulls from other beam * pushed changes to debugging errors, added pulls from other beam * pushed changes to debugging errors, added pulls from other beam * new read errors fixed * pushed changes to debugging errors, added pulls from other beam * consolidated schema transform files, fixed small issues and bugs * consolidated schema transform files, fixed small issues and bugs * consolidated schema transform files, fixed small issues and bugs * consolidated schema transform files, fixed small issues and bugs * pushed changes to debugging errors, added pulls from other beam * pushed changes from ahmed * pushed changes from ahmed * pushed changes from ahmed * pushed changes from ahmed * pushed changes from ahmed * pushed changes from ahmed * pushed changes from ahmed * pushed changes from ahmed * Following checkstyle tests * Following checkstyle tests * pushed new changes to BigTableRead, making it work with new functionality feature of allowing flatten (defaulted to true) * pushed new changes to BigTableRead, making it work with new functionality feature of allowing flatten (defaulted to true) and added a new test in IT and fixed formatting stuff * pushed new changes to BigTableRead, making it work with new functionality feature of allowing flatten (defaulted to true) and added a new test in IT and fixed formatting stuff * Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableReadSchemaTransformProvider.java Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com> * pushed new changes to BigTableRead, making it work with new functionality feature of allowing flatten (defaulted to true) and added a new test in IT and fixed formatting stuff * pushed new changes to BigTableRead, making it work with new functionality feature of allowing flatten (defaulted to true) and added a new test in IT and fixed formatting stuff * pushed new changes to BigTableRead, making it work with new functionality feature of allowing flatten (defaulted to true) and added a new test in IT and fixed formatting stuff * pushed new changes to BigTableRead, making it work with new functionality feature of allowing flatten (defaulted to true) and added a new test in IT and fixed formatting stuff * new mongo files in branch * fixed family_name to string * fixed family_name to string * fixed family_name to string * fixed family_name to string * fixed family_name to string * fixed family_name to string * fixed family_name to string * fixed family_name to string * fixed cmmit issues * commented assert test, everything should work now --------- Co-authored-by: Derrick Williams <myutat@gmail.com> Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com>

derrickaw · 2025-09-24T20:34:01Z

#28672
#33902

arnavarora2004 and others added 30 commits May 21, 2025 11:43

Refactored BigTableReadSchemaTransformConfiguration

ad54a58

changed scope, working on buffer class for making BigTable yaml fully…

0edf81d

… connected and actually look good on user end for mutations

Finished up a bit of standard_io.yaml

18c9395

Finished up a bit of standard_io.yaml

a25033e

Merge branch 'apache:master' into master

0e4fb14

Added bigTable test

c048dcf

Merge branch 'master' of github.com:arnavarora2004/ArnavBeamWork

4ebc7c8

changed some tests for BigTable

3bb3dfc

Added new IT file for simpleWrite and also made changes integration t…

a8b8196

…est debugging

Added new IT file for simpleWrite and also made changes integration t…

cf8bd8f

…est debugging

SetCell mutation test works, I want to see if this draft PR works CI …

a06d7c6

…test wise

Fixed a slight error

5760278

Merge branch 'apache:master' into master

5cb0dd7

Added way more changes to integrations test.py, BigTableSimpleWriteSc…

f2640ae

…hemaTransformProviderIT, and testing out new mutations etc

Merge branch 'master' of github.com:arnavarora2004/ArnavBeamWork

69467c5

BigTableSimpleWriteSchemaTransformProviderIT finished changes to muta…

121ddf6

…ted new user input, all mutations work correctly, put demo code for it

Merge branch 'apache:master' into master

82d3612

Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/be…

14ca717

…am/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com>

Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/be…

c36d864

…am/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com>

Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/be…

d6366dd

…am/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com>

Merge branch 'apache:master' into master

fcb5d03

Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/be…

38ff386

…am/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProvider.java Co-authored-by: Derrick Williams <myutat@gmail.com>

Update sdks/java/io/google-cloud-platform/src/test/java/org/apache/be…

a15e4ff

…am/sdk/io/gcp/bigtable/BigtableSimpleWriteSchemaTransformProviderIT.java Co-authored-by: Derrick Williams <myutat@gmail.com>

changed comments

c759a07

Merge branch 'master' of github.com:arnavarora2004/ArnavBeamWork

8e19a81

Added changes from derrick comments

b6d0157

Merge branch 'apache:master' into master

7ea3f76

Merge branch 'apache:master' into master

a35ced7

Added default schema maybe fixes the issues

50bb5a3

Added schema to every test specificly, will run tests to see if it works

426519d

arnavarora2004 added 5 commits July 30, 2025 16:23

fixed family_name to string

2b0806d

fixed family_name to string

77e4dd3

fixed family_name to string

b4ad9e4

Merge branch 'apache:master' into master

4b9ce38

Merge branch 'mongodb' into master

f56e50d

github-actions bot added the mongodb label Jul 31, 2025

derrickaw reviewed Jul 31, 2025

View reviewed changes

sdks/python/apache_beam/yaml/integration_tests.py Outdated Show resolved Hide resolved

arnavarora2004 added 2 commits July 31, 2025 12:59

fixed family_name to string

3386f94

fixed family_name to string

2fd82ea

github-actions bot removed the mongodb label Jul 31, 2025

arnavarora2004 added 2 commits July 31, 2025 13:03

fixed cmmit issues

3059504

Merge branch 'apache:master' into master

e0a2bd5

github-actions bot added the Next Action: Reviewers label Jul 31, 2025

derrickaw reviewed Aug 1, 2025

View reviewed changes

sdks/python/apache_beam/yaml/tests/bigtable.yaml Outdated Show resolved Hide resolved

mutianf reviewed Aug 1, 2025

View reviewed changes

arnavarora2004 added 4 commits August 1, 2025 18:47

Merge branch 'apache:master' into master

af0e80b

Merge branch 'apache:master' into master

188222f

Merge branch 'apache:master' into master

ef8b856

commented assert test, everything should work now

669b80d

ahmedabu98 reviewed Aug 6, 2025

View reviewed changes

ahmedabu98 approved these changes Aug 6, 2025

View reviewed changes

derrickaw mentioned this pull request Aug 6, 2025

[Bug]: BeamSchema issue #35790

Open

17 tasks

derrickaw approved these changes Aug 6, 2025

View reviewed changes

ahmedabu98 merged commit 4114f7c into apache:master Aug 6, 2025
104 of 105 checks passed

derrickaw mentioned this pull request Sep 24, 2025

[yaml]: Normalize BigtableIO #28672

Closed

15 tasks

Add BigTableRead connector and new feature implemented #35696

Add BigTableRead connector and new feature implemented #35696

Uh oh!

Conversation

arnavarora2004 commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

Uh oh!

github-actions bot commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mutianf Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

arnavarora2004 Aug 1, 2025

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derrickaw commented Aug 6, 2025

Uh oh!

ahmedabu98 Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arnavarora2004 Aug 6, 2025

Choose a reason for hiding this comment

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

arnavarora2004 commented Aug 6, 2025

Uh oh!

Uh oh!

derrickaw commented Sep 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

arnavarora2004 commented Jul 25, 2025 •

edited

Loading

ahmedabu98 Aug 6, 2025 •

edited

Loading

derrickaw commented Sep 24, 2025 •

edited

Loading