[BEAM-7246] Added Google Spanner Write Transform #10712

mszb · 2020-01-29T18:08:17Z

In addition to ReadFromSpanner (PR), this transform allows users to write on the google cloud spanner by using WriteToSpanner transform.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Gearpump	Samza	Spark
Go		---	---	---	---
Java
Python		---		---	---
XLang	---	---	---	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

lukecwik · 2020-01-29T18:28:49Z

ping for tests

mszb · 2020-01-30T07:37:53Z

R: @chamikaramj
R: @aaltay

charlesccychen · 2020-01-30T10:22:03Z

retest this please

charlesccychen · 2020-01-30T10:22:26Z

Run Python Precommit

mszb

Here I've added some explanations to help the review process. Feedbacks are always welcome!

mszb · 2020-01-30T09:04:13Z

sdks/python/apache_beam/io/gcp/experimental/spannerio.py

+
+  def process(self, element):
+    if element.primary().operation == 'delete':
+      # As delete mutations are not batchable.


As per java implementation, we can not determine the size of delete mutation, so we simply mark them as unbatchable mutations.

mszb · 2020-01-30T09:11:27Z

sdks/python/apache_beam/io/gcp/experimental/spannerio.py

+                       "provided: <%s: %s>" % (self.__class__.__name__,
+                                               str(self.__dict__)))
+
+  def __call__(self, *args, **kwargs):


Since in python we don't use builder pattern, I use this magic method to construct the mutation object! This class also have the static method which I thought is more convenient for the user to use.

mszb · 2020-01-30T09:15:59Z

sdks/python/apache_beam/io/gcp/experimental/spannerio.py

  from google.cloud.spanner import KeySet
+  from google.cloud.spanner_v1 import batch
  from google.cloud.spanner_v1.database import BatchSnapshot
+  from google.cloud.spanner_v1.proto.mutation_pb2 import Mutation


Since spanner package wont expose the Mutation and batch objects, so this is the only way to import it.

mszb · 2020-01-30T09:35:51Z

sdks/python/apache_beam/io/gcp/experimental/spannerio.py

+      values: Values to be modified.
+    """
+    return _Mutator(
+        mutation=Mutation(insert=batch._make_write_pb(table, columns, values)),


Since we need to calculate the mutation size to create the batches, we need to create Mutation object which has the method ByteSize() and we uses its value to determine how many mutations we can have in one batch on _BatchFn dofn. We are not passing these mutation object to spanner or anywhere, just using them to calculate the bytesize for write transform!

mszb · 2020-01-30T09:41:35Z

sdks/python/apache_beam/io/gcp/experimental/spannerio.py

+
+@with_input_types(MutationGroup)
+@with_output_types(MutationGroup)
+class _BatchableFilterFn(DoFn):


In this DoFn, we are just filtering the objects whos mutation size is greater the max_batch_size provided by the user. This helps us to focus on only batchable mutation which will further passed on the _BatchFn dofn to make batches.

iemejia · 2020-01-30T13:43:49Z

Retest this please

lukecwik · 2020-01-30T17:19:07Z

ping for test

chamikaramj · 2020-02-04T16:46:19Z

cc: @nithinsujir and @nielm

sdks/python/apache_beam/io/gcp/experimental/spannerio.py

kamilwu · 2020-02-06T12:34:19Z

Just to let you know that we've just introduced Python autoformatter. Your merge conflict might be a result of this.
Here you can find an instruction on how to run autoformatter: https://cwiki.apache.org/confluence/display/BEAM/Python+Tips, section Formatting.
Sorry for inconvenience.

mszb · 2020-02-06T15:14:44Z

Just to let you know that we've just introduced Python autoformatter. Your merge conflict might be a result of this.
Here you can find an instruction on how to run autoformatter: https://cwiki.apache.org/confluence/display/BEAM/Python+Tips, section Formatting.
Sorry for inconvenience.

No worries @kamilwu , i'll resolve the conflicts on my next commit. Thanks for the heads up :)

mszb · 2020-02-10T18:13:33Z

ping for test

aaltay · 2020-02-10T18:14:12Z

Trigger tests.

aaltay · 2020-02-10T18:14:48Z

Could you also add a new feature note in https://github.com/apache/beam/blob/master/CHANGES.md

mszb · 2020-02-10T18:32:47Z

Could you also add a new feature note in https://github.com/apache/beam/blob/master/CHANGES.md

Done. :)

mszb · 2020-02-10T18:40:25Z

ping for test

aaltay · 2020-02-10T18:47:41Z

Trigger tests.

iemejia · 2020-02-11T10:24:23Z

retest this please

aaltay · 2020-02-11T18:11:49Z

retest this please

aaltay · 2020-02-11T18:16:17Z

/cc @markflyhigh - Tests seems to be not triggering?

mszb · 2020-02-12T14:47:06Z

@aaltay @markflyhigh Seems like jobs are triggered and completed successfully but showing no activity on github!

https://builds.apache.org/job/beam_PreCommit_Python_Commit/11080/
https://builds.apache.org/job/beam_PreCommit_PythonFormatter_Commit/67/

aaltay · 2020-02-12T18:02:42Z

retest this please

mszb · 2020-02-12T19:18:11Z

@aaltay All three tests are failed due to ImportError: No module named 'pycodestyle' in avro-python3 package.

Portable_Python PreCommit
https://scans.gradle.com/s/3qf5sqnettmmq/console-log?task=:sdks:python:test-suites:portable:py35:installGcpTest#L299

PreCommit
https://scans.gradle.com/s/c5ncivj7k2pko/console-log?task=:sdks:python:test-suites:dataflow:py37:installGcpTest#L658

PythonFormatter PreCommit
https://scans.gradle.com/s/i6nvgyym5tfqk/console-log?task=:sdks:python:test-suites:tox:py37:formatter#L267

Could you please rerun these test, possibly it'll fix the issue!

aaltay · 2020-02-12T21:04:33Z

I believe this is fixed with #10844, you may need to rebase.

mszb · 2020-02-13T07:23:22Z

@aaltay I've rebased my branch, could you please trigger the tests!

Thanks.

aaltay · 2020-02-13T16:44:22Z

retest this please

aaltay · 2020-02-13T16:49:49Z

retest this please

mszb · 2020-02-15T18:37:04Z

@chamikaramj @nielm could you please verify the changes.

nielm

LGTM

mszb · 2020-02-18T19:18:22Z

Thanks @nielm .. @chamikaramj is there anything you would like to add or we are ready for merge?

aaltay · 2020-02-18T21:52:14Z

retest this please

aaltay · 2020-02-18T21:53:10Z

retest this please

chamikaramj · 2020-02-18T21:57:38Z

LGTM. Thanks.

Added Spanner Write Transform

c8831cb

mszb force-pushed the BEAM-7246_gcp_spannerio_write branch from 9bcaf69 to c8831cb Compare January 30, 2020 07:44

mszb commented Jan 30, 2020

View reviewed changes

Fix typo in _WriteGroup class.

fc0c5cf

aaltay requested a review from chamikaramj February 3, 2020 17:37

nielm reviewed Feb 4, 2020

View reviewed changes

sdks/python/apache_beam/io/gcp/experimental/spannerio.py Outdated Show resolved Hide resolved

nielm reviewed Feb 4, 2020

View reviewed changes

sdks/python/apache_beam/io/gcp/experimental/spannerio.py Show resolved Hide resolved

nielm reviewed Feb 4, 2020

View reviewed changes

sdks/python/apache_beam/io/gcp/experimental/spannerio.py Show resolved Hide resolved

mszb and others added 2 commits February 10, 2020 13:11

Merge branch 'master' into BEAM-7246_gcp_spannerio_write

0f5c578

added two batching parameters (max_number_rows, max_number_cells)

2e31bb0

adds io note on CHANGES.md

c51d9a7

Merge branch 'master' into BEAM-7246_gcp_spannerio_write

1f39f31

fix python code formate

fc3e056

Merge branch 'master' into BEAM-7246_gcp_spannerio_write

fdbd04b

probot-autolabeler bot added gcp io python labels Feb 13, 2020

Added docs for spanner write.

989cf4a

nielm approved these changes Feb 17, 2020

View reviewed changes

Merge branch 'master' into BEAM-7246_gcp_spannerio_write

8bc5223

chamikaramj merged commit e73e1d1 into apache:master Feb 18, 2020

[BEAM-7246] Added Google Spanner Write Transform #10712

[BEAM-7246] Added Google Spanner Write Transform #10712

Uh oh!

Conversation

mszb commented Jan 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

Uh oh!

lukecwik commented Jan 29, 2020

Uh oh!

mszb commented Jan 30, 2020

Uh oh!

charlesccychen commented Jan 30, 2020

Uh oh!

charlesccychen commented Jan 30, 2020

Uh oh!

mszb left a comment

Choose a reason for hiding this comment

Uh oh!

mszb Jan 30, 2020

Choose a reason for hiding this comment

Uh oh!

mszb Jan 30, 2020

Choose a reason for hiding this comment

Uh oh!

mszb Jan 30, 2020

Choose a reason for hiding this comment

Uh oh!

mszb Jan 30, 2020

Choose a reason for hiding this comment

Uh oh!

mszb Jan 30, 2020

Choose a reason for hiding this comment

Uh oh!

iemejia commented Jan 30, 2020

Uh oh!

lukecwik commented Jan 30, 2020

Uh oh!

chamikaramj commented Feb 4, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kamilwu commented Feb 6, 2020

Uh oh!

mszb commented Feb 6, 2020

Uh oh!

mszb commented Feb 10, 2020

Uh oh!

aaltay commented Feb 10, 2020

Uh oh!

aaltay commented Feb 10, 2020

Uh oh!

mszb commented Feb 10, 2020

Uh oh!

mszb commented Feb 10, 2020

Uh oh!

aaltay commented Feb 10, 2020

Uh oh!

iemejia commented Feb 11, 2020

Uh oh!

aaltay commented Feb 11, 2020

Uh oh!

aaltay commented Feb 11, 2020

Uh oh!

mszb commented Feb 12, 2020

Uh oh!

aaltay commented Feb 12, 2020

Uh oh!

mszb commented Feb 12, 2020

Uh oh!

aaltay commented Feb 12, 2020

Uh oh!

mszb commented Feb 13, 2020

Uh oh!

aaltay commented Feb 13, 2020

Uh oh!

aaltay commented Feb 13, 2020

Uh oh!

mszb commented Jan 29, 2020 •

edited

Loading