Support TIMESTAMP type in BigQueryIO with BEAM_ROW output type, and in read_gbq #23070

svetakvsundhar · 2022-09-07T20:42:09Z

This PR adds support for the TIMESTAMP time from BigQuery, and maps it to a BEAM_ROW type. This will be supported in Python's BigQuery IO implementation, as well as read_gbq.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2022-09-07T21:10:02Z

Codecov Report

Merging #23070 (1b15491) into master (25c6ed7) will decrease coverage by 0.26%.
The diff coverage is 57.14%.

@@            Coverage Diff             @@
##           master   #23070      +/-   ##
==========================================
- Coverage   73.67%   73.41%   -0.27%     
==========================================
  Files         714      719       +5     
  Lines       95220    95691     +471     
==========================================
+ Hits        70154    70248      +94     
- Misses      23769    24146     +377     
  Partials     1297     1297

Flag	Coverage Δ
python	`83.04% <57.14%> (-0.45%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...python/apache_beam/io/gcp/bigquery_schema_tools.py	`77.55% <57.14%> (-6.54%)`	⬇️
sdks/python/apache_beam/io/gcp/bigquery_tools.py	`73.34% <0.00%> (-12.39%)`	⬇️
...s/interactive/dataproc/dataproc_cluster_manager.py	`71.72% <0.00%> (-5.70%)`	⬇️
.../python/apache_beam/testing/test_stream_service.py	`88.09% <0.00%> (-4.77%)`	⬇️
sdks/python/apache_beam/internal/pickler.py	`92.00% <0.00%> (-3.46%)`	⬇️
sdks/python/apache_beam/io/gcp/bigquery.py	`71.31% <0.00%> (-2.94%)`	⬇️
sdks/python/apache_beam/internal/gcp/json_value.py	`85.50% <0.00%> (-2.90%)`	⬇️
...on/apache_beam/runners/dataflow/dataflow_runner.py	`80.80% <0.00%> (-2.20%)`	⬇️
sdks/python/apache_beam/typehints/schemas.py	`91.89% <0.00%> (-1.96%)`	⬇️
... and 62 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

github-actions · 2022-09-08T00:20:10Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @tvalentyn for label python.
R: @ahmedabu98 for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

tvalentyn · 2022-09-08T00:23:53Z

Run Python 3.8 PostCommit

tvalentyn · 2022-09-08T00:24:32Z

will defer to @ahmedabu98 for the review assuming tests pass.

sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py

svetakvsundhar · 2022-09-08T01:30:44Z

whoops, i didnt realize the reviewer would be auto-assigned.

anyways, this integration test will fail with AttributeError: 'datetime.datetime' object has no attribute 'micros' [while running 'ReadFromBigQuery/ParDo(BeamSchemaConversionDoFn)']

I wanted to ask, should apache_beam.utils.timestamp be using TimestampCoder? Looks like it's using RowCoder now.

svetakvsundhar · 2022-09-08T15:41:21Z

cc: @TheNeuralBit

ahmedabu98

Thanks @svetakvsundhar, left a couple comments

sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py

sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py

github-actions · 2022-09-23T12:40:22Z

Reminder, please take a look at this pr: @tvalentyn @ahmedabu98

github-actions · 2022-09-27T13:12:38Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @y1chi for label python.
R: @johnjcasey for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

tvalentyn · 2022-09-29T02:08:53Z

waiting on author

… into apache_beam.utils.timestamp.Timestamp

svetakvsundhar · 2022-09-29T22:48:09Z

Run Python 3.7 PostCommit

github-actions · 2022-10-15T12:14:10Z

Reminder, please take a look at this pr: @y1chi @johnjcasey

svetakvsundhar · 2022-10-18T21:36:26Z

Thanks all for the suggestions. Because the BQ Timestamp object was actually a datetime.datetime, we needed to modify the conversion ParDo logic to convert into an apache_beam_utils.timestamp.Timestamp

svetakvsundhar · 2022-10-19T00:21:23Z

Run Python 3.7 PostCommit

svetakvsundhar · 2022-10-19T12:34:17Z

Run Python PreCommit

svetakvsundhar · 2022-10-19T12:34:51Z

Run Python 3.7 PostCommit

ahmedabu98 · 2022-10-19T18:27:15Z

Run Python 3.8 PostCommit

tvalentyn · 2022-10-19T18:32:26Z

@AnandInguva is looking into py3.10 issue, looks unrelated, py3.8 has bq tests and should provide sufficient coverage, not sure why py3.7 failed, looks like infra error

tvalentyn · 2022-10-19T18:35:09Z

other errors:
apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers.test_batch_pardo_trigger_flush --> #21104
apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithExternalEnv.test_pardo_large_input -> #22321

ahmedabu98 · 2022-10-19T18:36:33Z

Py37 postcommits have been consistently timing out: Build timed out (after 240 minutes). Marking the build as aborted.

ahmedabu98 · 2022-10-19T18:37:12Z

Py38 more reliable as it isn't responsible for as many tests as Py37

ahmedabu98

This LGTM, let's wait for Py38 Postcommits though

svetakvsundhar · 2022-10-19T18:43:33Z

sg, looks like 3.7 ran the tests in question

tvalentyn · 2022-10-19T19:35:44Z

Py37 postcommits have been consistently timing out: Build timed out (after 240 minutes). Marking the build as aborted.

This is strange, I have increased the timeout in: #23595

why rowcoder?

277cb40

github-actions bot added gcp io python labels Sep 7, 2022

github-actions bot added the Next Action: Reviewers label Sep 8, 2022

tvalentyn reviewed Sep 8, 2022

View reviewed changes

sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py Outdated Show resolved Hide resolved

ahmedabu98 requested changes Sep 8, 2022

View reviewed changes

sdks/python/apache_beam/io/gcp/bigquery_schema_tools.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py Outdated Show resolved Hide resolved

svetakvsundhar added 4 commits September 14, 2022 17:56

registering datetime.datetime as schema type

a90cef7

registering datetime.datetime as schema type

509a164

registered type in schemas.py

7e4b417

registered type in schemas.py

30b0d99

github-actions bot added the slow-review label Sep 23, 2022

github-actions bot removed the slow-review label Sep 27, 2022

github-actions bot added Next Action: Author and removed Next Action: Reviewers labels Sep 29, 2022

convert Timestamp to datetime.datetime, which will then get converted…

7e0239a

… into apache_beam.utils.timestamp.Timestamp

github-actions bot added Next Action: Reviewers and removed Next Action: Author labels Sep 29, 2022

experiment with converting to datetime.datetime

ed3b317

Timestamp to datetime.datetime mapping

443ac88

github-actions bot added the slow-review label Oct 15, 2022

Timestamp conversion

72ca4ca

Timestamp conversion with lint

04969a7

Timestamp conversion with lint

1b15491

svetakvsundhar requested review from ahmedabu98 and tvalentyn and removed request for ahmedabu98 and tvalentyn October 19, 2022 15:09

tvalentyn approved these changes Oct 19, 2022

View reviewed changes

ahmedabu98 approved these changes Oct 19, 2022

View reviewed changes

github-actions bot added slow-review and removed slow-review labels Oct 19, 2022

tvalentyn merged commit 61b8677 into apache:master Oct 19, 2022

svetakvsundhar deleted the timestamp_branch branch October 19, 2022 22:21

Support TIMESTAMP type in BigQueryIO with BEAM_ROW output type, and in read_gbq #23070

Support TIMESTAMP type in BigQueryIO with BEAM_ROW output type, and in read_gbq #23070

Uh oh!

Conversation

svetakvsundhar commented Sep 7, 2022

GitHub Actions Tests Status (on master branch)

Uh oh!

codecov bot commented Sep 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Sep 8, 2022

Uh oh!

tvalentyn commented Sep 8, 2022

Uh oh!

tvalentyn commented Sep 8, 2022

Uh oh!

Uh oh!

svetakvsundhar commented Sep 8, 2022

Uh oh!

svetakvsundhar commented Sep 8, 2022

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Sep 23, 2022

Uh oh!

github-actions bot commented Sep 27, 2022

Uh oh!

tvalentyn commented Sep 29, 2022

Uh oh!

svetakvsundhar commented Sep 29, 2022

Uh oh!

github-actions bot commented Oct 15, 2022

Uh oh!

svetakvsundhar commented Oct 18, 2022

Uh oh!

svetakvsundhar commented Oct 19, 2022

Uh oh!

svetakvsundhar commented Oct 19, 2022

Uh oh!

svetakvsundhar commented Oct 19, 2022

Uh oh!

ahmedabu98 commented Oct 19, 2022

Uh oh!

tvalentyn commented Oct 19, 2022

Uh oh!

tvalentyn commented Oct 19, 2022

Uh oh!

ahmedabu98 commented Oct 19, 2022

Uh oh!

ahmedabu98 commented Oct 19, 2022

Uh oh!

ahmedabu98 left a comment

Choose a reason for hiding this comment

Uh oh!

svetakvsundhar commented Oct 19, 2022

Uh oh!

tvalentyn commented Oct 19, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Sep 7, 2022 •

edited

Loading