Skip to content

Conversation

@lostluck
Copy link
Contributor

@lostluck lostluck commented Jul 28, 2023

Re-enable single iteration "decode on-demand" stream for the Go SDK.

Originally disabled due to #22933, in #23042, the principle remains sound, and it's long past time to look at why it failed in the first place.

The issue was that we didn't drain the iterable properly if there were no reads at all, so we go with the slightly more ham fisted approach of iterating until io.EOF, or stream != nil. If the stream isn't nil, it means the reader got handed off, and it's up to that stream to close.

This avoids unnecessarily draining something off of the main reader which could be very large and costly.

Fixes #23043


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@github-actions github-actions bot added the go label Jul 28, 2023
@lostluck
Copy link
Contributor Author

Run Go Spark ValidatesRunner

@lostluck
Copy link
Contributor Author

Run Go Flink ValidatesRunner

@lostluck
Copy link
Contributor Author

Run Go Samza ValidatesRunner

@lostluck
Copy link
Contributor Author

Run Go PostCommit

@codecov
Copy link

codecov bot commented Jul 28, 2023

Codecov Report

Merging #27744 (7073e7c) into master (b668799) will increase coverage by 0.03%.
Report is 8 commits behind head on master.
The diff coverage is 85.71%.

❗ Current head 7073e7c differs from pull request most recent head 3ddbae0. Consider uploading reports for the commit 3ddbae0 to get more accurate results

@@            Coverage Diff             @@
##           master   #27744      +/-   ##
==========================================
+ Coverage   70.92%   70.95%   +0.03%     
==========================================
  Files         860      860              
  Lines      104860   104866       +6     
==========================================
+ Hits        74368    74406      +38     
+ Misses      28934    28901      -33     
- Partials     1558     1559       +1     
Flag Coverage Δ
go 53.66% <85.71%> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
sdks/go/pkg/beam/core/runtime/exec/fullvalue.go 84.10% <81.81%> (+6.85%) ⬆️
sdks/go/pkg/beam/core/runtime/exec/datasource.go 70.62% <100.00%> (+3.82%) ⬆️
sdks/go/pkg/beam/core/runtime/exec/plan.go 68.07% <100.00%> (+0.19%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions
Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @jrmccluskey for label go.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@lostluck
Copy link
Contributor Author

Whoops. Forgot to mark this as draft Jack. Just wanted to see if we could replicate the failure still (as listed in the description).

It doesn't occur on Prism (validated on laptop).

@lostluck
Copy link
Contributor Author

Run Go Spark ValidatesRunner

1 similar comment
@lostluck
Copy link
Contributor Author

Run Go Spark ValidatesRunner

@lostluck
Copy link
Contributor Author

It's very strange, since that element doesn't exist, it's just a windowed value header with no element, since there's only one element at all. Current idea is that element "always" exists, and it's being skipped somehow in the non-singleIterate path.

@lostluck
Copy link
Contributor Author

Run Go Spark ValidatesRunner

@lostluck
Copy link
Contributor Author

Ready for a look. Ready to submit if the Spark Validates runner passes.

@lostluck
Copy link
Contributor Author

Run Go Spark ValidatesRunner

@jrmccluskey
Copy link
Contributor

Looks like everything is good apart from the go fmt check

@lostluck
Copy link
Contributor Author

lostluck commented Aug 1, 2023

Thanks for the review!

@lostluck lostluck merged commit aeecddb into apache:master Aug 1, 2023
@lostluck lostluck deleted the beam23043 branch August 1, 2023 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task]: Reenable single iteration

2 participants