Avoid apiary submission of job graph when it is not needed. #15458
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Specifically if both Runner v2 and use_portable_job_submission are enabled.
Contributor
Author
|
R: @yifanmai |
Codecov Report
@@ Coverage Diff @@
## master #15458 +/- ##
=======================================
Coverage 83.76% 83.76%
=======================================
Files 442 443 +1
Lines 60050 60121 +71
=======================================
+ Hits 50302 50363 +61
- Misses 9748 9758 +10
Continue to review full report at Codecov.
|
Contributor
Author
|
Ping @yifanmai . |
Contributor
|
Could you explain the meaning of "Avoid apiary submission"? My understanding from this code is that this change will now cause the graph JSON to be uploaded here when it previously would not have been, if we are using unified worker, Relatedly, I recently noticed that setting |
Contributor
Author
|
On Fri, Sep 10, 2021 at 5:48 PM Yifan Mai ***@***.***> wrote:
Could you explain the meaning of "Avoid apiary submission"? My
understanding from this code is that this change will now cause the graph
JSON to be uploaded here
<https://github.com/apache/beam/blob/25c67cdbae3ed2b9158d1f18ff4d522422dd3886/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py#L671-L683>
when it previously would not have been, if we are using unified worker,
use_portable_job_submission is set, and upload_graph is unset.
The (potentially large) graph will not be uploaded via apiary, but instead
uploaded to GCS. (Eventually we could remove this as well, but keeping it
for now just in case.)
Relatedly, I recently noticed that setting upload_graph causes the graph
optimizer to stop running; will this PR change this behavior?
No. That sounds concerning, do you have more details?
… —
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#15458 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AADWVAP3RNFNBFQUL4ROFSTUBKRNFANCNFSM5DMURF6A>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
yifanmai
approved these changes
Sep 11, 2021
Contributor
|
Sounds good. I'll follow up separately re: the optimizer once I have a small repro. |
robertwb
added a commit
to robertwb/incubator-beam
that referenced
this pull request
Sep 24, 2021
…pache#15458)" This reverts commit a63e15f. This avoids regressions from missing display data that was being copied from the v1beta3 graph.
y1chi
added a commit
that referenced
this pull request
Sep 25, 2021
Revert "Avoid apiary submission of job graph when it is not needed. (#15458)"
dmitriikuzinepam
pushed a commit
to dmitriikuzinepam/beam
that referenced
this pull request
Nov 2, 2021
dmitriikuzinepam
pushed a commit
to dmitriikuzinepam/beam
that referenced
this pull request
Nov 2, 2021
…pache#15458)" This reverts commit a63e15f. This avoids regressions from missing display data that was being copied from the v1beta3 graph.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Specifically if both Runner v2 and use_portable_job_submission are enabled.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
ValidatesRunnercompliance status (on master branch)Examples testing status on various runners
Post-Commit SDK/Transform Integration Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI.