Skip to content

[BEAM-9291] Upload graph option in dataflow's python sdk #10829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
aaltay merged 2 commits into from
Feb 18, 2020
Merged

[BEAM-9291] Upload graph option in dataflow's python sdk #10829

aaltay merged 2 commits into from
Feb 18, 2020

Conversation

@stankiewicz
Copy link
Contributor

@stankiewicz stankiewicz commented Feb 11, 2020 •
edited
Loading

Added support for upload_graph option in dataflow's python sdk.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Apex Dataflow Flink Gearpump Samza Spark
Go Build Status --- --- Build Status --- --- Build Status
Java Build Status Build Status Build Status Build Status
Build Status
Build Status		 Build Status Build Status Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status		 --- Build Status
Build Status		 Build Status
Build Status		 --- --- Build Status
XLang --- --- --- Build Status --- --- ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website
Non-portable Build Status Build Status
Build Status		 Build Status Build Status
Portable --- Build Status --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@stankiewicz stankiewicz changed the title [BEAM-9288] Upload graph option in dataflow's python sdk [BEAM-9291] Upload graph option in dataflow's python sdk Feb 11, 2020
@stankiewicz stankiewicz requested a review from aaltay February 11, 2020 18:59
aaltay
aaltay requested changes Feb 11, 2020
View reviewed changes
@aaltay aaltay self-requested a review February 11, 2020 20:59
@aaltay
Copy link
Member

aaltay commented Feb 11, 2020

I added a few comments. Could we also verify that this works as expected on Dataflow?

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 12, 2020

I have one challenge with full e2e test - python version stages full apache-beam package.
pip download 2.20dev0 fails.
I can try to patch locally 2.19 tag just with this PR or maybe there is other way to test runner?
When I change versions to 2.19 then eventually I have error some errors on version clash (probably haven't changed everywhere), but I can see that pipeline.json is uploaded, it contains steps and graph is rendered properly in console.

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 12, 2020

@aaltay
Before: "The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs.", 400, during REST submit
After: "Workflow failed. Causes: The job graph is too large. Please try again with a smaller graph." on runtime..

@tneymanov tneymanov mentioned this pull request Feb 12, 2020
Open
aaltay
aaltay reviewed Feb 12, 2020
View reviewed changes
@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 12, 2020

removed empty lines.

@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

retest this please

@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

Not sure if tests will start or not. This has been an ongoing problem for a while.

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 13, 2020

retest this please

@stankiewicz stankiewicz force-pushed the upload_graph_python_dataflow branch from c44b60c to 9f61d0f Compare February 13, 2020 15:37
@stankiewicz
upload graph fuctionality
fff1d04 
lint, move functionality to apiclient
yapf
@stankiewicz stankiewicz force-pushed the upload_graph_python_dataflow branch from 9f61d0f to fff1d04 Compare February 13, 2020 15:39
@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 13, 2020

retest this please

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 13, 2020

I've squashed commits.

@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

retest this please

1 similar comment
@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

retest this please

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 13, 2020

retest this please

5 similar comments
@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

retest this please

@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

retest this please

@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

retest this please

@aaltay
Copy link
Member

aaltay commented Feb 13, 2020

retest this please

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 13, 2020

retest this please

@aaltay
Copy link
Member

aaltay commented Feb 14, 2020

Run Python PreCommit

@aaltay
Copy link
Member

aaltay commented Feb 14, 2020

Run Python2_PVR_Flink PreCommit

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 14, 2020

Run Python PreCommit

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 14, 2020

Run Python2_PVR_Flink PreCommit

@mwalenia
Copy link
Member

mwalenia commented Feb 14, 2020

Run Python PreCommit

@mwalenia
Copy link
Member

mwalenia commented Feb 14, 2020

Run Python2_PVR_Flink PreCommit

@mwalenia
Copy link
Member

mwalenia commented Feb 14, 2020

Run Python PreCommit

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 14, 2020

Precommit fails due to change from #10826
Fix will be applied in #10856 , after that retest is needed.

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 14, 2020

Run Python PreCommit

@rohdesamuel
Copy link
Contributor

rohdesamuel commented Feb 14, 2020

Precommit fails due to change from #10826
Fix will be applied in #10856 , after that retest is needed.

Sorry for the breakage!

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 14, 2020 •
edited
Loading

No worries:) from what @aaltay written, there are more reasons for failing tests

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 17, 2020

Run Python PreCommit

@mwalenia
Copy link
Member

mwalenia commented Feb 18, 2020

run python precommit

@stankiewicz
Copy link
Contributor Author

stankiewicz commented Feb 18, 2020

@aaltay , can we merge?

@aaltay aaltay merged commit b483ddb into apache:master Feb 18, 2020
@aaltay aaltay mentioned this pull request Feb 18, 2020
Merged
4 tasks
samanvp added a commit to samanvp/gcp-variant-transforms that referenced this pull request Feb 24, 2020
@samanvp
Move MergeVariants and ProcessVariants before sharding
5e28cc3 
This is to reduce the number of nodes in the Dataflow job and avoid this failure:

The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs.

We will revert this change when a new Beam releaes includes this PR:
apache/beam#10829
samanvp added a commit to samanvp/gcp-variant-transforms that referenced this pull request Feb 24, 2020
@samanvp
Move MergeVariants and ProcessVariants before sharding
6545cff 
This is to reduce the number of nodes in the Dataflow job and avoid this failure:

The job graph is too large. Please try again with a smaller job graph, or split your job into two or more smaller jobs.

We will revert this change when a new Beam releaes includes this PR:
apache/beam#10829
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aaltay aaltay aaltay approved these changes

Assignees

No one assigned

Labels

examples python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@stankiewicz @aaltay @mwalenia @rohdesamuel