[BEAM-6527] Use Gradle to parallel Python tox tests #7675

markflyhigh · 2019-01-30T18:52:39Z

This PR use Gradle to parallel Python 2 and 3 tox as well as integration tests (on DataflowRunner) in Python PreCommit by:

Group tests to Gradle tasks under multiple sub-projects:
- beam-sdks-python:preCommitPy2
- beam-sdks-python-test-suites-tox-py3:preCommitPy3
- beam-sdks-python-test-suites-dataflow:preCommitIT
Build sdk artifact based on copy of sdk sources to avoid a concurrent issue in setuptools when using same setup.py (details: Set sdk_location for Python 3 integration tests #7793 (comment))
Start Cython tests after other tox tests since the side effects may affect the test behavior and result.

Reduced ~30mins for the PreCommit build:

Before: 60mins: https://scans.gradle.com/s/irb2zhuymvnms
After: 31mins: https://scans.gradle.com/s/l7em7nt3nuxaq

Follow this checklist to help us incorporate your contribution quickly and easily:

Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go	---	---	---	---	---	---
Java
Python	---			---	---	---

markflyhigh · 2019-01-30T20:09:24Z

Run Python PreCommit

markflyhigh · 2019-01-30T21:39:42Z

Run Python PreCommit

markflyhigh · 2019-01-30T23:43:46Z

Run Python PreCommit

markflyhigh · 2019-01-31T00:12:28Z

Run Python PreCommit

markflyhigh · 2019-01-31T23:42:57Z

Run Python PreCommit

markflyhigh · 2019-02-02T00:15:48Z

One of tox test keep failing due to error: [Errno 2] No such file or directory: 'apache-beam-2.11.0.dev0/apache_beam/runners/common_test.py'. This happens during tox packaging the distribution though the time it happens may slightly different in each run.

The build execute multiple tox commands in parallel so it's possibly a race condition. However, I set toxworkdir in tox.ini based on name of the task to avoid shared directory among builds, but it's likely I missed something else.

Triggered this test few tests and consistently failed regardless of beam nodes. Out of idea how to move forward.

markflyhigh · 2019-03-07T23:42:24Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

          project.exec {
            executable 'sh'
-            args '-c', ". ${project.ext.envdir}/bin/activate && cd ${pythonRootDir} && python setup.py sdist --keep-temp --formats zip,gztar --dist-dir ${project.buildDir}"
+            args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedSrcDir}/sdks/python && python setup.py sdist --formats zip,gztar --dist-dir ${project.buildDir}"


Since we start to use copy of sdk sources to build artifact, there is no need to keep temp directory around after build. So remove --keep-temp flag here.

For my education, why did we need it before?

I added this flag to fix the parallel failure in integration tests since the temp directory is shared between different build processes and by default it's deleted after a build finish. However, it never works for tox tests.

markflyhigh · 2019-03-08T05:18:19Z

+R: @tvalentyn @charlesccychen
+cc: @RobbeSneyders @Juta

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

sdks/python/scripts/run_tox.sh

tvalentyn · 2019-03-09T03:26:37Z

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy

          project.exec {
            executable 'sh'
-            args '-c', ". ${project.ext.envdir}/bin/activate && cd ${pythonRootDir} && python setup.py sdist --keep-temp --formats zip,gztar --dist-dir ${project.buildDir}"
+            args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedSrcDir}/sdks/python && python setup.py sdist --formats zip,gztar --dist-dir ${project.buildDir}"


For my education, why did we need it before?

sdks/python/test-suites/tox/py3/build.gradle

sdks/python/scripts/run_tox.sh

sdks/python/test-suites/tox/py3/build.gradle

markflyhigh · 2019-03-12T00:35:08Z

PTAL @tvalentyn

markflyhigh · 2019-03-12T22:59:14Z

Run Python PreCommit

tvalentyn · 2019-03-12T23:34:05Z

Is this ready for review, @markflyhigh ?

markflyhigh · 2019-03-12T23:35:02Z

Yes, please take a look. @tvalentyn

tvalentyn

Thank you! This is great!

tvalentyn · 2019-03-14T01:37:10Z

R: @aaltay

charlesccychen · 2019-03-14T01:43:23Z

Thank you!

markflyhigh · 2019-03-15T18:07:55Z

BTW, the reason why I choose copy SDK sources to isolate build is explained in #7793 (comment).

markflyhigh mentioned this pull request Jan 30, 2019

[Do Not Merge] Use Gradle to parallel tox tests #7592

Closed

2 tasks

markflyhigh mentioned this pull request Feb 9, 2019

Set sdk_location for Python 3 integration tests #7793

Closed

2 tasks

markflyhigh force-pushed the py3-tox-2 branch 5 times, most recently from a2d80fd to 3c6979e Compare March 7, 2019 23:21

markflyhigh commented Mar 7, 2019

View reviewed changes

tvalentyn reviewed Mar 9, 2019

View reviewed changes

buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy Outdated Show resolved Hide resolved

tvalentyn reviewed Mar 9, 2019

View reviewed changes

adude3141 reviewed Mar 10, 2019

View reviewed changes

sdks/python/test-suites/tox/py3/build.gradle Outdated Show resolved Hide resolved

markflyhigh force-pushed the py3-tox-2 branch from 3c6979e to ac97927 Compare March 11, 2019 23:51

tvalentyn mentioned this pull request Mar 13, 2019

[BEAM-5663] Add Python 3.6 tox environment #7988

Merged

markflyhigh force-pushed the py3-tox-2 branch from 918cf82 to 755aa81 Compare March 14, 2019 00:21

[BEAM-6527] Use Gradle to parallel Python tox tests

8c3fbd6

markflyhigh force-pushed the py3-tox-2 branch from 755aa81 to 8c3fbd6 Compare March 14, 2019 00:58

tvalentyn approved these changes Mar 14, 2019

View reviewed changes

charlesccychen merged commit babd948 into apache:master Mar 14, 2019

aaltay mentioned this pull request Mar 14, 2019

Revert "[BEAM-6527] Use Gradle to parallel Python tox tests" #8059

Merged

markflyhigh mentioned this pull request Mar 15, 2019

[BEAM-6527] Use Gradle to parallel Python tox tests #8067

Merged

3 tasks

[BEAM-6527] Use Gradle to parallel Python tox tests #7675

[BEAM-6527] Use Gradle to parallel Python tox tests #7675

Uh oh!

Conversation

markflyhigh commented Jan 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Post-Commit Tests Status (on master branch)

Uh oh!

markflyhigh commented Jan 30, 2019

Uh oh!

markflyhigh commented Jan 30, 2019

Uh oh!

markflyhigh commented Jan 30, 2019

Uh oh!

markflyhigh commented Jan 31, 2019

Uh oh!

markflyhigh commented Jan 31, 2019

Uh oh!

markflyhigh commented Feb 2, 2019

Uh oh!

markflyhigh Mar 7, 2019

Choose a reason for hiding this comment

Uh oh!

tvalentyn Mar 9, 2019

Choose a reason for hiding this comment

Uh oh!

markflyhigh Mar 11, 2019

Choose a reason for hiding this comment

Uh oh!

markflyhigh commented Mar 8, 2019

Uh oh!

Uh oh!

Uh oh!

tvalentyn Mar 9, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markflyhigh commented Mar 12, 2019

Uh oh!

markflyhigh commented Mar 12, 2019

Uh oh!

tvalentyn commented Mar 12, 2019

Uh oh!

markflyhigh commented Mar 12, 2019

Uh oh!

tvalentyn left a comment

Choose a reason for hiding this comment

Uh oh!

tvalentyn commented Mar 14, 2019

Uh oh!

charlesccychen commented Mar 14, 2019

Uh oh!

markflyhigh commented Mar 15, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

markflyhigh commented Jan 30, 2019 •

edited

Loading