Skip to content

Set sdk_location for Python 3 integration tests #7793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
markflyhigh wants to merge 1 commit into from
Closed

Set sdk_location for Python 3 integration tests #7793

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1602,7 +1602,7 @@ class BeamModulePlugin implements Plugin<Project> {
doLast {
project.exec {
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate && python ${pythonRootDir}/setup.py sdist --formats zip,gztar --dist-dir ${project.buildDir}"
args '-c', ". ${project.ext.envdir}/bin/activate && cd ${pythonRootDir} && python setup.py sdist --formats zip,gztar --dist-dir ${project.buildDir}"
}
def collection = project.fileTree("${project.buildDir}"){ include '**/*.tar.gz' exclude '**/apache-beam.tar.gz'}
println "sdist archive name: ${collection.singleFile}"
Expand Down
6 changes: 5 additions & 1 deletion sdks/python/precommit/dataflow/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,14 @@ task preCommitIT(dependsOn: ['sdist', 'installGcpTest']) {
"--processes=2", // Number of tests running in parallel
"--process-timeout=1800", // Timeout of whole command execution
]
def cmdArgs = project.mapToArgString([
"test_opts": testOpts,
"sdk_location": "${project.buildDir}/apache-beam.tar.gz"
])

exec {
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh --test_opts \"${testOpts.join(' ')}\""
args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
}
}
}
7 changes: 6 additions & 1 deletion sdks/python/precommit/dataflow/py3/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,13 @@ task preCommitIT(dependsOn: ['sdist', 'installGcpTest']) {
def testOpts = [
"--tests=${tests.join(',')}",
"--nocapture", // Print stdout instantly
"--processes=1", // Number of tests running in parallel
Copy link
Contributor

@tvalentyn tvalentyn Feb 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have to set this now, and didn't have to in the past?

Copy link
Contributor Author

@markflyhigh markflyhigh Feb 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not required for the fix. The main reason to have this and timeout flag is to fail fast if pipeline hangs. Otherwise the build will hit Gradle timeout which is over 100mins.

Copy link
Contributor

@tvalentyn tvalentyn Feb 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I undesrtand timeout part, but not sure why we need to add --processes - is this a tox flag? What would happen in --processes=2? Do we ever want to do that?

"--process-timeout=1800", // Timeout of whole command execution
]
def cmdArgs = project.mapToArgString(["test_opts": testOpts])
def cmdArgs = project.mapToArgString([
"test_opts": testOpts,
"sdk_location": "${project.buildDir}/apache-beam.tar.gz"
])
exec {
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate && ${runScriptsDir}/run_integration_test.sh $cmdArgs"
Expand Down