Skip to content

Better manage transform service lifecycle for tests #28039

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chamikaramj merged 1 commit into from
Aug 19, 2023
Merged

Better manage transform service lifecycle for tests #28039

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ import org.gradle.api.tasks.PathSensitivity
import org.gradle.testing.jacoco.tasks.JacocoReport

import java.net.ServerSocket

import static java.util.UUID.randomUUID
/**
* This plugin adds methods to configure a module with Beam's defaults, called "natures".
*
Expand Down Expand Up @@ -2787,9 +2789,10 @@ class BeamModulePlugin implements Plugin<Project> {
def pythonDir = project.project(":sdks:python").projectDir
def externalPort = getRandomPort()
def launcherJar = project.project(':sdks:java:transform-service:launcher').shadowJar.archivePath
def groupId = project.name + randomUUID().toString()
def transformServiceOpts = [
"transform_service_launcher_jar": launcherJar,
"group_id": project.name,
"group_id": groupId,
"external_port": externalPort,
"beam_version": project.version
]
Expand All @@ -2807,9 +2810,17 @@ class BeamModulePlugin implements Plugin<Project> {
throw new GradleException(exceptionMessage)
}

// Transform service delivers transforms that refer to SDK harness containers with following sufixes.
def transformServiceJavaContainerSuffix = 'java11'
Copy link
Contributor

@Abacn Abacn Aug 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what caused this test requiring two java/python containers to work? Could it be able to clean up to make the test working for single java / py sdk container?

Copy link
Contributor Author

@chamikaramj chamikaramj Aug 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this setup method is (or rather will be) shared by PythonFromJava and JavaFromPython tests.

For external transforms, Transform Service will require Python3.8 and Java11 containers (transformServicePythonContainerSuffix and transformServiceJavaContainerSuffix).

The pipeline SDK, we could run tests with multiple Java and Python versions hence we have to build such containers based on the test ( pythonContainerSuffix and javaContainerSuffix).

For example, currently BT test is run using Python3.8 and 3.11 for pipeline SDK and Java 11 for external SDK.

Copy link
Contributor

@Abacn Abacn Aug 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. I see, for a single slang test, there are at least two SDK container is working

Copy link
Contributor

@Abacn Abacn Aug 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(though it is generally nice to have these version not hard coded)

Copy link
Contributor Author

@chamikaramj chamikaramj Aug 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Beam ships with these specific versions (Python 3.8 and Java 11) of expansion service containers. I hope to add documentation on developing expansion service containers with additional language versions for users who need that.

def transformServicePythonContainerSuffix = '38'

def setupTask = project.tasks.register(config.name+"Setup", Exec) {
// Containers for main SDKs when running tests.
dependsOn ':sdks:java:container:'+javaContainerSuffix+':docker'
dependsOn ':sdks:python:container:py'+pythonContainerSuffix+':docker'
// Containers for external SDKs used through the transform service.
dependsOn ':sdks:java:container:'+transformServiceJavaContainerSuffix+':docker'
dependsOn ':sdks:python:container:py'+transformServicePythonContainerSuffix+':docker'
dependsOn ':sdks:java:transform-service:controller-container:docker'
dependsOn ':sdks:python:expansion-service-container:docker'
dependsOn ':sdks:java:expansion-service:container:docker'
Expand Down
9 changes: 7 additions & 2 deletions sdks/python/scripts/run_transform_service.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,15 @@ TEMP_DIR=/tmp
case $STARTSTOP in
start)
echo "Starting the transform service for project $GROUP_ID at port $EXTERNAL_PORT for Beam version $BEAM_VERSION_DOCKER transform service startup jar is $TRANSFORM_SERVICE_LAUNCHER_JAR"
java -jar $TRANSFORM_SERVICE_LAUNCHER_JAR --project_name $GROUP_ID --port $EXTERNAL_PORT --beam_version $BEAM_VERSION_DOCKER --command up >$TEMP_DIR/$FILE_BASE-java1.log 2>&1 </dev/null &
java -jar $TRANSFORM_SERVICE_LAUNCHER_JAR --project_name $GROUP_ID --port $EXTERNAL_PORT --beam_version $BEAM_VERSION_DOCKER --command up >$TEMP_DIR/$FILE_BASE-java1.log 2>&1 </dev/null
;;
stop)
echo "Stopping the transform service for project $GROUP_ID at port $EXTERNAL_PORT for Beam version $BEAM_VERSION_DOCKER transform service startup jar is $TRANSFORM_SERVICE_LAUNCHER_JAR"
java -jar $TRANSFORM_SERVICE_LAUNCHER_JAR --project_name $GROUP_ID --port $EXTERNAL_PORT --beam_version $BEAM_VERSION_DOCKER --command down >$TEMP_DIR/$FILE_BASE-java2.log 2>&1 </dev/null &
java -jar $TRANSFORM_SERVICE_LAUNCHER_JAR --project_name $GROUP_ID --port $EXTERNAL_PORT --beam_version $BEAM_VERSION_DOCKER --command down >$TEMP_DIR/$FILE_BASE-java2.log 2>&1 </dev/null
TRANSFORM_SERVICE_TEMP_DIR=$TEMP_DIR/$GROUP_ID
if [[ -d ${TRANSFORM_SERVICE_TEMP_DIR} ]]; then
echo "Removing transform service temporary directory $TRANSFORM_SERVICE_TEMP_DIR"
rm -rf ${TRANSFORM_SERVICE_TEMP_DIR}
fi
;;
esac