Rewrite container docs for custom containers #13420
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,147 +17,252 @@ limitations under the License. | |
|
|
||
| # Container environments | ||
|
|
||
| The Beam SDK runtime environment can be [containerized](https://www.docker.com/resources/what-container) with [Docker](https://www.docker.com/) to isolate it from other runtime systems. To learn more about the container environment, read the Beam [SDK Harness container contract](https://s.apache.org/beam-fn-api-container-contract). | ||
|
|
||
| Prebuilt SDK container images are released per supported language during Beam releases and pushed to [Docker Hub](https://hub.docker.com/search?q=apache%2Fbeam&type=image). | ||
|
|
||
| ## Custom containers | ||
|
|
||
| You may want to customize container images for many reasons, including: | ||
|
|
||
| * Pre-installing additional dependencies | ||
| * Launching third-party software in the worker environment | ||
| * Further customizing the execution environment | ||
|
|
||
| This guide describes how to create and use customized containers for the Beam SDK. | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| * This guide requires building images using Docker. [Install Docker locally](https://docs.docker.com/get-docker/). Some CI/CD platforms like [Google Cloud Build](https://cloud.google.com/cloud-build/docs/building/build-containers) also provide the ability to build images using Docker. | ||
| * For remote execution engines/runners, have a container registry to host your custom container image. Options include [Docker Hub](https://hub.docker.com/) or a "self-hosted" repository, including cloud-specific container registries like [Google Container Registry](https://cloud.google.com/container-registry) (GCR) or [Amazon Elastic Container Registry](https://aws.amazon.com/ecr/) (ECR). Make sure your registry can be accessed by your execution engine or runner. | ||
|
|
||
| > **NOTE**: On Nov 20, 2020, Docker Hub put [rate limits](https://www.docker.com/increase-rate-limits) into effect for anonymous and free authenticated use, which may impact larger pipelines that pull containers several times. | ||
|
|
||
| For optimal user experience, we also recommend you use the latest released version of Beam. | ||
|
|
||
| ### Building and pushing custom containers | ||
|
|
||
| Beam [SDK container images](https://hub.docker.com/search?q=apache%2Fbeam&type=image) are built from Dockerfiles checked into the [Github](https://github.com/apache/beam) repository and published to Docker Hub for every release. You can build customized containers in one of two ways: | ||
|
|
||
| 1. **[Writing a new](#writing-new-dockerfiles) Dockerfile based on a released container image**. This is sufficient for simple additions to the image, such as adding artifacts or environment variables. | ||
| 2. **[Modifying](#modifying-dockerfiles) a source Dockerfile in [Beam](https://github.com/apache/beam)**. This method requires building from Beam source but allows for greater customization of the container (including replacement of artifacts or base OS/language versions). | ||
|
|
||
| #### Writing a new Dockerfile based on an existing published container image {#writing-new-dockerfiles} | ||
|
|
||
| 1. Create a new Dockerfile that designates a base image using the [FROM instruction](https://docs.docker.com/engine/reference/builder/#from). | ||
|
|
||
| ``` | ||
| FROM apache/beam_python3.7_sdk:2.25.0 | ||
|
|
||
| ENV FOO=bar | ||
| COPY /src/path/to/file /dest/path/to/file/ | ||
| ``` | ||
|
|
||
| This `Dockerfile` uses the prebuilt Python 3.7 SDK container image [`beam_python3.7_sdk`](https://hub.docker.com/r/apache/beam_python3.7_sdk) tagged at (SDK version) `2.25.0`, and adds an additional environment variable and file to the image. | ||
|
|
||
|
|
||
| 2. [Build](https://docs.docker.com/engine/reference/commandline/build/) and [push](https://docs.docker.com/engine/reference/commandline/push/) the image using Docker. | ||
|
|
||
| ``` | ||
| export BASE_IMAGE="apache/beam_python3.7_sdk:2.25.0" | ||
| export IMAGE_NAME="myremoterepo/mybeamsdk" | ||
| export TAG="latest" | ||
|
|
||
| # Optional - pull the base image into your local Docker daemon to ensure | ||
| # you have the most up-to-date version of the base image locally. | ||
| docker pull "${BASE_IMAGE}" | ||
|
|
||
| docker build -f Dockerfile -t "${IMAGE_NAME}:${TAG}" . | ||
| ``` | ||
|
|
||
| 3. If your runner is running remotely, retag and [push](https://docs.docker.com/engine/reference/commandline/push/) the image to the appropriate repository. | ||
|
|
||
| ``` | ||
| docker push "${IMAGE_NAME}:${TAG}" | ||
| ``` | ||
|
|
||
| 4. After pushing a container image, verify the remote image ID and digest matches the local image ID and digest, output from `docker build` or `docker images`. | ||
|
|
||
| #### Modifying a source Dockerfile in Beam {#modifying-dockerfiles} | ||
|
|
||
| This method requires building image artifacts from Beam source. For additional instructions on setting up your development environment, see the [Contribution guide](/contribute/#development-setup). | ||
|
|
||
| >**NOTE**: It is recommended that you start from a stable release branch (`release-X.XX.X`) corresponding to the same version of the SDK to run your pipeline. Differences in SDK version may result in unexpected errors. | ||
|
|
||
| 1. Clone the `beam` repository. | ||
|
|
||
| ``` | ||
| export BEAM_SDK_VERSION="2.26.0" | ||
| git clone https://github.com/apache/beam.git | ||
| cd beam | ||
|
|
||
| # Save current directory as working directory | ||
| export BEAM_WORKDIR=$PWD | ||
|
|
||
| git checkout origin/release-$BEAM_SDK_VERSION | ||
| ``` | ||
|
|
||
| 2. Customize the `Dockerfile` for a given language, typically `sdks/<language>/container/Dockerfile` directory (e.g. the [Dockerfile for Python](https://github.com/apache/beam/blob/master/sdks/python/container/Dockerfile). If you're adding dependencies from [PyPI](https://pypi.org/), use [`base_image_requirements.txt`](https://github.com/apache/beam/blob/master/sdks/python/container/base_image_requirements.txt) instead. | ||
|
|
||
| 3. Return to the root Beam directory and run the Gradle `docker` target for your image. | ||
|
|
||
| ``` | ||
| cd $BEAM_WORKDIR | ||
|
|
||
| # The default repository of each SDK | ||
| ./gradlew :sdks:java:container:java8:docker | ||
| ./gradlew :sdks:java:container:java11:docker | ||
| ./gradlew :sdks:go:container:docker | ||
| ./gradlew :sdks:python:container:py36:docker | ||
| ./gradlew :sdks:python:container:py37:docker | ||
| ./gradlew :sdks:python:container:py38:docker | ||
|
|
||
| # Shortcut for building all Python SDKs | ||
| ./gradlew :sdks:python:container buildAll | ||
| ``` | ||
|
|
||
| 4. Verify the images you built were created by running `docker images`. | ||
|
|
||
| ``` | ||
| $> docker images --digests | ||
| REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE | ||
| apache/beam_java8_sdk latest sha256:... ... 1 min ago ... | ||
| apache/beam_java11_sdk latest sha256:... ... 1 min ago ... | ||
| apache/beam_python3.6_sdk latest sha256:... ... 1 min ago ... | ||
| apache/beam_python3.7_sdk latest sha256:... ... 1 min ago ... | ||
| apache/beam_python3.8_sdk latest sha256:... ... 1 min ago ... | ||
| apache/beam_go_sdk latest sha256:... ... 1 min ago ... | ||
| ``` | ||
|
|
||
| 5. If your runner is running remotely, retag the image and [push](https://docs.docker.com/engine/reference/commandline/push/) the image to your repository. You can skip this step if you provide a custom repo/tag as [additional parameters](#additional-build-parameters). | ||
|
|
||
| ``` | ||
| export BEAM_SDK_VERSION="2.26.0" | ||
| export IMAGE_NAME="gcr.io/my-gcp-project/beam_python3.7_sdk" | ||
| export TAG="${BEAM_SDK_VERSION}-custom" | ||
|
|
||
| docker tag apache/beam_python3.7_sdk "${IMAGE_NAME}:${TAG}" | ||
| docker push "${IMAGE_NAME}:${TAG}" | ||
| ``` | ||
|
|
||
| 6. After pushing a container image, verify the remote image ID and digest matches the local image ID and digest output from `docker_images --digests`. | ||
|
|
||
| #### Additional build parameters{#additional-build-parameters} | ||
|
|
||
| The docker Gradle task defines a default image repository and [tag](https://docs.docker.com/engine/reference/commandline/tag/) is the SDK version defined at [gradle.properties](https://github.com/apache/beam/blob/master/gradle.properties). The default repository is the Docker Hub `apache` namespace, and the default tag is the [SDK version](https://github.com/apache/beam/blob/master/gradle.properties) defined at gradle.properties. | ||
|
|
||
| You can specify a different repository or tag for built images by providing parameters to the build task. For example: | ||
|
|
||
| ``` | ||
| ./gradlew :sdks:python:container:py36:docker -Pdocker-repository-root="example-repo" -Pdocker-tag="2.26.0-custom" | ||
| ``` | ||
|
|
||
| builds the Python 3.6 container and tags it as `example-repo/beam_python3.6_sdk:2.26.0-custom`. | ||
|
|
||
| From Beam 2.21.0 and later, a `docker-pull-licenses` flag was introduced to add licenses/notices for third party dependencies to the docker images. For example: | ||
|
|
||
| ``` | ||
| ./gradlew :sdks:java:container:java8:docker -Pdocker-pull-licenses | ||
| ``` | ||
| creates a Java 8 SDK image with appropriate licenses in `/opt/apache/beam/third_party_licenses/`. | ||
|
|
||
| By default, no licenses/notices are added to the docker images. | ||
|
|
||
|
|
||
| ## Running pipelines with custom container images {#running-pipelines} | ||
|
|
||
| The common method for providing a container image requires using the | ||
| PortableRunner flag `--environment_config` as supported by the Portable | ||
| Runner or by runners supported PortableRunner flags. | ||
| Other runners, such as Dataflow, support specifying containers with different flags. | ||
|
|
||
| {{< highlight class="runner-direct" >}} | ||
| export IMAGE="my-repo/beam_python_sdk_custom" | ||
| export TAG="X.Y.Z" | ||
| export IMAGE_URL = "${IMAGE}:${TAG}" | ||
|
|
||
| python -m apache_beam.examples.wordcount \ | ||
| --input=/path/to/inputfile \ | ||
| --output /path/to/write/counts \ | ||
| --runner=PortableRunner \ | ||
| --job_endpoint=embed \ | ||
| --environment_type="DOCKER" \ | ||
| --environment_config="${IMAGE_URL}" | ||
| {{< /highlight >}} | ||
|
|
||
| {{< highlight class="runner-flink-local" >}} | ||
| export IMAGE="my-repo/beam_python_sdk_custom" | ||
| export TAG="X.Y.Z" | ||
| export IMAGE_URL = "${IMAGE}:${TAG}" | ||
|
|
||
| # Run a pipeline using the FlinkRunner which starts a Flink job server. | ||
| python -m apache_beam.examples.wordcount \ | ||
| --input=/path/to/inputfile \ | ||
| --output=path/to/write/counts \ | ||
| --runner=FlinkRunner \ | ||
| --environment_type="DOCKER" \ | ||
| --environment_config="${IMAGE_URL}" | ||
| {{< /highlight >}} | ||
|
|
||
| {{< highlight class="runner-spark-local" >}} | ||
| export IMAGE="my-repo/beam_python_sdk_custom" | ||
| export TAG="X.Y.Z" | ||
| export IMAGE_URL = "${IMAGE}:${TAG}" | ||
|
|
||
| # Run a pipeline using the SparkRunner which starts the Spark job server | ||
| python -m apache_beam.examples.wordcount \ | ||
| --input=/path/to/inputfile \ | ||
| --output=path/to/write/counts \ | ||
| --runner=SparkRunner \ | ||
| --environment_type="DOCKER" \ | ||
| --environment_config="${IMAGE_URL}" | ||
| {{< /highlight >}} | ||
|
|
||
| {{< highlight class="runner-dataflow" >}} | ||
| export GCS_PATH="gs://my-gcs-bucket" | ||
| export GCP_PROJECT="my-gcp-project" | ||
| export REGION="us-central1" | ||
emilymye marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| # By default, the Dataflow runner has access to the GCR images | ||
| # under the same project. | ||
| export IMAGE="my-repo/beam_python_sdk_custom" | ||
| export TAG="X.Y.Z" | ||
| export IMAGE_URL = "${IMAGE}:${TAG}" | ||
|
|
||
| # Run a pipeline on Dataflow. | ||
| # This is a Python batch pipeline, so to run on Dataflow Runner V2 | ||
| # you must specify the experiment "use_runner_v2" | ||
|
||
|
|
||
| python -m apache_beam.examples.wordcount \ | ||
| --input gs://dataflow-samples/shakespeare/kinglear.txt \ | ||
| --output "${GCS_PATH}/counts" \ | ||
| --runner DataflowRunner \ | ||
| --project $GCP_PROJECT \ | ||
| --region $REGION \ | ||
| --temp_location "${GCS_PATH}/tmp/" \ | ||
| --experiment=use_runner_v2 \ | ||
| --worker_harness_container_image=$IMAGE_URL | ||
|
|
||
| {{< /highlight >}} | ||
|
|
||
|
|
||
| ### Troubleshooting | ||
|
|
||
| The following section describes some common issues to consider | ||
| when you encounter unexpected errors running Beam pipelines with | ||
| custom containers. | ||
|
|
||
| * Differences in language and SDK version between the container SDK and | ||
emilymye marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| pipeline SDK may result in unexpected errors due to incompatibility. For best | ||
| results, make sure to use the same stable SDK version for your base container | ||
| and when running your pipeline. | ||
| * If you are running into unexpected errors when using remote containers, | ||
| make sure that your container exists in the remote repository and can be | ||
| accessed by any third-party service, if needed. | ||
| * Local runners attempt to pull remote images and default to local | ||
| images. If an image cannot be pulled locally (by the docker daemon), | ||
| you may see an log message like: | ||
| ``` | ||
| Error response from daemon: manifest for remote.repo/beam_python3.7_sdk:2.25.0-custom not found: manifest unknown: ... | ||
| INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Unable to pull image... | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems unfortunate. WDYT about Dataflow using this flag as well in the future?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think in the future there is a plan for Portable runner to support Dataflow endpoint, not sure how far on the roadmap that is.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tvalentyn Can you send me more information on this plan if you have any?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 I'd like to know about this as well since we shouldn't go to GA with custom containers until the flags are settled.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ananvay @angoenka @robertwb may have input on this plan.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regardless of the plans for using the Portability API for submitting jobs, we should standardize on a single flag for specifying the image across runners. It could be argued that environment_config is not the best flag for this; maybe we should take it to the list?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 I agree that standardizing the flag is a worthwhile improvement, but it's out of scope for this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Standardizing the flag should be done before we document and adveritize this feature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Our plan was to keep these as is for preview and standardize for GA. This is also editing existing documentation which used --environment_config. I can send an email to dev re:flags.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're supporting old SDKs, we have to support the old flag anyway, so we should go ahead and get this in and standardizing going forward.