Rewrite container docs for custom containers #13420

emilymye · 2020-11-24T22:01:23Z

This rewrites the container guide as we move to launch custom containers for Dataflow runner. There are two methods that should be highlighted - one is using the released Dockerhub images as a base image, and one is building from Beam source itself.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

emilymye · 2020-11-24T22:38:06Z

R: @tysonjh
R: @tvalentyn
R: @alanmyrvold
R: @rosetn

Current page: https://beam.apache.org/documentation/runtime/environments/

website/www/site/content/en/documentation/runtime/environments.md

tysonjh · 2020-11-30T17:58:25Z

website/www/site/content/en/documentation/runtime/environments.md

This seems unfortunate. WDYT about Dataflow using this flag as well in the future?

I think in the future there is a plan for Portable runner to support Dataflow endpoint, not sure how far on the roadmap that is.

@tvalentyn Can you send me more information on this plan if you have any?

+1 I'd like to know about this as well since we shouldn't go to GA with custom containers until the flags are settled.

@ananvay @angoenka @robertwb may have input on this plan.

Regardless of the plans for using the Portability API for submitting jobs, we should standardize on a single flag for specifying the image across runners. It could be argued that environment_config is not the best flag for this; maybe we should take it to the list?

Regardless of the plans for using the Portability API for submitting jobs, we should standardize on a single flag for specifying the image across runners. It could be argued that environment_config is not the best flag for this; maybe we should take it to the list?

+1 I agree that standardizing the flag is a worthwhile improvement, but it's out of scope for this PR.

Standardizing the flag should be done before we document and adveritize this feature.

Our plan was to keep these as is for preview and standardize for GA. This is also editing existing documentation which used --environment_config. I can send an email to dev re:flags.

If we're supporting old SDKs, we have to support the old flag anyway, so we should go ahead and get this in and standardizing going forward.

website/www/site/content/en/documentation/runtime/environments.md

tysonjh · 2020-11-30T18:02:01Z

website/www/site/content/en/documentation/runtime/environments.md

Please put this into a code block or reformat.

The {{< highlight class="runner-X" >}} formats it into code-blocks, tabbed by runner type - see existing page https://beam.apache.org/documentation/runtime/environments/#testing-customized-images

Got it. I went to 'view file' in GitHub and it only showed the rendered markdown which made each of these H1 titles. Is there a way to preview?

The best way is to serve the website from source: ./gradlew :website:serveWebsite

You can also view the details of Website_Stage_GCS ("Run Website_Stage_GCS PreCommit") to find the staged version of the website. You can rerun the test if the link's gone stale.

website/www/site/content/en/documentation/runtime/environments.md

tvalentyn · 2020-11-30T23:10:17Z

website/www/site/content/en/documentation/runtime/environments.md

I think in the future there is a plan for Portable runner to support Dataflow endpoint, not sure how far on the roadmap that is.

tvalentyn · 2020-11-30T23:14:50Z

(some of my comments got hidden under Show Resolved due to new push, hope you can still find them :) )

website/www/site/content/en/documentation/runtime/environments.md

tvalentyn

LGTM, feel free to ping me when this is ready to merge.

website/www/site/content/en/documentation/runtime/environments.md

ibzib

LGTM.

It's exciting to finally have custom containers on Dataflow publicly documented.

rosetn

Thanks for rewriting this!

website/www/site/content/en/documentation/runtime/environments.md

rosetn · 2020-12-08T05:22:29Z

website/www/site/content/en/documentation/runtime/environments.md

Does "Navigate to the root directory where you've installed your local copy of the Beam SDK." work?

Tried to make it clearer (added some env var setting in instructions) - mostly worried installed might get confused for package installation (i.e. where pip installed Beam)

website/www/site/content/en/documentation/runtime/environments.md

tysonjh · 2020-12-08T23:39:25Z

website/www/site/content/en/documentation/runtime/environments.md

Can you add an anchor to this so I can link to it in cloud docs please?

added #running-pipelines

rosetn · 2020-12-09T21:43:38Z

website/www/site/content/en/documentation/runtime/environments.md

Typo here--I also think you can just call this section "Troubleshooting" or "Considerations"

website/www/site/content/en/documentation/runtime/environments.md

rosetn

One small typo but otherwise LGTM

rosetn · 2020-12-17T21:17:22Z

website/www/site/content/en/documentation/runtime/environments.md

+COPY /src/path/to/file /dest/path/to/file/
+```
+
+This `Dockerfile`: uses the prebuilt Python 3.7 SDK container image [`beam_python3.7_sdk`](https://hub.docker.com/r/apache/beam_python3.7_sdk) tagged at (SDK version) `2.25.0`, and adds an additional environment variable and file to the image.


Extra colon after Dockerfile

probot-autolabeler bot added the website label Nov 24, 2020

emilymye force-pushed the customcontainerdocs branch 2 times, most recently from 7259ff9 to c9c75ab Compare November 24, 2020 22:27

tysonjh reviewed Nov 30, 2020

View reviewed changes

emilymye force-pushed the customcontainerdocs branch from 07a91f0 to 96739ba Compare November 30, 2020 23:07

tvalentyn reviewed Nov 30, 2020

View reviewed changes

emilymye requested a review from tvalentyn November 30, 2020 23:27

tvalentyn reviewed Nov 30, 2020

View reviewed changes

website/www/site/content/en/documentation/runtime/environments.md Outdated Show resolved Hide resolved

tvalentyn approved these changes Dec 1, 2020

View reviewed changes

website/www/site/content/en/documentation/runtime/environments.md Outdated Show resolved Hide resolved

website/www/site/content/en/documentation/runtime/environments.md Outdated Show resolved Hide resolved

tysonjh approved these changes Dec 1, 2020

View reviewed changes

ibzib reviewed Dec 4, 2020

View reviewed changes

website/www/site/content/en/documentation/runtime/environments.md Outdated Show resolved Hide resolved

ibzib reviewed Dec 4, 2020

View reviewed changes

website/www/site/content/en/documentation/runtime/environments.md Outdated Show resolved Hide resolved

emilymye requested a review from ibzib December 4, 2020 22:01

emilymye force-pushed the customcontainerdocs branch from c9c883f to e1702d8 Compare December 7, 2020 22:15

ibzib approved these changes Dec 7, 2020

View reviewed changes

rosetn suggested changes Dec 8, 2020

View reviewed changes

tysonjh reviewed Dec 8, 2020

View reviewed changes

rosetn suggested changes Dec 9, 2020

View reviewed changes

emilymye added 10 commits December 15, 2020 11:13

rewrite containers docs for custom containers

e7da5e6

revisions

9c1dc3c

review edits

290e220

add gcr note

3b17a8c

more review edits

7244ba7

more fixes

8c3f467

fix title

2921f56

edits

ebb04f5

temp

b991bd3

update Flink/Spark runners

a632089

emilymye added 6 commits December 15, 2020 11:13

clean up Docker instructions

cbf75a0

add back a slash

1e1a84e

add notes

9c223a6

add troubleshooting section

062023a

add anchor

7c3086e

doc fixes

b7a4fb6

emilymye force-pushed the customcontainerdocs branch from 4244d56 to b7a4fb6 Compare December 15, 2020 19:28

rosetn approved these changes Dec 17, 2020

View reviewed changes

no should/will

4c35b8a

tvalentyn merged commit f8d326e into apache:master Dec 17, 2020

dxichen pushed a commit to linkedin/beam that referenced this pull request Aug 9, 2021

Rewrite container docs for custom containers (apache#13420)

c378cf4

Rewrite container docs for custom containers #13420

Rewrite container docs for custom containers #13420

Uh oh!

Conversation

emilymye commented Nov 24, 2020

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

emilymye commented Nov 24, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tvalentyn commented Nov 30, 2020

Uh oh!

Uh oh!

tvalentyn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ibzib left a comment

Choose a reason for hiding this comment

Uh oh!

rosetn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emilymye Dec 9, 2020 •

edited

Loading