+### Subsribe and Unsubscribe:
-| How to contact us | When to use it |
-| ----------------- | ---------------|
-| [user@](https://lists.apache.org/list.html?user@beam.apache.org) mailing list | User support and questions ([Subscribe](mailto:user-subscribe@beam.apache.org)[^1], [Unsubscribe](mailto:user-unsubscribe@beam.apache.org)[^1], [Archives](https://lists.apache.org/list.html?user@beam.apache.org)) |
-| [dev@](https://lists.apache.org/list.html?dev@beam.apache.org) mailing list | Development discussions ([Subscribe](mailto:dev-subscribe@beam.apache.org)[^1], [Unsubscribe](mailto:dev-unsubscribe@beam.apache.org)[^1], [Archives](https://lists.apache.org/list.html?dev@beam.apache.org)) |
-| [commits@](https://lists.apache.org/list.html?commits@beam.apache.org) mailing list | Firehose of commits, bugs, pull requests, etc. ([Subscribe](mailto:commits-subscribe@beam.apache.org)[^1], [Unsubscribe](mailto:commits-unsubscribe@beam.apache.org)[^1], [Archives](https://lists.apache.org/list.html?commits@beam.apache.org)) |
-| [builds@](https://lists.apache.org/list.html?builds@beam.apache.org) mailing list | Firehose of build notifications from Jenkins ([Subscribe](mailto:builds-subscribe@beam.apache.org)[^1], [Unsubscribe](mailto:builds-unsubscribe@beam.apache.org)[^1], [Archives](https://lists.apache.org/list.html?builds@beam.apache.org)) |
-| [JIRA bug tracker](https://issues.apache.org/jira/browse/BEAM) | Report bugs / discover known issues |
-| [StackOverflow](https://stackoverflow.com/questions/tagged/apache-beam) | Ask and answer user support questions |
-| [Slack](https://s.apache.org/beam-slack-channel) | Chat with users and developers in the ASF Slack. Note: Please [join the #beam channel](https://s.apache.org/beam-slack-channel) after you [created an account](https://s.apache.org/slack-invite). Please do not ask Beam questions in #general. |
+Prior to sending emails to these lists, you need to subscribe. To subscribe, send a blank email to [user-subscribe@beam.apache.org](user-subscribe@beam.apache.org) or [dev-subscribe@beam.apache.org](dev-subscribe@beam.apache.org) depending on the list you want to write to.
-
+### Useful Tips for Sending Emails
-If you have questions about how to use Apache Beam, we recommend you try out the [user@](https://lists.apache.org/list.html?user@beam.apache.org) mailing list, and [StackOverflow](https://stackoverflow.com/questions/tagged/apache-beam).
+Tip 1: Use tags in your subject line.
+A tag is a word within a pair of brackets [] that indicate the type of message you’re sending. For example: [Bug] or [Proposal] or [Question] or [Idea]. Tags help folks navigate emails easier.
-[^1]: To subscribe or unsubscribe, a blank email is fine.
+Tip 2: If you’re asking a troubleshooting question, provide as much information as possible to help others replicate your issue or find possible solutions.
-If you wish to report a security vulnerability, please contact [security@apache.org](mailto:security@apache.org). Apache Beam follows the typical [Apache vulnerability handling process](https://apache.org/security/committers.html#vulnerability-handling).
+Tip 3: Share complete links instead of hyperlinks. A common practice in Apache is adding a number tag like [1] to indicate a word is a link or an attachment, and use the tag as a footnote to add the complete link at the end of your message.
diff --git a/website/www/site/content/en/community/join-beam.md b/website/www/site/content/en/community/join-beam.md
new file mode 100644
index 000000000000..db65498f1307
--- /dev/null
+++ b/website/www/site/content/en/community/join-beam.md
@@ -0,0 +1,46 @@
+---
+title: "Join Beam Community"
+aliases:
+ - /community/
+ - /use/issue-tracking/
+ - /use/mailing-lists/
+ - /get-started/support/
+---
+
+
+
+# Join Beam Community!
+
+We’re so glad you want to join the Apache Beam community. Here’s a pathway you can follow to help you get started. This is not the only way to join, but it’s the recommended progression so you’re successful at reaching your objectives.
+
+## Join the Conversation
+
+{{< community/join_beam_columns number-of-data="0" >}}
+
+## Contribute to the Project
+
+{{< community/join_beam_columns number-of-data="1" >}}
+
+## Become a committer
+
+{{< community/join_beam_columns number-of-data="2" >}}
+
+## Become a PMC member
+
+{{< community/join_beam_columns number-of-data="3" >}}
+
+## Still have questions?
+
+Send us a note at user@beam.apache.org OR mention @ApacheBeam on Twitter OR ask us in the #beam slack channel. We look forward to seeing you in the community!
diff --git a/website/www/site/content/en/community/powered-by.md b/website/www/site/content/en/community/powered-by.md
deleted file mode 100644
index dd45ece15b4b..000000000000
--- a/website/www/site/content/en/community/powered-by.md
+++ /dev/null
@@ -1,27 +0,0 @@
----
-title: 'Powered by Apache Beam'
----
-
-# Projects and Products Powered by Apache Beam
-
-To add yourself to the list, please open a [pull request](https://github.com/apache/beam/edit/master/website/www/site/content/en/community/powered_by.md) adding your organization name, URL, a list of which Beam components you are using, and a short description of your use case.
-
-* **[Cloud Dataflow](https://cloud.google.com/dataflow):** Google Cloud Dataflow is a fully managed service for executing
- Apache Beam pipelines within the Google Cloud Platform ecosystem.
-* **[TensorFlow Extended (TFX)](https://www.tensorflow.org/tfx):** TensorFlow Extended (TFX) is an end-to-end platform
- for deploying production ML pipelines.
-* **[Apache Hop (incubating)](http://hop.apache.org):** Hop provides a complete data orchestration (ETL / DI) toolset with visual pipeline development. It supports execution on the main Apache Beam runners.
-
-
diff --git a/website/www/site/content/en/contribute/_index.md b/website/www/site/content/en/contribute/_index.md
index 333039f39a4f..a3a6c972d5bf 100644
--- a/website/www/site/content/en/contribute/_index.md
+++ b/website/www/site/content/en/contribute/_index.md
@@ -1,12 +1,15 @@
---
title: "Beam Contribution Guide"
+type: "contribute"
+layout: "arrow_template"
aliases:
- - /contribution-guide/
- - /contribute/contribution-guide/
- - /docs/contribute/
- - /contribute/source-repository/
- - /contribute/design-principles/
+ - /contribution-guide/
+ - /contribute/contribution-guide/
+ - /docs/contribute/
+ - /contribute/source-repository/
+ - /contribute/design-principles/
---
+
-# Apache Beam Contribution Guide
+# Contribution guide
+
+
+ {{< figure src="/images/arrow-icon_list.svg">}}
+
+## Overview
-The Apache Beam community welcomes contributions from anyone!
+
-If you have questions, please [reach out to the Beam community](/contribute/get-help).
+
-There are lots of opportunities to contribute:
+There are lots of opportunities to contribute. You can for example:
- - ask or answer questions on [user@beam.apache.org](/community/contact-us/) or
+- ask or answer questions on [user@beam.apache.org](/community/contact-us/) or
[stackoverflow](https://stackoverflow.com/questions/tagged/apache-beam)
- review proposed design ideas on [dev@beam.apache.org](/community/contact-us/)
- improve the documentation
@@ -49,17 +57,22 @@ There are lots of opportunities to contribute:
https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ)
- organize local meetups of users or contributors to Apache Beam
-Most importantly, if you have an idea of how to contribute, then do it!
+
+
+
+ {{< figure src="/images/arrow-icon_list.svg">}}
## Contributing code
+
+
+
+
Below is a tutorial for contributing code to Beam, covering our tools and typical process in
detail.
### Prerequisites
-To contribute code, you need
-
- a GitHub account
- a Linux, macOS, or Microsoft Windows development environment with Java JDK 8 installed
- [Docker](https://www.docker.com/) installed for some tasks including building worker containers and testing this website
@@ -71,6 +84,8 @@ To contribute code, you need
Agreement](https://www.apache.org/licenses/icla.pdf) (ICLA) to the Apache
Software Foundation (ASF).
+
+
To install these in a Debian-based distribution:
```
@@ -127,8 +142,7 @@ script which is part of the Beam repo:
1. Assign the issue to yourself. To get the permission to do so, email
the [dev@ mailing list](/community/contact-us)
to introduce yourself and to be added as a contributor in the Beam issue tracker including your
- ASF Jira Username. For example [this welcome email](
- https://lists.apache.org/thread.html/e6018c2aaf7dc7895091434295e5b0fafe192b975e3e3761fcf0cda7@%3Cdev.beam.apache.org%3E).
+ ASF Jira Username. For example [this welcome email](https://lists.apache.org/thread.html/e6018c2aaf7dc7895091434295e5b0fafe192b975e3e3761fcf0cda7@%3Cdev.beam.apache.org%3E).
1. If your change is large or it is your first change, it is a good idea to
[discuss it on the dev@ mailing list](/community/contact-us/).
1. For large changes create a design doc
@@ -233,6 +247,8 @@ script which is part of the Beam repo:
all commits in the PR during merge, however if a PR has a mixture of independent changes that should not be squashed, and fixup commits,
then the PR author should help squashing fixup commits to maintain a clean commmit history.
+
+
## When will my change show up in an Apache Beam release?
Apache Beam makes minor releases every 6 weeks. Apache Beam has a
diff --git a/website/www/site/content/en/contribute/runner-guide.md b/website/www/site/content/en/contribute/runner-guide.md
index 57457a68fa8b..6bc65ecd6eb5 100644
--- a/website/www/site/content/en/contribute/runner-guide.md
+++ b/website/www/site/content/en/contribute/runner-guide.md
@@ -27,183 +27,6 @@ Topics covered:
{{< toc >}}
-## Basics of the Beam model
-
-Suppose you have a data processing engine that can pretty easily process graphs
-of operations. You want to integrate it with the Beam ecosystem to get access
-to other languages, great event time processing, and a library of connectors.
-You need to know the core vocabulary:
-
- * [_Pipeline_](#pipeline) - A pipeline is a graph of transformations that a user constructs
- that defines the data processing they want to do.
- * [_PCollection_](#pcollections) - Data being processed in a pipeline is part of a PCollection.
- * [_PTransforms_](#ptransforms) - The operations executed within a pipeline. These are best
- thought of as operations on PCollections.
- * _SDK_ - A language-specific library for pipeline authors (we often call them
- "users" even though we have many kinds of users) to build transforms,
- construct their pipelines and submit them to a runner
- * _Runner_ - You are going to write a piece of software called a runner that
- takes a Beam pipeline and executes it using the capabilities of your data
- processing engine.
-
-These concepts may be very similar to your processing engine's concepts. Since
-Beam's design is for cross-language operation and reusable libraries of
-transforms, there are some special features worth highlighting.
-
-### Pipeline
-
-A pipeline in Beam is a graph of PTransforms operating on PCollections. A
-pipeline is constructed by a user in their SDK of choice, and makes its way to
-your runner either via the SDK directly or via the Runner API's (forthcoming)
-RPC interfaces.
-
-### PTransforms
-
-In Beam, a PTransform can be one of the five primitives or it can be a
-composite transform encapsulating a subgraph. The primitives are:
-
- * [_Read_](#implementing-the-read-primitive) - parallel connectors to external
- systems
- * [_ParDo_](#implementing-the-pardo-primitive) - per element processing
- * [_GroupByKey_](#implementing-the-groupbykey-and-window-primitive) -
- aggregating elements per key and window
- * [_Flatten_](#implementing-the-flatten-primitive) - union of PCollections
- * [_Window_](#implementing-the-window-primitive) - set the windowing strategy
- for a PCollection
-
-When implementing a runner, these are the operations you need to implement.
-Composite transforms may or may not be important to your runner. If you expose
-a UI, maintaining some of the composite structure will make the pipeline easier
-for a user to understand. But the result of processing is not changed.
-
-### PCollections
-
-A PCollection is an unordered bag of elements. Your runner will be responsible
-for storing these elements. There are some major aspects of a PCollection to
-note:
-
-#### Bounded vs Unbounded
-
-A PCollection may be bounded or unbounded.
-
- - _Bounded_ - it is finite and you know it, as in batch use cases
- - _Unbounded_ - it may be never end, you don't know, as in streaming use cases
-
-These derive from the intuitions of batch and stream processing, but the two
-are unified in Beam and bounded and unbounded PCollections can coexist in the
-same pipeline. If your runner can only support bounded PCollections, you'll
-need to reject pipelines that contain unbounded PCollections. If your
-runner is only really targeting streams, there are adapters in our support code
-to convert everything to APIs targeting unbounded data.
-
-#### Timestamps
-
-Every element in a PCollection has a timestamp associated with it.
-
-When you execute a primitive connector to some storage system, that connector
-is responsible for providing initial timestamps. Your runner will need to
-propagate and aggregate timestamps. If the timestamp is not important, as with
-certain batch processing jobs where elements do not denote events, they will be
-the minimum representable timestamp, often referred to colloquially as
-"negative infinity".
-
-#### Watermarks
-
-Every PCollection has to have a watermark that estimates how complete the
-PCollection is.
-
-The watermark is a guess that "we'll never see an element with an earlier
-timestamp". Sources of data are responsible for producing a watermark. Your
-runner needs to implement watermark propagation as PCollections are processed,
-merged, and partitioned.
-
-The contents of a PCollection are complete when a watermark advances to
-"infinity". In this manner, you may discover that an unbounded PCollection is
-finite.
-
-#### Windowed elements
-
-Every element in a PCollection resides in a window. No element resides in
-multiple windows (two elements can be equal except for their window, but they
-are not the same).
-
-When elements are read from the outside world they arrive in the global window.
-When they are written to the outside world, they are effectively placed back
-into the global window (any writing transform that doesn't take this
-perspective probably risks data loss).
-
-A window has a maximum timestamp, and when the watermark exceeds this plus
-user-specified allowed lateness the window is expired. All data related
-to an expired window may be discarded at any time.
-
-#### Coder
-
-Every PCollection has a coder, a specification of the binary format of the elements.
-
-In Beam, the user's pipeline may be written in a language other than the
-language of the runner. There is no expectation that the runner can actually
-deserialize user data. So the Beam model operates principally on encoded data -
-"just bytes". Each PCollection has a declared encoding for its elements, called
-a coder. A coder has a URN that identifies the encoding, and may have
-additional sub-coders (for example, a coder for lists may contain a coder for
-the elements of the list). Language-specific serialization techniques can, and
-frequently are used, but there are a few key formats - such as key-value pairs
-and timestamps - that are common so your runner can understand them.
-
-#### Windowing Strategy
-
-Every PCollection has a windowing strategy, a specification of essential
-information for grouping and triggering operations.
-
-The details will be discussed below when we discuss the
-[Window](#implementing-the-window-primitive) primitive, which sets up the
-windowing strategy, and
-[GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
-which has behavior governed by the windowing strategy.
-
-### User-Defined Functions (UDFs)
-
-Beam has seven varieties of user-defined function (UDF). A Beam pipeline
-may contain UDFs written in a language other than your runner, or even multiple
-languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
-definitions are language-independent (see the [Fn API](#the-fn-api)).
-
-The UDFs of Beam are:
-
- * _DoFn_ - per-element processing function (used in ParDo)
- * _WindowFn_ - places elements in windows and merges windows (used in Window
- and GroupByKey)
- * _Source_ - emits data read from external sources, including initial and
- dynamic splitting for parallelism (used in Read)
- * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
- in side inputs)
- * _WindowMappingFn_ - maps one element's window to another, and specifies
- bounds on how far in the past the result window will be (used in side
- inputs)
- * _CombineFn_ - associative and commutative aggregation (used in Combine and
- state)
- * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
-
-The various types of user-defined functions will be described further alongside
-the primitives that use them.
-
-### Runner
-
-The term "runner" is used for a couple of things. It generally refers to the
-software that takes a Beam pipeline and executes it somehow. Often, this is the
-translation code that you write. It usually also includes some customized
-operators for your data processing engine, and is sometimes used to refer to
-the full stack.
-
-A runner has just a single method `run(Pipeline)`. From here on, I will often
-use code font for proper nouns in our APIs, whether or not the identifiers
-match across all SDKs.
-
-The `run(Pipeline)` method should be asynchronous and results in a
-PipelineResult which generally will be a job descriptor for your data
-processing engine, providing methods for checking its status, canceling it, and
-waiting for it to terminate.
-
## Implementing the Beam Primitives
Aside from encoding and persisting data - which presumably your engine already
diff --git a/website/www/site/content/en/documentation/_index.md b/website/www/site/content/en/documentation/_index.md
index ffd6c21e7989..36812d53b630 100644
--- a/website/www/site/content/en/documentation/_index.md
+++ b/website/www/site/content/en/documentation/_index.md
@@ -4,6 +4,7 @@ aliases:
- /learn/
- /docs/learn/
---
+
+
+# Basics of the Beam model
+
+Suppose you have a data processing engine that can pretty easily process graphs
+of operations. You want to integrate it with the Beam ecosystem to get access
+to other languages, great event time processing, and a library of connectors.
+You need to know the core vocabulary:
+
+ * [_Pipeline_](#pipeline) - A pipeline is a graph of transformations that a user constructs
+ that defines the data processing they want to do.
+ * [_PCollection_](#pcollections) - Data being processed in a pipeline is part of a PCollection.
+ * [_PTransforms_](#ptransforms) - The operations executed within a pipeline. These are best
+ thought of as operations on PCollections.
+ * _SDK_ - A language-specific library for pipeline authors (we often call them
+ "users" even though we have many kinds of users) to build transforms,
+ construct their pipelines and submit them to a runner
+ * _Runner_ - You are going to write a piece of software called a runner that
+ takes a Beam pipeline and executes it using the capabilities of your data
+ processing engine.
+
+These concepts may be very similar to your processing engine's concepts. Since
+Beam's design is for cross-language operation and reusable libraries of
+transforms, there are some special features worth highlighting.
+
+### Pipeline
+
+A pipeline in Beam is a graph of PTransforms operating on PCollections. A
+pipeline is constructed by a user in their SDK of choice, and makes its way to
+your runner either via the SDK directly or via the Runner API's (forthcoming)
+RPC interfaces.
+
+### PTransforms
+
+In Beam, a PTransform can be one of the five primitives or it can be a
+composite transform encapsulating a subgraph. The primitives are:
+
+ * [_Read_](#implementing-the-read-primitive) - parallel connectors to external
+ systems
+ * [_ParDo_](#implementing-the-pardo-primitive) - per element processing
+ * [_GroupByKey_](#implementing-the-groupbykey-and-window-primitive) -
+ aggregating elements per key and window
+ * [_Flatten_](#implementing-the-flatten-primitive) - union of PCollections
+ * [_Window_](#implementing-the-window-primitive) - set the windowing strategy
+ for a PCollection
+
+When implementing a runner, these are the operations you need to implement.
+Composite transforms may or may not be important to your runner. If you expose
+a UI, maintaining some of the composite structure will make the pipeline easier
+for a user to understand. But the result of processing is not changed.
+
+### PCollections
+
+A PCollection is an unordered bag of elements. Your runner will be responsible
+for storing these elements. There are some major aspects of a PCollection to
+note:
+
+#### Bounded vs Unbounded
+
+A PCollection may be bounded or unbounded.
+
+ - _Bounded_ - it is finite and you know it, as in batch use cases
+ - _Unbounded_ - it may be never end, you don't know, as in streaming use cases
+
+These derive from the intuitions of batch and stream processing, but the two
+are unified in Beam and bounded and unbounded PCollections can coexist in the
+same pipeline. If your runner can only support bounded PCollections, you'll
+need to reject pipelines that contain unbounded PCollections. If your
+runner is only really targeting streams, there are adapters in our support code
+to convert everything to APIs targeting unbounded data.
+
+#### Timestamps
+
+Every element in a PCollection has a timestamp associated with it.
+
+When you execute a primitive connector to some storage system, that connector
+is responsible for providing initial timestamps. Your runner will need to
+propagate and aggregate timestamps. If the timestamp is not important, as with
+certain batch processing jobs where elements do not denote events, they will be
+the minimum representable timestamp, often referred to colloquially as
+"negative infinity".
+
+#### Watermarks
+
+Every PCollection has to have a watermark that estimates how complete the
+PCollection is.
+
+The watermark is a guess that "we'll never see an element with an earlier
+timestamp". Sources of data are responsible for producing a watermark. Your
+runner needs to implement watermark propagation as PCollections are processed,
+merged, and partitioned.
+
+The contents of a PCollection are complete when a watermark advances to
+"infinity". In this manner, you may discover that an unbounded PCollection is
+finite.
+
+#### Windowed elements
+
+Every element in a PCollection resides in a window. No element resides in
+multiple windows (two elements can be equal except for their window, but they
+are not the same).
+
+When elements are read from the outside world they arrive in the global window.
+When they are written to the outside world, they are effectively placed back
+into the global window (any writing transform that doesn't take this
+perspective probably risks data loss).
+
+A window has a maximum timestamp, and when the watermark exceeds this plus
+user-specified allowed lateness the window is expired. All data related
+to an expired window may be discarded at any time.
+
+#### Coder
+
+Every PCollection has a coder, a specification of the binary format of the elements.
+
+In Beam, the user's pipeline may be written in a language other than the
+language of the runner. There is no expectation that the runner can actually
+deserialize user data. So the Beam model operates principally on encoded data -
+"just bytes". Each PCollection has a declared encoding for its elements, called
+a coder. A coder has a URN that identifies the encoding, and may have
+additional sub-coders (for example, a coder for lists may contain a coder for
+the elements of the list). Language-specific serialization techniques can, and
+frequently are used, but there are a few key formats - such as key-value pairs
+and timestamps - that are common so your runner can understand them.
+
+#### Windowing Strategy
+
+Every PCollection has a windowing strategy, a specification of essential
+information for grouping and triggering operations.
+
+The details will be discussed below when we discuss the
+[Window](#implementing-the-window-primitive) primitive, which sets up the
+windowing strategy, and
+[GroupByKey](#implementing-the-groupbykey-and-window-primitive) primitive,
+which has behavior governed by the windowing strategy.
+
+### User-Defined Functions (UDFs)
+
+Beam has seven varieties of user-defined function (UDF). A Beam pipeline
+may contain UDFs written in a language other than your runner, or even multiple
+languages in the same pipeline (see the [Runner API](#the-runner-api)) so the
+definitions are language-independent (see the [Fn API](#the-fn-api)).
+
+The UDFs of Beam are:
+
+ * _DoFn_ - per-element processing function (used in ParDo)
+ * _WindowFn_ - places elements in windows and merges windows (used in Window
+ and GroupByKey)
+ * _Source_ - emits data read from external sources, including initial and
+ dynamic splitting for parallelism (used in Read)
+ * _ViewFn_ - adapts a materialized PCollection to a particular interface (used
+ in side inputs)
+ * _WindowMappingFn_ - maps one element's window to another, and specifies
+ bounds on how far in the past the result window will be (used in side
+ inputs)
+ * _CombineFn_ - associative and commutative aggregation (used in Combine and
+ state)
+ * _Coder_ - encodes user data; some coders have standard formats and are not really UDFs
+
+The various types of user-defined functions will be described further alongside
+the primitives that use them.
+
+### Runner
+
+The term "runner" is used for a couple of things. It generally refers to the
+software that takes a Beam pipeline and executes it somehow. Often, this is the
+translation code that you write. It usually also includes some customized
+operators for your data processing engine, and is sometimes used to refer to
+the full stack.
+
+A runner has just a single method `run(Pipeline)`. From here on, I will often
+use code font for proper nouns in our APIs, whether or not the identifiers
+match across all SDKs.
+
+The `run(Pipeline)` method should be asynchronous and results in a
+PipelineResult which generally will be a job descriptor for your data
+processing engine, providing methods for checking its status, canceling it, and
+waiting for it to terminate.
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix.md b/website/www/site/content/en/documentation/runners/capability-matrix/_index.md
similarity index 55%
rename from website/www/site/content/en/documentation/runners/capability-matrix.md
rename to website/www/site/content/en/documentation/runners/capability-matrix/_index.md
index 52cfd34aaa24..2897829ec9ec 100644
--- a/website/www/site/content/en/documentation/runners/capability-matrix.md
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/_index.md
@@ -1,6 +1,7 @@
---
type: runners
title: "Apache Beam Capability Matrix"
+layout: arrow_template
aliases:
- /learn/runners/capability-matrix/
- /capability-matrix/
@@ -22,21 +23,17 @@ limitations under the License.
# Beam Capability Matrix
Apache Beam provides a portable API layer for building sophisticated data-parallel processing pipelines that may be executed across a diversity of execution engines, or runners. The core concepts of this layer are based upon the Beam Model (formerly referred to as the [Dataflow Model](https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf)), and implemented to varying degrees in each Beam runner. To help clarify the capabilities of individual runners, we've created the capability matrix below.
-Individual capabilities have been grouped by their corresponding What / Where / When / How question:
+Individual capabilities have been grouped by their corresponding What / Where / When / How question:
-- What results are being calculated?
-- Where in event time?
-- When in processing time?
-- How do refinements of results relate?
+- What results are being calculated?
+- Where in event time?
+- When in processing time?
+- How do refinements of results relate?
-For more details on the What / Where / When / How breakdown of concepts, we recommend reading through the Streaming 102 post on O'Reilly Radar.
+For more details on the What / Where / When / How breakdown of concepts, we recommend reading through the Streaming 102 post on O'Reilly Radar.
Note that in the future, we intend to add additional tables beyond the current set, for things like runtime characterstics (e.g. at-least-once vs exactly-once), performance, etc.
-{{< capability-matrix-common >}}
-
-{{< capability-matrix cap-data="capability-matrix" cap-style="cap-summary" cap-view="summary" cap-other-view="full" cap-toggle-details=1 cap-display="block" >}}
+{{< documentation/capability-matrix-single cap-data="capability-matrix" cap-style="cap-summary" cap-view="summary" cap-toggle-details=1 cap-display="block" >}}
-
-{{< capability-matrix cap-data="capability-matrix" cap-style="cap" cap-view="full" cap-other-view="summary" cap-toggle-details=0 cap-display="none" >}}
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix/additional-common-features-not-yet-part-of-the-beam-model.md b/website/www/site/content/en/documentation/runners/capability-matrix/additional-common-features-not-yet-part-of-the-beam-model.md
new file mode 100644
index 000000000000..525e3def6f86
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/additional-common-features-not-yet-part-of-the-beam-model.md
@@ -0,0 +1,21 @@
+---
+type: runners
+title: "Additional common features not yet part of the Beam model"
+layout: "capability-template"
+---
+
+
+
+{{< documentation/capability-matrix-big cap-data="capability-matrix" cap-view="full" cap-index=6 >}}
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix/bounded-splittable-dofn-support-status.md b/website/www/site/content/en/documentation/runners/capability-matrix/bounded-splittable-dofn-support-status.md
new file mode 100644
index 000000000000..0163d270dae5
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/bounded-splittable-dofn-support-status.md
@@ -0,0 +1,21 @@
+---
+type: runners
+title: "Bounded Splittable DoFn Support Status"
+layout: "capability-template"
+---
+
+
+
+{{< documentation/capability-matrix-big cap-data="capability-matrix" cap-view="full" cap-index=1 >}}
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix/how-do-refinements-relate.md b/website/www/site/content/en/documentation/runners/capability-matrix/how-do-refinements-relate.md
new file mode 100644
index 000000000000..7e5fd22e62aa
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/how-do-refinements-relate.md
@@ -0,0 +1,21 @@
+---
+type: runners
+title: "How do refinements relate?"
+layout: "capability-template"
+---
+
+
+
+{{< documentation/capability-matrix-big cap-data="capability-matrix" cap-view="full" cap-index=5 >}}
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix/unbounded-splittable-dofn-support-status.md b/website/www/site/content/en/documentation/runners/capability-matrix/unbounded-splittable-dofn-support-status.md
new file mode 100644
index 000000000000..9a1395b4cdf2
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/unbounded-splittable-dofn-support-status.md
@@ -0,0 +1,21 @@
+---
+type: runners
+title: "Unbounded Splittable DoFn Support Status"
+layout: "capability-template"
+---
+
+
+
+{{< documentation/capability-matrix-big cap-data="capability-matrix" cap-view="full" cap-index=2 >}}
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix/what-is-being-computed.md b/website/www/site/content/en/documentation/runners/capability-matrix/what-is-being-computed.md
new file mode 100644
index 000000000000..cb15ebaea1d4
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/what-is-being-computed.md
@@ -0,0 +1,21 @@
+---
+type: runners
+title: "What is being computed?"
+layout: "capability-template"
+---
+
+
+
+{{< documentation/capability-matrix-big cap-data="capability-matrix" cap-view="full" cap-index=0 >}}
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix/when-in-processing-time.md b/website/www/site/content/en/documentation/runners/capability-matrix/when-in-processing-time.md
new file mode 100644
index 000000000000..a6b319501059
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/when-in-processing-time.md
@@ -0,0 +1,21 @@
+---
+type: runners
+title: "When in processing time?"
+layout: "capability-template"
+---
+
+
+
+{{< documentation/capability-matrix-big cap-data="capability-matrix" cap-view="full" cap-index=4 >}}
diff --git a/website/www/site/content/en/documentation/runners/capability-matrix/where-in-event-time.md b/website/www/site/content/en/documentation/runners/capability-matrix/where-in-event-time.md
new file mode 100644
index 000000000000..cf8c5c7cbee3
--- /dev/null
+++ b/website/www/site/content/en/documentation/runners/capability-matrix/where-in-event-time.md
@@ -0,0 +1,21 @@
+---
+type: runners
+title: "Where in event time?"
+layout: "capability-template"
+---
+
+
+
+{{< documentation/capability-matrix-big cap-data="capability-matrix" cap-view="full" cap-index=3 >}}
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/cogroupbykey.md b/website/www/site/content/en/documentation/transforms/python/aggregation/cogroupbykey.md
index f80c3b20affe..d5dac4ed9d73 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/cogroupbykey.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/cogroupbykey.md
@@ -38,7 +38,7 @@ Then, we apply `CoGroupByKey` to join both `PCollection`s using their keys.
`CoGroupByKey` expects a dictionary of named keyed `PCollection`s, and produces elements joined by their keys.
The values of each output element are dictionaries where the names correspond to the input dictionary, with lists of all the values found for that key.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py" cogroupbykey >}}
{{< /highlight >}}
@@ -50,9 +50,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py" >}}
-
## Related transforms
* [CombineGlobally](/documentation/transforms/python/aggregation/combineglobally) to combine elements.
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/combineglobally.md b/website/www/site/content/en/documentation/transforms/python/aggregation/combineglobally.md
index 75db253f32e7..1b27e1844bb8 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/combineglobally.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/combineglobally.md
@@ -36,7 +36,7 @@ Then, we apply `CombineGlobally` in multiple ways to combine all the elements in
We define a function `get_common_items` which takes an `iterable` of sets as an input, and calculates the intersection (common items) of those sets.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" combineglobally_function >}}
{{< /highlight >}}
@@ -48,14 +48,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally_test.py" common_items >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
-
### Example 2: Combining with a lambda function
We can also use lambda functions to simplify **Example 1**.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" combineglobally_lambda >}}
{{< /highlight >}}
@@ -67,9 +64,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally_test.py" common_items >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
-
### Example 3: Combining with multiple arguments
You can pass functions with multiple arguments to `CombineGlobally`.
@@ -77,7 +71,7 @@ They are passed as additional positional arguments or keyword arguments to the f
In this example, the lambda function takes `sets` and `exclude` as arguments.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" combineglobally_multiple_arguments >}}
{{< /highlight >}}
@@ -89,9 +83,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally_test.py" common_items_with_exceptions >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
-
### Example 4: Combining with a `CombineFn`
The more general way to combine elements, and the most flexible, is with a class that inherits from `CombineFn`.
@@ -110,7 +101,7 @@ The more general way to combine elements, and the most flexible, is with a class
* [`CombineFn.extract_output()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output):
It allows to do additional calculations before extracting a result.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" combineglobally_combinefn >}}
{{< /highlight >}}
@@ -122,9 +113,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally_test.py" percentages >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineglobally.py" >}}
-
## Related transforms
You can use the following combiner transforms:
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/combineperkey.md b/website/www/site/content/en/documentation/transforms/python/aggregation/combineperkey.md
index f139619a1f4a..b79c183286ef 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/combineperkey.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/combineperkey.md
@@ -38,7 +38,7 @@ We use the function
[`sum`](https://docs.python.org/3/library/functions.html#sum)
which takes an `iterable` of numbers and adds them together.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" combineperkey_simple >}}
{{< /highlight >}}
@@ -50,14 +50,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py" total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
-
### Example 2: Combining with a function
We define a function `saturated_sum` which takes an `iterable` of numbers and adds them together, up to a predefined maximum number.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" combineperkey_function >}}
{{< /highlight >}}
@@ -69,14 +66,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py" saturated_total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
-
### Example 3: Combining with a lambda function
We can also use lambda functions to simplify **Example 2**.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" combineperkey_lambda >}}
{{< /highlight >}}
@@ -88,9 +82,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py" saturated_total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
-
### Example 4: Combining with multiple arguments
You can pass functions with multiple arguments to `CombinePerKey`.
@@ -98,7 +89,7 @@ They are passed as additional positional arguments or keyword arguments to the f
In this example, the lambda function takes `values` and `max_value` as arguments.
-{{< highlight py >}}
+{{< highlight language="py" py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" combineperkey_multiple_arguments >}}
{{< /highlight >}}
@@ -110,9 +101,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py" saturated_total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
-
### Example 5: Combining with a `CombineFn`
The more general way to combine elements, and the most flexible, is with a class that inherits from `CombineFn`.
@@ -131,7 +119,7 @@ The more general way to combine elements, and the most flexible, is with a class
* [`CombineFn.extract_output()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output):
It allows to do additional calculations before extracting a result.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" combineperkey_combinefn >}}
{{< /highlight >}}
@@ -143,9 +131,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py" average >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py" >}}
-
## Related transforms
You can use the following combiner transforms:
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/combinevalues.md b/website/www/site/content/en/documentation/transforms/python/aggregation/combinevalues.md
index 1698a428d711..64c1e7f7ccb0 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/combinevalues.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/combinevalues.md
@@ -39,7 +39,7 @@ We use the function
[`sum`](https://docs.python.org/3/library/functions.html#sum)
which takes an `iterable` of numbers and adds them together.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" combinevalues_simple >}}
{{< /highlight >}}
@@ -51,9 +51,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues_test.py" total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
-
### Example 2: Combining with a function
We want the sum to be bounded up to a maximum value, so we use
@@ -61,7 +58,7 @@ We want the sum to be bounded up to a maximum value, so we use
We define a function `saturated_sum` which takes an `iterable` of numbers and adds them together, up to a predefined maximum number.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" combinevalues_function >}}
{{< /highlight >}}
@@ -73,14 +70,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues_test.py" saturated_total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
-
### Example 3: Combining with a lambda function
We can also use lambda functions to simplify **Example 2**.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" combinevalues_lambda >}}
{{< /highlight >}}
@@ -92,9 +86,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues_test.py" saturated_total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
-
### Example 4: Combining with multiple arguments
You can pass functions with multiple arguments to `CombineValues`.
@@ -102,7 +93,7 @@ They are passed as additional positional arguments or keyword arguments to the f
In this example, the lambda function takes `values` and `max_value` as arguments.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" combinevalues_multiple_arguments >}}
{{< /highlight >}}
@@ -114,9 +105,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues_test.py" saturated_total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
-
### Example 5: Combining with a `CombineFn`
The more general way to combine elements, and the most flexible, is with a class that inherits from `CombineFn`.
@@ -135,7 +123,7 @@ The more general way to combine elements, and the most flexible, is with a class
* [`CombineFn.extract_output()`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.CombineFn.extract_output):
It allows to do additional calculations before extracting a result.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" combinevalues_combinefn >}}
{{< /highlight >}}
@@ -147,9 +135,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues_test.py" percentages_per_season >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/combinevalues.py" >}}
-
## Related transforms
You can use the following combiner transforms:
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/count.md b/website/www/site/content/en/documentation/transforms/python/aggregation/count.md
index f4d332ed2fc9..5aa6e1b8252f 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/count.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/count.md
@@ -32,7 +32,7 @@ Then, we apply `Count` to get the total number of elements in different ways.
We use `Count.Globally()` to count *all* elements in a `PCollection`, even if there are duplicate elements.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" count_globally >}}
{{< /highlight >}}
@@ -44,14 +44,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/count_test.py" total_elements >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" >}}
-
### Example 2: Counting elements for each key
We use `Count.PerKey()` to count the elements for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" count_per_key >}}
{{< /highlight >}}
@@ -63,14 +60,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/count_test.py" total_elements_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" >}}
-
### Example 3: Counting all unique elements
We use `Count.PerElement()` to count the only the unique elements in a `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" count_per_element >}}
{{< /highlight >}}
@@ -82,9 +76,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/count_test.py" total_unique_elements >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/count.py" >}}
-
## Related transforms
N/A
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/distinct.md b/website/www/site/content/en/documentation/transforms/python/aggregation/distinct.md
index e0abca48ee7a..08db6bd987c5 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/distinct.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/distinct.md
@@ -29,7 +29,7 @@ In the following example, we create a pipeline with two `PCollection`s of produc
We use `Distinct` to get rid of duplicate elements, which outputs a `PCollection` of all the unique elements.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct.py" distinct >}}
{{< /highlight >}}
@@ -41,9 +41,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct_test.py" unique_elements >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/distinct.py" >}}
-
## Related transforms
* [Count](/documentation/transforms/python/aggregation/count) counts the number of elements within each aggregation.
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/groupby.md b/website/www/site/content/en/documentation/transforms/python/aggregation/groupby.md
index ad6c17b8ec3f..df281bdabe0a 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/groupby.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/groupby.md
@@ -32,7 +32,7 @@ In the following example, we create a pipeline with a `PCollection` of fruits.
We use `GroupBy` to group all fruits by the first letter of their name.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_expr >}}
{{< /highlight >}}
@@ -44,13 +44,9 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_expr_result >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
-
-
We can group by a composite key consisting of multiple properties if desired.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_two_exprs >}}
{{< /highlight >}}
@@ -65,10 +61,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_two_exprs_result >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
-
-
In the case that the property one wishes to group by is an attribute, a string
may be passed to `GroupBy` in the place of a callable expression. For example,
suppose I have the following data
@@ -79,7 +71,7 @@ suppose I have the following data
We can then do
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_attr >}}
{{< /highlight >}}
@@ -91,12 +83,9 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_attr_result >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
-
It is possible to mix and match attributes and expressions, for example
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_attr_expr >}}
{{< /highlight >}}
@@ -108,9 +97,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" groupby_attr_expr_result >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}.
-
## Aggregation
Grouping is often used in conjunction with aggregation, and the
@@ -122,7 +108,7 @@ by, and finally a field name in which to store the result.
For example, suppose one wanted to compute the amount of each fruit to buy.
One could write
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" simple_aggregate >}}
{{< /highlight >}}
@@ -134,13 +120,10 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" simple_aggregate_result >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}.
-
Similar to the parameters in `GroupBy`, one can also aggregate multiple fields
and by expressions.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" expr_aggregate >}}
{{< /highlight >}}
@@ -152,13 +135,10 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" expr_aggregate_result >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}.
-
One can, of course, aggregate the same field multiple times as well.
This example also illustrates a global grouping, as the grouping key is empty.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" global_aggregate >}}
{{< /highlight >}}
@@ -170,10 +150,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" global_aggregate_result >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupby_test.py" >}}.
-
-
## Related transforms
* [CombinePerKey](/documentation/transforms/python/aggregation/combineperkey) for combining with a single CombineFn.
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/groupbykey.md b/website/www/site/content/en/documentation/transforms/python/aggregation/groupbykey.md
index f313e9d0dbf4..06ff183f833b 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/groupbykey.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/groupbykey.md
@@ -32,7 +32,7 @@ In the following example, we create a pipeline with a `PCollection` of produce k
We use `GroupByKey` to group all the produce for each season.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey.py" groupbykey >}}
{{< /highlight >}}
@@ -44,9 +44,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey_test.py" produce_counts >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupbykey.py" >}}
-
## Related transforms
* [GroupBy](/documentation/transforms/python/aggregation/groupby) for grouping by arbitrary properties of the elements.
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/groupintobatches.md b/website/www/site/content/en/documentation/transforms/python/aggregation/groupintobatches.md
index d9917fd5ae4b..a7c9cd9b2348 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/groupintobatches.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/groupintobatches.md
@@ -29,7 +29,7 @@ In the following example, we create a pipeline with a `PCollection` of produce b
We use `GroupIntoBatches` to get fixed-sized batches for every key, which outputs a list of elements for every key.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupintobatches.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupintobatches.py" groupintobatches >}}
{{< /highlight >}}
@@ -41,9 +41,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupintobatches_test.py" batches_with_keys >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/groupintobatches.py" >}}
-
## Related transforms
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/latest.md b/website/www/site/content/en/documentation/transforms/python/aggregation/latest.md
index f9692b10f73a..613858c44752 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/latest.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/latest.md
@@ -33,7 +33,7 @@ We use `Latest` to get the element with the latest timestamp from the `PCollecti
We use `Latest.Globally()` to get the element with the latest timestamp in the entire `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest.py" latest_globally >}}
{{< /highlight >}}
@@ -45,14 +45,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest_test.py" latest_element >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest.py" >}}
-
### Example 2: Latest elements for each key
We use `Latest.PerKey()` to get the elements with the latest timestamp for each key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest.py" latest_per_key >}}
{{< /highlight >}}
@@ -64,9 +61,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest_test.py" latest_elements_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/latest.py" >}}
-
## Related transforms
* [Sample](/documentation/transforms/python/aggregation/sample) randomly takes some number of elements in a collection.
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/max.md b/website/www/site/content/en/documentation/transforms/python/aggregation/max.md
index 7769b814517c..37210888226c 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/max.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/max.md
@@ -32,7 +32,7 @@ Then, we get the element with the maximum value in different ways.
We use `Combine.Globally()` to get the maximum element from the *entire* `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/max.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/max.py" max_globally >}}
{{< /highlight >}}
@@ -44,14 +44,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/max_test.py" max_element >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/max.py" >}}
-
### Example 2: Maximum elements for each key
We use `Combine.PerKey()` to get the maximum element for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/max.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/max.py" max_per_key >}}
{{< /highlight >}}
@@ -63,9 +60,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/max_test.py" elements_with_max_value_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/max.py" >}}
-
## Related transforms
* [CombineGlobally](/documentation/transforms/python/aggregation/combineglobally)
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/mean.md b/website/www/site/content/en/documentation/transforms/python/aggregation/mean.md
index 69fa5ade447d..76a2d0082ed1 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/mean.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/mean.md
@@ -34,7 +34,7 @@ Then, we get the element with the average value in different ways.
We use `Mean.Globally()` to get the average of the elements from the *entire* `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean.py" mean_globally >}}
{{< /highlight >}}
@@ -46,14 +46,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean_test.py" mean_element >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean.py" >}}
-
### Example 2: Mean of elements for each key
We use `Mean.PerKey()` to get the average of the elements for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean.py" mean_per_key >}}
{{< /highlight >}}
@@ -65,9 +62,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean_test.py" elements_with_mean_value_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/mean.py" >}}
-
## Related transforms
* [CombineGlobally](/documentation/transforms/python/aggregation/combineglobally)
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/min.md b/website/www/site/content/en/documentation/transforms/python/aggregation/min.md
index 8f385d20af69..62f9d15d4904 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/min.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/min.md
@@ -32,7 +32,7 @@ Then, we get the element with the minimum value in different ways.
We use `Combine.Globally()` to get the minimum element from the *entire* `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/min.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/min.py" min_globally >}}
{{< /highlight >}}
@@ -44,14 +44,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/min_test.py" min_element >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/min.py" >}}
-
### Example 2: Minimum elements for each key
We use `Combine.PerKey()` to get the minimum element for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/min.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/min.py" min_per_key >}}
{{< /highlight >}}
@@ -63,9 +60,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/min_test.py" elements_with_min_value_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/min.py" >}}
-
## Related transforms
* [CombineGlobally](/documentation/transforms/python/aggregation/combineglobally)
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/sample.md b/website/www/site/content/en/documentation/transforms/python/aggregation/sample.md
index 33fc8f60235b..bd822ea89d4e 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/sample.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/sample.md
@@ -34,7 +34,7 @@ Then, we get a random sample of elements in different ways.
We use `Sample.FixedSizeGlobally()` to get a fixed-size random sample of elements from the *entire* `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample.py" sample_fixed_size_globally >}}
{{< /highlight >}}
@@ -46,14 +46,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample_test.py" sample >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample.py" >}}
-
### Example 2: Sample elements for each key
We use `Sample.FixedSizePerKey()` to get fixed-size random samples for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample.py" sample_fixed_size_per_key >}}
{{< /highlight >}}
@@ -65,9 +62,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample_test.py" samples_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sample.py" >}}
-
## Related transforms
* [Top](/documentation/transforms/python/aggregation/top) finds the largest or smallest element.
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/sum.md b/website/www/site/content/en/documentation/transforms/python/aggregation/sum.md
index a9aef38d4c21..9ceda5cbb084 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/sum.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/sum.md
@@ -32,7 +32,7 @@ Then, we get the sum of all the element values in different ways.
We use `Combine.Globally()` to get sum of all the element values from the *entire* `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum.py" sum_globally >}}
{{< /highlight >}}
@@ -44,14 +44,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum_test.py" total >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum.py" >}}
-
### Example 2: Sum of the elements for each key
We use `Combine.PerKey()` to get the sum of all the element values for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum.py" sum_per_key >}}
{{< /highlight >}}
@@ -63,9 +60,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum_test.py" totals_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/sum.py" >}}
-
## Related transforms
* [CombineGlobally](/documentation/transforms/python/aggregation/combineglobally)
diff --git a/website/www/site/content/en/documentation/transforms/python/aggregation/top.md b/website/www/site/content/en/documentation/transforms/python/aggregation/top.md
index 419c0f81fb58..8c93499d049b 100644
--- a/website/www/site/content/en/documentation/transforms/python/aggregation/top.md
+++ b/website/www/site/content/en/documentation/transforms/python/aggregation/top.md
@@ -34,7 +34,7 @@ Then, we get the largest or smallest elements in different ways.
We use `Top.Largest()` to get the largest elements from the *entire* `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" top_largest >}}
{{< /highlight >}}
@@ -46,14 +46,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top_test.py" largest_elements >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
-
### Example 2: Largest elements for each key
We use `Top.LargestPerKey()` to get the largest elements for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" top_largest_per_key >}}
{{< /highlight >}}
@@ -65,14 +62,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top_test.py" largest_elements_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
-
### Example 3: Smallest elements from a PCollection
We use `Top.Smallest()` to get the smallest elements from the *entire* `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" top_smallest >}}
{{< /highlight >}}
@@ -84,14 +78,11 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top_test.py" smallest_elements >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
-
### Example 4: Smallest elements for each key
We use `Top.SmallestPerKey()` to get the smallest elements for each unique key in a `PCollection` of key-values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" top_smallest_per_key >}}
{{< /highlight >}}
@@ -103,9 +94,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top_test.py" smallest_elements_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
-
### Example 5: Custom elements from a PCollection
We use `Top.Of()` to get elements with customized rules from the *entire* `PCollection`.
@@ -113,7 +101,7 @@ We use `Top.Of()` to get elements with customized rules from the *entire* `PColl
You can change how the elements are compared with `key`.
By default you get the largest elements, but you can get the smallest by setting `reverse=True`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" top_of >}}
{{< /highlight >}}
@@ -125,9 +113,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top_test.py" shortest_elements >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
-
### Example 6: Custom elements for each key
We use `Top.PerKey()` to get elements with customized rules for each unique key in a `PCollection` of key-values.
@@ -135,7 +120,7 @@ We use `Top.PerKey()` to get elements with customized rules for each unique key
You can change how the elements are compared with `key`.
By default you get the largest elements, but you can get the smallest by setting `reverse=True`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" top_per_key >}}
{{< /highlight >}}
@@ -147,9 +132,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/aggregation/top_test.py" shortest_elements_per_key >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py" >}}
-
## Related transforms
* [Sample](/documentation/transforms/python/aggregation/sample) to combine elements. Takes samples of the elements in a collection.
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/filter.md b/website/www/site/content/en/documentation/transforms/python/elementwise/filter.md
index 3f602441932b..3b1b46e35239 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/filter.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/filter.md
@@ -36,7 +36,7 @@ Then, we apply `Filter` in multiple ways to filter out produce by their duration
We define a function `is_perennial` which returns `True` if the element's duration equals `'perennial'`, and `False` otherwise.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py" notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py" filter_function >}}
{{< /highlight >}}
@@ -48,15 +48,12 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py" perennials >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
-
### Example 2: Filtering with a lambda function
We can also use lambda functions to simplify **Example 1**.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py" filter_lambda >}}
{{< /highlight >}}
@@ -68,10 +65,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py" perennials >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
-
### Example 3: Filtering with multiple arguments
You can pass functions with multiple arguments to `Filter`.
@@ -79,7 +72,8 @@ They are passed as additional positional arguments or keyword arguments to the f
In this example, `has_duration` takes `plant` and `duration` as arguments.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py" filter_multiple_arguments >}}
{{< /highlight >}}
@@ -91,10 +85,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py" perennials >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
-
### Example 4: Filtering with side inputs as singletons
If the `PCollection` has a single value, such as the average from another computation,
@@ -103,7 +93,8 @@ passing the `PCollection` as a *singleton* accesses that value.
In this example, we pass a `PCollection` the value `'perennial'` as a singleton.
We then use that value to filter out perennials.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py" filter_side_inputs_singleton >}}
{{< /highlight >}}
@@ -115,17 +106,14 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py" perennials >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
-
### Example 5: Filtering with side inputs as iterators
If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
This accesses elements lazily as they are needed,
so it is possible to iterate over large `PCollection`s that won't fit into memory.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py" filter_side_inputs_iter >}}
{{< /highlight >}}
@@ -137,10 +125,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py" valid_plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
-
> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
> but this requires that all the elements fit into memory.
@@ -151,7 +135,8 @@ Each element must be a `(key, value)` pair.
Note that all the elements of the `PCollection` must fit into memory for this.
If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py" filter_side_inputs_dict >}}
{{< /highlight >}}
@@ -163,10 +148,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py" perennials >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/filter" >}}
-
## Related transforms
* [FlatMap](/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/flatmap.md b/website/www/site/content/en/documentation/transforms/python/elementwise/flatmap.md
index 7c7baa117b76..15648b986e2f 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/flatmap.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/flatmap.md
@@ -37,7 +37,8 @@ where each of the output `iterable`'s elements is an element of the resulting `P
We use the function `str.split` which takes a single `str` element and outputs a `list` of `str`s.
This pipeline splits the input element using whitespaces, creating a list of zero or more elements.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_simple >}}
{{< /highlight >}}
@@ -49,15 +50,12 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
### Example 2: FlatMap with a function
We define a function `split_words` which splits an input `str` element using the delimiter `','` and outputs a `list` of `str`s.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_function >}}
{{< /highlight >}}
@@ -69,17 +67,14 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
### Example 3: FlatMap with a lambda function
For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
Each input element is already an `iterable`, where each element is what we want in the resulting `PCollection`.
We use a lambda function that returns the same input element it received.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_lambda >}}
{{< /highlight >}}
@@ -91,17 +86,14 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
### Example 4: FlatMap with a generator
For this example, we want to flatten a `PCollection` of lists of `str`s into a `PCollection` of `str`s.
We use a generator to iterate over the input list and yield each of the elements.
Each yielded result in the generator is an element in the resulting `PCollection`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_generator >}}
{{< /highlight >}}
@@ -113,16 +105,13 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
### Example 5: FlatMapTuple for key-value pairs
If your `PCollection` consists of `(key, value)` pairs,
you can use `FlatMapTuple` to unpack them into different function arguments.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_tuple >}}
{{< /highlight >}}
@@ -134,10 +123,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
### Example 6: FlatMap with multiple arguments
You can pass functions with multiple arguments to `FlatMap`.
@@ -145,7 +130,8 @@ They are passed as additional positional arguments or keyword arguments to the f
In this example, `split_words` takes `text` and `delimiter` as arguments.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_multiple_arguments >}}
{{< /highlight >}}
@@ -157,10 +143,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
### Example 7: FlatMap with side inputs as singletons
If the `PCollection` has a single value, such as the average from another computation,
@@ -169,7 +151,8 @@ passing the `PCollection` as a *singleton* accesses that value.
In this example, we pass a `PCollection` the value `','` as a singleton.
We then use that value as the delimiter for the `str.split` method.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_side_inputs_singleton >}}
{{< /highlight >}}
@@ -181,17 +164,14 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
### Example 8: FlatMap with side inputs as iterators
If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
This accesses elements lazily as they are needed,
so it is possible to iterate over large `PCollection`s that won't fit into memory.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_side_inputs_iter >}}
{{< /highlight >}}
@@ -203,10 +183,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" valid_plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
> but this requires that all the elements fit into memory.
@@ -217,7 +193,8 @@ Each element must be a `(key, value)` pair.
Note that all the elements of the `PCollection` must fit into memory for this.
If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py" flatmap_side_inputs_dict >}}
{{< /highlight >}}
@@ -229,10 +206,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py" valid_plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/flatmap" >}}
-
## Related transforms
* [Filter](/documentation/transforms/python/elementwise/filter) is useful if the function is just
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/keys.md b/website/www/site/content/en/documentation/transforms/python/elementwise/keys.md
index 1bfac3bb94ce..d30717d3e814 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/keys.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/keys.md
@@ -28,7 +28,8 @@ Takes a collection of key-value pairs and returns the key of each element.
In the following example, we create a pipeline with a `PCollection` of key-value pairs.
Then, we apply `Keys` to extract the keys and discard the values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/keys" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py" keys >}}
{{< /highlight >}}
@@ -40,10 +41,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py" icons >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/keys" >}}
-
## Related transforms
* [KvSwap](/documentation/transforms/python/elementwise/kvswap) swaps the key and value of each element.
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/kvswap.md b/website/www/site/content/en/documentation/transforms/python/elementwise/kvswap.md
index 259fe8902587..3c3a25518c82 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/kvswap.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/kvswap.md
@@ -29,7 +29,8 @@ which has each key and value swapped.
In the following example, we create a pipeline with a `PCollection` of key-value pairs.
Then, we apply `KvSwap` to swap the keys and values.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/kvswap" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py" kvswap >}}
{{< /highlight >}}
@@ -41,10 +42,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/kvswap" >}}
-
## Related transforms
* [Keys](/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/map.md b/website/www/site/content/en/documentation/transforms/python/elementwise/map.md
index a6f3bdab9f52..d31bbabea2e2 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/map.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/map.md
@@ -35,7 +35,8 @@ Then, we apply `Map` in multiple ways to transform every element in the `PCollec
We use the function `str.strip` which takes a single `str` element and outputs a `str`.
It strips the input element's whitespaces, including newlines and tabs.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_simple >}}
{{< /highlight >}}
@@ -47,15 +48,12 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
### Example 2: Map with a function
We define a function `strip_header_and_newline` which strips any `'#'`, `' '`, and `'\n'` characters from each element.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_function >}}
{{< /highlight >}}
@@ -67,15 +65,12 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
### Example 3: Map with a lambda function
We can also use lambda functions to simplify **Example 2**.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_lambda >}}
{{< /highlight >}}
@@ -87,10 +82,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
### Example 4: Map with multiple arguments
You can pass functions with multiple arguments to `Map`.
@@ -98,7 +89,8 @@ They are passed as additional positional arguments or keyword arguments to the f
In this example, `strip` takes `text` and `chars` as arguments.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_multiple_arguments >}}
{{< /highlight >}}
@@ -110,16 +102,13 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
### Example 5: MapTuple for key-value pairs
If your `PCollection` consists of `(key, value)` pairs,
you can use `MapTuple` to unpack them into different function arguments.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_tuple >}}
{{< /highlight >}}
@@ -131,10 +120,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
### Example 6: Map with side inputs as singletons
If the `PCollection` has a single value, such as the average from another computation,
@@ -143,7 +128,8 @@ passing the `PCollection` as a *singleton* accesses that value.
In this example, we pass a `PCollection` the value `'# \n'` as a singleton.
We then use that value as the characters for the `str.strip` method.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_side_inputs_singleton >}}
{{< /highlight >}}
@@ -155,17 +141,14 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
### Example 7: Map with side inputs as iterators
If the `PCollection` has multiple values, pass the `PCollection` as an *iterator*.
This accesses elements lazily as they are needed,
so it is possible to iterate over large `PCollection`s that won't fit into memory.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_side_inputs_iter >}}
{{< /highlight >}}
@@ -177,10 +160,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
> **Note**: You can pass the `PCollection` as a *list* with `beam.pvalue.AsList(pcollection)`,
> but this requires that all the elements fit into memory.
@@ -191,7 +170,8 @@ Each element must be a `(key, value)` pair.
Note that all the elements of the `PCollection` must fit into memory for this.
If the `PCollection` won't fit into memory, use `beam.pvalue.AsIter(pcollection)` instead.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py" map_side_inputs_dict >}}
{{< /highlight >}}
@@ -203,10 +183,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py" plant_details >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/map.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/map" >}}
-
## Related transforms
* [FlatMap](/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/pardo.md b/website/www/site/content/en/documentation/transforms/python/elementwise/pardo.md
index 16089701b5fd..5021e577a4c4 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/pardo.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/pardo.md
@@ -41,7 +41,8 @@ which stores the `delimiter` as an object field.
The `process` method is called once per element,
and it can yield zero or more output elements.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py" pardo_dofn >}}
{{}}
@@ -53,10 +54,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo" >}}
-
### Example 2: ParDo with timestamp and window information
In this example, we add new parameters to the `process` method to bind parameter values at runtime.
@@ -70,7 +67,8 @@ In this example, we add new parameters to the `process` method to bind parameter
[`apache_beam.transforms.window.*Window`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.window.html)
object.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py" pardo_dofn_params >}}
{{}}
@@ -82,10 +80,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py" dofn_params >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo" >}}
-
### Example 3: ParDo with DoFn methods
A [`DoFn`](https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.core.html#apache_beam.transforms.core.DoFn)
@@ -131,7 +125,8 @@ starts and finishes with `start_bundle` and `finish_bundle`.
Note that `teardown` is called as a *best effort* and is *not guaranteed*.
For example, if the worker crashes, `teardown` might not be called.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py" pardo_dofn_methods >}}
{{}}
@@ -143,10 +138,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py" results >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/pardo" >}}
-
> *Known issues:*
>
> * [[BEAM-7885]](https://issues.apache.org/jira/browse/BEAM-7885)
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/partition.md b/website/www/site/content/en/documentation/transforms/python/elementwise/partition.md
index 05897053455e..fbfe8a99ac52 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/partition.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/partition.md
@@ -46,7 +46,8 @@ and it must return an integer in the range `0` to `num_partitions-1`.
In the following example, we have a known list of durations.
We partition the `PCollection` into one `PCollection` for every duration type.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py" partition_function >}}
{{< /highlight >}}
@@ -58,15 +59,12 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py" partitions >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/partition" >}}
-
### Example 2: Partition with a lambda function
We can also use lambda functions to simplify **Example 1**.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py" partition_lambda >}}
{{< /highlight >}}
@@ -78,10 +76,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py" partitions >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/partition" >}}
-
### Example 3: Partition with multiple arguments
You can pass functions with multiple arguments to `Partition`.
@@ -114,7 +108,8 @@ identify the partition index to which that bucket corresponds.
This `split_dataset` function is generic enough to support any number of partitions by any ratio.
You might want to adapt the bucket assignment to use a more appropriate or randomized hash for your dataset.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/partition" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py" partition_multiple_arguments >}}
{{< /highlight >}}
@@ -126,10 +121,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py" train_test >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/partition" >}}
-
## Related transforms
* [Filter](/documentation/transforms/python/elementwise/filter) is useful if the function is just
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/regex.md b/website/www/site/content/en/documentation/transforms/python/elementwise/regex.md
index c31fa452e571..6947fd18252a 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/regex.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/regex.md
@@ -61,7 +61,8 @@ To match until the end of the string, add `'$'` at the end of the regular expres
To start matching at any point instead of the beginning of the string, use
[`Regex.find(regex)`](#example-4-regex-find).
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_matches >}}
{{< /highlight >}}
@@ -73,10 +74,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_matches >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 2: Regex match with all groups
`Regex.all_matches` keeps only the elements that match the regular expression,
@@ -90,7 +87,8 @@ To match until the end of the string, add `'$'` at the end of the regular expres
To start matching at any point instead of the beginning of the string, use
[`Regex.find_all(regex, group=Regex.ALL, outputEmpty=False)`](#example-5-regex-find-all).
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_all_matches >}}
{{< /highlight >}}
@@ -102,10 +100,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_all_matches >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 3: Regex match into key-value pairs
`Regex.matches_kv` keeps only the elements that match the regular expression,
@@ -120,7 +114,8 @@ To match until the end of the string, add `'$'` at the end of the regular expres
To start matching at any point instead of the beginning of the string, use
[`Regex.find_kv(regex, keyGroup)`](#example-6-regex-find-as-key-value-pairs).
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_matches_kv >}}
{{< /highlight >}}
@@ -132,10 +127,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_matches_kv >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 4: Regex find
`Regex.find` keeps only the elements that match the regular expression,
@@ -150,7 +141,8 @@ To match until the end of the string, add `'$'` at the end of the regular expres
If you need to match from the start only, consider using
[`Regex.matches(regex)`](#example-1-regex-match).
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_find >}}
{{< /highlight >}}
@@ -162,10 +154,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_matches >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 5: Regex find all
`Regex.find_all` returns a list of all the matches of the regular expression,
@@ -180,7 +168,8 @@ To match until the end of the string, add `'$'` at the end of the regular expres
If you need to match all groups from the start only, consider using
[`Regex.all_matches(regex)`](#example-2-regex-match-with-all-groups).
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_find_all >}}
{{< /highlight >}}
@@ -192,10 +181,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_find_all >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 6: Regex find as key-value pairs
`Regex.find_kv` returns a list of all the matches of the regular expression,
@@ -211,7 +196,8 @@ To match until the end of the string, add `'$'` at the end of the regular expres
If you need to match as key-value pairs from the start only, consider using
[`Regex.matches_kv(regex)`](#example-3-regex-match-into-key-value-pairs).
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_find_kv >}}
{{< /highlight >}}
@@ -223,10 +209,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_find_kv >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 7: Regex replace all
`Regex.replace_all` returns the string with all the occurrences of the regular expression replaced by another string.
@@ -234,7 +216,8 @@ You can also use
[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
on the `replacement`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_replace_all >}}
{{< /highlight >}}
@@ -246,10 +229,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_replace_all >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 8: Regex replace first
`Regex.replace_first` returns the string with the first occurrence of the regular expression replaced by another string.
@@ -257,7 +236,8 @@ You can also use
[backreferences](https://docs.python.org/3/library/re.html?highlight=backreference#re.sub)
on the `replacement`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_replace_first >}}
{{< /highlight >}}
@@ -269,16 +249,13 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_replace_first >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
### Example 9: Regex split
`Regex.split` returns the list of strings that were delimited by the specified regular expression.
The argument `outputEmpty` is set to `False` by default, but can be set to `True` to keep empty items in the output list.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py" regex_split >}}
{{< /highlight >}}
@@ -290,10 +267,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py" plants_split >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/regex" >}}
-
## Related transforms
* [FlatMap](/documentation/transforms/python/elementwise/flatmap) behaves the same as `Map`, but for
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/tostring.md b/website/www/site/content/en/documentation/transforms/python/elementwise/tostring.md
index 2409524dcd2d..4b3d79acada0 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/tostring.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/tostring.md
@@ -35,7 +35,8 @@ expect their input elements to be strings.
The following example converts a `(key, value)` pair into a string delimited by `','`.
You can specify a different delimiter using the `delimiter` argument.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py" tostring_kvs >}}
{{< /highlight >}}
@@ -47,16 +48,13 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring" >}}
-
### Example 2: Elements to string
The following example converts a dictionary into a string.
The string output will be equivalent to `str(element)`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py" tostring_element >}}
{{< /highlight >}}
@@ -68,10 +66,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py" plant_lists >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring" >}}
-
### Example 3: Iterables to string
The following example converts an iterable, in this case a list of strings,
@@ -79,7 +73,8 @@ into a string delimited by `','`.
You can specify a different delimiter using the `delimiter` argument.
The string output will be equivalent to `iterable.join(delimiter)`.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py" tostring_iterables >}}
{{< /highlight >}}
@@ -91,10 +86,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py" plants_csv >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/tostring" >}}
-
## Related transforms
* [Map](/documentation/transforms/python/elementwise/map) applies a simple 1-to-1 mapping function over each element in the collection
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/values.md b/website/www/site/content/en/documentation/transforms/python/elementwise/values.md
index 90f706ee8c7d..63e324c10eb4 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/values.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/values.md
@@ -28,7 +28,8 @@ Takes a collection of key-value pairs, and returns the value of each element.
In the following example, we create a pipeline with a `PCollection` of key-value pairs.
Then, we apply `Values` to extract the values and discard the keys.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/values" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py" values >}}
{{< /highlight >}}
@@ -40,10 +41,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py" plants >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/values.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/values" >}}
-
## Related transforms
* [Keys](/documentation/transforms/python/elementwise/keys) for extracting the key of each component.
diff --git a/website/www/site/content/en/documentation/transforms/python/elementwise/withtimestamps.md b/website/www/site/content/en/documentation/transforms/python/elementwise/withtimestamps.md
index 6fa4702ee636..51c9c64abc23 100644
--- a/website/www/site/content/en/documentation/transforms/python/elementwise/withtimestamps.md
+++ b/website/www/site/content/en/documentation/transforms/python/elementwise/withtimestamps.md
@@ -33,7 +33,8 @@ The elements themselves often already contain a timestamp field.
[Unix timestamp](https://en.wikipedia.org/wiki/Unix_time)
in the form of seconds.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py" withtimestamps_event_time >}}
{{< /highlight >}}
@@ -45,10 +46,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py" plant_timestamps >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps" >}}
-
To convert from a
[`time.struct_time`](https://docs.python.org/3/library/time.html#time.struct_time)
to `unix_time` you can use
@@ -75,7 +72,8 @@ If each element has a chronological number, these numbers can be used as a
[logical clock](https://en.wikipedia.org/wiki/Logical_clock).
These numbers have to be converted to a *"seconds"* equivalent, which can be especially important depending on your windowing and late data rules.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py" withtimestamps_logical_clock >}}
{{< /highlight >}}
@@ -87,10 +85,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py" plant_events >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps" >}}
-
### Example 3: Timestamp by processing time
If the elements do not have any time data available, you can also use the current processing time for each element.
@@ -99,7 +93,8 @@ Workers might have time deltas, so using this method is not a reliable way to do
By using processing time, there is no way of knowing if data is arriving late because the timestamp is attached when the element *enters* into the pipeline.
-{{< highlight py >}}
+{{< highlight language="py" file="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
+ notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps" >}}
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py" withtimestamps_processing_time >}}
{{< /highlight >}}
@@ -111,10 +106,6 @@ Output:
{{< code_sample "sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py" plant_processing_times >}}
{{< /highlight >}}
-{{< buttons-code-snippet
- py="sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py"
- notebook="examples/notebooks/documentation/transforms/python/elementwise/withtimestamps" >}}
-
## Related transforms
* [Reify](/documentation/transforms/python/elementwise/reify) converts between explicit and implicit forms of Beam values.
diff --git a/website/www/site/content/en/get-started/beam-overview.md b/website/www/site/content/en/get-started/beam-overview.md
index 40aacb01bc37..946d87177455 100644
--- a/website/www/site/content/en/get-started/beam-overview.md
+++ b/website/www/site/content/en/get-started/beam-overview.md
@@ -44,13 +44,13 @@ The Beam Pipeline Runners translate the data processing pipeline you define with
Beam currently supports the following runners:
- [Direct Runner](/documentation/runners/direct)
-- [Apache Flink Runner](/documentation/runners/flink) 
+- [Apache Flink Runner](/documentation/runners/flink) 
- [Apache Nemo Runner](/documentation/runners/nemo)
-- [Apache Samza Runner](/documentation/runners/samza) 
-- [Apache Spark Runner](/documentation/runners/spark) 
-- [Google Cloud Dataflow Runner](/documentation/runners/dataflow) 
-- [Hazelcast Jet Runner](/documentation/runners/jet) 
-- [Twister2 Runner](/documentation/runners/twister2) 
+- [Apache Samza Runner](/documentation/runners/samza)
+- [Apache Spark Runner](/documentation/runners/spark) 
+- [Google Cloud Dataflow Runner](/documentation/runners/dataflow) 
+- [Hazelcast Jet Runner](/documentation/runners/jet) 
+- [Twister2 Runner](/documentation/runners/twister2) 
**Note:** You can always execute your pipeline locally for testing and debugging purposes.
diff --git a/website/www/site/content/en/powered-by/_index.md b/website/www/site/content/en/powered-by/_index.md
new file mode 100644
index 000000000000..92836bef00c0
--- /dev/null
+++ b/website/www/site/content/en/powered-by/_index.md
@@ -0,0 +1,24 @@
+---
+title: "Projects and Products Powered by Apache Beam"
+layout: main
+---
+
+
+
+# Projects and Products Powered by Apache Beam
+
+To add yourself to the list, please open a pull request adding your organization name, URL, a list of which Beam components you are using, and a short description of your use case.
+
+{{< powered-by/powered-by-cards powered-by >}}
diff --git a/website/www/site/content/en/powered-by/commercial/_index.md b/website/www/site/content/en/powered-by/commercial/_index.md
new file mode 100644
index 000000000000..d3c86cacc178
--- /dev/null
+++ b/website/www/site/content/en/powered-by/commercial/_index.md
@@ -0,0 +1,19 @@
+---
+title: "Commercial projects"
+type: commercial
+icon: images/github-icon.svg
+---
+
+
diff --git a/website/www/site/content/en/powered-by/commercial/eBay.md b/website/www/site/content/en/powered-by/commercial/eBay.md
new file mode 100644
index 000000000000..feae3418fa93
--- /dev/null
+++ b/website/www/site/content/en/powered-by/commercial/eBay.md
@@ -0,0 +1,22 @@
+---
+title: "Feature Powered by Apache Beam - Beyond Lambda"
+type: commercial
+categories:
+ - Online & Traditional Retail
+icon: /images/logos/powered-by/eBay.png
+---
+
+
+eBay is an American e-commerce company that provides business-to-consumer and consumer-to-consumer sales through the online website. They build feature pipelines with Apache Beam: unify feature extraction and selection in online and offline, speed up E2E iteration for model training, evaluation and serving, support different types (streaming, runtime, batch) of features, etc. eBay leverages Apache Beam for the streaming feature SDK as a foundation to integrate with Kafka, Hadoop, Flink, Airflow and others in eBay.
diff --git a/website/www/site/content/en/powered-by/commercial/goga.md b/website/www/site/content/en/powered-by/commercial/goga.md
new file mode 100644
index 000000000000..ad83555b3f7b
--- /dev/null
+++ b/website/www/site/content/en/powered-by/commercial/goga.md
@@ -0,0 +1,22 @@
+---
+title: "Beam and Geocoding"
+type: commercial
+categories:
+ - Machine Learning
+icon: /images/logos/powered-by/goga.png
+---
+
+
+GOGA Data Analysis and Consulting is a company based in Japan that specializes in analytics of geospatial and mapping data. They use Apache Beam and Cloud Dataflow for a smooth data transformation process for analytical purposes. This use case focuses on handling multiple extractions, geocoding, and insertion process by wrangling and requesting API call of each data based on the location provided.
diff --git a/website/www/site/content/en/powered-by/commercial/mozilla.md b/website/www/site/content/en/powered-by/commercial/mozilla.md
new file mode 100644
index 000000000000..5ee0dfd4f550
--- /dev/null
+++ b/website/www/site/content/en/powered-by/commercial/mozilla.md
@@ -0,0 +1,22 @@
+---
+title: "The Nitty-Gritty of Moving Data with Beam"
+type: commercial
+categories:
+ - Enterprise
+icon: /images/logos/powered-by/mozilla.png
+---
+
+
+Mozilla is the non-profit Firefox browser. This use case focuses on complexity that comes from moving data from one system to another safely, modeling data as it passes from one transform to another, handling errors, testing the system, and organizing the code to make the pipeline configurable for different source and destination systems in their open source codebase for ingesting telemetry data from Firefox clients.
diff --git a/website/www/site/content/en/powered-by/commercial/neo4j.md b/website/www/site/content/en/powered-by/commercial/neo4j.md
new file mode 100644
index 000000000000..0f9f8500757c
--- /dev/null
+++ b/website/www/site/content/en/powered-by/commercial/neo4j.md
@@ -0,0 +1,23 @@
+---
+title: "Beam visual pipeline development with Hop"
+type: commercial
+categories:
+ - BI & Analytics
+icon: /images/logos/powered-by/neo4j.png
+---
+
+
+Neo4j is a graph database management system that leverages data relationships between data records to enable fast queries. They use Apache Beam support in Kettle to do everything and more in project Hop. The various Apache Beam runners are now directly supported as Hop pipeline engines.
+[Apache Hop (incubating)](http://hop.apache.org): Hop provides a complete data orchestration (ETL / DI) toolset with visual pipeline development. It supports execution on the main Apache Beam runners.
diff --git a/website/www/site/content/en/powered-by/commercial/oriel.md b/website/www/site/content/en/powered-by/commercial/oriel.md
new file mode 100644
index 000000000000..0827c3f2bc91
--- /dev/null
+++ b/website/www/site/content/en/powered-by/commercial/oriel.md
@@ -0,0 +1,23 @@
+---
+title: "From Apache Beam to Leukemia early detection"
+type: commercial
+categories:
+ - Machine Learning
+icon: /images/logos/powered-by/oriel.png
+---
+
+
+Oriel Research Therapeutics (ORT) is a startup company in the greater Boston area that provides early detection services for
+multiple medical conditions, utilizing cutting edge Artificial Intelligence technologies and Next Generation Sequencing (NGS). ORT utilizes Apache Beam pipelines to process over 1 million samples of genomics and clinical information. The processed data is used by ORT in detecting Leukemia, Sepsis and other medical conditions.
diff --git a/website/www/site/content/en/powered-by/commercial/ricardo.md b/website/www/site/content/en/powered-by/commercial/ricardo.md
new file mode 100644
index 000000000000..09e34c53ed3a
--- /dev/null
+++ b/website/www/site/content/en/powered-by/commercial/ricardo.md
@@ -0,0 +1,23 @@
+---
+title: "Four Apache Technologies Combined for Fun and Profit"
+type: commercial
+categories:
+ - Online & Traditional Retail
+icon: /images/logos/powered-by/ricardo.png
+---
+
+
+Ricardo, the largest online marketplace in Switzerland, uses Apache Beam to stream-process platform data and enables the
+Data Intelligence team to provide scalable data integration, analytics, and smart services.
diff --git a/website/www/site/content/en/powered-by/open-source/_index.md b/website/www/site/content/en/powered-by/open-source/_index.md
new file mode 100644
index 000000000000..bae5f3a50cbb
--- /dev/null
+++ b/website/www/site/content/en/powered-by/open-source/_index.md
@@ -0,0 +1,17 @@
+---
+title: "Open source projects"
+type: open-source
+---
+
diff --git a/website/www/site/content/en/powered-by/open-source/klio.md b/website/www/site/content/en/powered-by/open-source/klio.md
new file mode 100644
index 000000000000..9c45ba1efaa2
--- /dev/null
+++ b/website/www/site/content/en/powered-by/open-source/klio.md
@@ -0,0 +1,22 @@
+---
+title: "Klio"
+type: open-source
+categories:
+ - Open Source Projects Powered by Apache Beam
+icon: /images/logos/powered-by/klio.png
+---
+
+
+Developed at Spotify and built on top of Apache Beam for Python, Klio is an open source framework that lets researchers and engineers build smarter data pipelines for processing audio and other media files, easily and at scale.
diff --git a/website/www/site/content/en/powered-by/open-source/scio.md b/website/www/site/content/en/powered-by/open-source/scio.md
new file mode 100644
index 000000000000..549b5bbbab8e
--- /dev/null
+++ b/website/www/site/content/en/powered-by/open-source/scio.md
@@ -0,0 +1,20 @@
+---
+title: "Scio"
+type: open-source
+categories:
+ - Open Source Projects Powered by Apache Beam
+icon: /images/logos/powered-by/scio.png
+---
+
diff --git a/website/www/site/data/authors.yml b/website/www/site/data/authors.yml
index 24e90c5999f3..6f97254bc04a 100644
--- a/website/www/site/data/authors.yml
+++ b/website/www/site/data/authors.yml
@@ -213,4 +213,4 @@ ilyakozyrev:
alexkosolapov:
name: Alex Kosolapov
email: alex.kosolapov@akvelon.com
- twitter:
\ No newline at end of file
+ twitter:
diff --git a/website/www/site/data/capability_matrix.yaml b/website/www/site/data/capability_matrix.yaml
index dacddee82022..9b10feba3017 100644
--- a/website/www/site/data/capability_matrix.yaml
+++ b/website/www/site/data/capability_matrix.yaml
@@ -44,10 +44,12 @@ capability-matrix:
categories:
- description: What is being computed?
anchor: what
- color-b: "ca1"
- color-y: "ec3"
- color-p: "fe5"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: ParDo
values:
@@ -573,10 +575,12 @@ capability-matrix:
l3: ""
- description: Bounded Splittable DoFn Support Status
anchor: what
- color-b: "ca1"
- color-y: "ec3"
- color-p: "fe5"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Base
values:
@@ -870,10 +874,12 @@ capability-matrix:
l3:
- description: Unbounded Splittable DoFn Support Status
anchor: what
- color-b: "ca1"
- color-y: "ec3"
- color-p: "fe5"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Base
values:
@@ -1167,10 +1173,12 @@ capability-matrix:
l3:
- description: Where in event time?
anchor: where
- color-b: "37d"
- color-y: "59f"
- color-p: "8cf"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Global windows
values:
@@ -1525,10 +1533,12 @@ capability-matrix:
- description: When in processing time?
anchor: when
- color-b: "6a4"
- color-y: "8c6"
- color-p: "ae8"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Configurable triggering
values:
@@ -1941,10 +1951,12 @@ capability-matrix:
- description: How do refinements relate?
anchor: how
- color-b: "b55"
- color-y: "d77"
- color-p: "faa"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Discarding
values:
@@ -2101,10 +2113,12 @@ capability-matrix:
l3: ""
- description: Additional common features not yet part of the Beam model
anchor: misc
- color-b: "aaa"
- color-y: "bbb"
- color-p: "ccc"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Drain
values:
diff --git a/website/www/site/data/capability_matrix_snapshot.yaml b/website/www/site/data/capability_matrix_snapshot.yaml
index 74f5601c5e2a..9da8a705fc2c 100644
--- a/website/www/site/data/capability_matrix_snapshot.yaml
+++ b/website/www/site/data/capability_matrix_snapshot.yaml
@@ -23,10 +23,12 @@ capability-matrix-snapshot:
categories:
- description: What is being computed?
anchor: what
- color-b: "ca1"
- color-y: "ec3"
- color-p: "fe5"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: ParDo
values:
@@ -202,10 +204,12 @@ capability-matrix-snapshot:
- description: Where in event time?
anchor: where
- color-b: "37d"
- color-y: "59f"
- color-p: "8cf"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Global windows
values:
@@ -342,10 +346,12 @@ capability-matrix-snapshot:
- description: When in processing time?
anchor: when
- color-b: "6a4"
- color-y: "8c6"
- color-p: "ae8"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Configurable triggering
values:
@@ -503,10 +509,12 @@ capability-matrix-snapshot:
- description: How do refinements relate?
anchor: how
- color-b: "b55"
- color-y: "d77"
- color-p: "faa"
- color-n: "ddd"
+ color-y: "fff"
+ color-yb: "f6f6f6"
+ color-p: "f9f9f9"
+ color-pb: "d8d8d8"
+ color-n: "e1e0e0"
+ color-nb: "bcbcbc"
rows:
- name: Discarding
values:
diff --git a/website/www/site/data/en/cards.yaml b/website/www/site/data/en/calendar_events.yaml
similarity index 60%
rename from website/www/site/data/en/cards.yaml
rename to website/www/site/data/en/calendar_events.yaml
index a2981e7fba86..865542f9c637 100644
--- a/website/www/site/data/en/cards.yaml
+++ b/website/www/site/data/en/calendar_events.yaml
@@ -10,9 +10,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-- quote: "A framework that delivers the flexibility and advanced functionality our customers need."
- name: –Talend
-- quote: "Apache Beam has powerful semantics that solve real-world challenges of stream processing."
- name: –PayPal
-- quote: "Apache Beam represents a principled approach for analyzing data streams."
- name: –data Artisans
+- title: This is the title
+ place: Data Con LA
+ time: 2020/02/21
+ icon: icons/calendar-icon.svg
+ url: #
+- title: This is the title
+ place: Data Con LA
+ time: 2020/02/21
+ icon: icons/calendar-icon.svg
+ url: #
+- title: This is the title
+ place: Data Con LA
+ time: 2020/02/21
+ icon: icons/calendar-icon.svg
+ url: #
\ No newline at end of file
diff --git a/website/www/site/data/en/committer_traits.yaml b/website/www/site/data/en/committer_traits.yaml
new file mode 100644
index 000000000000..b9c83621f3fd
--- /dev/null
+++ b/website/www/site/data/en/committer_traits.yaml
@@ -0,0 +1,21 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- body: { text: "Learn more.", link: /contribute/attributes#knows-upholds-and-reinforces-the-apache-software-foundation-code-of-conduct }
+ bolded: Knows, upholds, and reinforces the Apache Software Foundation code of conduct.
+ icon: icons/contributor/become a committer/messages-icon.svg
+- body: { text: "Learn more.", link: /contribute/attributes#knows-upholds-and-reinforces-the-responsibilities-of-an-apache-software-foundation-committer }
+ bolded: Knows, upholds, and reinforces the responsibilities of an Apache Software Foundation committer.
+ icon: icons/contributor/become a committer/beam-logo-icon.svg
+- body: { text: "Learn more.", link: /contribute/attributes/#knows-upholds-and-reinforces-the-beam-communitys-practices }
+ bolded: Knows, upholds, and reinforces the Beam community’s practices.
+ icon: icons/contributor/become a committer/diamond-icon.svg
diff --git a/website/www/site/data/en/community_list.yaml b/website/www/site/data/en/community_list.yaml
new file mode 100644
index 000000000000..9a6b9e787489
--- /dev/null
+++ b/website/www/site/data/en/community_list.yaml
@@ -0,0 +1,27 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- title: Join Beam Community
+ body: Learn more about the contribution pathway you can take to become a committer or PMC member.
+ icon: icons/community/bee-icon.svg
+- title: Contact us
+ body: Here’s a list of channels that the Apache Beam community and Project Management Committee (PMC) monitor and interact in. We look forward to seeing you!
+ icon: icons/community/envelope-icon.svg
+- title: Keep up with Beam news and updates
+ body: Follow our official Twitter account and subscribe to our YouTube channel to stay up-to-date with project updates and the latest training.
+ icon: icons/community/message-icon.svg
+- title: Meet with the community
+ body: Check out our calendar of events in the homepage, we update it frequently so you know where to meet us.
+ icon: icons/community/calendar-icon.svg
+- title: Promotional assets
+ body: Looking for some marketing material for your next Beam talk or pitch? Here’s a list of our brand assets including logos, presentations, and the firefly mascot!
+ icon: icons/community/box-icon.svg
diff --git a/website/www/site/data/en/contact_us.yaml b/website/www/site/data/en/contact_us.yaml
new file mode 100644
index 000000000000..c5e09e35411f
--- /dev/null
+++ b/website/www/site/data/en/contact_us.yaml
@@ -0,0 +1,114 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- columns:
+ - purpose
+ - channel
+ - action
+ data:
+ - purpose: Get user support, ask general questions and stay up-to-date with project news
+ contact:
+ {
+ email: user@beam.apache.org,
+ link: "https://lists.apache.org/list.html?user@beam.apache.org",
+ add: "User mailing list:"
+ }
+ icon: icons/community/contact-us/question-mark.svg
+ actions:
+ [
+ {
+ action: Subscribe,
+ link: "mailto:user-subscribe@beam.apache.org"
+ },
+ {
+ action: Archives,
+ link: "https://lists.apache.org/list.html?user@beam.apache.org",
+ },
+ {
+ action: Read tips below,
+ }
+ ]
+ - purpose: Developer and contributor discussions
+ contact:
+ {
+ email: dev@beam.apache.org,
+ link: "https://lists.apache.org/list.html?dev@beam.apache.org",
+ add: "Developer mailing list:"
+ }
+ icon: icons/community/contact-us/discussion.svg
+ actions:
+ [
+ {
+ action: Subscribe,
+ link: "mailto:dev-subscribe@beam.apache.org"
+ },
+ {
+ action: Archives,
+ link: "https://lists.apache.org/list.html?dev@beam.apache.org",
+ },
+ {
+ action: Read tips below,
+ }
+ ]
+ - purpose: Report bugs, request features, follow known issues
+ contact:
+ {
+ email: Apache Beam Jira Tracker,
+ link: "https://issues.apache.org/jira/browse/BEAM",
+ }
+ icon: icons/community/contact-us/bug.svg
+ actions:
+ [
+ {
+ action: To create an issue you need a Jira account. Open one here.,
+ link: "https://issues.apache.org/jira/secure/Signup!default.jspa",
+ },
+ ]
+ - purpose: Ask and answer community questions
+ contact:
+ {
+ email: "Apache Beam category in Stackoverflow.",
+ link: "https://stackoverflow.com/questions/tagged/apache-beam",
+ }
+ icon: icons/community/contact-us/messages.svg
+ actions: [{ action: "‐" }]
+ - purpose: Contact users and contributors in real time through the ASF slack workspace.
+ contact:
+ {
+ email: The ASF slack workspace,
+ link: "https://s.apache.org/beam-slack-channel",
+ }
+ icon: icons/community/contact-us/knot.svg
+ actions: [{ action: "Join the #beam chanel and browse other available channels. Don’t ask Beam questions in #general, we don’t monitor it." }]
+ - purpose: Firehose of commits, bugs, pull requests, etc.
+ contact:
+ {
+ email: builds@beam.apache.org,
+ link: "https://lists.apache.org/list.html?builds@beam.apache.org",
+ add: Build mailing list,
+ }
+ icon: icons/community/contact-us/gitArrows.svg
+ actions: [{
+ action: Archives,
+ link: "https://lists.apache.org/list.html?commits@beam.apache.org",
+ }]
+ - purpose: 'Firehose of build notifications from Jenkins'
+ contact: {
+ email: commits@beam.apache.org,
+ link: "https://s.apache.org/beam-slack-channel",
+ add: Commits mailing list
+ }
+ icon: icons/community/contact-us/notification.svg
+ actions: [{
+ action: Archives,
+ link: "https://lists.apache.org/list.html?commits@beam.apache.org",
+ }]
diff --git a/website/www/site/data/en/documentation_runners.yaml b/website/www/site/data/en/documentation_runners.yaml
new file mode 100644
index 000000000000..0d5901a45253
--- /dev/null
+++ b/website/www/site/data/en/documentation_runners.yaml
@@ -0,0 +1,34 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: { text: "DirectRunner:", link: /documentation/runners/direct/ }
+ icon: icons/documentation/runners/beam-icon.svg
+ description: Runs locally on your machine – great for developing, testing, and debugging.
+- name: { text: "FlinkRunner:", link: /documentation/runners/flink/ }
+ icon: icons/documentation/runners/flink-icon.svg
+ description: Runs on Apache Flink.
+- name: { text: "SparkRunner:", link: /documentation/runners/spark/ }
+ icon: icons/documentation/runners/spark-icon.svg
+ description: Runs on Apache Spark.
+- name: { text: "DataflowRunner:", link: /documentation/runners/dataflow/ }
+ icon: icons/documentation/runners/dataflow-icon.svg
+ description: Runs on Google Cloud Dataflow, a fully managed service within Google Cloud Platform.
+- name: { text: "SamzaRunner:", link: /documentation/runners/samza/ }
+ icon: icons/documentation/runners/samza-icon.svg
+ description: Runs on Apache Samza.
+- name: { text: "NemoRunner:", link: /documentation/runners/nemo/ }
+ icon: icons/documentation/runners/nemo-icon.svg
+ description: Runs on Apache Nemo.
+- name: { text: "JetRunner:", link: /documentation/runners/jet/ }
+ description: Runs on Hazelcast Jet.
+- name: { text: "Twister2Runner:", link: /documentation/runners/twister2/ }
+ description: Runs on Twister2.
diff --git a/website/www/site/data/en/documentation_sdks.yaml b/website/www/site/data/en/documentation_sdks.yaml
new file mode 100644
index 000000000000..6086ed757be7
--- /dev/null
+++ b/website/www/site/data/en/documentation_sdks.yaml
@@ -0,0 +1,18 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: { text: Java SDK, link: /documentation/sdks/java/ }
+ icon: icons/documentation/sdks/java-icon.svg
+- name: { text: Python SDK, link: /documentation/sdks/python/ }
+ icon: icons/documentation/sdks/python-icon.svg
+- name: { text: Go SDK, link: /documentation/sdks/go/ }
+ icon: icons/documentation/sdks/go-icon.svg
diff --git a/website/www/site/data/en/graphic.yaml b/website/www/site/data/en/graphic.yaml
new file mode 100644
index 000000000000..03468d23dd68
--- /dev/null
+++ b/website/www/site/data/en/graphic.yaml
@@ -0,0 +1,22 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- title: Sources
+ body: Beam reads your data from a diverse set of supported sources, no matter if it’s on-prem or in the cloud.
+ icon: /images/model-sources-icon.svg
+- title: Processing
+ body: Beam executes your business logic for Batch and Streaming use cases.
+ icon: /images/model-processing-icon.svg
+- title: Sinks
+ body: Beam writes the results of your data processing logic to the most popular data destinations in the industry.
+ icon: /images/model-sinks-icon.svg
+
diff --git a/website/www/site/data/en/join_beam.yaml b/website/www/site/data/en/join_beam.yaml
new file mode 100644
index 000000000000..380b4d92b1ff
--- /dev/null
+++ b/website/www/site/data/en/join_beam.yaml
@@ -0,0 +1,29 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- list:
+ - { title: Mailing Lists, description: 'Join the official Apache Beam user and developer mailing lists. Learn how here' }
+ - { title: Slack Channel, description: 'For real-time communication, join us on the #beam channel in the ASF workspace.' }
+ - { title: Twitter account, description: 'Follow @ApacheBeam on Twitter for the most recent news on the project and community!' }
+ - { title: YouTube channel, description: 'Follow our YouTube channel for the latest training and use cases.' }
+ - { title: StackOverflow, description: 'Ask your Beam questions in StackOverflow using the apache-beam tag.' }
+ - { title: Events, description: 'Follow our calendar in the homepage which has the latest events organized by the community.' }
+- list:
+ - { title: Read our Contribution Guide and Policies, description: 'Get familiar with our contribution guide and policies.' }
+ - { title: Create a Jira account, description: 'Create a Jira account and send an email to dev@beam.apache.org to request comitter permissions.' }
+ - { title: Find an area to contribute, description: 'Beam recognizes code and non-code contributions. Check out this Jira board for some inspiration.' }
+ - { title: Contribute, description: 'To contribute code, you need to sign an Individual Contributor License Agreement (ICLA). Learn more here.' }
+ - { title: Stay engaged!, description: Finds ways to stay engaged in the project and community and help us shape it’s strategy and vision. }
+- list:
+ - { title: Learn about Apache Beam Committers, description: 'Read how to become a committer to learn about ways to get nominated for committership.' }
+- list:
+ - { title: Learn about the Project Management Committee (PMC), description: 'The Project Management Committee (PMC) is the highest recognition in Apache projects. If you’re interested in becoming part of it, learn more about it here.' }
diff --git a/website/www/site/data/en/pillars.yaml b/website/www/site/data/en/pillars.yaml
index d2138bdd8b38..f6e0d2383b89 100644
--- a/website/www/site/data/en/pillars.yaml
+++ b/website/www/site/data/en/pillars.yaml
@@ -12,7 +12,13 @@
- title: Unified
body: Use a single programming model for both batch and streaming use cases.
-- title: Portable
- body: Execute pipelines on multiple execution environments.
+ icon: icons/unified-icon.svg
- title: Extensible
body: Write and share new SDKs, IO connectors, and transformation libraries.
+ icon: icons/extensible-icon.svg
+- title: Portable
+ body: Execute pipelines on multiple execution environments.
+ icon: icons/portable-icon.svg
+- title: Open Source
+ body: Use a single programming model for both batch and streaming use cases.
+ icon: icons/open-source-icon.svg
diff --git a/website/www/site/data/en/pillars_social.yaml b/website/www/site/data/en/pillars_social.yaml
new file mode 100644
index 000000000000..2ed3ff6c03a9
--- /dev/null
+++ b/website/www/site/data/en/pillars_social.yaml
@@ -0,0 +1,21 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- name: pillars-github-icon
+ icon: icons/github-icon.svg
+ url: https://github.com/apache/beam
+- name: pillars-youtube-icon
+ icon: icons/youtube-icon.svg
+ url: https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ
+- name: pillars-twitter-icon
+ icon: icons/twitter-icon.svg
+ url: https://twitter.com/apachebeam
\ No newline at end of file
diff --git a/website/www/site/data/en/powered-by.yaml b/website/www/site/data/en/powered-by.yaml
new file mode 100644
index 000000000000..be4df14da1ec
--- /dev/null
+++ b/website/www/site/data/en/powered-by.yaml
@@ -0,0 +1,20 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- title: Commercial projects
+ body: Learn how these companies use Apache Beam to make their work easier.
+ icon: /icons/commercial-icon.svg
+ link: /powered-by/commercial
+- title: Open source projects
+ body: Find other open source projects that use Apache Beam.
+ icon: /icons/open_source-icon.svg
+ link: /powered-by/open-source
diff --git a/website/www/site/data/en/quotes.yaml b/website/www/site/data/en/quotes.yaml
new file mode 100644
index 000000000000..38700e5099c6
--- /dev/null
+++ b/website/www/site/data/en/quotes.yaml
@@ -0,0 +1,21 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- text: eBay, an e-commerce company, uses Apache Beam in their streaming pipelines to integrate with other OSS services such as Apache Kafka and Apache Airflow.
+ icon: icons/quote-icon.svg
+ logoUrl: images/logos/powered-by/eBay.png
+- text: Developed at Spotify and built on top of Apache Beam, Klio is an open source framework for data processing pipelines for audio and other media files.
+ icon: icons/quote-icon.svg
+ logoUrl: images/logos/powered-by/scio.png
+- text: Oriel Research Therapeutics (ORT) is a startup company that utilizes Apache Beam to process over 1 million samples of genomic data to detect Leukemia, Sepsis and other medical conditions.
+ icon: icons/quote-icon.svg
+ logoUrl: images/logos/powered-by/oriel.png
diff --git a/website/www/site/data/en/ways_of_contribution.yaml b/website/www/site/data/en/ways_of_contribution.yaml
new file mode 100644
index 000000000000..ccf870682608
--- /dev/null
+++ b/website/www/site/data/en/ways_of_contribution.yaml
@@ -0,0 +1,28 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- title: Code contributions
+ body: "Ways you can contribute code to the project include:"
+ list:
+ - Building a new feature like a new IO connector or PTransform
+ - Fixing a Bug
+ - Sharing a code example or template with the community
+ - Testing a release candidate
+ icon: icons/contributor/become a committer/code-icon.svg
+- title: Non-code contributions
+ body: "Non-code contributions are ways to add value to the projects outside code, here are some examples of things that you can contribute:"
+ list:
+ - Organize a meetup
+ - Give a talk about Beam at a conference
+ - Update a typo in the documentation
+ - Write a blog post with useful tips about Beam
+ icon: icons/contributor/become a committer/file-icon.svg
diff --git a/website/www/site/i18n/blog/en.yaml b/website/www/site/i18n/blog/en.yaml
index 4eef374dc95c..98492abf9a2a 100644
--- a/website/www/site/i18n/blog/en.yaml
+++ b/website/www/site/i18n/blog/en.yaml
@@ -9,10 +9,12 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-
+
- id: blog-title
- translation: "Apache Beam Blog"
+ translation: "Blog"
- id: blog-subtitle
translation: "This is the blog for the Apache Beam project. This blog contains news and updates for the project."
- id: blog-readmore
translation: "Read more"
+- id: blog-loadmore
+ translation: "Load more"
diff --git a/website/www/site/i18n/capability/en.yaml b/website/www/site/i18n/capability/en.yaml
new file mode 100644
index 000000000000..24d608145865
--- /dev/null
+++ b/website/www/site/i18n/capability/en.yaml
@@ -0,0 +1,16 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: capability-back
+ translation: "back to collapsed details"
+- id: capability-go-big
+ translation: "SEE DETAILS AND FULL VERSION HERE."
diff --git a/website/www/site/i18n/feedback/en.yaml b/website/www/site/i18n/feedback/en.yaml
new file mode 100644
index 000000000000..70b9dbfbc135
--- /dev/null
+++ b/website/www/site/i18n/feedback/en.yaml
@@ -0,0 +1,19 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: feedback-title
+ translation: "Have you found everything you were looking for?"
+- id: feedback-description
+ translation: "Was it all useful and clear? Is there anything that you would like to change? Let us know!"
+- id: feedback-button
+ translation: "SEND FEEDBACK"
+
diff --git a/website/www/site/i18n/footer/en.yaml b/website/www/site/i18n/footer/en.yaml
index 5ebd5554db48..e89f539957c2 100644
--- a/website/www/site/i18n/footer/en.yaml
+++ b/website/www/site/i18n/footer/en.yaml
@@ -9,7 +9,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-
+
- id: footer-start
translation: "Start"
- id: footer-overview
diff --git a/website/www/site/i18n/home/calendar/en.yaml b/website/www/site/i18n/home/calendar/en.yaml
new file mode 100644
index 000000000000..cb1e5f95d2ec
--- /dev/null
+++ b/website/www/site/i18n/home/calendar/en.yaml
@@ -0,0 +1,17 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: home-calendar-title
+ translation: "Stay up to date with Beam"
+- id: home-calendar-card-events-title
+ translation: "Upcoming events"
+
diff --git a/website/www/site/i18n/home/en.yaml b/website/www/site/i18n/home/en.yaml
index cbb3f23a78c3..2d9157be2c1a 100644
--- a/website/www/site/i18n/home/en.yaml
+++ b/website/www/site/i18n/home/en.yaml
@@ -9,11 +9,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-
-- id: home-hero-title
- translation: "Apache Beam: An advanced unified programming model"
-- id: home-hero-subtitle
- translation: "Implement batch and streaming data processing jobs that run on any execution engine."
+
- id: home-learn-more
translation: "Learn more"
- id: home-try-beam
@@ -28,10 +24,6 @@
translation: "Go Quickstart"
- id: home-hero-blog-title
translation: "The latest from the blog"
-- id: home-pillars-title
- translation: "All about Apache Beam"
-- id: home-logos-title
- translation: "Works with"
- id: home-cards-title
translation: "Testimonials"
- id: home-cards-body
@@ -41,4 +33,18 @@
- id: home-section
translation: "section"
- id: home-ctas-title
- translation: "Get started"
\ No newline at end of file
+ translation: "Get started"
+- id: home-model-title1
+ translation: "How Does it work?"
+- id: home-model-title2
+ translation: "Write it in your language of choice - run it anywhere"
+- id: home-graphic-more
+ translation: "& MORE"
+- id: home-graphic-runner-title
+ translation: "Choose your runner"
+- id: home-graphic-runner-body
+ translation: "A Beam pipeline can execute in the most popular distributed data processing systems such as Spark, Flink or Samza."
+- id: home-graphic-language-title
+ translation: "Choose your language"
+- id: home-graphic-language-body
+ translation: "You can write Apache Beam pipelines in your programming language of choice: Java, Python and Go. Learn More."
diff --git a/website/www/site/i18n/home/hero/en.yaml b/website/www/site/i18n/home/hero/en.yaml
new file mode 100644
index 000000000000..85f4728d86be
--- /dev/null
+++ b/website/www/site/i18n/home/hero/en.yaml
@@ -0,0 +1,20 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: home-hero-title
+ translation: Introducing Apache Beam
+- id: home-hero-heading
+ translation: An advanced unified programming model
+- id: home-hero-subheading
+ translation: Implement batch and streaming data processing jobs that run on any execution engine.
+- id: home-hero-button
+ translation: Install Beam
diff --git a/website/www/site/i18n/home/logos/en.yaml b/website/www/site/i18n/home/logos/en.yaml
new file mode 100644
index 000000000000..7ed599a42ab5
--- /dev/null
+++ b/website/www/site/i18n/home/logos/en.yaml
@@ -0,0 +1,14 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: home-logos-title
+ translation: "Supported runners"
\ No newline at end of file
diff --git a/website/www/site/i18n/home/pillars/en.yaml b/website/www/site/i18n/home/pillars/en.yaml
new file mode 100644
index 000000000000..ab8a6d1ff002
--- /dev/null
+++ b/website/www/site/i18n/home/pillars/en.yaml
@@ -0,0 +1,16 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: home-pillars-title
+ translation: "All about Apache Beam"
+- id: home-pillars-social-text
+ translation: "Check out our social media to learn more about the community!"
\ No newline at end of file
diff --git a/website/www/site/i18n/home/quotes/en.yaml b/website/www/site/i18n/home/quotes/en.yaml
new file mode 100644
index 000000000000..95c8e85b6453
--- /dev/null
+++ b/website/www/site/i18n/home/quotes/en.yaml
@@ -0,0 +1,14 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: home-quotes-title
+ translation: "They tried it out"
\ No newline at end of file
diff --git a/website/www/site/i18n/navbar/en.yaml b/website/www/site/i18n/navbar/en.yaml
index fd22c3a73821..74cc78caf1f3 100644
--- a/website/www/site/i18n/navbar/en.yaml
+++ b/website/www/site/i18n/navbar/en.yaml
@@ -9,17 +9,19 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-
+
- id: nav-toggle-navigation
translation: "Toggle navigation"
- id: nav-get-started
translation: "Get Started"
- id: nav-documentation
translation: "Documentation"
+- id: nav-documentation-general
+ translation: "General"
- id: nav-languages
translation: "Languages"
- id: nav-runners
- translation: "RUNNERS"
+ translation: "Runners"
- id: nav-roadmap
translation: "Roadmap"
- id: nav-contribute
@@ -40,3 +42,5 @@
translation: "Sponsorship"
- id: nav-code-of-conduct
translation: "Code of Conduct"
+- id: nav-powered-by
+ translation: "Powered by"
diff --git a/website/www/site/i18n/powered-by/en.yaml b/website/www/site/i18n/powered-by/en.yaml
new file mode 100644
index 000000000000..8cd24f3a7f81
--- /dev/null
+++ b/website/www/site/i18n/powered-by/en.yaml
@@ -0,0 +1,16 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+- id: powered-read
+ translation: "READ MORE"
+- id: powered-check
+ translation: "CHECK OUT"
diff --git a/website/www/site/layouts/_default/baseof.html b/website/www/site/layouts/_default/baseof.html
index f8fbfb6c04b1..2a75558f5d85 100644
--- a/website/www/site/layouts/_default/baseof.html
+++ b/website/www/site/layouts/_default/baseof.html
@@ -17,14 +17,19 @@
{{ partial "header.html" . }}
-
+
{{ block "hero-section" . }}{{ end }}
+ {{ block "ctas-section" . }}{{ end }}
{{ block "pillars-section" . }}{{ end }}
{{ block "graphic-section" . }}{{ end }}
+ {{ block "calendar-section" . }}{{ end }}
+ {{ block "quotes-section" . }}{{ end }}
+ {{ block "quotes-mobile-section" . }}{{ end }}
{{ block "logos-section" . }}{{ end }}
- {{ block "cards-section" . }}{{ end }}
- {{ block "ctas-section" . }}{{ end }}
- {{ partial "footer.html" . }}
+
+ {{ partial "footer.html" . }}
+
+ {{ partial "hooks/body-end.html"}}