Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
<!--
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these comments show up in the github pull request description field?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it's hidden in github (markdown).

Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

Expand Down
17 changes: 17 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

language: java

sudo: false
Expand Down
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Apache Beam

[Apache Beam](http://beam.incubator.apache.org) is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends like [Apache Spark](http://spark.apache.org/), [Apache Flink](http://flink.apache.org), and [Google Cloud Dataflow](http://cloud.google.com/dataflow).
Expand Down
19 changes: 19 additions & 0 deletions examples/java/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Example Pipelines

The examples included in this module serve to demonstrate the basic
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# "Complete" Examples

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# "Cookbook" Examples

Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# 'Gaming' examples

Expand Down
42 changes: 42 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -219,8 +219,49 @@
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<version>0.11</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
<configuration>
<reportFile>${project.build.directory}/${project.build.finalName}.rat</reportFile>
<excludeSubProjects>false</excludeSubProjects>
<useDefaultExcludes>true</useDefaultExcludes>
<excludes>
<!-- Keep exclude sync with .gitignore -->
<exclude>**/target/**/*</exclude>
<exclude>**/dependency-reduced-pom.xml</exclude>
<exclude>**/*.iml</exclude>
<exclude>**/package-list</exclude>
<exclude>**/user.avsc</exclude>
<exclude>**/test/resources/**/*.txt</exclude>
<exclude>**/test/**/.placeholder</exclude>
<exclude>.repository/**/*</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</pluginManagement>

<plugins>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty line between pluginManagement and plugins sections?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
</plugin>
</plugins>
</build>
</profile>

Expand All @@ -233,6 +274,7 @@
<dataflow.javadoc_opts>-Xdoclint:-missing</dataflow.javadoc_opts>
</properties>
</profile>

</profiles>

<dependencyManagement>
Expand Down
19 changes: 19 additions & 0 deletions runners/flink/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

Flink Beam Runner (Flink-Runner)
-------------------------------

Expand Down
19 changes: 19 additions & 0 deletions runners/spark/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

Spark Beam Runner (Spark-Runner)
================================

Expand Down
15 changes: 0 additions & 15 deletions sdks/java/build-tools/src/main/resources/beam/checkstyle.xml
Original file line number Diff line number Diff line change
Expand Up @@ -25,21 +25,6 @@ page at http://checkstyle.sourceforge.net/config.html -->

<module name="Checker">

<!-- Requires the Apache copyright header in each file.
Code intended to be open-sourced may have a multi-line copyright
notice, so that this required text appears on the second line:
<pre>
/*
* Licensed to the Apache Software Foundation (ASF) under one
* ...
</pre>
See http://checkstyle.sf.net/config_header.html -->
<module name="Header">
<property name="headerFile" value="${checkstyle.header.file}"/>
<property name="ignoreLines" value="2"/>
<property name="fileExtensions" value="java, scala"/>
</module>

<module name="FileTabCharacter">
<!-- Checks that there are no tab characters in the file. -->
</module>
Expand Down
17 changes: 0 additions & 17 deletions sdks/java/build-tools/src/main/resources/beam/header-file.txt

This file was deleted.

23 changes: 22 additions & 1 deletion sdks/java/core/src/main/proto/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,29 @@
## Protocol Buffers in Google Cloud Dataflow
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

This directory contains the Protocol Buffer messages used in Google Cloud
Dataflow.

## Protocol Buffers in Apache Beam
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be Google Dataflow specific, based on the below maven artifact reference.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, does it mean that it should be removed from the Apache repo ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like the only thing here (proto2_coder_test_messages.proto) is for testing. If that's the case, I would suggest updating the verbage of this README with the test details and removing dataflow-specific bits. At any rate, it might be easier to save for a separate PR as to not hold up this one.

Adding a few people that might know more: @dhalperi @kennknowles

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have a few options forward here, but yes I think that this can wait for another PR.

Right now we have not yet re-deployed the proto jar under org.apache.beam

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jbonofre : this is needed for testing ProtoCoder, which is currently part of the SDK core


This directory contains the Protocol Buffer messages used in Apache Beam.

They aren't, however, used during the Maven build process, and are included here
for completeness only. Instead, the following artifact on Maven Central contains
the binary version of the generated code from these Protocol Buffers:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,21 @@
# SDK source version.
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# SDK source version

version=${pom.version}

build.date=${timestamp}
Expand Down
Loading