Skip to content
This repository was archived by the owner on Nov 11, 2022. It is now read-only.
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
f1e2580
pom.xml
Feb 16, 2016
7454816
most of the implemetation
Feb 18, 2016
af9b887
remove java 1.7 restriction in pom
Feb 18, 2016
88767fd
Revert "remove java 1.7 restriction in pom"
Feb 19, 2016
464ad1e
simplify consumedOffset management with PartitionState
Feb 19, 2016
ed9f92a
minor
Feb 19, 2016
e42533b
minor
Feb 19, 2016
d419594
TopHashtagsExample (just does global count for now)
Feb 19, 2016
74a27cc
set maxNumRecords to be able to run with DirectRunner
Feb 19, 2016
773ba2b
add log4j dependency (temp)
Feb 19, 2016
5990e93
some fixes and some temp code
Feb 19, 2016
7d33c59
Add KafkaRecord (a Serializable version of kafka.ConsumerRecord)
Feb 20, 2016
3a1dc97
Builder for ValueSource
Feb 23, 2016
8580ed2
make identityFn a static val
Feb 23, 2016
a00625d
remove full kafka depenency (was used for old consumer)
Feb 23, 2016
7e8393d
serialization fixes and update example to track top hashtags
Feb 24, 2016
49fd42f
serializer fix
Feb 24, 2016
a86f11e
offset fix
Feb 24, 2016
f8fb50d
testing with timestampFn. getWatermark() odd behaviour
Feb 25, 2016
2e494ec
more getWatermark() debugging
Feb 25, 2016
ef2620d
minor
Feb 25, 2016
5e0a485
minor
Feb 25, 2016
a0ee9e4
KafkaRecord
Feb 25, 2016
25b761c
big update to app
Feb 26, 2016
cab32fc
minor
Feb 26, 2016
bf41bec
use coders rather than functions to encode kafka bytes to user types
Feb 26, 2016
c02f002
stay with java 1.7 (both for source and target)
Feb 26, 2016
88df84b
minor
Feb 26, 2016
c397dfa
tweak timeout for cosumer.poll (longer timeout initially). remove deb…
Feb 27, 2016
e953c84
tried consumer metadata fetch rather than longer timeout for first fe…
Feb 27, 2016
1872275
minor
Feb 27, 2016
ebc7d57
minor
Feb 27, 2016
7f7ec7b
move examples to its own directory. mvn dependencies don't work prope…
Feb 29, 2016
c5cf5ee
meven changes for examples directory
Mar 1, 2016
16182ee
javadoc update
Mar 1, 2016
025f047
watermark should be Instant(Long.MIN_VALUE) when the reader hasn't re…
Mar 1, 2016
cb20dc9
Factory method for creating consumer.
Mar 2, 2016
29e8c0a
unit tests
Mar 2, 2016
f544556
Merge remote-tracking branch 'upstream/master' into kafka
Mar 8, 2016
a14a9ea
minor
Mar 2, 2016
73691b7
fix serialization issue
Mar 3, 2016
f7746b8
minor
Mar 4, 2016
c9d4a69
KafkaSource -> KafkaIO refactor
Mar 7, 2016
ec2e74e
unit tests work
Mar 7, 2016
79298c7
add manual partitions test
Mar 7, 2016
56411da
update example app
Mar 7, 2016
900d339
rename Reader to Read. more javadoc
Mar 8, 2016
0ea9e7a
more javadoc
Mar 8, 2016
b44a201
more javadoc
Mar 8, 2016
8db3458
fix javadoc and checkstyle errors
Mar 8, 2016
686f608
use KV in KafkaRecord
Mar 8, 2016
dc2ac1a
minor
Mar 8, 2016
b7a9c91
remove uses of Lists.newArrayList()
Mar 8, 2016
fff4a76
minor:formatting fix
Mar 10, 2016
8aea5fa
update comment for getSplitBacklogBytes
Mar 16, 2016
f18e175
Merge remote-tracking branch 'upstream/master' into kafka
Mar 16, 2016
4f93270
increase send buffer size for kafka consumer.
Mar 17, 2016
432ff26
poll kafka consumer in a seperate thread to improve throughput. see c…
Mar 17, 2016
d408d0b
minor
Mar 17, 2016
f0957eb
minor
Mar 17, 2016
b531283
micro -> milli
Mar 18, 2016
c1f069c
minor clean up
Mar 18, 2016
0db0a37
review comments
Mar 18, 2016
c2b4c3e
fetch latest offsets in a thread. some review somments
Mar 18, 2016
e1b0d0d
offset consumer : comment on hack, disable auto_offset_commit
Mar 18, 2016
25609ca
minor
Mar 18, 2016
ecc6376
minutes -> seconds
Mar 18, 2016
43973e7
fix warning in tests. a few more minor diffs
Mar 18, 2016
32e7362
review comments
Mar 22, 2016
70a0093
Update KafkaIO interface based on sugestions from Dan.
Mar 31, 2016
3d56073
KafkaIO JavaDoc
Apr 5, 2016
5e3b54e
review comments
Apr 5, 2016
4e432bd
review comments
Apr 5, 2016
95912d1
revert a small fix
Apr 5, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 189 additions & 0 deletions contrib/examples/kafka/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ Copyright (C) 2015 Google Inc.
~
~ Licensed under the Apache License, Version 2.0 (the "License"); you may not
~ use this file except in compliance with the License. You may obtain a copy of
~ the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
~ WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
~ License for the specific language governing permissions and limitations under
~ the License.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-contrib-kafka-examples</artifactId>
<name>Google Cloud Dataflow Kafka Examples</name>
<description>Examples apps using Kafka Source in Google Cloud Dataflow</description>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

<licenses>
<license>
<name>Apache License, Version 2.0</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<build>
<plugins>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<version>2.12</version>
<dependencies>
<dependency>
<groupId>com.puppycrawl.tools</groupId>
<artifactId>checkstyle</artifactId>
<version>6.6</version>
</dependency>
</dependencies>
<configuration>
<configLocation>../../../checkstyle.xml</configLocation>
<consoleOutput>true</consoleOutput>
<failOnViolation>true</failOnViolation>
<includeTestSourceDirectory>true</includeTestSourceDirectory>
</configuration>
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>

<!-- Source plugin for generating source and test-source JARs. -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.4</version>
<executions>
<execution>
<id>attach-sources</id>
<phase>compile</phase>
<goals>
<goal>jar</goal>
</goals>
</execution>
<execution>
<id>attach-test-sources</id>
<phase>test-compile</phase>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.10.3</version>
<configuration>
<windowtitle>Google Cloud Dataflow Kafka Contrib</windowtitle>
<doctitle>Google Cloud Dataflow Kafka Contrib</doctitle>

<subpackages>com.google.cloud.dataflow.contrib.kafka</subpackages>
<use>false</use>
<bottom><![CDATA[<br>]]></bottom>

<offlineLinks>
<offlineLink>
<url>https://cloud.google.com/dataflow/java-sdk/JavaDoc/</url>
<location>${basedir}/../../javadoc/dataflow-sdk-docs</location>
</offlineLink>
<offlineLink>
<url>http://docs.guava-libraries.googlecode.com/git-history/release18/javadoc/</url>
<location>${basedir}/../../javadoc/guava-docs</location>
</offlineLink>
</offlineLinks>
</configuration>
<executions>
<execution>
<goals>
<goal>jar</goal>
</goals>
<phase>package</phase>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>${project.artifactId}-bundled-${project.version}</finalName>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>

</plugins>
</build>

<dependencies>
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-contrib-kafka</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.7.7</version>
<scope>runtime</scope>
</dependency>
</dependencies>
</project>
Loading