This repository was archived by the owner on Nov 11, 2022. It is now read-only.

[BEAM-52] Kafka custom source #121

Closed

rangadi wants to merge 74 commits intoGoogleCloudPlatform:masterfrom

Commits on Feb 16, 2016

pom.xml
Raghu Angadi
committed

Commits on Feb 18, 2016

most of the implemetation
Raghu Angadi
committed
remove java 1.7 restriction in pom
Raghu Angadi
committed

Commits on Feb 19, 2016

Revert "remove java 1.7 restriction in pom"
Raghu Angadi
committed
simplify consumedOffset management with PartitionState
Raghu Angadi
committed
minor
Raghu Angadi
committed
minor
Raghu Angadi
committed
TopHashtagsExample (just does global count for now)
Raghu Angadi
committed
set maxNumRecords to be able to run with DirectRunner
Raghu Angadi
committed
add log4j dependency (temp)
Raghu Angadi
committed
some fixes and some temp code
Raghu Angadi
committed

Commits on Feb 20, 2016

Add KafkaRecord (a Serializable version of kafka.ConsumerRecord)
Raghu Angadi
committed

Commits on Feb 23, 2016

Builder for ValueSource
Raghu Angadi
committed
make identityFn a static val
Raghu Angadi
committed
remove full kafka depenency (was used for old consumer)
Raghu Angadi
committed

Commits on Feb 24, 2016

serialization fixes and update example to track top hashtags
Raghu Angadi
committed
serializer fix
Raghu Angadi
committed
offset fix
Raghu Angadi
committed

Commits on Feb 25, 2016

testing with timestampFn. getWatermark() odd behaviour
Raghu Angadi
committed
more getWatermark() debugging
Raghu Angadi
committed
minor
Raghu Angadi
committed
minor
Raghu Angadi
committed
KafkaRecord
Raghu Angadi
committed

Commits on Feb 26, 2016

big update to app
Raghu Angadi
committed
minor
Raghu Angadi
committed
use coders rather than functions to encode kafka bytes to user types
Raghu Angadi
committed
stay with java 1.7 (both for source and target)
Raghu Angadi
committed
minor
Raghu Angadi
committed

Commits on Feb 27, 2016

tweak timeout for cosumer.poll (longer timeout initially). remove debug logging
Raghu Angadi
committed
tried consumer metadata fetch rather than longer timeout for first fetch. It didn't work. some more logging updates
Raghu Angadi
committed
minor
Raghu Angadi
committed
minor
Raghu Angadi
committed

Commits on Feb 29, 2016

move examples to its own directory. mvn dependencies don't work properly yet.
Raghu Angadi
committed

Commits on Mar 1, 2016

meven changes for examples directory
Raghu Angadi
committed
javadoc update
Raghu Angadi
committed
watermark should be Instant(Long.MIN_VALUE) when the reader hasn't read any records.
Raghu Angadi
committed

Commits on Mar 2, 2016

Factory method for creating consumer.
Raghu Angadi
committed
unit tests
Raghu Angadi
committed

Commits on Mar 8, 2016

Merge remote-tracking branch 'upstream/master' into kafka
Raghu Angadi
committed
minor
Raghu Angadi
committed
fix serialization issue
Raghu Angadi
committed
minor
Raghu Angadi
committed
KafkaSource -> KafkaIO refactor
Raghu Angadi
committed
unit tests work
Raghu Angadi
committed
add manual partitions test
Raghu Angadi
committed
update example app
Raghu Angadi
committed
rename Reader to Read. more javadoc
Raghu Angadi
committed
more javadoc
Raghu Angadi
committed
more javadoc
Raghu Angadi
committed
fix javadoc and checkstyle errors
Raghu Angadi
committed
use KV in KafkaRecord
Raghu Angadi
committed
minor
Raghu Angadi
committed
remove uses of Lists.newArrayList()
Raghu Angadi
committed

Commits on Mar 10, 2016

minor:formatting fix
Raghu Angadi
committed

Commits on Mar 16, 2016

update comment for getSplitBacklogBytes
Raghu Angadi
committed
Merge remote-tracking branch 'upstream/master' into kafka
Raghu Angadi
committed

Commits on Mar 17, 2016

increase send buffer size for kafka consumer.
Raghu Angadi
committed
poll kafka consumer in a seperate thread to improve throughput. see comment for consumerPollThread
Raghu Angadi
committed
minor
Raghu Angadi
committed
minor
Raghu Angadi
committed

Commits on Mar 18, 2016

micro -> milli
Raghu Angadi
committed
minor clean up
Raghu Angadi
committed
review comments
Raghu Angadi
committed
fetch latest offsets in a thread. some review somments
Raghu Angadi
committed
offset consumer : comment on hack, disable auto_offset_commit
Raghu Angadi
committed
minor
Raghu Angadi
committed
minutes -> seconds
Raghu Angadi
committed
fix warning in tests. a few more minor diffs
Raghu Angadi
committed

Commits on Mar 22, 2016

review comments
Raghu Angadi
committed

Commits on Mar 31, 2016

Update KafkaIO interface based on sugestions from Dan.
Raghu Angadi
committed

Commits on Apr 5, 2016

KafkaIO JavaDoc
Raghu Angadi
committed
review comments
Raghu Angadi
committed
review comments
Raghu Angadi
committed
revert a small fix
Raghu Angadi
committed