Skip to content

Supporting firehoseV2 in RealtimeIndexTask#2649

Closed
sirpkt wants to merge 2 commits intoapache:masterfrom
sirpkt:firehoseV2_realtime_index
Closed

Supporting firehoseV2 in RealtimeIndexTask#2649
sirpkt wants to merge 2 commits intoapache:masterfrom
sirpkt:firehoseV2_realtime_index

Conversation

@sirpkt
Copy link
Copy Markdown
Contributor

@sirpkt sirpkt commented Mar 14, 2016

Only realtimeManager, used in realtime node, supports firehoseV2 now.
So I modified RealtimeIndexTask to enable firehose V2 in indexing service.

@fjy
Copy link
Copy Markdown
Contributor

fjy commented Mar 14, 2016

@sirpkt Are you using Kafka to ingest data? If so, I strongly suggest you look at the Kafka Index Task from #2220. We have a few more PRs coming after that is merged which will provide exactly-once consumption of data from Kafka.

@fjy
Copy link
Copy Markdown
Contributor

fjy commented Mar 14, 2016

@gianm do u mind taking a look at this?

@sirpkt
Copy link
Copy Markdown
Contributor Author

sirpkt commented Mar 15, 2016

@fjy Yes, I use Kafka and felt some limitation on Kafka firehose. But #2220 looks great to me! Thank you for the information. I'll check that more.

@fjy
Copy link
Copy Markdown
Contributor

fjy commented Jun 28, 2016

@sirpkt with the new kafka indexing task, which supports exactly once ingestion from Kafka, is this PR still needed?

@sirpkt
Copy link
Copy Markdown
Contributor Author

sirpkt commented Jun 29, 2016

@fjy Oh, I forgot to close this issue.
Thanks.

@sirpkt sirpkt closed this Jun 29, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants