Web console, adding Apache Kafka and AWS Kinesis to the data loader#7643
Web console, adding Apache Kafka and AWS Kinesis to the data loader#7643clintropolis merged 7 commits intoapache:masterfrom
Conversation
|
If you want to have a look at this in action: https://youtu.be/tAEp5BXVHYE |
clintropolis
left a comment
clintropolis
left a comment
There was a problem hiding this comment.
overall lgtm 👍
To test I pulled branch the and merged in #7566 and ran through loader with local running kafka broker, started a supervisor, got data* 🤘
One thing that I'm not certain is intended or not, useEarliestOffset: true is set by default on the sampler, but not in the final spec unless you hit the option. So out of the box, for a new supervisor, the supervisor spec you post will start from the end offset of the stream instead of the beginning like the data you sampled to create your spec with. I'm not sure if this is a big deal, so approving anyway, but maybe it would be nice to detect if the supervisor already exists or not, and if not start from the beginning?
|
That is a good point about |
…pache#7643) * adding kafka and kinesis to the data loader * feature detect * copy fixes * wording fixes * added missing spec type * increase timeout * Call it Google Cloud Storage
…pache#7643) * adding kafka and kinesis to the data loader * feature detect * copy fixes * wording fixes * added missing spec type * increase timeout * Call it Google Cloud Storage
3 participants
This PR is part of #7502 and the UI counterpart for (and depends on) #7566
Adding the flow for the streaming datasources: