changing segment granularity on realtime nodes causes ingestion to stop

[pdeva]

discussed and described in detail on
https://groups.google.com/forum/#!msg/druid-development/P3xjuX9xp30/HWGFubDjf0sJ

The solution given, to delete the directory of the realtime node is incorrect since it will result in loss of data

[fjy]

Actually the workaround is to start another real-time node with the changed granularity. The solution is to re-evaluate how sinks are selected.

[pdeva]

so at some point s3 will end up with 2 segments for a given time period, one from the hourly node and other from the daily node. how will it decide which one to pick?

[fjy]

MVCC

On Tue, Nov 18, 2014 at 8:57 PM, pdeva notifications@github.com wrote:

so at some point s3 will end up with 2 segments for a given time period,
one from the hourly node and other from the daily node. how will it decide
which one to pick?

—
Reply to this email directly or view it on GitHub
https://github.com/metamx/druid/issues/888#issuecomment-63592343.

[pdeva]

is it possible to have a different granularity that, day, hour, etc?
can i say '2hour' for example?

[pdeva]

I tried spinning up a 3rd realtime node with 'day' as granularity and it died with an exception:
https://dl.dropboxusercontent.com/u/152963/druid/realtime3.txt

(the log is bottom up btw, so the last event is on top).

all the other nodes were working btw..

[pdeva]

ok i have tried installing multiple times and each time the realtime node fails with that. all other nodes are able to connect to mysql

[drcrallen]

@pdeva : there are two things odd there. 1) the log4j stuff is freaking out, which makes me wonder if it's picking up settings from an unexpected location. 2) The sql driver is failing to make a connection. Are you able to connect through a "normal" mysql client from that node?

[pdeva]

1. Don't worry about that. That's our DripStat agent. We should have
   suppressed that exception in fact in the agent itself.
2. Yup all other nodes can connect just fine and they have the exact same
   setting.

On Wednesday, November 19, 2014, Charles Allen notifications@github.com
wrote:

@pdeva https://github.com/pdeva : there are two things odd there. 1)
the log4j stuff is freaking out, which makes me wonder if it's picking up
settings from an unexpected location. 2) The sql driver is failing to make
a connection. Are you able to connect through a "normal" mysql client from
that node?

—
Reply to this email directly or view it on GitHub
https://github.com/metamx/druid/issues/888#issuecomment-63757236.

Prashant

[pdeva]

ok so it turns out the sql connection issue was due to rds' security group settings. i was able to make it work.

[pdeva]

so @fjy the suggested workaround doesnt work.

I have 2 existing realtime nodes with hourly segments.
based on your suggestion, i added a 3rd one in parallel with day level segments.

if i switch off the first 2 (hourly) nodes and just leave the 3rd one running, none of my queries on realtime nodes work.

[pdeva]

also when i restarted the realtime nodes a full hour of data was lost. even though the 2 nodes were only down for a few minutes and the 3rd node was always running.

Screenshot:
http://screencast.com/t/kwSYLwjL

[fjy]

Hi Prashant, it looks like the node you spun up isn't even ingesting
events. Can you double check the configuration? What message bus are you
using?

On Thu, Nov 20, 2014 at 1:14 PM, pdeva notifications@github.com wrote:

also when i restarted the realtime nodes a full hour of data was lost.
even though the 2 nodes were only down for a few minutes and the 3rd node
was always running.

Screenshot:
http://screencast.com/t/kwSYLwjL

—
Reply to this email directly or view it on GitHub
https://github.com/metamx/druid/issues/888#issuecomment-63880844.

[fjy]

Hi Prashant, looking over some Druid code, if you have two real-time nodes
ingesting data with different segment granularities completely overlapping,
queries will fail. You need to shut down one of the real-time nodes before
things resolve themselves. This was an oversight on my part in suggesting
the 2 nodes in parallel. If you want to change segment granularity, it
cannot be done without downtime.

On Thu, Nov 20, 2014 at 1:20 PM, Fangjin Yang fangjin@metamarkets.com
wrote:

Hi Prashant, it looks like the node you spun up isn't even ingesting
events. Can you double check the configuration? What message bus are you
using?

On Thu, Nov 20, 2014 at 1:14 PM, pdeva notifications@github.com wrote:

also when i restarted the realtime nodes a full hour of data was lost.
even though the 2 nodes were only down for a few minutes and the 3rd node
was always running.

Screenshot:
http://screencast.com/t/kwSYLwjL

—
Reply to this email directly or view it on GitHub
https://github.com/metamx/druid/issues/888#issuecomment-63880844.

[pdeva]

here is the code for the last 60 mins of the 3rd node (daily segment)
https://dl.dropboxusercontent.com/u/152963/druid/realtime3-last60mins.txt

we use ansible automated deployment for the nodes so config is the same for all nodes. just the kafaka group id is separate for each node

[pdeva]

If you want to change segment granularity, it
cannot be done without downtime.

How much downtime? just one hour to do the switch or a full day?