start/stop of individual clustered realtime nodes causes data loss

[pdeva]

I have 4 realtime nodes. Each on a separate kafka consumer group.
I am using druid 0.6.160

2 of those nodes (1 &2) were running from before.
I started 3 last night.
4 was started today at 14:06 PST (timezone of machines are all in utc).

After starting 4, I shutdown nodes 1 & 2. (seeing cpu usage it seems node 2 was serving all requests).

Note that at all times, atleast 2 realtime nodes were active, so there should be no data loss whatsoever.

Now when I query for last 30 or 60 mins of data, sometimes I see the whole data, but sometimes I see missing gap between 14:00 and 14:08.

Here is a video showing the issue:
http://screencast.com/t/cs0osqHJ2

Is this because the broker is sometimes querying node 4, which doesnt have all the data cause it was started at 14:06 (and then took a couple minutes to initialize)?

Either way, this looks like a bug.

Portions of my realtime spec file:

"config": {
      "maxRowsInMemory": 50000,
      "intermediatePersistPeriod": "PT10m"
    },
    "firehose": {
      "type": "kafka-0.8",
      "consumerProps": {
        "zookeeper.connect": "xx:2181,xx:2181,xx2181",
        "zookeeper.connection.timeout.ms": "15000",
        "zookeeper.session.timeout.ms": "15000",
        "zookeeper.synctime.ms": "5000",
        "group.id": "druid-dripstat",
        "fetch.size": "1048586",
        "auto.offset.reset": "largest",
        "auto.commit.enable": "false"
      },
      "feed": "dripstat",
      "parser": {
        "timestampSpec": {
          "column": "timestamp"
        },
        "data": {
          "format": "json"
        }
      }
    },
    "plumber": {
      "type": "realtime",
      "windowPeriod": "PT10m",
      "segmentGranularity": "hour",
      "basePersistDirectory": "/data/realtime/basePersist"
    }
  }

[pdeva]

update: now that the segment has gone to historical node, we now see a permanent data loss from 14:00 to 14:08

[fjy]

Hi @pdeva ,

As you may already be aware, the Kafka firehose is a Kafka high level consumer, and within the firehose are a set of configuration options for Kafka. One of the configurations is auto.offset.reset. According to the Kafka docs for this config:

What to do when there is no initial offset in Zookeeper or if an offset is out of range:

• smallest : automatically reset the offset to the smallest offset
• largest : automatically reset the offset to the largest offset
• anything else: throw exception to the consumer. If this is set to largest, the consumer may lose some messages when the number of partitions, for the topics it subscribes to, changes on the broker. To prevent data loss during partition addition, set auto.offset.reset to smallest

Given that Kafka has no real notion of timestamps, Druid has no way of reading messages from exactly at the beginning of an hour. The firehose needs to start at some offset in Kafka, and in this case, it started reading messages from an offset corresponding to events that were ingested at around the time the node started up. I agree we should do a better job of documenting this possibility.

[pdeva]

even in the case where node 4 couldnt read data from the begining of the hour, node 3 clearly had the entire data for the full hour.
This is evidenced in the video i linked where refreshing sometimes shows the full data.

in that case, why didnt druid pick the node that has all the data to persist to deep storage?

[pdeva]

this is a reproducible issue.
i can add a realtime node in the middle of the hour.
the broker is then free to pass it any query it for the hour, even though it doesnt have the data for the full hour.

this all happens when there are other replica realtime nodes containing the full hour's data

[fjy]

@pdeva, yes, it is a known issue

FWIW, we don't run realtime nodes in production because of the difficulty of migrating them and making schema changes. We run all of our ingestion through the indexing service.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.