Broker double-counting HA rt-segments

[sixtus]

Hi,

I got a mixed setup of RTs with linear sharding. Not all shards are created equal, i.e. some I keep double, e.g.

rt1:
1
100
rt2:
2
100

In this case, brokers will ask both rts for segment 100 and thus double count. Once the segment hits historical nodes, all is good.

[fjy]

Hi sixtus, let me try and understand the issue a bit more. So you have two rt nodes, one with linear shard spec 1 and the other with linear shard spec 2. Are these same linear shards replicated across multiple nodes? It seems like you are saying the are. So the problem you are having is that the broker seems to be adding the results for linear shard spec 1 and the copy? If this is the case, perhaps we should chat about the numbered shard spec, which is what we use in production. If this is not the case, perhaps you can help me understand the problem a bit more.

[sixtus]

ok, so I got two kafka topics:

"big" - which is just massive (95% of all events)
"important" - which contains the really important events (the important 5%)

they are supposed to go into the same data source. So I use linear sharding to have a setup like this:
feed: "big" / partition:1 / kafka-group: big-read
feed: "big" / partition:2 / kafka-group: big-read
...
feed: "big" / partition:7 / kafka-group: big-read
feed: "important" / partition: 100 / kafka-group: important-ha-1
feed: "important" / partition: 100 / kafka-group: important-ha-2

which makes perfect sense to me. I really need that many rts for "big", and I really don't want "important" to ever go down.

However, the broker gets confused and includes both rts serving partition 100. Once the segments reach historical nodes, all is good.

[sixtus]

The problem with numbered shard is that they all must be available or the segment is not up, right?

That's an issue, once a feed gets bigger than a single cpu can take.

[sixtus]

Which class does the request fan out, the I see into a PR to get this fixed. Tnx

[sixtus]

@cheddar says: RealtimeManager is the culprit, it builds a Map<String, FireChief> where String is the data source name that should be data source + shard Then, on query, it needs to look at what shards are available and what was asked for in the query and route accordingly

the lookup is then
https://github.com/metamx/druid/blob/master/server/src/main/java/io/druid/segment/realtime/RealtimeManager.java#L114
https://github.com/metamx/druid/blob/master/server/src/main/java/io/druid/segment/realtime/RealtimeManager.java#L122

insertion is
https://github.com/metamx/druid/blob/master/server/src/main/java/io/druid/segment/realtime/RealtimeManager.java#L85

[vogievetsky]

I believe this is no longer an issue with the Kafka indexing service