Ability to skip Incremental Index during query using query context

[nishantmonu51]

• add adds the ability to skip incremental index when querying results from realtime nodes.
• default behaviour is to include incrementalIndex in queries.

[fjy]

can you explain the logic behind this PR?

[drcrallen]

I'm guessing its because this context setting causes the query to only hit persisted segments, which should be faster. If you don't care about absolute-up-to-the-second data you can skip hitting the incremental index which might be heavily hit due to ingestion.

@nishantmonu51 can you get some performance metrics around this?

[nishantmonu51]

@fjy, @drcrallen, yeah its correct, the idea behind this is to help with query performance when you don't want data upto the latest second and can tolerate delay upto the intermediatePersistPeriod,
the general idea on how it will help is -

• the already persisted segments are faster to query then IncrementalIndex.
• With caching of results for persisted segments (Enable caching on intermediate realtime persists #1943) and this change, queries can be tuned to always be cached.

[nishantmonu51]

@drcrallen will look into getting some performance numbers.

[xvrl]

@fjy tldr; it's the difference between real-time and real²time :)

[drcrallen]

I think this will fail with class cast exception if the value is a string instead of a boolean.

[himanshug]

also, it might be useful to get it once on top and reuse instead of getting for each sink.

[nishantmonu51]

I think we only support Strings for backwards compatibility, do we have to support strings for new properties ?

[himanshug]

can you document the new attribute?

[fjy]

@nishantmonu51 @xvrl I don't really understand the purpose of this PR? Is it so you guys can avoid querying the in memory component of Druid's realtime? Why not look into why the in-memory component is slow?

[gianm]

👍

[xvrl]

👍 it's helpful to have to investigate performance problems. Probably should not need to be documented, since it's behavior is dependent on real-time ingestion implementation details. If it proves very useful, we can promote it.