Ingest: need a way to determine when new data is queryable

[danprince1]

We use druid 0.10, but recently upgraded to 0.11 and still see the same issue. We have a use case in which we (a) ingest data via HDFS indexing, then (b) wait until the data is available to be queried, then (c) ping a separate system, which then queries the data to do something with it. The problem is in step b. At first we tried this:

1. Poll the Overlord status endpoint /druid/indexer/v1/task//status until it returns SUCCESS or FAILED. This lets us know when the indexing job is complete.
2. Poll the Coordinator loadstatus endpoint /druid/coordinator/v1/loadstatus until it returns 100% for our datasource.

But then we discovered that for a pre-existing datasource there is a period of time after the indexing job completes in which loadstatus returns 100%, before it starts to return something less than 100%. So we added another step:

  1.5.  Poll the Coordinator loadstatus endpoint every second until it returns non-100%.

This polling has to be every second because if the datasource is small it takes very little time to distribute through the cluster. If we poll every 10 seconds, we might miss seeing a non-100% number, and then we have no way of knowing if the new data is available or not. Even at 1 second it's possible to miss this window.

It would be great to have a reliable way of knowing when ingested data is queryable. Thanks!

[josephglanville]

You could use segment metadata queries if you know ahead of time which intervals you are indexing for.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[danprince1]

Commenting to keep this alive, as this is still an issue for us. The segment metadata queries is a good suggestion, but for most of our use cases it would be ungainly to manage, as we ingest hundreds of segments at a time.

[stale]

This issue is no longer marked as stale.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[maytasm]

loadstatus API reads segments from SqlSegmentsMetadataManager of the Coordinator which caches segments in memory and periodically updates them. Hence, there can be a race condition as this API implementation compares segments metadata from cache with segments in historicals. Particularly, when number of segment changes (new ingestion) after the initial load of the datasource, the cache still only contains the metadata of old segments. I notice this too when trying to verify integration tests. The workaround I did (which may not be applicable to real life production) is to verify the number of segments matches expected from metadata/datasources/{DATASOURCE}/segments API before calling this method (since, that would wait until the cache is updated with expected data).

[stale]

This issue is no longer marked as stale.

[Mantrapps]

verify the number of segments only works on no parallel ingestion job running on the same time.
interesting to found this

druid/examples/bin/post-index-task-main

Line 171 in c5bfe36

await_load_completion(args, datasource, load_timeout_at)