add note on consistency of results for sys.segments queries by surekhasaharan · Pull Request #7034 · apache/druid

surekhasaharan · 2019-02-07T20:12:34Z

For the sys.segments queries, it seems broker randomly chooses one of the replicas, so if there are more than one replica for a segment, then the fields like size num_rows etc. can have different values based on which realtime replica, the broker queries. The results will be eventually consistent once, the segment is served by a historical server.Adding this note to the docs. This may not be a problem once this issue is addressed.

justinborromeo · 2019-02-07T20:26:45Z


 ### SEGMENTS table
 Segments table provides details on all Druid segments, whether they are published yet or not.
+Note that if a segment is served by more than one realtime tasks(multiple realtime replicas), then the results may vary between the sys.segments queries for columns such as `size`, `num_rows` etc., until the segment is served by a historical eventually.


There should be a space between tasks and (multiple

This got changed.

jihoonson · 2019-02-07T21:58:32Z


 ### SEGMENTS table
 Segments table provides details on all Druid segments, whether they are published yet or not.
+Note that if a segment is served by more than one realtime tasks(multiple realtime replicas), then the results may vary between the sys.segments queries for columns such as `size`, `num_rows` etc., until the segment is served by a historical eventually.


The purpose of this note is to make people less confused, and thus it should be detailed as much as possible.

Please add more details about when this can happen and why, and what columns can vary. I think it's worth to add a new section for this caveat.

Sometimes more details can be more confusing :)). Tried to add more details, let me know if it's less confusing. Not sure if it needs it's own section and what should be the title of that section. Added a caveat subheading.

…doc-update

fjy · 2019-02-08T22:36:01Z

 Segments table provides details on all Druid segments, whether they are published yet or not.

+#### CAVEAT
+Note that a segment can be served by more than one realtime or historical servers, in that case it would have multiple replicas. These replicas are weakly consistent with each other when served by multiple realtime tasks, until a segment is eventually served by a historical, at that point the segment is immutable. And broker prefers to query a segment from historical over realtime server. But if a segment has multiple realtime replicas, for eg. kafka index tasks, and one task is slower than other, then the sys.segments query results can vary for the duration of the tasks. The columns of segments table that can have inconsistent values during this period include `size`, `num_replicas`, `num_rows`.


There are no such things are realtime or historical servers

please ensure consistent capitalization for Historicals, Brokers, etc

There are no such things are realtime or historical servers

I see mention of Historical Node, Real-time Node in docs. So what should I write historical node ? process ?

IMO, "Historical process" and "stream ingestion tasks"

corrected the capitalization and changed to correct terminology

jihoonson · 2019-02-08T23:44:54Z

 Segments table provides details on all Druid segments, whether they are published yet or not.

+#### CAVEAT
+Note that a segment can be served by more than one realtime or historical servers, in that case it would have multiple replicas. These replicas are weakly consistent with each other when served by multiple realtime tasks, until a segment is eventually served by a historical, at that point the segment is immutable. And broker prefers to query a segment from historical over realtime server. But if a segment has multiple realtime replicas, for eg. kafka index tasks, and one task is slower than other, then the sys.segments query results can vary for the duration of the tasks. The columns of segments table that can have inconsistent values during this period include `size`, `num_replicas`, `num_rows`.


Would you explain why size and num_replica vary? It looks that they are not getting from segmentMetadataQuery.

Please add why this happens. The root cause is that system schema uses segmentMetadatQuery to retrieve some information, and the broker randomly picks one of the realtime tasks for query processing if there's no published segments, and thus it's not guaranteed that the same task serves segmentMetadataQuery every time.

I think it's worth to link #5915 here too.

hmm, should there be mention of segmentMetadatQuery and RandomServerSelectorStrategy in user facing docs. I tried to explain without adding internal code details. I feel such details should be in github issues or in javadocs. And do we generally link to github issues in user documentation, are there any similar examples in druid docs?

SegmentMetdataQuery is a documented query type (http://druid.io/docs/latest/querying/segmentmetadataquery.html). I don't think it's worth to mention the class name of RandomServerSelectorStrategy but the configuration for it is also documented (http://druid.io/docs/latest/configuration/index.html#query-prioritization).

Well, but my above comment about random selection may not be appropriate because it can give a wrong intuition to users. Probably better to not say about random selection at all. But, I think it's still needed to say about only one of the realtime tasks is selected if multiple replicas are running.

And do we generally link to github issues in user documentation, are there any similar examples in druid docs?

Why not? Here're some examples: https://cse.google.com/cse?cx=000162378814775985090%3Amolvbm0vggm&q=github&oq=github&gs_l=partner-generic.3...1401.2048.0.2184.6.6.0.0.0.0.102.536.5j1.6.0.gsnos%2Cn%3D13...0.606j90652j6...1.34.partner-generic..5.1.102.mApbmyfw_Jw.

Would you explain why size and num_replica vary? It looks that they are not getting from segmentMetadataQuery.

I think size would not vary between ingestion tasks, since they all would show 0, but it can vary if a segment is queried from Historical vs realtime task. But given that, Broker prefers Historical, may be size is not an issue. For num_replica, it can change if a segment gets added or removed from TimelineServerView.TimelineCallback in DruidSchema, and it's value can vary between the queries.

Hmm. For num_replica, it sounds like it's a valid result because it reflects the changes which actually happened. I think it's different from varying num_rows and doesn't have to be noted here.

In that case, it seems num_rows is the only col affected.

…doc-update

jihoonson

Thanks for the update!

jihoonson · 2019-02-14T01:54:11Z

 Segments table provides details on all Druid segments, whether they are published yet or not.

+#### CAVEAT
+Note that a segment can be served by more than one stream ingestion tasks or Historical processes, in that case it would have multiple replicas. These replicas are weakly consistent with each other when served by multiple ingestion tasks, until a segment is eventually served by a Historical, at that point the segment is immutable. Broker prefers to query a segment from Historical over a ingestion task. But if a segment has multiple realtime replicas, for eg. kafka index tasks, and one task is slower than other, then the sys.segments query results can vary for the duration of the tasks because only one of the ingestion tasks is queried by the Broker and it is not gauranteed that the same task gets picked everytime. The columns of segments table that can have inconsistent values during this period include `num_replicas` and `num_rows`. There is an open [issue](https://github.com/apache/incubator-druid/issues/5915) about this inconsistency with stream ingestion tasks.


a ingestion task -> an ingestion task.

…doc-update

jihoonson · 2019-02-15T18:04:37Z

@surekhasaharan thanks! LGTM.

) * add doc * change docs * PR comments * few more changes

…7101) * add doc * change docs * PR comments * few more changes

add doc

b1e3455

justinborromeo reviewed Feb 7, 2019

View reviewed changes

surekhasaharan added the Area - Documentation label Feb 7, 2019

jihoonson reviewed Feb 7, 2019

View reviewed changes

Surekha Saharan added 2 commits February 7, 2019 16:29

change docs

9672aea

Merge branch 'master' of github.com:druid-io/druid into sys-segments-…

0a71937

…doc-update

fjy reviewed Feb 8, 2019

View reviewed changes

jihoonson reviewed Feb 8, 2019

View reviewed changes

jon-wei added this to the 0.14.0 milestone Feb 13, 2019

Surekha Saharan added 2 commits February 13, 2019 16:23

PR comments

2cdbb1d

Merge branch 'master' of github.com:druid-io/druid into sys-segments-…

9b42012

…doc-update

jihoonson reviewed Feb 14, 2019

View reviewed changes

Surekha Saharan added 2 commits February 13, 2019 21:38

few more changes

bb55eb9

Merge branch 'master' of github.com:druid-io/druid into sys-segments-…

0154bd2

…doc-update

jihoonson approved these changes Feb 15, 2019

View reviewed changes

jihoonson merged commit 2b04e6d into apache:master Feb 19, 2019

surekhasaharan pushed a commit to surekhasaharan/druid that referenced this pull request Feb 20, 2019

add note on consistency of results for sys.segments queries (apache#7034

ce1d1ee

) * add doc * change docs * PR comments * few more changes

surekhasaharan mentioned this pull request Feb 20, 2019

[Backport] add note on consistency of results for sys.segments queries (#7034) #7101

Merged

fjy pushed a commit that referenced this pull request Feb 20, 2019

add note on consistency of results for sys.segments queries (#7034) (#…

9990e04

…7101) * add doc * change docs * PR comments * few more changes

surekhasaharan deleted the sys-segments-doc-update branch February 20, 2019 18:33

Conversation

surekhasaharan commented Feb 7, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonson Feb 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gianm Feb 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Feb 15, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jihoonson Feb 7, 2019 •

edited

Loading

gianm Feb 12, 2019 •

edited

Loading