real time node lead to wrong result(empty result) for the request with regexp filter when real time node is ingesting kafka msg.

[res-life]

increase the kafka message send speed gradually, real time query result will be wrong for the request with regex filter.
request is:

"filter": {
    "type": "regex",
    "dimension": "sub5",
    "pattern": "B"
}
...
"orderBy": {
    "type": "default",
    "columns": [
        {
            "dimension": "clicks",
            "direction": "DESCENDING"
        }
    ],
    "limit": 3
}

result is:
[ {
"version" : "v1",
"timestamp" : "2015-01-06T05:37:00.000Z",
"event" : {
"clicks" : 28
}
}, {
"version" : "v1",
"timestamp" : "2015-01-06T05:37:00.000Z",
"event" : {
"clicks" : 3,
"sub5" : "BB1"
}
}
...
]
Note, I did not send empty value for the dimension I queried. But result show one null value for the sub5 dimension.
query later, the empty result disappear. It seems thread race condition leads to this error.
If not solve the problem, druid will not be really real time system.
both tested on 0.6.121.1 and 0.6.160

[fjy]

@res-life Just to be clear, is the error around the result
{
"version" : "v1",
"timestamp" : "2015-01-06T05:37:00.000Z",
"event" : {
"clicks" : 28
}

showing up initially and then disappearing later on in the query?

[res-life]

{
"event" : {
// not have sub5 dimension. All the sub5 value in Kafka Msg is not empty.
"clicks" : 28
}

The query is same, but the result is different. At first have null sub5 dimension when ingesting, later on this result is:
{
"clicks" : 3,
"sub5" : "BB1"
}

[fjy]

@res-life I believe others in the community have hit this problem and #995 fixes it.

[res-life]

@fjy seems not the same problem.
I specified the regexp filter:
"filter": {
"type": "regex",
"dimension": "sub5",
"pattern": "B"
}
but result is:
"event" : {
// not have sub5 dimension. All the sub5 value in Kafka Msg is not empty.
"clicks" : 28
}
it's not functional issue but performance issue. Only occurs under high throughput ingesting.

[druid-robot]

@res-life, what you are seeing is a result of how Druid handled nulls very
inconsistency. From what I've been told, the situation you described has
been exactly seen before and the pull request is meant to address the
problem.

On Tue, Jan 6, 2015 at 9:52 PM, res-life notifications@github.com wrote:

@fjy https://github.com/fjy seems not the same problem.
I specified the regexp filter:
"filter": {
"type": "regex",
"dimension": "sub5",
"pattern": "B"
}
but result is:
"event" : {
// not have sub5 dimension. All the sub5 value in Kafka Msg is not empty.
"clicks" : 28
}
it's not functional issue but performance issue. Only occurs under high
throughput ingesting.

—
Reply to this email directly or view it on GitHub
#1014 (comment).

[cheddar]

Is this query a topN query? A groupBy?

[res-life]

it's groupBy query

[vogievetsky]

I think this has been addressed. If not I think it should be a 0.15.0 release blocker. @jihoonson

[jihoonson]

Since the first RC is about to come out, I think only regression bugs or very critical bugs could be release blockers. This issue looks related to the real time node which is deprecated and should be removed in the near future. I think we can just remove the real time node in 0.16.0.