Critical Calcite StackOverflowError in 1.21.0 causing failure in WHERE with too many conditions

[l15k4]

Affected Version

0.19.0

Description

Any SQL query that has ~ hundreds of dimensions in WHERE clause x = foo OR y = bar will fail due to
https://issues.apache.org/jira/browse/CALCITE-2792 on StackOverflowError

Solution

Bumping calcice to at least 1.22.0 but there might be some guava conflicts

[l15k4]

Do you guys know how to hot fix it? I'm trying to figure out how to put Calcite 1.22.0 on classpath in docker image such that it will take precedence over 1.21.0

[suneet-s]

@l15k4 I had tried to update Calcite to 1.22.0 in the past, but got stuck as we are using guava 16.0.1 and Calcite dropped support for older versions of guava here - https://issues.apache.org/jira/browse/CALCITE-2191

It looks like 1.22.0 was the first version where APIs that are incompatible with the version of guava we use were introduced to Calcite. So this requires some work in Druid to sort out the dependencies. If you're able to solve this - a PR would be greatly appreciated 😄

I believe there were some other issues that would be fixed by upgrading the calcite version, but I can't seem to find the specific issues right now.

[l15k4]

Yeah I suspected something like this, otherwise it would be bumped. Ou my, we're having ugly production issue with this and I'm afraid to rollback. I believe this is going to hit a lot of people who upgrade to 0.19.0, because it starts from a few hundreds of dimensions. This is a repro :
https://gist.github.com/l15k4/149064987862831c8aaad5e3ce81331c

[gianm]

Sorry, this looks like it's frustrating.

Would downgrading Calcite be a good workaround for you in the short run? Or, potentially, rewrite your query to use an "IN" filter rather than a bunch of "OR"s?

I don't think we'd want to downgrade Calcite for an official release (we updated in #8566, which fixed other issues, including #8266). We've been on the newer Calcite version since Druid 0.17.0. But maybe it'd work to do it locally?

Ultimately for the project the best way to fix all of this stuff is to keep upgrading Calcite as they fix bugs. We'll need to deal with the Guava situation somehow. Maybe shade it, or some other solution.