[backport to 2.19-dev] Backport calcite prs#3752

Merged

penghuo merged 178 commits intoopensearch-project:2.19-devfrom

xinyual:backportForSignoff

Jun 16, 2025

Commits on Jun 12, 2025

Add CIDR function to PPL (#3036) (#3110)

currantw
authored and
xinyual
committed
Add FILLNULL command in PPL (#3032) (#3075)

authored and
xinyual
committed
Add trendline PPL command (#3071)

authored and
xinyual
committed
* Add `ExprIpValue` and `IP` data type

currantw
authored and
xinyual
committed
Framework of Calcite Engine: Parser, Catalog Binding and Plan Converter (#3249)

LantaoJin
authored and
xinyual
committed
[POC] Make PPL execute successfully on Calcite engine (#3258)

qianheng-aws
authored and
xinyual
committed
Make basic aggregation working (part 1) (#3318)

LantaoJin
authored and
xinyual
committed
[Calcite Engine] Push down project and filter operator into index scan (#3327)

qianheng-aws
authored and
xinyual
committed
[Calcite] Build integration test framework (#3342)

LantaoJin
authored and
xinyual
committed
Followup: Register OpenSearchTypeSystem to OpenSearchTypeFactory (#3349)

LantaoJin
authored and
xinyual
committed
Make basic aggregation working (part 2) : adding tests (#3355)

LantaoJin
authored and
xinyual
committed
Fix execution errors caused by plan gap (#3350)

qianheng-aws
authored and
xinyual
committed
Implement ppl join command with Calcite (#3364)

LantaoJin
authored and
xinyual
committed
Enable push down optimization by default (#3366)

qianheng-aws
authored and
xinyual
committed
Fix CalcitePPLJoinIT (#3369)

qianheng-aws
authored and
xinyual
committed
Implement ppl `IN` subquery command with Calcite (#3371)

LantaoJin
authored and
xinyual
committed
fix java 21 syntax
xinyual
committed
fix IT for syntax 21
xinyual
committed
[BugFix] support push down text field correctly. (#3376)

qianheng-aws
authored and
xinyual
committed
Add udf interface (#3374)
xinyual
committed
[BugFix] Fix the join condition resolving bug introduced by IN subquery impl (#3377)

LantaoJin
authored and
xinyual
committed
Implement ppl relation subquery command with Calcite (#3378)

LantaoJin
authored and
xinyual
committed
Implement ppl exists subquery command with Calcite (#3388)

LantaoJin
authored and
xinyual
committed
Implement ppl scalar subquery command with Calcite (#3392)

LantaoJin
authored and
xinyual
committed
Calcite enable pushdown aggregation (#3389)

qianheng-aws
authored and
xinyual
committed
Add string builtin functions (#3393)
xinyual
committed
Keep aggregation in Calcite consistent with current PPL behavior (#3405)

LantaoJin
authored and
xinyual
committed
Support multiple table and index pattern (#3409)

qianheng-aws
authored and
xinyual
committed
Separate with/without pushdown ITs (#3413)

LantaoJin
authored and
xinyual
committed
Add condition UDFs (#3412)
xinyual
committed
Fix IT and CI, revert alias change (#3423)

penghuo
authored and
xinyual
committed
Revert result ordering of `stats-by` (#3427)

LantaoJin
authored and
xinyual
committed
Implement lookup command (#3419)

authored and
xinyual
committed
Support ppl BETWEEN operator within Calcite (#3433)

LantaoJin
authored and
xinyual
committed
Correct the precedence for logical operators (#3435)

LantaoJin
authored and
xinyual
committed
Implement ppl `dedup` command with Calcite (#3416)

LantaoJin
authored and
xinyual
committed
Use entire shadow jar to fix IT (#3447)

xinyual
and
penghuo
committed
use entireShadowJar
xinyual
committed
[Calcite Engine] Support In expression (#3429)

qianheng-aws
authored and
xinyual
committed
Support ppl CAST function with Calcite (#3439)

LantaoJin
authored and
xinyual
committed
Support group by span over time based column with Span UDF (#3421)

songkant-aws
authored and
xinyual
committed
Add math udf (#3390)

xinyual
and
yuancu
committed
Fix flaky tests (#3456)

yuancu
authored and
xinyual
committed
fix syntax in java 21
xinyual
committed
fix syntax for java 21
xinyual
committed
json-valid PPL function (#3230)

authored and
xinyual
committed
json-valid PPL function (#3230)

authored and
xinyual
committed
remove geo ip

acarbonetto
authored and
xinyual
committed
Enable Calcite by default and refactor all related ITs (#3468)

LantaoJin
authored and
xinyual
committed
remove geo ip
xinyual
committed
Unified OpenSearch PPL Data Type (#3345)

penghuo
authored and
xinyual
committed
Add missing text function (#3471)
xinyual
committed
[Calcite engine] Support nested field (#3476)

qianheng-aws
authored and
xinyual
committed
[Followup] fix text function IT for locate and strcmp (#3482)
xinyual
committed
Support `TYPEOF` function with Calcite (#3446)

LantaoJin
authored and
xinyual
committed
Support `parse` command with Calcite (#3474)

LantaoJin
authored and
xinyual
committed
[Calcite Engine]Support UDT for date, time, timestamp (#3483)

qianheng-aws
authored and
xinyual
committed
Support for OpenSearch alias type (#3246)

qianheng-aws
authored and
xinyual
committed
[Calcite Engine] Support UDT for IP (#3504)

qianheng-aws
authored and
xinyual
committed
Execute Calcite PPL query in thread pool (#3508)

LantaoJin
authored and
xinyual
committed
Add datetime functions (#3473)

committed
use correct timezone name (#3517)
xinyual
committed
Improved patterns command with new algorithm (#3263) (#3335)

songkant-aws
authored and
xinyual
committed
Fix antlr4 parser issues (#3492)

LantaoJin
authored and
xinyual
committed
Fix return types of MOD and DIVIDE UDFs (#3513)

yuancu
authored and
xinyual
committed
Support line comment and block comment in PPL (#2806)

LantaoJin
authored and
xinyual
committed
fix varchar bug (#3518)
xinyual
committed
[Calcite Engine] Function framework refactoring (#3522)

qianheng-aws
authored and
xinyual
committed
New output for explain endpoint with Calcite engine (#3521)

LantaoJin
authored and
xinyual
committed
unified settings

penghuo
authored and
xinyual
committed
Fix GET_FORMAT UDF (#3543)

yuancu
authored and
xinyual
committed
[Calcite Engine] Support alias type field (#3538)

qianheng-aws
authored and
xinyual
committed
[Calcite Engine] Only enable fallback for tests that need to fall back (#3544)

qianheng-aws
authored and
xinyual
committed
remove geo ip it
xinyual
committed
Support UDT for BINARY (#3549)

qianheng-aws
authored and
xinyual
committed
fix timestamp bug (#3542)
xinyual
committed
[Calcite Engine]Support metadata field (#3445)

qianheng-aws
authored and
xinyual
committed
Support `explain` PPL command (#3560)

LantaoJin
authored and
xinyual
committed
New added commands should throw exception when calcite disabled (#3571)

LantaoJin
authored and
xinyual
committed
Support `CASE` function with Calcite (#3558)

LantaoJin
authored and
xinyual
committed
Support parsing documents with flattened value (#3577)

qianheng-aws
authored and
xinyual
committed
Revert stream pattern method in V2 and implement SIMPLE_PATTERN in Calcite (#3553)

songkant-aws
authored and
xinyual
committed
[DOC] Remove the duplicated timestamp row in data type mapping table (#2617)

authored and
xinyual
committed
[DOC] Documentation for PPL new engine (V3) and limitations of 3.0.0 Beta (#3488)

LantaoJin
authored and
xinyual
committed
Support `eventstats` command with Calcite (#3585)

LantaoJin
authored and
xinyual
committed
Implement cryptographic hash UDFs (#3574)

yuancu
authored and
xinyual
committed
Add a new row count estimation mechanism for CalciteIndexScan (#3605)

qianheng-aws
authored and
xinyual
committed
Migrate existing UDFs to PPLFuncImpTable (#3576)

yuancu
and
xinyual
committed
Add UT for PredicateAnalyzer and AggregateAnalyzer (#3612)

qianheng-aws
authored and
xinyual
committed
Add a trimmed project before aggregate to avoid NPE in Calcite (#3621)

LantaoJin
authored and
xinyual
committed
Add a TPC-H PPL query suite (#3622)

LantaoJin
authored and
xinyual
committed
Support functions `isempty`, `isblank` and `ispresent` with Calcite (#3627)

LantaoJin
authored and
xinyual
committed
Support `describe` command with Calcite (#3624)

LantaoJin
authored and
xinyual
committed
Support `fillnull` command with Calcite (#3634)

LantaoJin
authored and
xinyual
committed
Support function `coalesce` with Calcite (#3628)

yuancu
authored and
xinyual
committed
Implement `cidrmatch` udf with Calcite (#3603)

yuancu
authored and
xinyual
committed
Fix pushing down filter with nested filed of the text type (#3645)

qianheng-aws
authored and
xinyual
committed
Fix alias type points to nested field (#3674)

qianheng-aws
authored and
xinyual
committed
Support `top`, `rare` commands with Calcite (#3647)

LantaoJin
authored and
xinyual
committed
Support match_only_text field type (#3663)

penghuo
authored and
xinyual
committed
Update PPL Limitation Docs (#3656)

penghuo
authored and
xinyual
committed
Make query.size_limit only affect the final results (#3623)

qianheng-aws
authored and
xinyual
committed
fix missing conflict
xinyual
committed
Switch percentile implementation to MergingDigest to align with OpenSearch (#3698)

LantaoJin
authored and
xinyual
committed
Support Limit pushdown (#3615)

yuancu
authored and
xinyual
committed
fix java 21 syntax and apply spotless on develop code
xinyual
committed
fix syntax for IT
xinyual
committed
restore useless change
xinyual
committed
fix IT
xinyual
committed
remove useless change
xinyual
committed
fix IT
xinyual
committed
Revert "Switch percentile implementation to MergingDigest to align with OpenSearch (#3698)"
xinyual
committed
fix IT
xinyual
committed
fix part of IT
xinyual
committed
Support GEO_POINT type (#3511)

qianheng-aws
authored and
xinyual
committed
fix IT
xinyual
committed
fix IT by add timestamp value
xinyual
committed
revert change
xinyual
committed
ignore Prometheus with calcite as previous
xinyual
committed
fix Patterns IT
xinyual
committed
fix UT
xinyual
committed
fix IT by workaround
xinyual
committed
fix IT and doctest
xinyual
committed
fix UT and doctest
xinyual
committed
fix UT and revert change
xinyual
committed
remove useless UT
xinyual
committed
Support Grok command in Calcite engine (#3678)

songkant-aws
authored and
xinyual
committed
Support `appendcol` command with Calcite (#3729)

LantaoJin
authored and
xinyual
committed
Support decimal literal with Calcite (#3673)

LantaoJin
authored and
xinyual
committed
Support ResourceMonitor with Calcite (#3738)

LantaoJin
authored and
xinyual
committed
Implement Parameter Validation for PPL functions on Calcite (#3626)

authored and
xinyual
committed
fix IT
xinyual
committed
fix bwc
xinyual
committed
fix 21 syntax
xinyual
committed
avoid resolve safe
xinyual
committed
add json functions (#3559)
xinyual
committed
Support merging object-type fields when fetching the schema from the index (#3653)
xinyual
committed
Support trendline command in Calcite (#3741)

songkant-aws
authored and
xinyual
committed
Add DISTINCT_COUNT_APPROX function (#3654)
xinyual
committed
use access controller
xinyual
committed
fix 21 syntax
xinyual
committed
Support `expand` command with Calcite (#3745)

yuancu
authored and
xinyual
committed
Add earliest and latest in condition function (#3640)
xinyual
committed
fix syntax 21 code
xinyual
committed
Add lambda function and array related functions (#3584)
xinyual
committed
fix syntax 21 code
xinyual
committed
Return double with correct precision for `UNIX_TIMESTAMP` (#3679)

yuancu
authored and
xinyual
committed
[BugFix] Prevent push down limit with offset reach maxResultWindow (#3713)

qianheng-aws
authored and
xinyual
committed
Calcite patterns command brain pattern method (#3570)

songkant-aws
authored and
xinyual
committed
Support `flatten` command with Calcite (#3747)

yuancu
authored and
xinyual
committed
Fix: correct ATAN(x, y) and CONV(x, a, b) functions bug (#3748)

ishaoxy
authored and
xinyual
committed
fix IT and ignore information schema IT
xinyual
committed
Fix field not found issue in join output when column names are ambiguous (#3760)

LantaoJin
authored and
xinyual
committed
fix IT with syntax 21
xinyual
committed
remove test IT
xinyual
committed
revert useless change
xinyual
committed
Enable update calcite setting through _plugins/_query/settings API (#3531)

penghuo
authored and
xinyual
committed
Fix error when pushing down script filter with struct field (#3693)

qianheng-aws
authored and
xinyual
committed
fix explain IT
xinyual
committed
fix link checker
xinyual
committed
CI: Geo-Spatial Plugin integration test (#3244)

andy-k-improving
authored and
xinyual
committed
PPL command expression implementation for `geoip` (#3228)

authored and
xinyual
committed
Implement `geoip` udf with Calcite (#3604)

yuancu
authored and
xinyual
committed
revert useless change
xinyual
committed
Update PPL parse command doc with limitation (#3274)

dai-chen
authored and
xinyual
committed
Fixed week of week based year handling (#3299)

normanj-bitquill
authored and
xinyual
committed
Add other functions to SQL query validator (#3304)

ykmr1224
authored and
xinyual
committed
Add SQLQuery Utils support for Vaccum queries (#3269)

authored and
xinyual
committed
Fix: Long IN-lists causes crash (#3660)

Swiddis
authored and
xinyual
committed
CVE-2024-57699: Fix json-smart vulnerability (#3484)

LantaoJin
authored and
xinyual
committed
[Fix CVE-2025-47273]Bump setuptools to 78.1.1 (#3671)

RyanL1997
authored and
xinyual
committed
Fix issue 2489 (#3442)

penghuo
authored and
xinyual
committed
fix conflict
xinyual
committed
fix conflict
xinyual
committed
fix bwc IT
xinyual
committed
fix flaky IT
xinyual
committed
delete publish to maven
xinyual
committed
follow the latest parse logic
xinyual
committed