Skip to content

Add MAP_APPEND internal function to Calcite PPL #4515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
penghuo merged 4 commits into from
Oct 15, 2025
Merged

Add MAP_APPEND internal function to Calcite PPL #4515

penghuo merged 4 commits into from
Oct 15, 2025

Conversation

@ykmr1224
Copy link
Collaborator

@ykmr1224 ykmr1224 commented Oct 10, 2025

Description

  • Add MAP_APPEND internal function to Calcite PPL
    • This function is needed to implement spath function to merge existing dynamic fields and newly extracted fields by spath
    • Internally share logic with mvappend to align the behavior
    • All the items will be array for consistency
  • This function is internal and won't be exposed as PPL function

Related Issues

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ykmr1224 ykmr1224 added PPL Piped processing language feature calcite calcite migration releated backport 2.19-dev labels Oct 10, 2025
ykmr1224
ykmr1224 commented Oct 10, 2025
View reviewed changes
...test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLRelNodeIntegTestCase.java
import org.opensearch.sql.executor.QueryType;

/** Base class for integration test based on RelNode tree. Mainly for testing internal functions */
public abstract class CalcitePPLRelNodeIntegTestCase extends CalcitePPLIntegTestCase {
Copy link
Collaborator Author

@ykmr1224 ykmr1224 Oct 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class is overwrapping with other PRs, and will be eliminated by rebase once one is merged.

@ykmr1224 ykmr1224 marked this pull request as ready for review October 11, 2025 00:01
penghuo
penghuo previously approved these changes Oct 13, 2025
View reviewed changes
@ykmr1224 ykmr1224 force-pushed the map_append branch 2 times, most recently from 0378928 to 6c7c15c Compare October 13, 2025 19:18
penghuo
penghuo previously approved these changes Oct 14, 2025
View reviewed changes
@ykmr1224
Add MAP_APPEND internal function to Calcite PPL
a6c9441 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Minor fix
0a52d19 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Address comment
05ff8dd 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224
Rebase and fix IT issue
0d8ac14 
Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@penghuo penghuo merged commit cba8d02 into opensearch-project:main Oct 15, 2025
33 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 15, 2025
@github-actions
Add MAP_APPEND internal function to Calcite PPL (#4515)
5c5f1ab 
* Add MAP_APPEND internal function to Calcite PPL

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Minor fix

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Address comment

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Rebase and fix IT issue

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

---------

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
(cherry picked from commit cba8d02)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Oct 15, 2025
Merged
@ykmr1224 ykmr1224 deleted the map_append branch October 15, 2025 22:14
ykmr1224 added a commit to ykmr1224/sql that referenced this pull request Oct 15, 2025
@ykmr1224
Merge main to map-remove:
6acef56 
commit cba8d02
Author: Tomoyuki MORITA <moritato@amazon.com>
Date:   Wed Oct 15 13:08:05 2025 -0700

    Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515)

    * Add MAP_APPEND internal function to Calcite PPL

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

    * Address comment

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

    * Rebase and fix IT issue

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

    ---------

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

commit 3388dc7
Author: Lantao Jin <ltjin@amazon.com>
Date:   Thu Oct 16 01:45:29 2025 +0800

    Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569)

    * Use _shard_doc as sort tiebreaker

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * _doc as a part of tie-breaker have better performance

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    ---------

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

commit 5630119
Author: qianheng <qianheng@amazon.com>
Date:   Wed Oct 15 16:40:41 2025 +0800

    Fix sort push down into agg after project already pushed (opensearch-project#4546)

    * Fix sort push down into agg

    Signed-off-by: Heng Qian <qianheng@amazon.com>

    * Change some json files to yaml format

    Signed-off-by: Heng Qian <qianheng@amazon.com>

    ---------

    Signed-off-by: Heng Qian <qianheng@amazon.com>

commit 1e62fba
Author: Tomoyuki MORITA <moritato@amazon.com>
Date:   Tue Oct 14 17:20:38 2025 -0700

    Fix JsonExtractAllFunctionIT failure (opensearch-project#4556)

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

commit 02ee33e
Author: Kai Huang <105710027+ahkcs@users.noreply.github.com>
Date:   Tue Oct 14 14:28:53 2025 -0700

    Add more examples to the `where` command doc (opensearch-project#4457)

    Co-authored-by: Manasvini B S <manasvis@amazon.com>

commit 0b7e86c
Author: Jialiang Liang <jiallian@amazon.com>
Date:   Tue Oct 14 10:46:01 2025 -0700

    [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434)

    Co-authored-by: Simeon Widdis <sawiddis@amazon.com>

commit 9c97cfb
Author: Tomoyuki MORITA <moritato@amazon.com>
Date:   Tue Oct 14 08:36:43 2025 -0700

    Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489)

    * Add JSON_EXTRACT_ALL internal function for Calcite PPL

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

    * Address comments

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

    ---------

    Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

commit 89dbc31
Author: Lantao Jin <ltjin@amazon.com>
Date:   Tue Oct 14 18:24:52 2025 +0800

    Check server status before starting Prometheus (opensearch-project#4537)

    * Check server status before starting Prometheus

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * Change to func call

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * Fix doc

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    ---------

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

commit fe62472
Author: Lantao Jin <ltjin@amazon.com>
Date:   Tue Oct 14 18:10:27 2025 +0800

    Update request builder after pushdown sort into agg buckets (opensearch-project#4541)

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

commit 42a415f
Author: qianheng <qianheng@amazon.com>
Date:   Tue Oct 14 17:42:45 2025 +0800

    Including metadata fields type when doing agg/filter script push down (opensearch-project#4522)

    * Including metadata fields type when doing agg/filter script push down

    Signed-off-by: Heng Qian <qianheng@amazon.com>

    * Fix IT

    Signed-off-by: Heng Qian <qianheng@amazon.com>

    ---------

    Signed-off-by: Heng Qian <qianheng@amazon.com>

commit 8de0386
Author: Xinyuan Lu <xinyual@amazon.com>
Date:   Tue Oct 14 16:41:08 2025 +0800

    Fix percentile bug (opensearch-project#4539)

    * fix percentile bug

    Signed-off-by: xinyual <xinyual@amazon.com>

    * add IT

    Signed-off-by: xinyual <xinyual@amazon.com>

    * optimize it

    Signed-off-by: xinyual <xinyual@amazon.com>

    ---------

    Signed-off-by: xinyual <xinyual@amazon.com>

commit de2fdc8
Author: Lantao Jin <ltjin@amazon.com>
Date:   Tue Oct 14 12:29:03 2025 +0800

    [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534)

    * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * fix doctest

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * Fix conflicts

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    ---------

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

commit 977b7ab
Author: Simeon Widdis <sawiddis@gmail.com>
Date:   Mon Oct 13 20:23:10 2025 -0700

    Update stalled action (opensearch-project#4485)

commit fddbb70
Author: Lantao Jin <ltjin@amazon.com>
Date:   Tue Oct 14 10:23:12 2025 +0800

    Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501)

    * Add configurable sytem limitations for subsearch and join command

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * Fix IT

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * typo

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * fix IT

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * remove rollback in doc

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * address comments

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * fix typo

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    * Fix IT

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

    ---------

    Signed-off-by: Lantao Jin <ltjin@amazon.com>

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
ykmr1224 pushed a commit that referenced this pull request Oct 15, 2025
@opensearch-trigger-bot @github-actions
Add MAP_APPEND internal function to Calcite PPL (#4515) (#4577)
f3b8fb0 
* Add MAP_APPEND internal function to Calcite PPL



* Minor fix



* Address comment



* Rebase and fix IT issue



---------


(cherry picked from commit cba8d02)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@penghuo penghuo penghuo approved these changes

@dai-chen dai-chen dai-chen approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@LantaoJin LantaoJin Awaiting requested review from LantaoJin LantaoJin is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@qianheng-aws qianheng-aws Awaiting requested review from qianheng-aws qianheng-aws is a code owner

@yuancu yuancu Awaiting requested review from yuancu yuancu is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 is a code owner

Assignees

No one assigned

Labels

backport 2.19-dev calcite calcite migration releated feature PPL Piped processing language

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ykmr1224 @penghuo @dai-chen