Skip to content

Conversation

@ykmr1224
Copy link
Collaborator

@ykmr1224 ykmr1224 commented Oct 9, 2025

Description

  • Add MAP_CONCAT internal function for dynamic fields implementation
    • It will be used to merge dynamic fields map among inputs (mainly join), where same key will be overwritten by later input.
  • It is referring Spark library implementation
  • Added integration test to verify its behavior and access to the function (manual RelNode generation to avoid exposing function in PPL syntax)

Related Issues

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
Signed-off-by: Tomoyuki MORITA <moritato@amazon.com>
penghuo
penghuo previously approved these changes Oct 10, 2025
Copy link
Collaborator

@penghuo penghuo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure. Do we need map_concat or map_zip_with?

spark-sql (default)> SELECT map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> array(v1, v2)) as res;
res
{1:["a","x"],2:["b","y"]}

@ykmr1224
Copy link
Collaborator Author

ykmr1224 commented Oct 10, 2025

Not sure. Do we need map_concat or map_zip_with?

spark-sql (default)> SELECT map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) -> array(v1, v2)) as res;
res
{1:["a","x"],2:["b","y"]}

That sounds more related to MAP_APPEND which I am planning to implement, and that will be used to merge dynamic fields when spath command is executed (because spath command needs to append instead of replace).
MAP_CONAT is different because it will simply overwrite value in case of key conflict. This is needed mainly for join operation where we don't merge values.

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
@ykmr1224 ykmr1224 merged commit ef783f1 into opensearch-project:main Oct 13, 2025
33 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 13, 2025
* Add MAP_CONCAT internal function

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

* Minor fix

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>

---------

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
Signed-off-by: Tomoyuki MORITA <moritato@amazon.com>
(cherry picked from commit ef783f1)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
penghuo pushed a commit that referenced this pull request Oct 13, 2025
* Add MAP_CONCAT internal function



* Minor fix



---------



(cherry picked from commit ef783f1)

Signed-off-by: Tomoyuki Morita <moritato@amazon.com>
Signed-off-by: Tomoyuki MORITA <moritato@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev calcite calcite migration releated feature PPL Piped processing language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants