Publish internal modules separately for downstream reuse #4484

Swiddis · 2025-10-09T19:17:37Z

Description

Copy of #4385 but merged with main, trying to see what's broken with tests

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…project#3763) * Add common Gradle task for all published modules Signed-off-by: Chen Dai <daichen@amazon.com> * Add publish workflow file Signed-off-by: Chen Dai <daichen@amazon.com> * Rename group ID from sql to query Signed-off-by: Chen Dai <daichen@amazon.com> --------- Signed-off-by: Chen Dai <daichen@amazon.com>

* Add api module with API and UT Signed-off-by: Chen Dai <daichen@amazon.com> * Refactor catalog API and clean up build.gradle Signed-off-by: Chen Dai <daichen@amazon.com> * Add cache schema API and refactor UT Signed-off-by: Chen Dai <daichen@amazon.com> * Add readme Signed-off-by: Chen Dai <daichen@amazon.com> * Add comment for hardcoding query size limit Signed-off-by: Chen Dai <daichen@amazon.com> * Add default namespace API with more UTs Signed-off-by: Chen Dai <daichen@amazon.com> --------- Signed-off-by: Chen Dai <daichen@amazon.com>

…-project#4015) * add datasources submodule Signed-off-by: cnoramut <cnoramut@amazon.com> * add legacy since datasources need it Signed-off-by: cnoramut <cnoramut@amazon.com> --------- Signed-off-by: cnoramut <cnoramut@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Louis Chu <clingzhi@amazon.com>

Signed-off-by: Simeon Widdis <sawiddis@gmail.com>

Swiddis · 2025-10-09T20:18:13Z

api/src/main/java/org/opensearch/sql/api/EmptyDataSourceService.java

+/**
+ * A DataSourceService that assumes no access to data sources
+ */
+public class EmptyDataSourceService implements DataSourceService {


Since CalciteRelNodeVisitor needs a data source service now, but this notion doesn't make any sense outside of a cluster, let's just fill in an empty one that always returns no results and reports any given source as nonexistent.

Better solution long-term would be to make CalciteRelNodeVisitor not have this dependency to begin with, seems like it crosses boundaries.

Signed-off-by: Simeon Widdis <sawiddis@gmail.com>

dai-chen · 2025-10-22T18:23:54Z

.github/workflows/maven-publish-modules.yml

+      - 2.*
+
+env:
+  SNAPSHOT_REPO_URL: https://aws.oss.sonatype.org/content/repositories/snapshots/


Is this already deprecated? If so, probably you need a PR for previous feature branch to verify the publish?

agh, sonatype is the gift that keeps on giving...

Interesting that tests pass when this shouldn't be able to build

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

.github/workflows/maven-publish-modules.yml

Swiddis · 2025-10-23T16:10:08Z

~~Don't merge -- trying to test the publish flow on the feature branch currently, it says "success" but no artifact shows up in central~~

Fixed, successful publish: https://github.com/opensearch-project/sql/actions/runs/18763687236/job/53534148205

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Chen Dai <daichen@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com> (cherry picked from commit 05d6594) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* default-main: (34 commits) Enhance dynamic source clause to support only metadata filters (opensearch-project#4554) Make nested alias type support referring to outer context (opensearch-project#4673) Update big5 ppl queries and check plans (opensearch-project#4668) Support push down sort after limit (opensearch-project#4657) Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670) Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621) Fix bin nested fields issue (opensearch-project#4606) Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531) Pushdown sort aggregate metrics (opensearch-project#4603) Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648) Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646) Allow renaming group-by fields to existing field names (opensearch-project#4586) Publish internal modules separately for downstream reuse (opensearch-project#4484) Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643) Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599) Replace all dots in fields of table scan's PhysType (opensearch-project#4633) Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629) Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623) Pushdown case function in aggregations as range queries (opensearch-project#4400) Update GEOIP function to support IP types as input (opensearch-project#4613) ... # Conflicts: # docs/user/ppl/functions/conversion.rst

* default-main: (34 commits) Enhance dynamic source clause to support only metadata filters (opensearch-project#4554) Make nested alias type support referring to outer context (opensearch-project#4673) Update big5 ppl queries and check plans (opensearch-project#4668) Support push down sort after limit (opensearch-project#4657) Use table scan rowType in filter pushdown could fix rename issue (opensearch-project#4670) Fix: Support Alias Fields in MIN, MAX, FIRST, LAST, and TAKE Aggregations (opensearch-project#4621) Fix bin nested fields issue (opensearch-project#4606) Add `per_minute`, `per_hour`, `per_day` function support (opensearch-project#4531) Pushdown sort aggregate metrics (opensearch-project#4603) Followup: Change ComparableLinkedHashMap to compare Key than Value (opensearch-project#4648) Mitigate the CI failure caused by 500 Internal Server Error (opensearch-project#4646) Allow renaming group-by fields to existing field names (opensearch-project#4586) Publish internal modules separately for downstream reuse (opensearch-project#4484) Revert "Update grammar files and developer guide (opensearch-project#4301)" (opensearch-project#4643) Support Automatic Type Conversion for REX/SPATH/PARSE Command Extractions (opensearch-project#4599) Replace all dots in fields of table scan's PhysType (opensearch-project#4633) Return comparable LinkedHashMap in `valueForCalcite()` of ExprTupleValue (opensearch-project#4629) Refactor JsonExtractAllFunctionIT and MapConcatFunctionIT (opensearch-project#4623) Pushdown case function in aggregations as range queries (opensearch-project#4400) Update GEOIP function to support IP types as input (opensearch-project#4613) ... Signed-off-by: Asif Bashar <asif.bashar@gmail.com>

Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Chen Dai <daichen@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com>

…project#4484) Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Chen Dai <daichen@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com>

…project#4484) Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Chen Dai <daichen@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com> Signed-off-by: Chen Dai <daichen@amazon.com>

…m reuse (#4723) * Publish internal modules separately for downstream reuse (#4484) Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Chen Dai <daichen@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com> Signed-off-by: Chen Dai <daichen@amazon.com> * Add test branch and downgrade JDK in maven publish workflow Signed-off-by: Chen Dai <daichen@amazon.com> * Add 2.19-dev branch to workflow and bump Calcite version in api module Signed-off-by: Chen Dai <daichen@amazon.com> --------- Signed-off-by: Chen Dai <daichen@amazon.com> Co-authored-by: Simeon Widdis <sawiddis@gmail.com> Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com>

…project#4484) Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Chen Dai <daichen@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com>

dai-chen and others added 5 commits June 16, 2025 08:56

add datasources submodule (opensearch-project#3935)

99724d9

Merge branch 'main' into feature/unified-ppl

ec5e8fe

Swiddis added the enhancement New feature or request label Oct 9, 2025

Swiddis changed the title ~~Unified PPL as a library~~ Publish internal modules separately for downstream reuse Oct 9, 2025

Swiddis added 4 commits October 9, 2025 12:37

Unlock Spotless version

87396b7

Signed-off-by: Simeon Widdis <sawiddis@gmail.com>

Don't run publish on feature fork

99f181b

Signed-off-by: Simeon Widdis <sawiddis@gmail.com>

Merge remote-tracking branch 'upstream/main' into feature/unified-ppl

8c428c6

Fill in empty data source service for external API

f6af494

Signed-off-by: Simeon Widdis <sawiddis@gmail.com>

Swiddis commented Oct 9, 2025

View reviewed changes

Run spotless

7dc10bf

Signed-off-by: Simeon Widdis <sawiddis@gmail.com>

Swiddis marked this pull request as ready for review October 9, 2025 20:54

Swiddis requested review from GumpacG, MaxKsyunz, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, penghuo, ps48, seankao-az and vamsimanohar as code owners October 9, 2025 20:54

Swiddis requested review from LantaoJin, RyanL1997, noCharger, qianheng-aws and yuancu as code owners October 9, 2025 20:54

dai-chen reviewed Oct 22, 2025

View reviewed changes

Swiddis added 4 commits October 22, 2025 18:53

Switch sonatype repo

2bb0223

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Fix url again (weird it didn't push the first time)

42b9dd5

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge remote-tracking branch 'upstream/main' into HEAD

6195a56

Deal with upstream breaking stuff again

05dd1a4

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

dai-chen previously approved these changes Oct 23, 2025

View reviewed changes

.github/workflows/maven-publish-modules.yml Outdated Show resolved Hide resolved

Attempt to publish to S3 again

eed569d

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis dismissed dai-chen’s stale review via eed569d October 23, 2025 22:52

Tweak publish branch rules

98e16bc

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

dai-chen approved these changes Oct 23, 2025

View reviewed changes

RyanL1997 approved these changes Oct 23, 2025

View reviewed changes

Swiddis merged commit 05d6594 into opensearch-project:main Oct 23, 2025
33 checks passed

Swiddis added the backport 3.3 label Oct 24, 2025

opensearch-trigger-bot bot mentioned this pull request Oct 24, 2025

[Backport 3.3] Publish internal modules separately for downstream reuse #4661

Closed

dai-chen added a commit that referenced this pull request Oct 30, 2025

Publish internal modules separately for downstream reuse (#4484)

4314feb

Co-authored-by: Louis Chu <clingzhi@amazon.com> Co-authored-by: Chen Dai <daichen@amazon.com> Co-authored-by: Mebsina <cnoramut@gmail.com>

dai-chen mentioned this pull request Nov 3, 2025

[Backport 2.19-dev] Publish internal modules separately for downstream reuse #4723

Merged

dai-chen mentioned this pull request Nov 4, 2025

[FEATURE] Export PPL-Calcite engine as reusable library #3734

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish internal modules separately for downstream reuse #4484

Publish internal modules separately for downstream reuse #4484

Uh oh!

Swiddis commented Oct 9, 2025

Uh oh!

Swiddis Oct 9, 2025 •

edited

Loading

Uh oh!

dai-chen Oct 22, 2025

Uh oh!

Swiddis Oct 22, 2025 •

edited

Loading

Uh oh!

Uh oh!

Swiddis commented Oct 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Publish internal modules separately for downstream reuse #4484

Publish internal modules separately for downstream reuse #4484

Uh oh!

Conversation

Swiddis commented Oct 9, 2025

Description

Related Issues

Check List

Uh oh!

Swiddis Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dai-chen Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Swiddis Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Swiddis commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Swiddis Oct 9, 2025 •

edited

Loading

Swiddis Oct 22, 2025 •

edited

Loading

Swiddis commented Oct 23, 2025 •

edited

Loading