Feature addtotals and addcoltotals

[asifabashar]

Description

Is your feature request related to a problem?
addtotals command to show total of all columns of each row as a new column , and also have option to show total of all rows of each column values to show at the end of rows.
Fixes issue #4607
From roadmap #4287

addcoltotals command to show total of each column's all rows values to show at the end of rows.
From roadmap #4287

What solution would you like?
command: addtotals ,addcoltotals
addtotals: Add totals across rows by default and also calculate total across columns when col=true
The addtotals command adds together the numeric fields in each search result.

You may specify which fields to include rather than summing all numeric fields.
The final total is stored in a new field.

The addtotals command's behavior is as follows:

When col=true, it computes the sum for every column and adds a summary row at the end containing those totals.

To label this final summary row, specify a labelfield and assign it a value using the label option.

Alternatively, instead of using the addtotals col=true command, you can use the addcoltotals command to calculate a summary event.

labelfield, if specified, is a field that will be added at the last row of the column specified by labalfield with the value set by the 'label' option.

Command Syntax:
addtotals [row=<bool>] [col=<bool>] [labelfield=<field>] [label=<string>] [fieldname=<field>] [<field-list>]
arguments description:
row: Syntax: row=<bool> . Indicates whether to compute the sum of the for each event. This works like generating a total for each row in a table. The result is stored in a new field, which is named Total by default. To use a different field name, provide the fieldname argument. Default value is true.

col : Syntax: col=<bool> . Indicates whether to append a new event—called a summary event—to the end of the event list. This summary event shows the total for each field across all events, similar to calculating column totals in a table. Default is false.

fieldname : Syntax: fieldname=<field> . Specifies the name of the field that stores the calculated sum of the field-list for each event. This argument is only applicable when row=true. Default is Total

field-list : Syntax: <field> ... . One or more numeric fields separated by spaces. Only the fields listed in the are included in the sum. If no is provided, all numeric fields are summed by default.

labelfield : Syntax: labelfield=<field> . Specifies a field to use as the label for the summary event. This argument is only applicable when col=true."

To use an existing field from your result set, provide its name as the value for the labelfield argument. For example, if the field is named salary, specify labelfield=salary. If no existing field matches the labelfield value, a new field is created using that value.

label: Syntax: label=<string>. Specifies a row label for the summary event.

If the labelfield argument refers to an existing field in the result set, the label value appears in that field for the summary row.

If the labelfield argument creates a new field, the label is placed in that new field in the summary event row. Default label is Total.

command addcoltotals: Add totals across columns of each row to show total in a new field.

addcoltotals: options
Optional Arguments
<field-list>
Syntax: <field> ... . A space-delimited list of valid field names. addcoltotals calculates sums only for the fields you include in this list. By default, the command calculates the sum for all fields.

labelfield: Syntax: labelfield=<fieldname>. Field name to add to the result set.

label : Syntax: label=<string> . Used together with the labelfield argument to add a label to the summary event. If labelfield is not specified, the label argument has no effect. Default label is Total.

Related Issues

Resolves #4607 [#4607 ]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[asifabashar]

@penghuo Please review

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (2)

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java (2)

77-118: Refactor to honor single-responsibility and 20-line guideline.

The verifyColTotals() method is 41 lines and performs multiple responsibilities: accumulating totals from data rows, verifying computed totals match expected values, and optionally checking summary event labels.

Per coding guidelines, methods should stay under 20 lines with single responsibility.

Consider extracting separate methods:

+  private BigDecimal[] computeColumnTotals(
+      org.json.JSONArray dataRows, List<Integer> fieldIndexes) {
+    BigDecimal[] columnTotals = new BigDecimal[fieldIndexes.size()];
+    for (int i = 0; i < dataRows.length() - 1; i++) {
+      var row = dataRows.getJSONArray(i);
+      for (int j = 0; j < fieldIndexes.size(); j++) {
+        int colIndex = fieldIndexes.get(j);
+        if (columnTotals[j] == null) {
+          columnTotals[j] = new BigDecimal(0);
+        }
+        columnTotals[j] = columnTotals[j].add(extractNumericValue(row, colIndex));
+      }
+    }
+    return columnTotals;
+  }
+
+  private BigDecimal extractNumericValue(org.json.JSONArray row, int colIndex) {
+    Object value = row.isNull(colIndex) ? 0 : row.get(colIndex);
+    if (value instanceof Integer) {
+      return new BigDecimal((Integer) value);
+    } else if (value instanceof Double) {
+      return new BigDecimal((Double) value);
+    } else if (value instanceof BigDecimal) {
+      return (BigDecimal) value;
+    } else if (value instanceof String && isNumeric((String) value)) {
+      return new BigDecimal((String) value);
+    }
+    return BigDecimal.ZERO;
+  }
+
   private void verifyColTotals(
-      org.json.JSONArray dataRows, List<Integer> field_indexes, String finalSummaryEventLevel) {
-
-    BigDecimal[] cColTotals = new BigDecimal[field_indexes.size()];
-    for (int i = 0; i < dataRows.length() - 1; i++) {
-      var row = dataRows.getJSONArray(i);
-
-      // Iterate through each field in the row
-      for (int j = 0; j < field_indexes.size(); j++) {
-
-        int colIndex = field_indexes.get(j);
-        if (cColTotals[j] == null) {
-          cColTotals[j] = new BigDecimal(0);
-        }
-        Object value = row.isNull(colIndex) ? 0 : row.get(colIndex);
-        if (value instanceof Integer) {
-          cColTotals[j] = cColTotals[j].add(new BigDecimal((Integer) (value)));
-        } else if (value instanceof Double) {
-          cColTotals[j] = cColTotals[j].add(new BigDecimal((Double) (value)));
-        } else if (value instanceof BigDecimal) {
-          cColTotals[j] = cColTotals[j].add((BigDecimal) value);
-
-        } else if (value instanceof String) {
-          if (org.opensearch.sql.calcite.remote.CalciteAddColTotalsCommandIT.isNumeric(
-              (String) value)) {
-            cColTotals[j] = cColTotals[j].add(new BigDecimal((String) (value)));
-          }
-        }
-      }
-    }
+      org.json.JSONArray dataRows, List<Integer> fieldIndexes, String finalSummaryEventLevel) {
+    BigDecimal[] columnTotals = computeColumnTotals(dataRows, fieldIndexes);
     var total_row = dataRows.getJSONArray((dataRows.length() - 1));
-    for (int j = 0; j < field_indexes.size(); j++) {
-      int colIndex = field_indexes.get(j);
+    for (int j = 0; j < fieldIndexes.size(); j++) {
+      int colIndex = fieldIndexes.get(j);
       BigDecimal foundTotal = total_row.getBigDecimal(colIndex);
-      assertEquals(foundTotal.doubleValue(), cColTotals[j].doubleValue(), 0.000001);
+      assertEquals(foundTotal.doubleValue(), columnTotals[j].doubleValue(), 0.000001);
     }
     if (finalSummaryEventLevel != null) {
       String foundSummaryEventLabel = total_row.getString(total_row.length() - 1);
-
       assertEquals(foundSummaryEventLabel, finalSummaryEventLevel);
     }
   }

As per coding guidelines, keep methods under 20 lines with single responsibility.

---

48-49: Fix variable naming to follow camelCase convention.

The variable field_indexes uses snake_case instead of camelCase throughout the file. Additionally, line 100 uses an unnecessary fully qualified class name.

Per coding guidelines, use camelCase for variable names.

Apply these changes:

1. Rename field_indexes to fieldIndexes throughout
2. Rename cColTotals to columnTotals for clarity
3. Remove fully qualified class name on line 100

-    ArrayList<Integer> field_indexes = new ArrayList<>();
-    field_indexes.add(0);
+    ArrayList<Integer> fieldIndexes = new ArrayList<>();
+    fieldIndexes.add(0);

   private void verifyColTotals(
-      org.json.JSONArray dataRows, List<Integer> field_indexes, String finalSummaryEventLevel) {
+      org.json.JSONArray dataRows, List<Integer> fieldIndexes, String finalSummaryEventLevel) {
 
-    BigDecimal[] cColTotals = new BigDecimal[field_indexes.size()];
+    BigDecimal[] columnTotals = new BigDecimal[fieldIndexes.size()];

-        } else if (value instanceof String) {
-          if (org.opensearch.sql.calcite.remote.CalciteAddColTotalsCommandIT.isNumeric(
-              (String) value)) {
-            cColTotals[j] = cColTotals[j].add(new BigDecimal((String) (value)));
-          }
-        }
+        } else if (value instanceof String && isNumeric((String) value)) {
+          columnTotals[j] = columnTotals[j].add(new BigDecimal((String) value));
+        }

As per coding guidelines, use camelCase for method and variable names.

Also applies to: 67-68, 78-80, 100-102, 133-134, 156-157, 193-194

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c874251 and 2d18f7e.

📒 Files selected for processing (3)

• core/src/main/java/org/opensearch/sql/ast/tree/AddColTotals.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (2)

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java
• core/src/main/java/org/opensearch/sql/ast/tree/AddColTotals.java

🧰 Additional context used

📓 Path-based instructions (5)

**/*.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

**/*.java: Use PascalCase for class names (e.g., QueryExecutor)
Use camelCase for method and variable names (e.g., executeQuery)
Use UPPER_SNAKE_CASE for constants (e.g., MAX_RETRY_COUNT)
Keep methods under 20 lines with single responsibility
All public classes and methods must have proper JavaDoc
Use specific exception types with meaningful messages for error handling
Prefer Optional<T> for nullable returns in Java
Avoid unnecessary object creation in loops
Use StringBuilder for string concatenation in loops
Validate all user inputs, especially queries
Sanitize data before logging to prevent injection attacks
Use try-with-resources for proper resource cleanup in Java
Maintain Java 11 compatibility when possible for OpenSearch 2.x
Document Calcite-specific workarounds in code

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

---

⚙️ CodeRabbit configuration file

**/*.java: - Verify Java naming conventions (PascalCase for classes, camelCase for methods/variables)

• Check for proper JavaDoc on public classes and methods
• Flag redundant comments that restate obvious code
• Ensure methods are under 20 lines with single responsibility
• Verify proper error handling with specific exception types
• Check for Optional usage instead of null returns
• Validate proper use of try-with-resources for resource management

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

integ-test/**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

End-to-end scenarios need integration tests in integ-test/ module

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

---

⚙️ CodeRabbit configuration file

integ-test/**/*IT.java: - Verify integration tests are in correct module (integ-test/)

• Check tests can be run with ./gradlew :integ-test:integTest
• Ensure proper test data setup and teardown
• Validate end-to-end scenario coverage

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

Name integration tests with *IT.java suffix in OpenSearch SQL

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

**/test/**/*.java

⚙️ CodeRabbit configuration file

**/test/**/*.java: - Verify test coverage for new business logic

• Check test naming follows conventions (*Test.java for unit, *IT.java for integration)
• Ensure tests are independent and don't rely on execution order
• Validate meaningful test data that reflects real-world scenarios
• Check for proper cleanup of test resources

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

**/calcite/**/*.java

⚙️ CodeRabbit configuration file

**/calcite/**/*.java: - Follow existing patterns in CalciteRelNodeVisitor and CalciteRexNodeVisitor

• Verify SQL generation and optimization paths
• Document any Calcite-specific workarounds
• Test compatibility with Calcite version constraints

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

🧠 Learnings (5)

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*IT.java : Name integration tests with `*IT.java` suffix in OpenSearch SQL

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test SQL generation and optimization paths for Calcite integration changes

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to integ-test/**/*IT.java : End-to-end scenarios need integration tests in `integ-test/` module

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : Document Calcite-specific workarounds in code

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : All public classes and methods must have proper JavaDoc

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (16)

• GitHub Check: security-it-linux (25)
• GitHub Check: security-it-linux (21)
• GitHub Check: security-it-windows-macos (macos-14, 21)
• GitHub Check: security-it-windows-macos (windows-latest, 21)
• GitHub Check: security-it-windows-macos (windows-latest, 25)
• GitHub Check: security-it-windows-macos (macos-14, 25)
• GitHub Check: build-windows-macos (macos-14, 25, doc)
• GitHub Check: build-windows-macos (macos-14, 25, integration)
• GitHub Check: build-windows-macos (macos-14, 25, unit)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 21, unit)
• GitHub Check: build-windows-macos (macos-14, 21, integration)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, unit)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 21, doc)
• GitHub Check: test-sql-cli-integration (21)

[coderabbitai]

Actionable comments posted: 0

♻️ Duplicate comments (1)

integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1)

252-265: Fix stale query_string copy‑paste comments for the new tests

Both new tests exercise addtotals/addcoltotals, but the inline comments still say “Test query_string without fields parameter on remote cluster”, which is misleading and was already flagged in an earlier review. Updating them (and slightly tightening the Javadoc) will make the intent clear:

   @Test
   public void testCrossClusterAddTotals() throws IOException {
-    // Test query_string without fields parameter on remote cluster
+    // Verify addtotals works on the remote bank index with row totals over age.
@@
-  /** CrossClusterSearchIT Test for addcoltotals. */
+  /** CrossClusterSearchIT test for addcoltotals on the remote bank index. */
   @Test
   public void testCrossClusterAddColTotals() throws IOException {
-    // Test query_string without fields parameter on remote cluster
+    // Verify addcoltotals works on the remote bank index with column totals for age and balance.

🧹 Nitpick comments (1)

integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1)

250-259: Cross‑cluster addtotals scenario looks solid; optionally assert schema

The query exercises addtotals on the remote bank index and verifyDataRows(result, rows("Nanette", 28, 28)) confirms the extra totals column is present and populated as expected for the first row. If you want to harden this further against regressions, consider also asserting the column names (including the totals field) with verifyColumn(...), similar to neighbouring tests.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2d18f7e and ad94472.

📒 Files selected for processing (1)

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1 hunks)

🧰 Additional context used

📓 Path-based instructions (4)

**/*.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

**/*.java: Use PascalCase for class names (e.g., QueryExecutor)
Use camelCase for method and variable names (e.g., executeQuery)
Use UPPER_SNAKE_CASE for constants (e.g., MAX_RETRY_COUNT)
Keep methods under 20 lines with single responsibility
All public classes and methods must have proper JavaDoc
Use specific exception types with meaningful messages for error handling
Prefer Optional<T> for nullable returns in Java
Avoid unnecessary object creation in loops
Use StringBuilder for string concatenation in loops
Validate all user inputs, especially queries
Sanitize data before logging to prevent injection attacks
Use try-with-resources for proper resource cleanup in Java
Maintain Java 11 compatibility when possible for OpenSearch 2.x
Document Calcite-specific workarounds in code

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

---

⚙️ CodeRabbit configuration file

**/*.java: - Verify Java naming conventions (PascalCase for classes, camelCase for methods/variables)

• Check for proper JavaDoc on public classes and methods
• Flag redundant comments that restate obvious code
• Ensure methods are under 20 lines with single responsibility
• Verify proper error handling with specific exception types
• Check for Optional usage instead of null returns
• Validate proper use of try-with-resources for resource management

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

integ-test/**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

End-to-end scenarios need integration tests in integ-test/ module

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

---

⚙️ CodeRabbit configuration file

integ-test/**/*IT.java: - Verify integration tests are in correct module (integ-test/)

• Check tests can be run with ./gradlew :integ-test:integTest
• Ensure proper test data setup and teardown
• Validate end-to-end scenario coverage

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

Name integration tests with *IT.java suffix in OpenSearch SQL

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

**/test/**/*.java

⚙️ CodeRabbit configuration file

**/test/**/*.java: - Verify test coverage for new business logic

• Check test naming follows conventions (*Test.java for unit, *IT.java for integration)
• Ensure tests are independent and don't rely on execution order
• Validate meaningful test data that reflects real-world scenarios
• Check for proper cleanup of test resources

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

🧠 Learnings (2)

📓 Common learnings

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Verify AST generation for new PPL parser syntax

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test SQL generation and optimization paths for Calcite integration changes

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*IT.java : Name integration tests with `*IT.java` suffix in OpenSearch SQL

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (28)

• GitHub Check: bwc-tests-full-restart (25)
• GitHub Check: build-linux (21, integration)
• GitHub Check: build-linux (25, unit)
• GitHub Check: build-linux (21, unit)
• GitHub Check: build-linux (21, doc)
• GitHub Check: build-linux (25, integration)
• GitHub Check: bwc-tests-rolling-upgrade (25)
• GitHub Check: build-linux (25, doc)
• GitHub Check: bwc-tests-rolling-upgrade (21)
• GitHub Check: bwc-tests-full-restart (21)
• GitHub Check: security-it-linux (21)
• GitHub Check: security-it-linux (25)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, unit)
• GitHub Check: security-it-windows-macos (macos-14, 25)
• GitHub Check: security-it-windows-macos (macos-14, 21)
• GitHub Check: security-it-windows-macos (windows-latest, 25)
• GitHub Check: security-it-windows-macos (windows-latest, 21)
• GitHub Check: build-windows-macos (macos-14, 21, doc)
• GitHub Check: build-windows-macos (macos-14, 25, unit)
• GitHub Check: build-windows-macos (macos-14, 25, integration)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 25, doc)
• GitHub Check: build-windows-macos (macos-14, 21, unit)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, unit)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 21, integration)
• GitHub Check: CodeQL-Scan (java)
• GitHub Check: test-sql-cli-integration (21)

[dai-chen]

As I understand, we're translating to SQL Union query, could you explain bit where the most complexity comes from?

[asifabashar]

Calcite auto converts a int to long and float to double for aggregation result type. So union to a existing data of int or float type causes a type cast exception as same column with two different types are being computed to union. So this method get the type we need to cast to existing dataset so a union of two dataset a) original b) aggregated total row works without a cast exception. The cast exception can be reproduced if we remove method getAggregateDataTypeFieldRef and run CrossClusterSearchIT test.

[dai-chen]

Could you give me an example? Based on my test with Calcite SQL CLI below, I wonder is this problem only in RelNode?

0: jdbc:calcite:model=src/test/resources/mode> WITH q AS (SELECT gender, COUNT(*) AS cnt FROM account GROUP BY gender)
. . . . . . . . . . . . . . . . . . semicolon> SELECT gender, cnt FROM q
. . . . . . . . . . . . . . . . . . semicolon> UNION ALL
. . . . . . . . . . . . . . . . . . semicolon> SELECT 'Total', SUM(cnt) FROM q;
+--------+-----+
| GENDER | CNT |
+--------+-----+
| F      | 1   |
| M      | 3   |
| Total  | 4   |
+--------+-----+
3 rows selected (0.053 seconds)

[asifabashar]

@dai-chen Please check a my forked branch Pull request to test no casting use case , which I just created to test the scenerio where I commented casting for int, float type. The error is coming form CrossClusterSearchIT test, so basically it only appears when multi node results are combined where it goes through a different code path. The server side exception for the run is here .

git workflow for CrosClusterSearchIT

Exception stacktrace is below
=== Standard error of node node{:integ-test:integTestWithSecurity-0} ===
» ERROR][o.o.s.p.r.RestPPLQueryAction] [integTestWithSecurity-0] Error happened during query handling
» java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long (java.lang.Integer and java.lang.Long are in module java.base of loader 'bootstrap')
» at org.apache.calcite.avatica.util.AbstractCursor$LongAccessor.getLong(AbstractCursor.java:633) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.apache.calcite.avatica.AvaticaSite.get(AvaticaSite.java:310) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.apache.calcite.avatica.AvaticaResultSet.getObject(AvaticaResultSet.java:395) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.util.JdbcOpenSearchDataTypeConvertor.getExprValueFromSqlType(JdbcOpenSearchDataTypeConvertor.java:69) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.buildResultSet(OpenSearchExecutionEngine.java:236) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$1(OpenSearchExecutionEngine.java:208) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:223) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:204) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.QueryService.lambda$executeWithCalcite$0(QueryService.java:108) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.calcite.CalcitePlanContext.run(CalcitePlanContext.java:121) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.QueryService.executeWithCalcite(QueryService.java:98) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.QueryService.execute(QueryService.java:75) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:66) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$schedule$0(OpenSearchQueryManager.java:64) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$0(OpenSearchQueryManager.java:89) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) [opensearch-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
» at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) [?:?]
» at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) [?:?]
» at java.base/java.lang.Thread.run(Thread.java:1474) [?:?]

[LantaoJin]

I am okey for current workaround. Let me create a followup issue to optimize it. @coderabbitai can you help to create a github issue for tracking?

[coderabbitai]

🧩 Analysis chain

@LantaoJin, I'll create a GitHub issue to track this optimization. Let me gather the necessary information and create it for you.

---

🏁 Script executed:

#!/bin/bash

# Create a GitHub issue for tracking the type casting workaround
gh issue create \
  --repo opensearch-project/sql \
  --title "Optimize type casting workaround in addtotals/addcoltotals aggregation logic" \
  --body "## Description

A workaround was implemented in PR #4754 to handle type casting issues when performing UNION operations between original data and aggregated totals in the addtotals/addcoltotals commands.

## Current Implementation

The method \`getAggregateDataTypeFieldRef\` (line 2529 in \`CalciteRelNodeVisitor.java\`) explicitly casts:
- INTEGER → BIGINT
- FLOAT/REAL → DOUBLE

This workaround prevents \`ClassCastException\` that occurs during UNION operations when Calcite automatically converts aggregation result types.

## Problem

When combining original data (with int/float types) and aggregated totals (Calcite converts to long/double), a \`ClassCastException\` occurs in multi-node scenarios:

\`\`\`
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long
  at org.apache.calcite.avatica.util.AbstractCursor\$LongAccessor.getLong
  at org.opensearch.sql.opensearch.util.JdbcOpenSearchDataTypeConvertor.getExprValueFromSqlType
\`\`\`

This issue only manifests in CrossClusterSearchIT tests when multi-node results are combined.

## Expected Optimization

Investigate whether:
1. This is a Calcite bug that should be reported upstream
2. There's a better way to handle type compatibility in UNION operations
3. The type conversion can be handled more elegantly at a different layer

## Context

- Introduced in: PR #4754
- Discussion: https://github.com/opensearch-project/sql/pull/4754#discussion_r2612169694
- Test case demonstrating the issue: https://github.com/asifabashar/sql/pull/8/files

## Related Code

\`core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java\`:
- \`getAggregateDataTypeFieldRef\` method (line 2529)
- \`buildAddRowTotalAggregate\` method usage" \
  --assignee LantaoJin

Length of output: 53

---

@LantaoJin, I've created GitHub issue #4952 to track the optimization of the type casting workaround in the addtotals/addcoltotals aggregation logic.

The issue includes:

• Description of the current workaround in getAggregateDataTypeFieldRef
• Details about the ClassCastException that occurs in multi-node scenarios
• Context from this PR and discussion
• Suggestions for potential optimization approaches
• Links back to this PR and the test case demonstrating the issue

The issue has been assigned to you for follow-up investigation.

---

🧬 Code Graph Analysis Results

• File: core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

  • Snippet/summary: forceProjectExcept (private static)

    • Purpose: Implements a forced-exclusion projection by removing specific expressions from the project, ensuring the remaining fields align with the expected output after modifications. Used as a workaround to ensure correct field ordering when nested or deduplicated fields appear in the Calcite plan.
    • Parameters: (RelBuilder relBuilder, Iterable expressions)
      • Behavior: Builds a full projection list, removes the provided expressions from that list, and applies project(...) to enforce the desired field set.
    • Returns/Effects: Modifies the RelBuilder state to project only the desired fields, aiding correctness when Calcite’s default projection could misalign fields after transformations.
    • Key details: Used in scenarios where dedup/nesting changes require explicit control over which fields are emitted in the final projection.
    • Location hint: Related to a workaround for a Calcite projection ordering issue within the class (non-exported helper method).

  • Snippet/summary: visitDedupe (override)

    • Purpose: Handles the deduplication operation (Dedupe node) by delegating to internal buildDedup* helpers and validating options.
    • Behavior:
      • Extracts options: allowedDuplication, keepEmpty, consecutive.
      • Validates constraints (e.g., allowedDuplication > 0, non-consecutive is required).
      • Delegates to either buildDedupOrNull or buildDedupNotNull depending on keepEmpty, producing a windowed/filtered plan that deduplicates rows.
    • Exceptions/Errors: Throws IllegalArgumentException for invalid dup count or unsupported options (e.g., consecutive dedup not allowed).
    • Key details: Integrates with the rest of the plan-building to ensure dedup results are produced with correct null handling and ordering.
    • Location hint: Central logic for the Calcite-based deduplication path.

  • Snippet/summary: buildDedupOrNull (private static)

    • Purpose: Builds a deduplication plan that keeps nulls (keepempty behavior) by inserting a ROW_NUMBER window, filtering on nulls or row_number, and dropping the helper column.
    • Parameters: (CalcitePlanContext context, List dedupeFields, Integer allowedDuplication)
    • Behavior:
      • Adds a ROW_NUMBER() OVER (PARTITION BY dedupeFields ORDER BY dedupeFields) as a dedup marker.
      • Filters: isNull(a) OR isNull(b) OR row_number_dedup <= n to implement dedup with possible null retention.
      • Drops the helper column (row_number_dedup) after filtering.
    • Return/Effects: Emits a projected plan that preserves nulls per configuration and enforces dedup limits.
    • Location hint: Part of the deduplication workaround logic in CalciteRelNodeVisitor.

  • Snippet/summary: buildDedupNotNull (private static)

    • Purpose: Builds a deduplication plan that enforces non-null constraints on dedup fields and uses a ROW_NUMBER window for dedup.
    • Parameters: (CalcitePlanContext context, List dedupeFields, Integer allowedDuplication, boolean fromJoinMaxOption)
    • Behavior:
      • Applies isNotNull filters on the dedupe fields (and on any aggregated/related fields when applicable).
      • Creates a ROW_NUMBER() OVER (PARTITION BY dedupeFields ORDER BY dedupeFields) aliasing to either ROW_NUMBER_COLUMN_FOR_JOIN_MAX_DEDUP or ROW_NUMBER_COLUMN_FOR_DEDUP depending on fromJoinMaxOption.
      • Filters by row_number_dedup <= allowedDuplication, then drops the dedup helper column.
    • Return/Effects: Produces a plan that deduplicates while enforcing non-null constraints and maintains correct field output.
    • Location hint: Part of the same dedup workaround path as buildDedupOrNull.

  • Snippet/summary: containsSubqueryExpression (private)

    • Purpose: Utility to determine if any part of a given Node tree contains a SubqueryExpression.
    • Parameters: Node expr
    • Return: boolean
    • Behavior: Recursively traverses the node and its children to detect SubqueryExpression nodes.
    • Location hint: Used by dedup/regex/eval-related paths to decide if subquery correlation handling is needed.

  • Snippet/summary: visitRegex (override)

    • Purpose: Builds a regex-based filter by translating a field and a pattern into a RexNode condition.
    • Notable interplay: Validates field type, constructs REGEXP_CONTAINS call, supports negation, and applies a filter.
    • Location hint: Part of the visitor methods that translate high-level patterns into Calcite Rex operations, related to the user’s note about a possible bug/workaround around conversion behavior.

  • Snippet/summary: visitAppendPipe (override)

    • Purpose: Merges a subquery with a main plan, by resolving the subquery plan, visiting, building a sub-pipeline, and then merging.
    • Behavior: Attaches child sub-pipeline to main pipeline, then merges with mergeTableAndResolveColumnConflict (which unifies schemas and resolves conflicts).
    • Location hint: Demonstrates a path where a workaround in plan alignment might be needed when combining sub-queries and main queries, relevant to discussion about conversion/PR impact.

  • Snippet/summary: mergeTableAndResolveColumnConflict (private)

    • Purpose: Reuses a shared schema unifier (SchemaUnifier) to align two RelNode schemas and then unions them.
    • Behavior: Builds unified schema with conflict resolution, pushes projections, and unions.
    • Location hint: Central to how plan components with potential schema conflicts are merged, potentially impacted by conversion behavior or bug-workaround discussions.

Note: The above entries refer to conceptual snippets within CalciteRelNodeVisitor.java that are relevant to the user’s comment about a workaround/bug and a potential follow-up issue. If you’d like, I can extract or reformat exact code blocks from the file with precise line numbers.

---

🧠 Learnings used

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Follow existing patterns in `CalciteRelNodeVisitor` and `CalciteRexNodeVisitor` for Calcite integration

---

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test SQL generation and optimization paths for Calcite integration changes

---

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : Document Calcite-specific workarounds in code

[dai-chen]

Hi @LantaoJin Just to clarify, I'm not concerned about this specific cast issue. I’m trying to understand the complexity in this RelNode translation and would like to collect some data points. As you can see from the example above, the PPL → SQL translation is very straightforward, but the current RelNode visitor logic is hard to maintain. Just wonder do we already have an issue tracking the discussion around PPL → SqlNode vs. the current PPL → RelNode discussion? Probably we can do some overall evaluation later.

[asifabashar]

@dai-chen I have applied the recommended changes, Please review. Please check the use case #4754 (comment)