parsing error when reading from filesystem with the SQL api

[zkaoudi]

When trying to run a simple SQL query to read data from a CSV file I am getting a parsing error because of this check:

wayang/wayang-api/wayang-api-sql/src/main/java/org/apache/wayang/api/sql/sources/fs/JavaCSVTableSource.java

Line 127 in 29bde26

if (tokens.length != fieldTypes.size())

After running on debugging mode I see that in the WayangTableScanVisitor the list of fieldTypes has size equal to 1 although I have four columns in my csv file.

wayang/wayang-api/wayang-api-sql/src/main/java/org/apache/wayang/api/sql/calcite/converter/WayangTableScanVisitor.java

Line 67 in 29bde26

final List<RelDataType> fieldTypes = wayangRelNode.getRowType().getFieldList().stream()

[Prathamesh9284]

Hi @zkaoudi

My name is Prathamesh Dhanashri, and I'd like to introduce myself as a new contributor to Apache Wayang. I'm interested in contributing to the project as part of Google Summer of Code (GSoC) and am looking to familiarize myself with the codebase by working on open issues.

As a starting point, I have been investigating this issue, and here is what I found:

---

Root Cause

In WayangTableScanVisitor.java (line 67), the fieldTypes list is built using:

wayangRelNode.getRowType()

This returns the RelNode’s row type, which in certain cases may contain fewer fields than the actual table schema (e.g., when Calcite optimizes away unused columns).

However, the CSV source always reads all columns from disk, leading to a mismatch between tokens.length and fieldTypes.size(), which causes runtime issues.

---

Proposed Fix

Update line 67 in WayangTableScanVisitor.java:

From:

final List<RelDataType> fieldTypes =
    wayangRelNode.getRowType().getFieldList().stream()

To:

final List<RelDataType> fieldTypes =
    wayangRelNode.getTable().getRowType().getFieldList().stream()

Using getTable().getRowType() ensures we always retrieve the full table schema, which aligns with how getColumnNames() is implemented in WayangTableScan.java (line 98).

Column pruning is still correctly handled downstream by the WayangProject operator via a MapOperator.

---

Testing

I added a regression test using Mockito that:

• Simulates a WayangTableScan with a trimmed row type (1 field)
• While the underlying table schema contains 4 fields
• Reproduces the exact scenario described in this issue

The test:

• Fails before the fix
• Passes after applying the fix
• All existing tests continue to pass

---

If this approach looks correct, I’ll proceed with opening a PR including the fix and the regression test.

Looking forward to feedback!

[zkaoudi]

Hi @Prathamesh9284 , thank you for your interest in our project. It seems reasonable what you propose.
Maybe @mspruc who has been working on the sql api can provide more input.

[zkaoudi]

for completeness, here is my sample csv file:

1;Alice Johnson;alice@example.com;USA
2;Bob Lee;bob@example.com;Canada
3;Carla Díaz;carla@example.com;Spain
4;David Kim;david@example.com;Korea
5;Emma Brown;emma@example.com;UK
6;John Miller;john@example.com;Germany

[mspruc]

@Prathamesh9284 I think based off of @zkaoudi 's CSV, this error happens because Calcite needs types in the CSV header, although I agree that the error message is unclear. Maybe you could figure out some way to make the exception clearer? And perhaps guard against CSV files not having headers or headers with the correct schema that Calcite expects?

[zkaoudi]

thanks @mspruc. I added this header

id:int,name:string,email:string,country:string

and it worked!

We also need to homogenize the separators as well. The default is ; but for the header we are expecting a comma.

[Prathamesh9284]

Thanks @mspruc and @zkaoudi for the additional context! Here's my plan:

1. Improve the error message at JavaCSVTableSource.java line 127 to include expected vs actual column count and a hint about the required Calcite header format (name:type).
2. Add early header validation to detect missing or malformed headers before parsing begins.
3. Address the separator inconsistency — The header is parsed by Calcite's CsvEnumerator which hardcodes commas, while data rows use Wayang's configurable separator. I'll add validation to detect this mismatch early with a clear error message.

Does this approach sound right?

[zkaoudi]

sounds great! thanks a lot @Prathamesh9284 :)

[Prathamesh9284]

Hi @zkaoudi @mspruc,

I've opened a PR with the changes we discussed.

Could you please review it when you have a chance? I'm happy to make any adjustments based on your feedback.

Thanks!