fix: improve CSV header validation and error messages

[Prathamesh9284]

Summary

Closes #690

Improves CSV error handling in the SQL API filesystem source (JavaCSVTableSource) to provide clear, actionable error messages when CSV files are malformed or misconfigured.

Changes

• Refactored streamLines to read and validate header in a single file open: The header line is consumed via iterator.next() and validated before streaming data rows, avoiding opening the file twice.
• Added validateHeaderLine method (static): Validates the CSV header before data parsing — checks that each column follows the name:type format and that the comma-separated column count matches the number of name:type pairs (detecting wrong separator usage).
• Improved error message in parseLine: Distinct data row error showing expected vs actual column count, the separator used, and the offending line.
• Added empty file detection in streamLines: Throws a clear error if the CSV file has no lines at all.

Context

Calcite's CSV adapter requires a typed header row (e.g., id:int,name:string,email:string) using commas, while data rows use Wayang's configurable separator (default ;). Since the header always uses commas, header validation is static and doesn't need the instance-level separator.

Without proper validation, the previous errors were unclear and came too late (during data parsing). The new errors clearly explain the issue at the right stage:

• CSV file '...' is empty. Expected a header row (e.g., 'id:int,name:string').
• CSV file '...': header column 'NAMEA' missing required type. Expected 'name:type' format (e.g., 'id:int'). Header: 'NAMEA,NAMEB,NAMEC'.
• CSV file '...': column count mismatch. Expected 4 comma-separated 'name:type' columns but found 1. Header: 'id:int;name:string;email:string;country:string'.
• CSV file '...': data row has 2 columns but expected 3 (separator ';'). Line: 'test1;1'.

[mspruc]

Thanks for your contribution! Left some comments for your PR, I prefer if we keep the accessors as-is so we don't accidentally deprecate anything people might be using.

[Prathamesh9284]

Hi @mspruc @zkaoudi

I’ve updated the PR based on your feedback to keep the existing methods and signatures the same. I moved the validateHeaderLine logic to createStream instead of changing streamLines.

Please let me know if anything else needs to be improved.

[mspruc]

@zkaoudi can you verify this with your example?

[zkaoudi]

I checked with the right file and it works.

I also checked with an incorrect file which has this heading:
id:int;name:string;email:string;country:string

And got this error message: "Column count mismatch in CSV file 'file:///Users/zoi/Work/WAYANG/wayang-examples/src/main/resources/input/customers.csv': expected 1 columns but found 4 (separator ';'). Line: '1;Alice Johnson;alice@example.com;USA'. Ensure the header uses 'name:type' format with commas and data rows use ';' as delimiter."

The line it prints is the second line of my file, not the header line. Can we print the header line?

[Prathamesh9284]

Hi @zkaoudi @mspruc,

I've updated the PR to handle this case. With the CSV file you shared using the header id:int;name:string;email:string;country:string, the error now correctly identifies the header issue:

CSV file 'file:///Users/zoi/Work/WAYANG/wayang-examples/src/main/resources/input/customers.csv': header uses ';' as separator, but Calcite requires commas. Header: 'id:int;name:string;email:string;country:string'. Expected format: 'id:int,name:string,email:string,country:string'.

It now prints the header line instead of the data line and clearly tells the user what to fix.

[zkaoudi]

Would it be possible to do the check directly when we are reading the file? We are now opening the file twice which could be costly?

[Prathamesh9284]

@zkaoudi since streamLines() is static and is the place where file opening and iterator creation are already defined, I considered it the appropriate location to perform header validation. However, because it is static, it cannot access the instance-level separator, and we cannot modify its signature or behavior to pass the separator or expose the header.

Given these constraints, performing header validation within the same file-open operation would require changing streamLines(), which @mspruc wanted to avoid to preserve the existing definition. As a result, the file is currently opened twice.

Is there something we can do here to avoid the double file open while keeping the existing structure intact? I’d appreciate your guidance.

[zkaoudi]

why do we need to access the separator? the separator for the header should always be a comma, right?

[Prathamesh9284]

You're right, I got confused here the header separator is always a comma, so the core validation doesn't actually need the instance-level separator. I was using it only for a more descriptive error message, but that's not essential.

[Prathamesh9284]

Hi @zkaoudi @mspruc,

I've refactored validateHeaderLine to be static and moved it into streamLines(), so the file is only opened once. The header is consumed via the iterator before streaming data rows.

Here are the error messages for each case:

1. Empty CSV file

CSV file 'customers.csv' is empty. Expected a header row (e.g., 'id:int,name:string').

2. Header missing types (e.g., NAMEA,NAMEB,NAMEC)

CSV file 'customers.csv': header column 'NAMEA' missing required type. Expected 'name:type' format (e.g., 'id:int'). Header: 'NAMEA,NAMEB,NAMEC'.

3. Header uses wrong separator (e.g., id:int;name:string;email:string;country:string)

CSV file 'customers.csv': column count mismatch. Expected 4 comma-separated 'name:type' columns but found 1. Header: 'id:int;name:string;email:string;country:string'.

4. Data row has wrong number of columns (e.g., test1;1 in a 3-column table)

CSV file 'customers.csv': data row has 2 columns but expected 3 (separator ';'). Line: 'test1;1'.

[zkaoudi]

looks good! thank you :)