Files without `.parquet`, `.csv` extension inferred as having no schema

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I wrote a test approximating

let file = tempfile::tempfile();

// ... write parquet data ...

let mut context = ExecutionContext::new();
context.register_parquet("t", file.path().as_str())
context.sql("select column from t");

This would result in "Invalid identifier" errors, effectively claiming the column didn't exist. I verified the file existed, had the correct columns, etc... I was very confused 😆

Eventually I tracked this down to the schema being inferred as empty if the extension is not ".parquet", this feels unexpected

Describe the solution you'd like

Either register_parquet should return an error if the extension is missing, or FileFormat::infer_schema should be more agnostic to file extensions.

[alamb]

Change this description to be a bug, which I think better reflects what is going on

[Ted-Jiang]

@tustvold may i ask you how to write a tmp parquet file, IMHP i think all parquet data file should end .parquet.

[tustvold]

@Ted-Jiang Remove this line in the SQL benchmark https://github.com/apache/arrow-datafusion/pull/1738/files#diff-d1dbff8af63c3a3fe4d918432f982181b40fa4b7e1641522a6a48904f521fc89R143 and that was what caused issues.

I don't feel strongly whether the .parquet file should be mandatory or not, but I do think silently inferring an empty schema makes for a pretty poor UX 😅

[alamb]

but I do think silently inferring an empty schema makes for a pretty poor UX 😅

I agree -- an error about "can not infer schema" seems much better than silently ignoring

[andygrove]

I plan on fixing this for 14.0.0

[andygrove]

Some notes from investigating this.

This is where the EmptyExec gets introduced. I ran into this issue in Ballista as well in apache/datafusion-ballista#414

        // if no files need to be read, return an `EmptyExec`
        if partitioned_file_lists.is_empty() {
            let schema = self.schema();
            let projected_schema = project_schema(&schema, projection.as_ref())?;
            return Ok(Arc::new(EmptyExec::new(false, projected_schema)));
        }

The problem is I am registering a CSV data source and the filename does not end with .csv and we have hard-coded assumptions that CSV files must end with ".csv", so we cannot support .tsv or .tbl (without the user explicitly specifying this, and it is not possible to specify this in SQL as far as I know).

I think we just need to add an error check for this case and prompt the user to specify the file extension they want rather than use the default.

[andygrove]

I propose that the error check we add is that at least one file exists with the specified extension.