Is your feature request related to a problem or challenge?
The spec defines metadata columns, one of which is the _file column. This column seems instrumental for write operations positional deletes in future, as well as for having some ways for users to identify rows (combination of filepath + position)
Describe the solution you'd like
The solution should allow building a TableScan with option to return metadata columns (in this case _file). When that happens, the library should return batches that include this extra column. The column should be encoded in such a way that it doesn't add much memory overhead. Since arrow doesn't support constant vectors, perhaps RLE could be used.
Willingness to contribute
I can contribute to this feature independently
Is your feature request related to a problem or challenge?
The spec defines metadata columns, one of which is the
_filecolumn. This column seems instrumental for write operations positional deletes in future, as well as for having some ways for users to identify rows (combination of filepath + position)Describe the solution you'd like
The solution should allow building a TableScan with option to return metadata columns (in this case
_file). When that happens, the library should return batches that include this extra column. The column should be encoded in such a way that it doesn't add much memory overhead. Since arrow doesn't support constant vectors, perhaps RLE could be used.Willingness to contribute
I can contribute to this feature independently