fread feather

[mattdowle]

As suggested here to avoid needing to use or wrap with setDT :
https://twitter.com/bennetvoorhees/status/830070242659414016
(I guess that rio returns a data.frame or tibble, so making fread do it is perhaps clearer as people use fread to return data.table.)

[MichaelChirico]

I think a separate package like rio is better suited for this -- minimally, a package of simple wrappers to communicate among data.table and other non-CSV data types (e.g. Parquet #2505). Worth considering to build such a package maybe as an add-on within Rdatatable org but not in core data.table.

Of course feel free to re-open if you think otherwise @mattdowle :)

[mattdowle]

Yes I agree in general rio is better place for this. But the wrinkle is that rio returns a data.frame. So it's inconvenience to users who like fread returning a data.table by default. I got the impression that was the gist motivating Bernard's tweet.

[DrMaphuse]

I would really love to see more support for this. We see ourselves increasingly moving away from R simply because our preferred dataframe package (data.table) needs to use setDT before working with parquet files.

The way things are now, freading .csv files is faster than reading and converting parquet files, despite the latter being the superior format in every aspect. In the age of delta lakes and similar modern data science infrastructure, this feels somewhat anachronistic and it represents a significant bottleneck when working with large datasets.