CSVY wishlist

[MichaelChirico]

Filing a follow-up to #1701 & #3534 which handled basic implementation of CSVY support to centralize a wish list for more advanced functionality for CSVY:

Both fread and fwrite

• Support for writing a factor's levels & then reading them
• yaml = '/path/to/file' -- store/read metadata stored outside the file itself
• yaml = 'auto' by default -- read/write csvy files automatically by detecting the csvy extension

fread

• Fully sync fread arguments with what's output from fwrite by default
• Expand recognized colClasses following Various colClasses enhancements #2545
• auto yaml detection using file contents even if extension is not .csvy

fwrite

• yaml = list() -- supply custom key/value pairs to be included in the metadata

[rossholmberg]

I'd love to see more user-oriented metadata. Description/note options would be fantastic for storing something along the lines of a brief data readme. I played with something like this myself a while ago (it's all on GitHub), and used an object attribute to store these things in the R object. Here's a section of my readme showing what I mean:
https://github.com/rossholmberg/rcsv/blob/master/README.md#notes-an-added-detail-to-your-datasets

I only had "notes" for the table as a whole, but per-column would be even better.

[MichaelChirico]

as in, how to provide customized metadata right? if so yes!! (if not please correct me 😁) something I've thought about but I'm not sure the best API for enabling this. could be that we force user to provide an attribute but that feels inconvenient.

[jangorecki]

I would lean towards generalising to store DT/cols attributes in yaml metadata. Then we don't need to worry if it is a column attribute, or a table attribute, what is its purpose. This way we handle factor levels, datetime origin/timezone etc. with column attributes, and sorted attribute of a data.table.
Handling indexes is possible (adding technical __index_cols integer columns) but probably is not worth as it adds relatively a lot of space, some read/write time, and at the same time it is something that can be easily and pretty fast computed.

[MichaelChirico]

Makes sense. Agree we shouldn't be writing the index vector, but it would be useful to have a sortedBy attribute (or similar) that can indicate the data is already sorted by some columns.

[rossholmberg]

I agree Jan, that would be great to be able to store attributes at both DT and col levels.
Yes Michael, customised metadata is exactly what I mean. A few use cases are data source attribution, method notes, measurement units, notes about equipment used, explanations of coded variables, etc. I use the description attributes in SQL a lot for these things, and I find having them stored alongside the data itself to be more tidy than in a separate metadata file.

[eliocamp]

I've recently stumbled upon the need to store custom fields in csvy metadata and fully support this. I think something like a yaml.exta parameter which accepts a list to be appended to the current automatically-generated list would make sense.

[eliocamp]

This code works as advertised.

eliocamp@dc4c578

library(data.table)

data <- data.table(x = 1, y = 100)
file <- tempfile()

fwrite(data, file, yaml =  list(something_else = "o hai!"))

x <- fread(file, yaml = TRUE)
attr(x, "yaml_metadata")$something_else
#> [1] "o hai!"

^{Created on 2019-12-16 by the reprex package (v0.3.0)}

I can start a PR, but I would need some guidance to write tests for this functionality, since I don't understand how tests work in data.table.

[MichaelChirico]

@eliocamp that would be great! Something like that is on my radar to add when I get back to this.

Have you seen the section on testing in the Contributing wiki?

https://github.com/Rdatatable/data.table/wiki/Contributing

[eliocamp]

Thanks! There it is.