fread to recognize "mdy" dates?

[andersmik]

Would there be interest in making 'fread' recognize dates stored in the format "month-day-year" as dates? The format is very common the datasets I work with. At the moment fread imports these as strings.
Thanks for a great package.

-Anders

[eddelbuettel]

It is a very slippery slope. For formatless parsing ISO 8601 is preferred by base R too. With "month-day-year" how do you know it is not "day-month-year" from another part of the world? What is "01-02-2020"? Jan 2 or Feb 1?

[andersmik]

I see your point. Possibly an optional argument specifying the date format to look for: ymd, mdy, dmy?

[MichaelChirico]

I don't think we'll do this for the reason Dirk mentioned. Basically, I think it's out of scope for data.table. What tool is producing these data sets? I would try and focus on getting better-normalized data as input to the extent possible.

please see also #3279 which will help to improve the performance of as.IDate in the event fread can't do it.

[andersmik]

Thanks for the consideration. So the dataset are from national administrative data in Denmark stored in .sas7bdat format, then converted to .csv using StatTransfer. This is not optimal, but i am not in a position to change the format in which these data are stored. Although when they are imported, DT is great for handling these datasets, some are >100gb.
Anders

[jangorecki]

I think the best way to go, which you are probably already using, is to parse into character and then convert to date using some fast date parsing R package.

[andersmik]

I use the lubridate::mdy() function, then the as.IDate function. This works fine and is fast enough. Though not effortless when working with many datasets, with many date columns.

[MichaelChirico]

Have you tried haven::read_sas to skip the StatTransfer step? I am not sure about the columns' data types in SAS, but it's possible the StatTransfer step is responsible for creating the wonky date format, and that haven would get around this by reading a Date directly.