Add parameter to fread to read numbers with leading zeros as character

[marc-outins]

Add a parameter or option to not ignore leading zeros when reading data with fread. I have data that contains numeric looking data that has leading zeros that I would like fread to read as character. I've seen other people asking about this on stack overflow and the some of the main responses are to set colClasses = "character" so all columns are read as character or specifically call out the columns that need to be read as character. These options aren't great if there are lots of columns with this issue along with other columns that should be read in as non-character. I've never dealt with data that looks like "0300" that really represents 300 so I would by default like to read columns that contain data with leading 0's as character.

# Minimal Reproducible Example

library(data.table)

dt <- fread(input = "0300\n")
str(dt)

Classes ‘data.table’ and 'data.frame':	1 obs. of  1 variable:
 $ V1: int 300
 - attr(*, ".internal.selfref")=<externalptr>

# Output of sessionInfo()

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] doSNOW_1.0.16     snow_0.4-2        iterators_1.0.10  foreach_1.4.4     purrr_0.2.5       stringi_1.2.4    
 [7] stringr_1.3.1     fuzzyjoin_0.1.4   data.table_1.11.4 rvest_0.3.2       xml2_1.2.0       

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.18     compiler_3.5.1   pillar_1.3.0     bindr_0.1.1      tools_3.5.1      digest_0.6.15   
 [7] evaluate_0.11    tibble_1.4.2     pkgconfig_2.0.1  rlang_0.2.1      rstudioapi_0.7   curl_3.2        
[13] yaml_2.1.19      bindrcpp_0.2.2   dplyr_0.7.6      httr_1.3.1       knitr_1.20       hms_0.4.2       
[19] rprojroot_1.3-2  tidyselect_0.2.4 glue_1.3.0       R6_2.2.2         rmarkdown_1.10   readr_1.1.1     
[25] selectr_0.4-1    magrittr_1.5     backports_1.1.2  codetools_0.2-15 htmltools_0.3.6  assertthat_0.2.0
[31] crayon_1.3.4    

[gsgxnet]

i can second that request from my experience with CSV files. Quite often numeric ids/keys in a source database are exported with leading zeros. Because thats their data dictionary definition in the database. E.g. a customer number is 123456 and it is defined as 10 digits in the db. In the CSV export this would be done as 0000123456. Most times there is no use at all for a conversion into a numeric column of a customer number when imported.

Only culprit i can see is a column containing just a 0. Should be treated as numeric? Or is this a leading 0 for ?

[jangorecki]

are you able to insert a row having id 9999999999+1 and see what csv extract will look like? characters are safer to store ID as you won't hit int32 limit. They are safer in terms of in/out data from db-other tools, inside system/db integers should be preferred.

[st-pasha]

If there is an int that is too large to fit into int64, the entire column will be converted into character type (and in that case leading zeros are preserved):

> fread("A\n001\n12345678901234567890")
                      A
                 <char>
1:                  001
2: 12345678901234567890

The only time when the problem arises is when all IDs are sufficiently small, and have leading 0s.

[marc-outins]

The issue is not only related to id's. We deal with data that includes NDC codes which contain leading zeros. One of the data sets we get from the National Cancer Institute has a table that contains over 2000 variables. Some of those variables contain coded data with leading 0's (01, 02, etc) so manually adding colClasses to all them would be difficult

@st-pasha For all the data I've encountered I don't think I've run into ids bigger than int64 allows so for all of my cases I'll never have fread pick the right column type for my data with leading zero's.

It just seems like if fread is going to guess column types it shouldn't remove data (leading zero's) especially without warning the user that it did that. I'm curious if any of you have run into data with leading zero's that you would want fread to read in as a numeric?

[st-pasha]

We thought about these issues, and even contemplated adding an option, but in the end, it fell off from our radars somehow.
The use case in favor of retaining 0s are zip-codes and user IDs. The use case in favor of removing zeros is line numbers in text files (001, 002, etc), and numbers formatted with printf("%0Nd"). I believe the "remove zeros" viewpoint won because a) if it looks like a number then we should treat it like a number; and b) the integers are faster to parse and also take less space in memory -- so for users who don't care about the representation, the integer choice is more preferable.

I've added a [beginner-task] label because this is a task that is actually quite easy to do: look into fread.c for functions StrtoI32 and StrtoI64. There is even a comment there about leading 0s, but it seems to be outdated.

[jangorecki]

IMO this should not be default behavior. Or at least it should be easy to keep leading zero with colClasses=c(id="character"). Whoever will implement that please include test for that.

[gsgxnet]

I do understand many pros and cons. Right an int64 is principally okay as format for a numeric code/id/ when read, and probably it needs less storage most times.
If it just were this one csv. But in real life we have to cope with many sources which handle codes/ids different. E.g. if you read some data by jdbc they will probably be transferred as character, other data from xls it depends, may be number, may be character. If these codes/ids were needed for joins between the different data sources the format has to be unified. No fun.

csv is is a quite poor data format. Much metadata is lost when db tables are exported into it. Leading 0 might be a hint for the right reconstruction of missing metadata.

@marc-outins - I can agree in the last years of all columns with leading 0 I had seen, none were meant to be numeric.

[MichaelChirico]

@gsgxnet your last point I think is the best take, and the best motivation for this feature/argument/improvement.

Regarding the inconsistency across many sources, one is always free to store my_colclasses = list(character = 'my_id_col') and pass that to any number of input CSVs.

And I think more important is better upstream data management (wherever possible).

[marc-outins]

I'm working on implementing this feature. I've come around to the idea of keeping the current default behavior of dropping the leading zeros and reading as numeric since the base R readers read.table, read.csv, etc have this behavior.

I've implemented a solution that allows the user to set an option data.table.fread.keepLeadingZeros to be TRUE (default will be FALSE) if they want fread to read data with leading zeros as character. (see marc-outins@60d6653). Still needs more work but this is the general idea. I also still need to add tests and would like to add a warning to let the user know something happened when data.table.fread.keepLeadingZeros==FALSE and a column with leading zero's is stored as numeric.

I'm curious if people prefer this solution or adding a specific parameter to fread? There is the logical01 parameter which is in the same vain as the leading zero option so it may make sense.

[gsgxnet]

@marc-outins many thanks for your good effort. Yes, please keep the current default behavior, otherwise existing code might be broken.
I would love to have a new parameter to fread because that way it could be used very flexible with every fread statement within the coding of one script/notebook/... containing several freads. But using an option should work there too, may be with some extra effort.
There are many parameters in fread() already, may be one more might be regarded as overload. If not my vote is for option and a parameter with default = option.

[johncassil]

This just caused an issue for me when trying to read in a file including zip codes and then broke the join with data from a database. Cutting off leading zeroes is one of the reasons I don't use Excel. I typically expect R to not have fickle behavior such as changing my data by making assumptions about it, so I'm lending my voice to the "this should be default behavior" position. For now, I switched to read_csv() for this process.

[MichaelChirico]

@johncassil use colClasses to force that column to be read as a character for now please :)