fwrite encoding problem with file names

[dpprdan]

fwrite() cannot handle umlauts (and presumably all non-ASCII chars) in file names and paths on Windows (here with LC_COLLATE=German_Germany.1252 but from my experience this will also be a problem in other non-UTF-8 locales).

When the umlaut is in the file name, fwrite writes the file, but with a faulty file name.

library(data.table)
setwd(tempdir())
DF = data.frame(A=1:3, B=c("foo","A,Name","baz"))
fwrite(DF, "töst.csv")
list.files(pattern = "\\.csv")
#> [1] "töst.csv"

When the umlaut is in the path, fwrite cannot write the file at all.

dir.create("ä")
data.table::fwrite(DF, "ä/test.csv")
#> Error in data.table::fwrite(DF, "ä/test.csv"): No such file or directory: 'ä/test.csv'. Unable to create new file for writing (it does not exist already). Do you have permission to write here, is there space on the disk and does the path exist?

I looked at and debug-ed the R code and it seems to me that up until line 67 the file argument is encoded as “UTF-8” (as it should IMO) and looks fine. So my guess would be that the file path’s encoding goes wrong in the CfwriteR code.

From looking at the characters that should be “ö” or “ä” respectively, the problem seems to be that CfwriteR get’s a UTF-8 string but handles it as if it were encoded as latin-1, see this table.

If the error were in the R code, I would solve it with a file <- Encoding("UTF-8") line, but I do not know how this is done in C.

Session info

devtools::session_info()
#> Session info -------------------------------------------------------------
#>  setting  value                       
#>  version  R version 3.5.1 (2018-07-02)
#>  system   x86_64, mingw32             
#>  ui       RTerm                       
#>  language en                          
#>  collate  German_Germany.1252         
#>  tz       Europe/Berlin               
#>  date     2018-09-27
#> Packages -----------------------------------------------------------------
#>  package    * version date       source                            
#>  base       * 3.5.1   2018-07-02 local                             
#>  compiler     3.5.1   2018-07-02 local                             
#>  data.table * 1.11.6  2018-09-19 CRAN (R 3.5.1)                    
#>  datasets   * 3.5.1   2018-07-02 local                             
#>  devtools     1.13.6  2018-06-27 CRAN (R 3.5.1)                    
#>  digest       0.6.17  2018-09-12 CRAN (R 3.5.1)                    
#>  evaluate     0.11    2018-07-17 CRAN (R 3.5.1)                    
#>  graphics   * 3.5.1   2018-07-02 local                             
#>  grDevices  * 3.5.1   2018-07-02 local                             
#>  htmldeps     0.1.1   2018-07-30 Github (rstudio/htmldeps@c1023e0) 
#>  htmltools    0.3.6   2017-04-28 CRAN (R 3.5.1)                    
#>  knitr        1.20    2018-02-20 CRAN (R 3.5.1)                    
#>  magrittr     1.5     2014-11-22 CRAN (R 3.5.1)                    
#>  memoise      1.1.0   2017-04-21 CRAN (R 3.5.1)                    
#>  methods    * 3.5.1   2018-07-02 local                             
#>  Rcpp         0.12.18 2018-07-23 CRAN (R 3.5.1)                    
#>  rmarkdown    1.10.13 2018-09-04 Github (rstudio/rmarkdown@19008bf)
#>  stats      * 3.5.1   2018-07-02 local                             
#>  stringi      1.2.4   2018-07-20 CRAN (R 3.5.1)                    
#>  stringr      1.3.1   2018-05-10 CRAN (R 3.5.1)                    
#>  tools        3.5.1   2018-07-02 local                             
#>  utils      * 3.5.1   2018-07-02 local                             
#>  withr        2.1.2   2018-03-15 CRAN (R 3.5.1)                    
#>  yaml         2.2.0   2018-07-25 CRAN (R 3.5.1)

[MichaelChirico]

Nice catch!

[MichaelChirico]

@dpprdan is mingw32 a Windows system or Unix-alike?

[dpprdan]

@MichaelChirico It is Windows (10). mingw is short for "Minimalist GNU for Windows" (I think).

[kiesner]

Any news on this? We have the same problem and unfortunately have to resort to write.csv2 for now.

[MichaelChirico]

@kiesner can't you just write with fwrite and use file.rename?

[dpprdan]

@MichaelChirico No, not if @kiesner's problem is with umlauts in the path (and not only the file name), see my second example.

[jangorecki]

@dpprdan file.rename is not only for renaming files but directories also

[MichaelChirico]

@jangorecki I'm starting work on a hacky solution that just uses file.rename internally, what do you think?

[jangorecki]

@MichaelChirico there is generally much more interesting stuff to do than such tedious workarounds for filename encoding :) but if it won't bloat the code, will be just single chunk in R fwrite then I don't see any problem.

[MichaelChirico]

Hmm actually I'm not able to reproduce the original issue on macOS... so not sure how to test any solution...

[st-pasha]

You could make a PR with a test, mark it [WIP], and then just wait for the results from AppVeyor. This is rather slow, but at least will get the job done.

[MichaelChirico]

@dpprdan does file.rename work on your OS?

f = "ä/test.csv"
f_ascii = iconv(f, from = 'UTF-8', to = 'ASCII', sub = 'XXX')

dir.create(dirname(f_ascii), showWarnings = FALSE, recursive = TRUE)
fwrite(DT, f_ascii)
file.rename(f_ascii, f)

[dpprdan]

@MichaelChirico Your example probably works, but the issue with umlauts in the path name extends to existing directories, possibly in the user’s home directory or on network drives. And TBH I fail to see where temporarily renaming existing paths that could potentially be used by other processes and users is a good idea.

I dug a little deeper and this is what I discovered.

1. fread cannot handle UTF-8 encoded file names/paths either:

library(data.table)
setwd(tempdir())
DF = data.frame(A=1:3, B=c("foo","A,Name","baz"))
fwrite(DF, "test.csv")
file.rename("test.csv", "töst.csv")
#> [1] TRUE
list.files(pattern = "\\.csv")
#> [1] "töst.csv"
fname <- "töst.csv"
Encoding(fname)
#> [1] "latin1"
fname_utf8 <- enc2utf8(fname)
Encoding(fname_utf8)
#> [1] "UTF-8"
fread(fname)
#>    A      B
#> 1: 1    foo
#> 2: 2 A,Name
#> 3: 3    baz
fread(fname_utf8)
#> Error in fread(fname_utf8): File not found: töst.csv

This is probably not intended, at least lines 56 ff. of fread.h reads: “Name of the file to open (a \0-terminated C string). If the file name contains non-ASCII characters, it should be UTF-8 encoded (however fread will not validate the encoding).” (emphasis mine) The same comment can be found on lines 31 ff. of fwrite.h
This also has practical significance, because e.g. basename(), dirname, or path.expand all convert the path to UTF-8 (at least on R 3.5.1 on Windows 10 and with non-ASCII characters in the path).

Encoding(basename(fname))
#> [1] "UTF-8"
Encoding(dirname("ö/ä"))
#> [1] "UTF-8"
Encoding(path.expand(fname))
#> [1] "UTF-8"

fread(path.expand(fname))
#> Error in fread(path.expand(fname)) : File not found: töst.csv

2. path.expand converting to UTF-8 is actually the reason why fwrite fails in my original post. Contrary to what I wrote there, the file argument is encoded as latin1 until line 44 of fwrite.R (or rather in the “native encoding” which is latin1 on Windows with a cp1252 locale). In line 44 path.expand converts it to UTF-8. Now it is not in the native encoding anymore, which is why fwrite fails, because file or filename is not explicitly marked as UTF-8 and thus assumed to be in the native encoding in the C part of the function.

BTW, on macOS the native encoding is UTF-8, which is why you cannot reproduce this, @MichaelChirico, because everything stays in UTF-8 there.

From my point of view there are two options to fix this:

1. Change the encoding of the file parameter in the R code with enc2native(file), so that file is passed to fwriteR in the native encoding.
2. It would be safer though, IMHO, and in accordance with the comment mentioned above to stick to UTF-8 encoded file paths in both fread and fwrite (file would have to be encoded as UTF-8 first with enc2utf8 in fread, however) and mark them as such when passing it to the C code (with something like mkCharCE(filename, CE_UTF8)) if my google fu serves me right).

[MichaelChirico]

yes, my solution was only intended as duct tape. it sounds like you're pretty close to understanding the issue well enough to maybe file a PR? if so that'd be great and put the issue to rest more firmly.

…

On Sun, Nov 11, 2018, 4:36 PM Daniel Possenriede ***@***.*** wrote: @MichaelChirico <https://github.com/MichaelChirico> Your example probably works, but the issue with umlauts in the path name extends to existing directories, possibly in the user’s home directory or on network drives. And TBH I fail to see where temporarily renaming existing paths that could potentially be used by other processes and users is a good idea. I dug a little deeper and this is what I discovered. 1. fread cannot handle UTF-8 encoded file names/paths either: library(data.table) setwd(tempdir())DF = data.frame(A=1:3, B=c("foo","A,Name","baz")) fwrite(DF, "test.csv") file.rename("test.csv", "töst.csv")#> [1] TRUE list.files(pattern = "\\.csv")#> [1] "töst.csv"fname <- "töst.csv" Encoding(fname)#> [1] "latin1"fname_utf8 <- enc2utf8(fname) Encoding(fname_utf8)#> [1] "UTF-8" fread(fname)#> A B#> 1: 1 foo#> 2: 2 A,Name#> 3: 3 baz fread(fname_utf8)#> Error in fread(fname_utf8): File not found: töst.csv This is probably not intended, at least lines 56 ff. of fread.h <https://github.com/Rdatatable/data.table/blob/d3ccd8139d3d592201e0f085c5f37a4fc5a426e3/src/fread.h#L56> reads: “Name of the file to open (a \0-terminated C string). If the file name contains non-ASCII characters, it *should be UTF-8 encoded* (however fread will not validate the encoding).” (emphasis mine) The same comment can be found on lines 31 ff. of fwrite.h <https://github.com/Rdatatable/data.table/blob/d3ccd8139d3d592201e0f085c5f37a4fc5a426e3/src/fwrite.h#L31> This also has practical significance, because e.g. basename(), dirname, or path.expand all convert the path to UTF-8 (at least on R 3.5.1 on Windows 10 and with non-ASCII characters in the path). Encoding(basename(fname))#> [1] "UTF-8" Encoding(dirname("ö/ä"))#> [1] "UTF-8" Encoding(path.expand(fname))#> [1] "UTF-8" fread(path.expand(fname))#> Error in fread(path.expand(fname)) : File not found: töst.csv 1. path.expand converting to UTF-8 is actually the reason why fwrite fails in my original post. Contrary to what I wrote there, the file argument is encoded as latin1 until line 44 of fwrite.R <https://github.com/Rdatatable/data.table/blob/df4e56f5d960741c864204a713aed43d7867f17f/R/fwrite.R#L44> (or rather in the “native encoding” which is latin1 on Windows with a cp1252 locale). In line 44 path.expand converts it to UTF-8. Now it is not in the native encoding anymore, which is why fwrite fails, because file or filename is not explicitly marked as UTF-8 and thus assumed to be in the native encoding in the C part of the function. BTW, on macOS the native encoding is UTF-8, which is why you cannot reproduce this, @MichaelChirico <https://github.com/MichaelChirico>, because everything stays in UTF-8 there. From my point of view there are two options to fix this: 1. Change the encoding of the file parameter in the R code with enc2native(file), so that file is passed to fwriteR in the native encoding. 2. It would be safer though, IMHO, and in accordance with the comment mentioned above to stick to UTF-8 encoded file paths in both fread and fwrite (file would have to be encoded as UTF-8 first with enc2utf8 in fread, however) and mark them as such when passing it to the C code (with something like mkCharCE(filename, CE_UTF8)) <https://github.com/wch/r-source/blob/7f0ae7735816eccba5e2e507543f0486c264bc28/src/main/envir.c#L3839> if my google fu serves me right <https://cran.r-project.org/doc/manuals/R-exts.html#Character-encoding-issues> ). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3078 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHQQdQSeJUGQNKbL7bniBe4xDdZfMYR4ks5ut-F7gaJpZM4W8GH_> .

[dpprdan]

I only know how to do the enc2native(file) solution, which, IMO, is duct tape as well, just a different brand.

The second option should be a quick fix as well, for someone who knows how to do that in C and would be more robust, I think.

But I can surely do a PR with the first option if that's what you want.