Support for writing to gzip (.gz) with fwrite

[MichaelChirico]

write.csv can handle the following:

library(data.table)
DT = data.table(a = 1:10, b = 11:20)

tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
write.csv(DT, gzf)
close(gzf)

But fwrite cannot:

tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
fwrite(DT, gzf)
close(gzf)

Error: is.character(file) && length(file) == 1 && !is.na(file) is not TRUE

(because the gzf object is more complicated than just being a file path).

fwrite(DT, tmp.gz), as expected, does not perform compression -- the output is the same size as fwrite(DT, tempfile()) (which is different from write.csv).

I'm using fwrite to pipe some output to Vowpal Wabbit, trying to use the --compressed argument to save on some disk space. Workaround is to fwrite then gzip, but this seems sub-optimal.

[JhossePaul]

+1

[rargelaguet]

+1

[msgoussi]

Dear Mr.
When I export data.table using fwrite, the size of the file is 31.3GB
then when I use gzip in order to compress the file, I get:
Error in writeBin(bfr, con = out, size = 1L) :
'Calloc' could not allocate memory (10000000 of 1 bytes)

How can I solve this problem?
Thanks

[Griefberg]

+1

[wookietreiber]

Please don't post comments with simply "+1". This adds absolutely no value. Either write something of value that adds something to the discussion or use the thumbs up emoticon / reaction on the original post to express interest in the issue and a solution to it.

I'm in favor of deleting such comments, including mine BTW, because it is of no consequence to the technical discussion of this issue as writing "+1". Please, people, please ;)