Handling of zlib fwrite dependency

[MichaelChirico]

#include <zlib.h> might fail on some machines. In this case it would be helpful to fail more helpfully.

Currently on rocker/r-ver:3.5.3 (e.g.), we get the following installation failure for dev data.table:

install.packages('data.table', type = 'source', repos='http://Rdatatable.github.io/data.table')
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.12.3.tar.gz'
Content type 'application/gzip' length 4960958 bytes (4.7 MB)
==================================================
downloaded 4.7 MB

* installing *source* package ‘data.table’ ...
** libs
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c assign.c -o assign.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c between.c -o between.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c bmerge.c -o bmerge.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c chmatch.c -o chmatch.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c cj.c -o cj.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c coalesce.c -o coalesce.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c dogroups.c -o dogroups.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fastmean.c -o fastmean.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fcast.c -o fcast.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fifelse.c -o fifelse.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fmelt.c -o fmelt.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c forder.c -o forder.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c frank.c -o frank.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fread.c -o fread.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c freadR.c -o freadR.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c froll.c -o froll.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c frollR.c -o frollR.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c frolladaptive.c -o frolladaptive.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fsort.c -o fsort.o
gcc -I"/usr/local/lib/R/include" -DNDEBUG   -I/usr/local/include  -fopenmp -fpic  -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c fwrite.c -o fwrite.o
fwrite.c:22:48: fatal error: zlib.h: No such file or directory
 #include "zlib.h"      // for writing gzip file
                                                ^
compilation terminated.
/usr/local/lib/R/etc/Makeconf:163: recipe for target 'fwrite.o' failed
make: *** [fwrite.o] Error 1
ERROR: compilation failed for package ‘data.table’
* removing ‘/usr/local/lib/R/site-library/data.table’
* restoring previous ‘/usr/local/lib/R/site-library/data.table’

The downloaded source packages are in
	‘/tmp/Rtmp7Po4yU/downloaded_packages’
Warning message:
In install.packages("data.table", type = "source", repos = "http://Rdatatable.github.io/data.table") :
  installation of package ‘data.table’ had non-zero exit status

(this doesn't affect the CRAN version since there's no fwrite+gz support there yet)

By contrast, here's how units package fails:

# install.packages('Rcpp') first on clean image
install.packages('units')
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
trying URL 'https://mran.microsoft.com/snapshot/2019-04-26/src/contrib/units_0.6-2.tar.gz'
Content type 'application/octet-stream' length 917316 bytes (895 KB)
==================================================
downloaded 895 KB

* installing *source* package ‘units’ ...
** package ‘units’ successfully unpacked and MD5 sums checked
configure: units: 0.6-2
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking how to run the C++ preprocessor... g++ -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdbool.h that conforms to C99... no
checking for _Bool... no
checking for error_at_line... yes
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for XML_ParserCreate in -lexpat... no
checking udunits2.h usability... no
checking udunits2.h presence... no
checking for udunits2.h... no
checking udunits2/udunits2.h usability... no
checking udunits2/udunits2.h presence... no
checking for udunits2/udunits2.h... no
checking for ut_read_xml in -ludunits2... no
configure: error: in `/tmp/Rtmp78wlOS/R.INSTALL3f75541cfe/units':
configure: error: 
--------------------------------------------------------------------------------
  Configuration failed because libudunits2.so was not found. Try installing:
    * deb: libudunits2-dev (Debian, Ubuntu, ...)
    * rpm: udunits2-devel (Fedora, EPEL, ...)
    * brew: udunits (OSX)
  If udunits2 is already installed in a non-standard location, use:
    --configure-args='--with-udunits2-lib=/usr/local/lib'
  if the library was not found, and/or:
    --configure-args='--with-udunits2-include=/usr/include/udunits2'
  if the header was not found, replacing paths with appropriate values.
  You can alternatively set UDUNITS2_INCLUDE and UDUNITS2_LIBS manually.
--------------------------------------------------------------------------------

See `config.log' for more details
ERROR: configuration failed for package ‘units’
* removing ‘/usr/local/lib/R/site-library/units’

The downloaded source packages are in
	‘/tmp/RtmpJB7VSi/downloaded_packages’
Warning message:
In install.packages("units") :
  installation of package ‘units’ had non-zero exit status

Quite useful & actionable!

I'm not sure how ubiquitous zlib is in the wild...

[MichaelChirico]

It seems to do so we should add a configure script which can detect the missing zlib header:

• configure in R-exts: https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Configure-and-cleanup
• configure script used by units: https://github.com/r-quantities/units/blob/7da0e9fcf9d8e732a0ac372f69e6a1fb3357f7e2/configure

[jangorecki]

Could we compile without zlib support if it is not present? Then fwrite would error when attempting to use gz. This reduce OS software dependency, this is very important to stay lightweight.

[st-pasha]

We had a similar problem in (py) datatable, and ended up just shipping the zlib.h and zconf.h with the source distribution (h2oai/datatable#1957).

[jangorecki]

@st-pasha way of shipping zlib together with data.table is another option. Definitely better than current error. Yet IMO it is better to not ship it together, but escape support of gz fwrite when zlib was not present during compilation. I can imagine that not every user uses fwrite, and among them not many uses gz, ultimately zlib might be needed by a tiny fraction of data.table users.

[MichaelChirico]

Agree w Jan it's best if we can essentially make zlib a "Suggests" (optional), I just don't know how to do that at the C level. Seems the way we handle openmp could be a template but I couldn't quite make it work. Exploratory stuff here in case anyone sees how to take it over the line:

https://github.com/MichaelChirico/data.table/tree/zlib_suggests

[philippechataignon]

zlib is required for R : that's why I think that there wasn't a problem for gzip support. But you're right: the headers (zlib1g-dev on Debian/Ubuntu) are not always present and are needed for compiling. On Debian zlib1g-dev is a r-base-dev dependency: that's why it's often present. In the initial example, I had a look at the dockerfile and zlib1g-dev is in BUILDDEPS : why zlib header is not found a compile time ?

I don't think that including zlib headers is a good idea because distribution binaries from zlib and included headers in data.table may be incompatibles. In this case, data.table must include headers and source of the zlib library.

I think configure is the best idea to detect if zlib is present and compile without zlib if not. OK for suggest. A warning can be displayed at compile time like package units do (I think curl, git2r and xml2 act in similar way).

Another advantage of configure is that it could be possible to add another compression method like lz4 and xz (which need external libs too). I've tested lz4 and on my machine : lz4 fwrite is fastest than csv fwrite. xz is interesting to have better ratio compressed (never implemented).

I don't know how to write a configure script. If  this solution is retained, I can try to see how to modify fwrite.c to use a ZLIB_PRESENT macro to have a fwrite version with or without gzip support.

[philippechataignon]

This branch is an attempt to write a configure script to detect zlib. Feel free to test. Not a PR because it introduces changes in current data.table build process.

[jangorecki]

@philippechataignon any chance for a unit test? It has to be conditionally escaped of course.
Also it seems more appropriate to error than attempt to write non-gzip file.

[philippechataignon]

I can't go further on the branch fwrite_zlib_present et for me it's OK but perhaps too late/too dangerous for 1.12.4. I've replaced warning by stop as suggests. I've manually tested fwrite on data.table compiled with/without HAVE_ZLIB flag forced in fwrite.c and it's OK but I don't know how to write a unit test to verify the message error when there's no zlib headers on the machine.