fwrite gzip on Solaris error

[mattdowle]

Now that 1.12.8 is on CRAN, we have the output from the extra tracing in #4028.
https://www.r-project.org/nosvn/R.check/r-patched-solaris-x86/data.table-00check.html

using R version 3.6.2 RC (2019-12-06 r77544)
using platform: i386-pc-solaris2.10 (32-bit)
using session charset: UTF-8
using option ‘--no-stop-on-test-error’
checking for file ‘data.table/DESCRIPTION’ ... OK
this is package ‘data.table’ version ‘1.12.8’
checking package namespace information ... OK
checking package dependencies ... OK
checking if this is a source package ... OK
checking if there is a namespace ... OK
checking for executable files ... OK
checking for hidden files and directories ... OK
checking for portable file names ... OK
checking for sufficient/correct file permissions ... OK
checking whether package ‘data.table’ can be installed ... OK
checking installed package size ... OK
checking package directory ... OK
checking ‘build’ directory ... OK
checking DESCRIPTION meta-information ... OK
checking top-level files ... OK
checking for left-over files ... OK
checking index information ... OK
checking package subdirectories ... OK
checking R files for non-ASCII characters ... OK
checking R files for syntax errors ... OK
checking whether the package can be loaded ... OK
checking whether the package can be loaded with stated dependencies ... OK
checking whether the package can be unloaded cleanly ... OK
checking whether the namespace can be loaded with stated dependencies ... OK
checking whether the namespace can be unloaded cleanly ... OK
checking loading without being on the library search path ... OK
checking use of S3 registration ... OK
checking dependencies in R code ... OK
checking S3 generic/method consistency ... OK
checking replacement functions ... OK
checking foreign function calls ... OK
checking R code for possible problems ... [34s/41s] OK
checking Rd files ... OK
checking Rd metadata ... OK
checking Rd cross-references ... OK
checking for missing documentation entries ... OK
checking for code/documentation mismatches ... OK
checking Rd \usage sections ... OK
checking Rd contents ... OK
checking for unstated dependencies in examples ... OK
checking line endings in shell scripts ... OK
checking line endings in C/C++/Fortran sources/headers ... OK
checking line endings in Makefiles ... OK
checking compilation flags in Makevars ... OK
checking for GNU extensions in Makefiles ... OK
checking for portable use of $(BLAS_LIBS) and $(LAPACK_LIBS) ... OK
checking include directives in Makefiles ... OK
checking compilation flags used ... OK
checking compiled code ... OK
checking installed files from ‘inst/doc’ ... OK
checking files in ‘vignettes’ ... OK
checking examples ... OK
checking for unstated dependencies in ‘tests’ ... OK
checking tests ... [102s/115s] ERROR
  Running ‘autoprint.R’
  Comparing ‘autoprint.Rout’ to ‘autoprint.Rout.save’ ... OK
  Running ‘froll.R’ [35s/38s]
  Running ‘knitr.R’
  Comparing ‘knitr.Rout’ to ‘knitr.Rout.save’ ... OK
  Running ‘main.R’ [58s/66s]
  Running ‘nafill.R’
  Running ‘other.R’
  Running ‘types.R’
Running the tests in ‘tests/main.R’ failed.
Complete output:
  > require(data.table)
  Loading required package: data.table
  >
  > test.data.table() # runs the main test suite of 5,000+ tests in /inst/tests/tests.Rraw
  getDTthreads(verbose=TRUE):
    omp_get_num_procs() 16
    R_DATATABLE_NUM_PROCS_PERCENT unset (default 50)
    R_DATATABLE_NUM_THREADS unset
    omp_get_thread_limit() 2
    omp_get_max_threads() 2
    OMP_THREAD_LIMIT 2
    OMP_NUM_THREADS unset
    RestoreAfterFork true
    data.table is using 2 threads. See ?setDTthreads.
  test.data.table() running: /home/ripley/R/Lib32/data.table/tests/tests.Rraw.bz2
  Test 1658.421 produced 1 errors but expected 0
  Expected:
  Observed: zlib 1.2.11 (zlib.h 1.2.11) deflate() returned error -2 with z_stream->msg=="" Z_FINISH=4 Z_BLOCK=5. Please include the full output above and below this message in your data.table bug report.
  Output captured before unexpected warning/error/message:
    omp_get_num_procs() 16
    R_DATATABLE_NUM_PROCS_PERCENT unset (default 50)
    R_DATATABLE_NUM_THREADS unset
    omp_get_thread_limit() 2
    omp_get_max_threads() 2
    OMP_THREAD_LIMIT 2
    OMP_NUM_THREADS unset
    RestoreAfterFork true
    data.table is using 2 threads. See ?setDTthreads.
  No list columns are present. Setting sep2='' otherwise quote='auto' would quote fields containing sep2.
  Column writers: 3 3
  args.doRowNames=0 args.rowNames=0 doQuote=-128 args.nrow=200 args.ncol=2 eolLen=1
  maxLineLen=51. Found in 0.000s
  Writing bom (false), yaml (0 characters) and column names (true) ... deflate input stream: e15b240 39 afdc2f8 4
  deflate returned 1 with stream->total_out==24; Z_FINISH==4, Z_OK==0, Z_STREAM_END==1
  done in 0.000s
  Writing 200 rows in 1 batches of 200 rows (each buffer size 8MB, showProgress=0, nth=1)
  deflate input stream: d259448 8391193 f9f9fd8 800
  deflate returned -2 with stream->total_out==0; Z_FINISH==4, Z_OK==0, Z_STREAM_END==1
  <simpleError in fwrite(DT, file = f1 <- tempfile(fileext = ".gz"), verbose = TRUE): zlib 1.2.11 (zlib.h 1.2.11) deflate() returned error -2 with z_stream->msg=="" Z_FINISH=4 Z_BLOCK=5. Please include the full output above and below this message in your data.table bug report.>
  Test 1658.43 ran without errors but failed check that x equals y:
  > x = fread(f1)
  Empty data.table (0 rows and 2 cols): a,b
   [Key= Types=log,log Classes=log,log]
  > y = DT
       a b [Key= Types=int,int Classes=int,int]
    1: 1 1
    2: 1 1
   ---
  199: 2 4
  200: 2 4
  Different number of rows
  Error in fwrite(DT, file = f3 <- tempfile(), compress = "gzip") :
    zlib 1.2.11 (zlib.h 1.2.11) deflate() returned error -2 with z_stream->msg=="" Z_FINISH=4 Z_BLOCK=5. Please retry fwrite() with verbose=TRUE and include the full output with your data.table bug report.
  Error in test.data.table() :
    Failed after test 1658.43 before the next test() call in /home/ripley/R/Lib32/data.table/tests/tests.Rraw.bz2
  Execution halted
checking for unstated dependencies in vignettes ... OK
checking package vignettes in ‘inst/doc’ ... OK
checking re-building of vignette outputs ... [139s/159s] OK
checking PDF version of manual ... OK
checking for detritus in the temp directory ... OK
DONE
Status: 1 ERROR

[mattdowle]

The only thing I see so far to follow up on is in the 2nd deflate call :
deflate input stream: d259448 8391193 f9f9fd8 800
where that 8391193 comes from. That is the 8MB buffer (8 * 1024^2 = 8,388,608) - 2,585. Why *destLen would be 8MB - 2585 at that stage I'm not sure. Shouldn't it be the whole 8MB?

A: by reading the code it looks like 8391193 is returned by deflateBound when passed 8MB. Maybe it knows that no matter what the contents of the 8MB, it can always reduce a little bit (2,585 bytes). I'm not sure why we're using deflateBound for the 2nd part, because our buffers are fixed. The idea of deflateBound I thought was to provide an upper bound for the result of the compression, so that just that amount can be allocated. Important for the header, because for the very small header, the compressed output is most often larger. But not sure why for the data chunks.

Running locally I get the same output, absolutely no difference, including that 8391193. So that was barking up the wrong tree.

No list columns are present. Setting sep2='' otherwise quote='auto' would quote fields containing sep2.
Column writers: 3 3 
args.doRowNames=0 args.rowNames=0 doQuote=-128 args.nrow=200 args.ncol=2 eolLen=1
maxLineLen=51. Found in 0.000s
Writing bom (false), yaml (0 characters) and column names (true) ... deflate input stream: 0x55df63204b70 39 0x55df5feff690 4
deflate returned 1 with stream->total_out==24; Z_FINISH==4, Z_OK==0, Z_STREAM_END==1
done in 0.000s
Writing 200 rows in 1 batches of 200 rows (each buffer size 8MB, showProgress=1, nth=1)
deflate input stream: 0x55df69ade4b0 8391193 0x55df692de4a0 800
deflate returned 1 with stream->total_out==50; Z_FINISH==4, Z_OK==0, Z_STREAM_END==1

[mattdowle]

nth==1 in this test (output confirms that), otherwise it could be a thread safety issue.

[mattdowle]

I asked on S.O. here :
https://stackoverflow.com/questions/59259256/z-stream-error-from-deflate-on-solaris-only

I fear that will get closed for being too vague. Just a quick review of the code or some pointers from @madler would be awesome! He's by far the top user in that tag too, so fingers crossed!

[madler]

Your arithmetic needs a little work. 8,391,193 is greater than 8,388,608.

[mattdowle]

Thanks, @madler. Arithmetic was never my strong point :-) Just an error here in comments. I rechecked our code in fwrite.c and it does allocate that 8,391,193 size (slightly larger than 8MB) and looks ok, afaics.

[mattdowle]

Now extra tracing is merged, will revisit after next release to CRAN. See comments at the top of #4107. Hence bump to 1.12.11 milestone.