Crash in fread in dev when under-estimating the number of rows

[st-pasha]

require(data.table)
DT1 = data.table(A=rep(123, 100), B=456)
DT2 = data.table(A=rep(1, 200), B=2)
DT = rbind(DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT2, DT1, DT1)
fwrite(DT, f<-tempfile())
ff = fread(f)

Normally this produces

Error in fread(f) : 
  Internal error: DTi(2900) > allocnrow(2208) but nrows=9223372036854775807 (not limited)

but occasionally also

R(22039,0x7fff7504e000) malloc: *** error for object 0x7f957ec29608: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

[mattdowle]

Great find Pasha!

[st-pasha]

Solution outline:

1. Introduce function DTrealloc(int64_t nrows) (similar to DTalloc()). The new function's job will be to re-alloc every column in the datatable to the new length, however preserving all the data read so far.
2. Set the maxNrows parameter in freadMain to be the minimum of args.maxNrows and allocnrows.
3. If after the main loop we find that allocnrows rows were read, but more exist afterwards -- then guess how many rows are remaining and call the DTrealloc() function. After that, go back to the top of the data reading loop and continue reading the data from where we ended.