Fails torture

[mattdowle]

(Writing with future reference in mind in case we hit this again.)

Motivated by #2767, we're looking for a memory fault in data.table. At least, we have to utterly rule out everything on the data.table side, first. Experience shows that R-devel-only problems are in fact revealing underlying problems in data.table. Particularly on 32bit which seems to expose memory problems faster, or when R changes internal memory layout, as happened recently in R 3.5.0.

All rchk messages have been dealt with in #2836. All were false positives I believe. Still, it was useful to flag, consider and make rchk-compliant so we don't have to consider those lines again. rchk is currently set to be 'permissive' according to notes on CRAN and its documentation. Perhaps now we pass 'permissive' (when 1.11.2 goes to CRAN), we can ask for a stricter rchk setting for data.table, or run it ourselves. We could come back to that.

However, with latest R-devel, compiled with ASAN and --strict-barrier and torture2(step=100), we fail with the following log. The runtime was somewhere between 12h and 18h. (It was still running when I checked at 12h, and had failed when I checked again at 18h).

Log

> gctorture2(step=100)
> print(Sys.time()); started.at<-proc.time(); try(test.data.table()); print(Sys.time()); print(timetaken(started.at))
[1] "2018-05-04 22:41:26 PDT"
Running /home/mdowle/build/R-devel/library/data.table/tests/tests.Rraw 
Running test id 819      Test 819 ran without errors but failed check that x equals y:
> x = system.time(X[Y, allow.cartesian = TRUE])["user.self"] < 10 
First 6 of 1 (type 'logical'): user.self 
    FALSE 
> y = TRUE 
First 6 of 1 (type 'logical'): [1] TRUE
1 element mismatch
Running test id 820      Test 820 ran without errors but failed check that x equals y:
> x = system.time(X[Y, mult = "first"])["user.self"] < 10 
First 6 of 1 (type 'logical'): user.self 
    FALSE 
> y = TRUE 
First 6 of 1 (type 'logical'): [1] TRUE
1 element mismatch
Running test id 895.0013      Test 895.0013  should not fail but failed with error:  unprotected object (0x6250123aa1a8) encountered (was STRSXP) 
Running test id 896.0013      Test 896.0013 produced 1 warnings but expected 0 
Expected: 
Observed: Previous fread() session was not cleaned up properly. Cleaned up ok at the beginning of this fread() call.
Running test id 895.0114      Test 895.0114  should not fail but failed with error:  unprotected object (0x6250123aa170) encountered (was STRSXP) 
Running test id 896.0114      Test 896.0114 produced 1 warnings but expected 0 
Expected: 
Observed: Previous fread() session was not cleaned up properly. Cleaned up ok at the beginning of this fread() call.
Running test id 896.0211      Test 896.0211 ran without errors but failed check that x equals y:
> x = fread(f, na.strings = "") 
     a  [Key= Types=int Classes=int]
1: 435                              
2: 713                              
3:  NA                              
> y = if (nc == 1) rbind(headDT, list(NA)) else headDT 
First 6 of 1 (type 'character'): [1] "Error in rbindlist(l, use.names, fill, idcol) : \n  unprotected object (0x6250123aa250) encountered (was INTSXP)\n"
Running test id 896.0311      Test 896.0311 ran without errors but failed check that x equals y:
> x = fread(f, na.strings = "") 
     a  [Key= Types=int Classes=int]
1: 435                              
2: 713                              
3: 400                              
4:  NA                              
> y = if (nc == 1) rbind(headDT, list(NA)) else headDT 
First 6 of 1 (type 'character'): [1] "Error in rbindlist(l, use.names, fill, idcol) : \n  unprotected object (0x6250123aa4f0) encountered (was LGLSXP)\n"
Running test id 896.0312      Test 896.0312 ran without errors but failed check that x equals y:
> x = fread(f, na.strings = "") 
     a  [Key= Types=int Classes=int]
1: 435                              
2: 713                              
3: 400                              
4:  NA                              
> y = if (nc == 1) rbind(headDT, list(NA)) else headDT 
First 6 of 1 (type 'character'): [1] "Error in rbindlist(l, use.names, fill, idcol) : \n  Answer requires 2 columns whereas one or more item(s) in the input list has only 1 columns. This could be because the items in the list may not all have identical column names or some of the items may have duplicate names. In either case, if you're aware of this and would like to fill those missing columns, set the argument 'fill=TRUE'.\n"
Running test id 896.0314      Test 896.0314 ran without errors but failed check that x equals y:
> x = fread(f, na.strings = "") 
     a  [Key= Types=int Classes=int]
1: 435                              
2: 713                              
3: 400                              
4:  NA                              
> y = if (nc == 1) rbind(headDT, list(NA)) else headDT 
First 6 of 1 (type 'character'): [1] "Error in rbindlist(l, use.names, fill, idcol) : \n  unprotected object (0x6250123a9f40) encountered (was LGLSXP)\n"
Running test id 896.0315      =================================================================
==21858==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6020002a1a78 at pc 0x7fb141f7760a bp 0x7ffd25c18ad0 sp 0x7ffd25c18ac0
WRITE of size 4 at 0x6020002a1a78 thread T0
    #0 0x7fb141f77609 in push /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:44
    #1 0x7fb141f854ba in csorted /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:1031
    #2 0x7fb141f872b2 in forder /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:1167
    #3 0x7fb141fe825f in fast_order /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:402
    #4 0x7fb141fe89e5 in match_names /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:442
    #5 0x7fb141feb4b1 in preprocess /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:549
    #6 0x7fb141fecf6e in rbindlist /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:631
    #7 0x5574971f53e3 in R_doDotCall /home/mdowle/build/R-devel/src/main/dotcode.c:579
    #8 0x55749731c717 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:7280
    #9 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #10 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #11 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #12 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #13 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #14 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #15 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #16 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #17 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #18 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #19 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #20 0x55749712de7e in do_bind /home/mdowle/build/R-devel/src/main/bind.c:1117
    #21 0x5574973dc5b8 in do_internal /home/mdowle/build/R-devel/src/main/names.c:1371
    #22 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #23 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #24 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #25 0x5574972c2553 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:747
    #26 0x5574972c8dff in do_if /home/mdowle/build/R-devel/src/main/eval.c:2131
    #27 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #28 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #29 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #30 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #31 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #32 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #33 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #34 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #35 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #36 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #37 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #38 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #39 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #40 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #41 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #42 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #43 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #44 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #45 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #46 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #47 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #48 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #49 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #50 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #51 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #52 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #53 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #54 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #55 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #56 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #57 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #58 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #59 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #60 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #61 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #62 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #63 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #64 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #65 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #66 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #67 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #68 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #69 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #70 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #71 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #72 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #73 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #74 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #75 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #76 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #77 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #78 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #79 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #80 0x5574972c2553 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:747
    #81 0x5574972ce431 in do_begin /home/mdowle/build/R-devel/src/main/eval.c:2395
    #82 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #83 0x5574972cca5f in do_for /home/mdowle/build/R-devel/src/main/eval.c:2281
    #84 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #85 0x5574972ce431 in do_begin /home/mdowle/build/R-devel/src/main/eval.c:2395
    #86 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #87 0x5574972cca5f in do_for /home/mdowle/build/R-devel/src/main/eval.c:2281
    #88 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #89 0x5574972ce431 in do_begin /home/mdowle/build/R-devel/src/main/eval.c:2395
    #90 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #91 0x5574972cca5f in do_for /home/mdowle/build/R-devel/src/main/eval.c:2281
    #92 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #93 0x5574972d39d5 in do_eval /home/mdowle/build/R-devel/src/main/eval.c:3181
    #94 0x5574973dc5b8 in do_internal /home/mdowle/build/R-devel/src/main/names.c:1371
    #95 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #96 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #97 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #98 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #99 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #100 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #101 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #102 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #103 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #104 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #105 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #106 0x5574972c2553 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:747
    #107 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #108 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #109 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #110 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #111 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #112 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #113 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #114 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #115 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #116 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #117 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #118 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #119 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #120 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #121 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #122 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #123 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #124 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #125 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #126 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #127 0x5574972c115d in forcePromise /home/mdowle/build/R-devel/src/main/eval.c:520
    #128 0x5574972da0bd in FORCE_PROMISE /home/mdowle/build/R-devel/src/main/eval.c:4971
    #129 0x5574972da362 in getvar /home/mdowle/build/R-devel/src/main/eval.c:5013
    #130 0x5574972e994b in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6504
    #131 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #132 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #133 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #134 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #135 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #136 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #137 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #138 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #139 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #140 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #141 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #142 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #143 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #144 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #145 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #146 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #147 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #148 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #149 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #150 0x5574972c2553 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:747
    #151 0x55749737ba9f in Rf_ReplIteration /home/mdowle/build/R-devel/src/main/main.c:258
    #152 0x55749737befc in R_ReplConsole /home/mdowle/build/R-devel/src/main/main.c:308
    #153 0x55749737fcc0 in run_Rmainloop /home/mdowle/build/R-devel/src/main/main.c:1082
    #154 0x55749737fcd6 in Rf_mainloop /home/mdowle/build/R-devel/src/main/main.c:1089
    #155 0x5574970b6fd2 in main /home/mdowle/build/R-devel/src/main/Rmain.c:29
    #156 0x7fb1510781c0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x211c0)
    #157 0x5574970b6e89 in _start (/home/mdowle/build/R-devel/bin/exec/R+0x605e89)

0x6020002a1a78 is located 0 bytes to the right of 8-byte region [0x6020002a1a70,0x6020002a1a78)
allocated by thread T0 here:
    #0 0x7fb153f02f40 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdef40)
    #1 0x7fb141f76dd0 in growstack /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:36
    #2 0x7fb141f77361 in push /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:43
    #3 0x7fb141f854ba in csorted /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:1031
    #4 0x7fb141f872b2 in forder /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:1167
    #5 0x7fb141fe825f in fast_order /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:402
    #6 0x7fb141fe89e5 in match_names /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:442
    #7 0x7fb141feb4b1 in preprocess /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:549
    #8 0x7fb141fecf6e in rbindlist /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/rbindlist.c:631
    #9 0x5574971f53e3 in R_doDotCall /home/mdowle/build/R-devel/src/main/dotcode.c:579
    #10 0x55749731c717 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:7280
    #11 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #12 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #13 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #14 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #15 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #16 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #17 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #18 0x5574972f0d96 in bcEval /home/mdowle/build/R-devel/src/main/eval.c:6740
    #19 0x5574972c171d in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:624
    #20 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #21 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #22 0x55749712de7e in do_bind /home/mdowle/build/R-devel/src/main/bind.c:1117
    #23 0x5574973dc5b8 in do_internal /home/mdowle/build/R-devel/src/main/names.c:1371
    #24 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699
    #25 0x5574972c67d0 in R_execClosure /home/mdowle/build/R-devel/src/main/eval.c:1764
    #26 0x5574972c5cb6 in Rf_applyClosure /home/mdowle/build/R-devel/src/main/eval.c:1692
    #27 0x5574972c2553 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:747
    #28 0x5574972c8dff in do_if /home/mdowle/build/R-devel/src/main/eval.c:2131
    #29 0x5574972c1eb2 in Rf_eval /home/mdowle/build/R-devel/src/main/eval.c:699

SUMMARY: AddressSanitizer: heap-buffer-overflow /tmp/RtmpEFQ0wK/R.INSTALL54eb25d930bf/data.table/src/forder.c:44 in push
Shadow bytes around the buggy address:
  0x0c048004c2f0: fa fa fa fa fa fa fd fa fa fa fd fa fa fa fa fa
  0x0c048004c300: fa fa fa fa fa fa fd fd fa fa fd fd fa fa fd fa
  0x0c048004c310: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c048004c320: fa fa fd fa fa fa fd fa fa fa fa fa fa fa fd fa
  0x0c048004c330: fa fa fd fd fa fa fd fa fa fa fa fa fa fa fd fa
=>0x0c048004c340: fa fa fa fa fa fa fd fa fa fa fd fa fa fa 00[fa]
  0x0c048004c350: fa fa fa fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c048004c360: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fa fa
  0x0c048004c370: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fd fa
  0x0c048004c380: fa fa fa fa fa fa fd fa fa fa fd fa fa fa fd fa
  0x0c048004c390: fa fa fd fa fa fa fa fa fa fa fd fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==21858==ABORTING

[HughParsonage]

In particular, gctorture2 appears to mess up the ordering of rbind:

> library("data.table", lib.loc = tempd)
data.table 1.11.1 IN DEVELOPMENT built 2018-05-05 01:21:17 UTC; travis
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
> dt = data.table(a = rep(as.Date("2010-01-01"), 4), b = rep("a",4))
> identical(rbind(dt,dt), data.table(a = rep(as.Date("2010-01-01"), 8), b = rep("a",8)))
[1] TRUE
> gctorture2(step = 1)
[1] 0
> dt = data.table(a = rep(as.Date("2010-01-01"), 4), b = rep("a",4))
> identical(rbind(dt,dt), data.table(a = rep(as.Date("2010-01-01"), 8), b = rep("a",8)))
[1] FALSE
> rbind(dt, dt)
   b          a
1: a 2010-01-01
2: a 2010-01-01
3: a 2010-01-01
4: a 2010-01-01
5: a 2010-01-01
6: a 2010-01-01
7: a 2010-01-01
8: a 2010-01-01

R version 3.5.0 (2018-04-23)
Platform: i386-w64-mingw32/i386 (32-bit)
Running under: Windows 10 x64 (build 16299)

Matrix products: default

locale:
[1] LC_COLLATE=English_Australia.1252  LC_CTYPE=English_Australia.1252    LC_MONETARY=English_Australia.1252
[4] LC_NUMERIC=C                       LC_TIME=English_Australia.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.11.1

loaded via a namespace (and not attached):
[1] compiler_3.5.0 tools_3.5.0  

[mattdowle]

Great. That rbind example survives torture for me now with ce61409 in PR #2843.
But the PR still fails on R-devel AppVeyor. Maybe there's another fault after test 896. I'll add skip.test= to test.data.table (to save 12hrs) and rerun. The earliest I remember seeing the Memory exhausted is around 1253. So, somewhere between 896 and 1253, perhaps.