I've managed to reproduce this on Windows. Here's a MRE.
require(data.table) # v1.9.8, R version 3.3.1, Windows 7
.Call("CsubsetVector", character(0), 1:0)
# Error: attempt to set index 1/1 in SET_STRING_ELT
Running it again hangs up the R session. The actual usage was subsetting of an empty table, but I tracked it down to subsetVector. Note that both .Call("CsubsetVector", character(0), 1L) and .Call("CsubsetVector", character(0), 0L), from a fresh session, work fine.
I've managed to reproduce this on Windows. Here's a MRE.
Running it again hangs up the R session. The actual usage was subsetting of an empty table, but I tracked it down to
subsetVector. Note that both.Call("CsubsetVector", character(0), 1L)and.Call("CsubsetVector", character(0), 0L), from a fresh session, work fine.