Failed to allocate counts or TMP when assigning g in gforce

[renkun-ken]

I'm working with a data.table of 1423657324 rows and 14 columns.

There's an integer group (grp) with 3790 unique values.

When I do the following

stats <- ft[, .(
      col1 = median(col1, na.rm = TRUE),
      col2 = median(col2, na.rm = TRUE),
      col3 = median(col3, na.rm = TRUE)
    ), keyby = grp]

The following error occurs:

Error in gforce(thisEnv, jsub, o__, f__, len__, irows) :
  Internal error: Failed to allocate counts or TMP when assigning g in gforce

This does not occur when a shorter version of the data (1/3 of the size) is done with the same aggregation.

[MichaelChirico]

I would imagine median could be very memory hungry, this is why you usually see approx_quantile in "big data" SQL and median itself is dropped entirely.

Do you get the same issue if you try and sort the data by col1? If not, it may be that gmedian is trying to do all three columns at once.

As a workaround, do ft[ , .N, keyby = .(grp, col1)] and get the median from the frequency table, since you said there are far fewer unique values.

[shrektan]

Will changing the median to stats::median help? It should prevent data.table from optimizing with gforce.

[renkun-ken]

Disabling gforce or using stats::median will not trigger this error. But I'm still curious about why this occurs?

My server memory is 1TB and should be enough to process.

[MichaelChirico]

how about stats::median(c(col1, col2, col3)) (just to check if tripling the memory footprint has an impact)

[renkun-ken]

stats::median(c(col1, col2, col3)) works quite normally and nothing special occurs.

[shrektan]

Well, the error is thrown from

data.table/src/gsumm.c

Lines 114 to 115 in b1b1832

	int *counts = calloc(nBatch*highSize, sizeof(int)); // TODO: cache-line align and make highSize a multiple of 64
	int *TMP = malloc(nrow*2*sizeof(int));

However, I also see the nrow is declared as an int

data.table/src/gsumm.c

Line 6 in b1b1832

static int nrow = 0; // length of underlying x; same as length(ghigh) and length(glow)

It looks like it will lead to an overflow for 1423657324 * 2...

[shrektan]

@renkun-ken Would you mind to have a test? Just cut the row count of your data to 1073741823L / 1073741824L respectively and try your original code.

I'm expecting the first case (1073741823L row) works but the second will fail.

[shrektan]

I believe that's the cause. The overflow leads to an infinite memory allocation...

Rcpp::cppFunction("size_t test(int x) {
                    return x*2*sizeof(int);
                  }")
test(1073741823L)
#> [1] 8589934584
test(1073741824L)
#> [1] 1.844674e+19
test(1423657324L)
#> [1] 1.844674e+19

^{Created on 2020-03-10 by the reprex package (v0.3.0)}

[renkun-ken]

Yes, 1073741823L case works perfectly while 1073741824L case does not work as you expected.

[shrektan]

@renkun-ken It would be great if you can verify that PR #4297 does fix this issue, when you have time, of course.

[renkun-ken]

@shrektan Thanks! I'll verify it soon.

[renkun-ken]

@shrektan sadly too hard to fetch the repo

remote: Enumerating objects: 1431, done.
remote: Counting objects: 100% (1431/1431), done.
remote: Compressing objects: 100% (181/181), done.
Timeout, server github.com not responding. KiB | 2.00 KiB/s  
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

tried many times but no luck

[shrektan]

The network issue I encountered before... so I packaged the source and have sent it to your email (I assume you are able to access your personal email).