fread "segfault from C stack overflow" using 1M+ columns

[malbianco]

I'm using "fread" function to read a file with 160 rows and columns 1141430.
The function will stop reporting an error "Error: segfault from C stack overflow" and closing the R session.
The file contains letters. Only the first two columns contain names.
This problem occurs only in the Mac system, and works well on Linux and Windows environments.

Mac OSX: 10.12.2
Version: 1.10.0 data.table
Files: http://datadryad.org/bitstream/handle/10255/dryad.104088/Dryad_Submission.7z?sequence=1
Info files: http://datadryad.org/resource/doi:10.5061/dryad.1p7sf/1
R script:
genotype_path='/Users/Gabriele/Desktop/Dryad_Submission-0/4H_160indivs_Final.ped'
pops <- data.table::fread(genotype_path, sep = " ", header = FALSE, verbose=TRUE)

verbose output :

Input contains no \n. Taking this to be a filename to open
File opened, filesize is 0.340174 GB.
Memory mapping ... ok
Detected eol as \r\n (CRLF) in that order, the Windows standard.
Positioned on line 1 after skip or autostart
This line is the autostart and not blank so searching up for the last non-blank ... line 1
Using supplied sep ' ' ... found ok
Detected 1141430 columns. Longest stretch was from line 1 to line 30
Starting data input on line 1 (either column names or first row of data). First 10 characters: Jacobs H70
'header' changed by user from 'auto' to FALSE
Count of eol: 161 (including 1 at the end)
Count of sep: 182628640
nrow = MIN( nsep [182628640] / (ncol [1141430] -1), neol [161] - endblanks [1] ) = 160
Type codes (point 0): 44111144444444444444444444444444444444444444404444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444444000044444444440044444444000044444444444444444444444444004444444444444444444444444444444444004400440044444444444444444444444444004444444444444400444444444444444444444444444444444444444444444400444400440044444444444444444444444444444444444444444444444400444400444444444444444444444444444444444444444444004400440044004400444444444444444444444444444444440000004444004400444444444444440044000044444444444444444444444444440044004400444400444444440000444444444444444444444444440044444444444444444444444444444444004444440044004444444400444444444400444444004444444444004444004400444444444444444444004444004444444400444444444444444444440044444444004444444444444444444444000044000044444400440044444444444444444444004444444444444444444444444444444444444444444444444400444444444400444444004444444...

Could you help me with this?

[jangorecki]

Reproducible on 1.10.1 and Ubuntu. Not sure why you were not able to reproduce on linux, did you try the same version?
Looks like the issue is related to amount of columns present in your file.

[bunop]

Hi @jangorecki, thank you for your reply. I think I found the problem; Is not related on OS or library version (I've tried in Linux and Windows with the last library version 1.10.0) The main point could be stack_size in R environment. For example, in (my) windows environment is:

> Cstack_info()
      size    current  direction eval_depth 
  19922944      15128          1          2 

While on my Debian is:

> Cstack_info()
      size    current  direction eval_depth 
   7969177       8848          1          2

And is a little lower than the amount needed to handle the example file. In linux (but I think also in OSX, @malbianco could you try it?) you have to increase size with ulimit (before starting R). You need execute ulimit -a to determine if size is specified in Kb or bytes. Then you could increase stack size (in your current terminal) with:

$ ulimit -s 65535

To increase size stack to 64Mb (the amount in expressed in Kb in my linux distribution). Now, by entering R, you should see a different stack size:

> Cstack_info()
      size    current  direction eval_depth 
  63753420      12992          1          2 

Now you can load the file provide in the example. More info on R memory could be found here.

Cheers,

Paolo

[danimfernandes]

@bunop, unfortunately that solution does not work for me. I am using exactly the same type of files as @malbianco.

I am able to change the stack size to 16MB, but was unable to do so for 64MB:
bash: ulimit: stack size: cannot modify limit: Operation not permitted

But even when it was changed to 16MB, Rstudio did not recognize the change when I tried Cstack_info()...

[danimfernandes]

@malbianco, if in the meantime you want a temporary fix, you can read a PED file with this transposable function:
http://stackoverflow.com/questions/17288197/reading-a-csv-file-organized-horizontally

It is faster then the normal read.csv or read.table, but still fread is faster than anything else.

[bunop]

Hi @danimag , did you get the same error type (Error: segfault from C stack overflow)? you haven't modified the stack size: see your error message: 'Operation not permitted': You should see the change with ulimit -a (from terminal) and Cstack_info() (from R). There's no need of root privileges to decrease a resource with ulimit, but you need it to increase a resource. Try to do the same command with sudo. You won't see any error message. Consider that ulimit will change limits on current terminal, so if you open another session (or if you launch Rstudio by clicking the icon on your desktop) you shouldn't see the changes. You need to open a R session (or Rstudio) from the same terminal modified with ulimit.
You can also try to change limits permanently (but I suggest you to change them when you needed it) but you need to reboot your machine to see the effect. Take a look here

Cheers

[danimfernandes]

@bunop, I use Rstudio in Linux, so did not see any error description whatsoever.
I was able to increase the stack size to 16MB without 'sudo', but after that it didn't allow me to change it another time (in the same terminal). So I see I should have opened Rstudio through that terminal, might try that later. But that being a temporary solution, I won't be able to include it on my script, which I plan to release in the future... Or add your link to the manual.

[malbianco]

I solved the problem "temporarily". My stack size was:
stack size (kbytes, -s) 8192
using.
$ ulimit -s 16384
my stack size has increased. But this is a temporary solution and I do not know the maximum limit of my stack size. Rstudio continues to always have the same problem.
If I need open a very large file I must use the terminal, use the ulimit command and after, run the R script.
The only solution is to make permanent the stack size value to the entire system.

[danimfernandes]

One could possibly include a "system" command inside Rstudio to change the stack when desired, but it might not work since you have to restart it. So for me this option does not seem viable, unfortunately, which might be the only one.

[bunop]

@danimag, you can't change this value with a system (system2) command, because when this function is called, a new shell environment is created, then the command is executed and finally the terminal is closed and all its variables are lost. This environment is different from Rstudio, so you can't change it, even by calling a system and put a very long process in background. The only way is by changing it permanently (but unfortunately I don't know how to change it on Mac OSX) or temporarly (using ulimit). Then all R and Rstudio session need to start from the same terminal (I found here a tip on how to launch Rstudio from a terminal on OSX, but I have no OSX so I have no idea if this trick works; if you want to try it on linux, simply type rstudio in terminal). IMHO is better to use R terminal in dedicated infrastructures when dealing with big data.

[danimfernandes]

@bunop That is a good point, I did not realise that about system. Opening R or Rstudio from the terminal in linux is just like you said, so no issues there. I would gladly do that in-house, but I don't feel like seeing that as an option if I eventually publish this script. If I do maybe I could put an optional argument to whether use data.table for this or not though.

[bunop]

I understand. My interest in this discussion is because I didn't found a solution to this problem. The main point is that such problem is not related to data.table itself but on your system. I realize also that such solutions are not familiar to the basic user, and need to be avoided if possible. Take a look also at bigtable (you can find an intro here). In our case this solution doesn't work because our ped files may contain letters and this is not supported at the moment by the library. Moreover, all data need to be of the same type (like a matrix). Good luck!

[danimfernandes]

Letters is all my ped files are made of, but thanks for the suggestion! :) It seems there are a few great alternatives to r-base for large datasets.