Master task for fread bugs / proposals

[st-pasha]

Bugs (critical)

• Occasionally fread garbles the input data (Race condition causes fread to read the data incorrectly occasionally #2260)
• Crash when fread underestimates number of rows in a file (Crash in fread in dev when under-estimating the number of rows #2246)
• Crash when nrows argument is too small relative to the size of the file (fread fails when max_nrows argument is too small #2243)
• Crash when reading a file whose size is a multiple of 4096 (fread segfaults when reading a file with size a multiple of 4k #2194, fread on particular file crashes R #2228, Segmentation fault when reading CSV #2238, Reproducible crash in fread #1464)
• Crash when using parameter skip with files whose size is a multiple of 4096 (fread crashes when using option skip with some data files #2201)
• Potential crash when error happens on a line containing % character ([fread] Error message is not displayed when a field ends with percent sign #2287)
• Crash on Windows when seeing less columns than expected, and fill is false (Segfault with fread #2299)
• Possible crash when a line has extra sep at the end (Fread: possible segfault when reading a file with extra sep on out-of-sample row #2523)

Bugs (core)

• unable to handle mis-quoted fields if they first appear after the first 100 lines (fread Unable to handle mis-quoted field if it is out-of-sample #2265)
• big integers are read incorrectly (fread garbles integers that are too big #2250, fread could read integers > 2^63 as character #1618)
• when one of na.strings is a number, it may fail to work properly (na.strings may fail to convert numbers into NAs in dev #2222)
• eliminate message "can be slowed down by other apps" (fread: remove "can be slowed down by other apps" scold message #2197)
• wrong heuristic in predicting num cols: quote rule should take precedence (fread heuristic predicts wrong column count on a valid file #2196)
• fread may report negative waiting time in verbose mode (fread verbose waiting time can be negative #2175)
• fread may produce error "sampling jump point is before the last jump ended" (fread error: Sampling jump point is before the last jump ended #2173)
• fread may produce error "could not find first good line start after jump point 1" (Unexpected "Could not find first good line start after jump point 1 when sampling" #2134)
• fread may fail to "find first good line start" even with fill=True (fread fails on file with inconsistent # columns #2267)
• better handle NAs in single-column input (fwrite/fread single column input with NA -vs- empty lines #2106)
• fread reports warning about "found last consistent line but text exists afterwards" even when nrows argument is used (fread warning on "stopped reading" should interact with nrows #1671)
• fread discards column names if there are less names than data columns (fread incorrectly skips header on this file #1625)
• fread cannot read a file if it ends with 0x1A character (Control-Z as EOF #1612)
• if incomplete row occurs at the top of the file, fread skips it rather than suggesting to use "fill=TRUE" (fread ignores first 2 lines of file with tabs after data in several lines  #1416)
• type detector should take into account parameters skip and nrows ([fread] broken functionality in 1.9.6 #1267)
• no longer reads all files from the extended test suite ([Regression] fread can no longer read all files from the extended test suite #2285)
• unable to parse a file if it contains very long multiline text fields (fread with many new lines in string #2395, Unable to parse Kaggle "toxic comments" dataset #2600)
• fread's select should advise using drop if receiving negative indices (Negative select in fread #2423)
• Cannot parse file containing embedded NULs (fread error reading Latin-1 file containing NUL byte <0x00> #2435, is it possible to ignore error and coninue reading from file #2496)
• unable to read file containing quoted last field, which is excluded via select (Error in fread when selecting a subset of columns, and one of the fields contains a sep  #2464)
• strip.white=FALSE is not respected (fread's skip.white=FALSE behavior #2492)
• NA values are accidentally omitted in a 1-column file (1-column file with NAs is read incorrectly #2515)
• 1x1 DT with NA value is read should be read as logical, not character (File containing a single NA value is read as chr instead of logi #2516)
• Number of rows is estimated incorrectly for small files (fread: invalid estimation of DT size for small files #2518)
• File with quoted integer fields may be read as character (File with quoted integer fields may be read as character #2520)
• fread fails if maxnrow=0 and file has just 1 header row (fread's nrows=0 errors if input is an empty table #2512)
• erroneous detection of column names when they are not present (Error in logic for column name detection #2526)
• \r newlines are squashed in a single-column file (\r newlines are squashed in a single-column file #2542)
• last line with whitespace messes up sep detection for small files (Separator not detected in trivial file with whitespace at the end #2543)
• Error "jump X did not finish counting rows exactly where jump X+1 found its first good line start" (Jump 0 did not finish counting rows exactly where jump 1 found its first good line start #2561)
• quoted na strings are not recognized properly (na.strings is too literal when column is quoted on file #2586)
• error when int columns becomes string containing sep on an out-of-sample row (fread error if integer column bumps into quoted field containing comma out-of-sample #2614)
• some strings are erroneously parsed as floats (fread parses some invalid tokens as numeric #2625)
• explicitly specified sep may sometimes be ignored (fread fill=true and sep= provided could still read as 1-column #2666)
• fread may improperly detect sep within quoted fields (fread improperly detects sep within quoted fields #2667)
• NAs may not be recognized correctly if surrounded by whitespace (in fread, NAs do not interact properly with whitespace  #2697)
• logical01 parameter may prevent column name detection (logical01 parameter may prevent column name detection #2735)
• quote rule may be detected wrongly for a 1-column file (Bad quote rule detection for a backslash-escaped test file #2744)
• sep is not detected when nrows=0 (nrows=0 interferes with proper sep detection #2747)
• dec=',' may cause file to be parsed incorrectly (fread may fail to parse valid file when dec=',' #2750)
• na.strings="0" is not permitted (na.strings="0" is not permitted in fread #2927)
• self-healing regression for fill=TRUE with unmatched quotes (fread from 1.11: self healing regression for fill = T when unmatched quote occurs #2859)

Bugs (R)

• Memory leak on Windows when reading file with many subnormal floats (fread memory leak when small number bumped up to character #918)
• escaped / doubled quote marks should be unescaped / undoubled (fread should un-escape escaped quotes in fields #1109, fread: quotes in quoted string fields #1299)
• add option to customize location where temporary files are stored ([Request] allow tmpDir to be supplied as argument: fread can run out of tmpfs space on unix during preprocessing #1139)
• using select = NULL causes an error (fread error with explicit 'select = NULL' #2007)
• Crash when using fread in conjunction with xlsx package (stack overflow error while executing fread after loading xlsx package #2220)
• colClasses=integer no longer working (colClasses=integer no longer working in fread #2251)
• fread wastes too much time to match URL regexp when input is an in-memory buffer (fread spends too much time in is_url/is_secureurl/is_file for long in memory input #2531)
• integer64 parameter does not work out-of-sample (in fread integer64 argument is ignored if int64 type-bump occurs out-of-sample #2749, fread(integer64 = "double") not working for some data #2607)
• colClasses="logical" may produce a spurious error (colClasses=logical is no longer working #2766)
• fread session is not cleaned up after a try-catch statement (fread fails when warning is caught: "Previous fread() session was not cleaned up properly. Cleaned up ok at the beginning of this fread() call" #2904)

Feature requests (core)

• support hexadecimal floating point numbers (Hex floating-point numbers no longer parse in 1.10.5 #2316)
• refactor code so that file reading is done through a single common procedure (consolidate code for reading data from file #2261)
• better report file size in verbose mode (Better report file size in verbose output #2252)
• fread should distinguish between empty string and n/a string (fread/fwrite default options for NA mismatch #2281, fread should distinguish between quoted and unquoted empty strings #2217, Irreversible empty string handling by fread() and fwrite() #2214, fread reads in empty fields as logical NA #1159)
• add ability to skip "broken" lines and report them to the user in the end (Ability to export lines fread thinks are incorrectly classed #2163, fread: need more flexible behavior when encountering a broken line. #2263, fread option to skip certain rows or rows with reading errors #2765)
• auto-detect NA strings in a column (fread - if there is one string in a numeric column, guess it represents NA #2100)
• add option to skip initial rows but not the header (fread skip doesn't get the header names. #2080)
• allow multi-character field separators (Support multi-character sep argument #2011 )
• detect newlines more reliably if file contains different newlines (fread should be more robust in detecting newlines #2248, fread crashes on files with mixed Windows and Unix line endings #1183)
• allow fread to understand files where first column contains row indices and have no name (for example such file may be produced by write.table) (fread row.names option #1818, fread does not handle row.names produced by write.table #763)
• make fread able to process files with irregular number of columns, where some rows may have more columns than originally anticipated (fread problem with different number of columns #1812)
• fread should understand special float values 1#.INF, 1.#IND, 1.#QNAN, and 1.#SNAN (Handle 1.#INF, 1.#IND, 1.#QNAN and 1.#SNAN  #1800)
• allow fread to process input file in chunks, i.e. add ability to stop/resume scanning (Chunked fread #1721)
• support for CSVY format (Add CSVY support for fread() #1701)
• allow nrow argument to be negative, meaning skip that many lines from the end (fread's nrow argument could accept -ve values to skip last 'n' rows #1643)
• add support for numbers with thousand separator ([Request] fread to support thousand separator #1636, [Request] Strip commas from columns declared numeric in fread #1723)
• add simple option to read file without splitting it into fields (FR: Robust option for fread not to split lines #1616, Document that fread can readLines with sep='@' (non present character) #1627, readLines with fread #1518)
• fread should be able to understand fixed-width format files (fread for huge fixed width files with smart handing of trailing spaces #1345)
• when detecting columns the case of single column should also be considered ( [Request] Please include "\n" as additional default separator for parameter sep in fread() for improved backwards-compatibility #1073)
• make warning messages less intimidating ([Request] [Low] Shortening fread Warning Output #1042)
• allow reading part of a file >4Gb on a 32-bit machine ([Request] With fread, do not throw 32 bit large file support error if nrows is specified #949)
• add option to skip lines starting with a comment character (Implement comment.char argument in fread #856 )
• Using nrows= could be parallel again by just issuing a (rare) warning if any out-of-sample type bumps occurred, since they could occur after the nrow'th row, as mentioned in PR Better skip= and nrow= #2623. This warning would be acceptable and would avoid extra code complexity and memory usage.
• Should always warn when 'healing' quote rule is used, even in-sample. And no need for warning if no quotes are in sample and then it bumps from rule 0 to rule 1 (unambiguous and robust). Follow up to fread Unable to handle mis-quoted field if it is out-of-sample #2265.
• fread should be able to parse data.table's printout (fread parse standard data.table print output #2720)
• detect binary files as input, and issue proper error to the user (fread does not stop when input is unsupported (e.g. binary file) #2834)

Feature requests (R)

• allow fread to work on character vectors ([Request] add text= argument to fread, to read from multi-line character vectors #1423)
• if select is a list, then colClasses should accept a list of same length (FR for fread: if select is used, colClasses need only correspond to the columns in select #1426)
• allow colClasses for columns not present in the file (FR: in colClasses of fread, allow column names not present in actual table #1602)
• allow stringsAsFactors parameter to be a fraction between 0 and 1 (fread stringsAsFactors=0.2 #2025)
• read feather files with fread (fread feather #2026)
• more flexible parameters select / col.names / colClasses ([Request] Ability to use OR statements within fread #2185, select via pattern in fread #2066, [R-Forge #5323] fread: colClasses does not convert to non-builtin types #491)
• Support UTF-16 encoded files (Support UTF-16 encoded files in fread #2560)
• fread directly from the clipboard on windows (fread from "clipboard" or "clipboard-128" (on Windows) fails #1292 )
• auto-detect and parse JSON fields within a CSV file (autodetect json fields and use jsonlite to parse them into lists #1240)
• implement sep2 (when will sep2 in fread be implemented? #1162)
• apply filtering function to detect which rows to read ([R-Forge #2605] add filtering option to fread so it can load less than all rows #583 )
• add option to control destination for verbose messages (verbose=T - con feature request #775)
• allow reading .gz files directly (Support .gz file format for fread #717)
• read multiple files (specified as a list or a glob) (fread could accept multiple files #586, fread for directories #2582)
• support reading files with more than 2B rows ([R-Forge #4649] fread files with more than 4 billion rows #570)
• support file connections as an input source ([R-Forge #4931] Support file connections for fread #561)
• drop could drop all duplicated column names (e.g. many "filler" or "" column names) (Improve how fread's drop/select deal with duplicate column names #1899)
• Do not use curl::download in R 3.2.2+, since base::download.file is more robust ([bug] fread does not work with https url behind a proxy #1686)
• Parse date types natively (fread/fwrite *base data types* directly for efficiency #1656, fread should read columns of Date/POSIXct types directly #1450)
• Check that colClasses have valid types (fread doesn't check colClasses to be valid type #1634)
• Skip sampling when all colClasses supplied (fread skip sampling when all colClasses provided #3024)
• Free over-allocated space (fread free over-allocated space #3025)
• fills character fields with empty strings or NAs (fread(fill=TRUE) fills character fields with empty strings instead of NAs #2524)

Miscellaneous

• Performance test on huge CSV files (fread with large csv (44 GB) takes a lot of RAM in latest data.table dev version #2073)
• Crash report on an .xz file (fread crashes when reading large file with binary data #1895)

                        open  closed
critical                   0       8
core bug                   8      32
bugs R                     4       6
                          ==========
                          12      46   /58
                                                   
feature requests (core)   16       8 
feature requests (R)      22       0
misc                       2       2
                          ==========
                          40      10   /50