fread zipped files stored online

[rafapereirabr]

Hi all. You have done a fantastic work with fread and I know it can both read .csv files stored online and zipped .csv files stored locally. In the first case, fread automatically downloads the file, while in the second case it automatically unzips the file. My question is whether it would be possible to read a zipped .csv file that is stored online in a similar way, with a simple fread call .

reproducible example:

library(data.table)

online_file <- 'https://www.gov.br/anac/pt-br/assuntos/regulados/empresas-aereas/envio-de-informacoes/microdados/basica2000-02.zip'

# download to a local file
local_file <- tempfile('basica2010-02.zip')
utils::download.file(url = online_file, destfile = local_file)

# read zipped file stored locally
read_local <- fread( cmd = paste0('unzip -p ', local_file) )

# attempts to read zipped file stored online
read_online1 <- fread( online_file )
read_online1 <- fread( cmd = paste0('unzip -p ', online_file) )

sessionInfo:

> sessionInfo()
R version 4.1.1 (2021-08-10)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.14.2

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.8         pillar_1.6.4       compiler_4.1.1     r5r_0.6.0         
 [5] prettyunits_1.1.1  remotes_2.4.2      class_7.3-19       easypackages_0.1.0
 [9] tools_4.1.1        testthat_3.1.1     pkgbuild_1.3.1     pkgload_1.2.4     
[13] tibble_3.1.6       memoise_2.0.1      lifecycle_1.0.1    pkgconfig_2.0.3   
[17] rlang_0.4.12       cli_3.1.0          DBI_1.1.2          curl_4.3.2        
[21] fastmap_1.1.0      rJava_1.0-6        e1071_1.7-9        dplyr_1.0.7       
[25] withr_2.4.3        generics_0.1.1     vctrs_0.3.8        desc_1.4.0        
[29] fs_1.5.2           devtools_2.4.3     sfheaders_0.4.0    tidyselect_1.1.1  
[33] grid_4.1.1         classInt_0.4-3     rprojroot_2.0.2    glue_1.6.0        
[37] sf_1.0-5           R6_2.5.1           processx_3.5.2     fansi_1.0.2       
[41] sessioninfo_1.2.2  callr_3.7.0        purrr_0.3.4        magrittr_2.0.1    
[45] units_0.7-2        ps_1.6.0           ellipsis_0.3.2     usethis_2.1.5     
[49] assertthat_0.2.1   utf8_1.2.2         KernSmooth_2.23-20 proxy_0.4-26      
[53] cachem_1.0.6       crayon_1.4.2  

[ben-schwen]

Sounds like a duplicate of what #5240 does. Could you try to update to dev-version with data.table:::update.dev.pkg() and run the following reprex?

If it's not working then it might be a windows related issue or something related to curl.

Reprex:

library(data.table)
url = 'https://www.gov.br/anac/pt-br/assuntos/regulados/empresas-aereas/envio-de-informacoes/microdados/basica2000-02.zip'
fread(url, verbose=TRUE)
#>   data.table is using 6 threads with throttle==1024. See ?setDTthreads.
#> freadR.c has been passed a filename: /tmp/RtmpWZkn9m/basica2000-02.txt
#> [01] Check arguments
#>   Using 6 threads (omp_get_max_threads()=12, nth=6)
#>   NAstrings = [<<NA>>]
#>   None of the NAstrings look like numbers.
#>   show progress = 0
#>   0/1 column will be read as integer
#> [02] Opening the file
#>   Opening file /tmp/RtmpWZkn9m/basica2000-02.txt
#>   File opened, size = 56.43MB (59169409 bytes).
#>   Memory mapped ok
#> [03] Detect and skip BOM
#> [04] Arrange mmap to be \0 terminated
#>   \n has been found in the input and different lines can end with different line endings (e.g. mixed \n and \r\n in one file). This is common and ideal.
#> [05] Skipping initial rows if needed
#>   Positioned on line 1 starting: <<"id_basica";"id_empresa";"sg_e>>
#> [06] Detect separator, quoting rule, and ncolumns
#>   Detecting sep automatically ...
#>   sep=';'  with 100 lines of 110 fields using quote rule 0
#>   Detected 110 columns on line 1. This line is either column names or first data row. Line starts as: <<"id_basica";"id_empresa";"sg_e>>
#>   Quote rule picked = 0
#>   fill=false and the most number of columns found is 110
#> [07] Detect column types, good nrow estimate and whether first row is column names
#>   Number of sampling jump points = 100 because (59169407 bytes from row 1 to eof) / (2 * 91228 jump0size) == 324
#>   Type codes (jump 000)    : 66DDDDD6666DDB66D6D6D6D666DDDDDDB66D6D6D6D666DDDDDDDD6DB66D6D6D6D666DD16DDDDDDDD...6666666D6C  Quote rule 0
#>   Type codes (jump 001)    : 66DDDDD6666DDB66D6D6D6D666DDDDDDB66D6D6D6D666DDDDDDDD6DB66D6D6D6D666DDD6DDDDDDDD...6666666D6C  Quote rule 0
#>   Type codes (jump 100)    : 66DDDDD6666DDB66D6D6D6D666DDDDDDB66D6D6D6D666DDDDDDDD6DB66D6D6D6D666DDD6DDDDDDDD...6666666D6C  Quote rule 0
#>   'header' determined to be true due to column 1 containing a string on row 1 and a lower type (int32) in the rest of the 10051 sample rows
#>   =====
#>   Sampled 10051 rows (handled \n inside quoted fields) at 101 jump points
#>   Bytes from first data row on line 2 to the end of last row: 59167309
#>   Line length: mean=884.43 sd=53.07 min=577 max=1050
#>   Estimated number of rows: 59167309 / 884.43 = 66900
#>   Initial alloc = 76022 rows (66900 + 13%) using bytes/max(mean-2*sd,min) clamped between [1.1*estn, 2.0*estn]
#>   =====
#> [08] Assign column names
#> [09] Apply user overrides on column types
#>   After 0 type and 0 drop user overrides : 66DDDDD6666DDB66D6D6D6D666DDDDDDB66D6D6D6D666DDDDDDDD6DB66D6D6D6D666DDD6DDDDDDDD...6666666D6C
#> [10] Allocate memory for the datatable
#>   Allocating 110 column slots (110 - 0 dropped) with 76022 rows
#> [11] Read the data
#>   jumps=[0..60), chunk_size=986121, total_size=59167309
#> Read 66887 rows x 110 columns from 56.43MB (59169409 bytes) file in 00:00.216 wall clock time
#> [12] Finalizing the datatable
#>   Type counts:
#>          1 : bool8     '1'
#>         57 : int32     '6'
#>          3 : int32     'B'
#>          1 : float64   'C'
#>         48 : string    'D'
#> =============================
#>    0.000s (  0%) Memory map 0.055GB file
#>    0.017s (  8%) sep=';' ncol=110 and header detection
#>    0.000s (  0%) Column type detection using 10051 sample rows
#>    0.010s (  5%) Allocation of 76022 rows x 110 cols (0.045GB) of which 66887 ( 88%) rows used
#>    0.189s ( 87%) Reading 60 chunks (0 swept) of 0.940MB (each chunk 1114 rows) using 6 threads
#>    +    0.026s ( 12%) Parse to row-major thread buffers (grown 0 times)
#>    +    0.109s ( 51%) Transpose
#>    +    0.053s ( 25%) Waiting
#>    0.000s (  0%) Rereading 0 columns due to out-of-sample type exceptions
#>    0.216s        Total
#>        id_basica id_empresa sg_empresa_icao sg_empresa_iata
#>            <int>      <int>          <char>          <char>
#>     1:    775299    1000357             SLM              PY
#>     2:    775298    1000357             SLM              PY
#>     3:    775326    1000357             SLM              PY
#>     4:    775327    1000357             SLM              PY
#>     5:    775328    1000357             SLM              PY

[rafapereirabr]

Hi @ben-schwen , it works perfectly ! Thanks! Do you know when this fix will be incorporated in the CRAN version of data.table?

[ben-schwen]

Ok then closing since duplicate.

@rafapereirabr
AFAIK there is no fixed timeline and there are still some things that need to be patched up for the next CRAN release. As @mattdowle pointed out in another thread it could take weeks or even months before this release (which is growing pretty fat) is on CRAN.