R - read_feather() error: embedded nul in string ...

[ben519]

First off - thanks for the feather package! I use it all the time to quickly transfer large data between R and Python, and just to read and write data because it's FAST.

Unfortunately I frequently run into the same issue in R and debugging it has been tricky. I usually work with data.tables, writing and reading them as feather files with something like

write_feather(transactions, "Data/transactions1.feather")
transactions1 <- data.table(read_feather("Data/transactions1.feather"))

Unfortunately, I frequently get the error "embedded nul in string". In this case,

Error in coldataFeather(x, i) : 
  embedded nul in string: 'Acoustic\0\0\0\0\0\0\031\xa5\x9b\001)\xa5\x9b\001礛\001\006\xa5\x9b\0016\xa4\x9b\0017\xa4\x9b\001\x9d\xa4\x9b\001\x96\xa4\x9b\001\xf3\xa2\x9b\001M\xa3\x9b\001K\xa3\x9b\001P\xa3\x9b\001a\xa3\x9b\001\xba\xa9\x9b\001\xed\xa3\x9b\001\x8a\xa3\x9b\001\xff\xa2\x9b\0011\xa3\x9b\001\xb1\xa3\x9b\001a\xa2\x9b\001\b\xa3\x9b\001\ua89b\001\xf2\xa2\x9b\001\x98\xa9\x9b\001\xaa\xa2\x9b\001\xa9\xa2\x9b\001\xb0\xa2\x9b\001١\x9b\001R\xa1\x9b\001[\xa1\x9b\001H\xa1\x9b\001\xb6\xa2\x9b\001s\xa1\x9b\001Ѡ\x9b\001\x87\xa9\x9b\001\x96\xa0\x9b\001\x99\xa1\x9b\001\x9c\xa1\x9b\001-\xa1\x9b\001|\xa0\x9b\001\xa6\xa0\x9b\001\xab\xa0\x9b\001f\xa0\x9b\001h\xa0\x9b\001(\xa0\x9b\001\x81\xa9\x9b\0017\xa0\x9b\001\x80\xa9\x9b\001\a\xa1\x9b\001ӟ\x9b\001\xbb\x9f\x9b\001\xbc\x9f\x9b\001m\x9f\x9b'

Debugging is epecially weird - if I try slicing my data in half, sometimes each half of the dataset will read and write as feather format just fine. Needless to say I haven't been able to build a reproducible example of this error and I can't share my large transactions dataset. Any tips to help me figure out what's going wrong?

str(transactions1)
Classes ‘data.table’ and 'data.frame':  3000001 obs. of  6 variables:
 $ ArticleID    : int  13516378 13516378 13516378 13516379 13516379 13516379 13516379 13516379 13516379 13516379 ...
 $ ArticleTags  : int  34 34 34 24 24 24 24 24 24 24 ...
 $ Tagset       : chr  "Apex predator|Autonomy|City|Climate|Ethics|Exercise|Human resource management|Jacksonville Jaguars|Jacksonville, Florida|Jaguar"| __truncated__ "Apex predator|Autonomy|City|Climate|Ethics|Exercise|Human resource management|Jacksonville Jaguars|Jacksonville, Florida|Jaguar"| __truncated__ "Apex predator|Autonomy|City|Climate|Ethics|Exercise|Human resource management|Jacksonville Jaguars|Jacksonville, Florida|Jaguar"| __truncated__ "AFC North|Baltimore Ravens|Blood|Cincinnati|Cincinnati Bengals|Cleveland Browns|Discrimination|Emotion|Hatred|Heinz|Hematology|"| __truncated__ ...
 $ TransactionID: int  153089414 153089435 153089428 153089444 153089445 153089448 153089450 153089446 153089447 153089453 ...
 $ TagID        : int  23892 26058 26229 344 1977 2776 4828 4829 4963 7076 ...
 $ Tag          : chr  "Stained glass" "Trousers" "U.S. state" "AFC North" ...
 - attr(*, ".internal.selfref")=<externalptr> 

[keithgw]

I am having a similar issue with sufficiently large data sets. I have a feather file that is 64.7 MB

str(raw)
Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   90650 obs. of  58 variables:

if I write the feather file to disk, then reread it, I get the error:

raw <- read_feather('data_for_model.feather')
Error in coldataFeather(x, i) : embedded nul in string: '1\0'

As suggested by ben519, I can split the data_frame in half and read and write both halves separately with success.
Last thing, the data_frame does contain character vectors.

[jayflatland]

I'm getting the same "embedded nul in string:" error. I'm running the latest from CRAN - feather 0.3.0. I'm trying to reproduce the problem reliably, but am also noticing that it seems to go away with minor perturbations - reinstalling the library, running on different datasets of similar size/content.

[jkaupp]

I'm getting the same error on read_feather, but the dataset is only 6.8 Mb.

R version 3.3.1 (2016-06-21)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.6 (El Capitan)

locale:
[1] en_CA.UTF-8/en_CA.UTF-8/en_CA.UTF-8/C/en_CA.UTF-8/en_CA.UTF-8

attached base packages:
[1] grid stats graphics grDevices utils datasets methods base

other attached packages:
[1] gtable_0.2.0 gridExtra_2.2.1 feather_0.3.0 stringr_1.1.0 jkmisc_0.1.0 janitor_0.2.0
[7] jailbreakr_0.0.1 rexcel_0.0.1 dplyr_0.5.0.9000 purrr_0.2.2 readr_1.0.0 tidyr_0.6.0
[13] tibble_1.2 ggplot2_2.1.0.9001 tidyverse_1.0.0

loaded via a namespace (and not attached):
[1] Rcpp_0.12.7 xml2_1.0.0.9001 magrittr_1.5 hms_0.2 munsell_0.4.3
[6] lattice_0.20-34 colorspace_1.2-6 R6_2.2.0 rematch_1.0.1 plyr_1.8.4
[11] tools_3.3.1 linen_0.0.3 DBI_0.5-1 lazyeval_0.2.0 assertthat_0.1
[16] stringi_1.1.1 cellranger_1.1.0.9000 scales_0.4.0 zoo_1.7-13

[jayflatland]

I'm not 100% sure this is a factor, but it seems like most of the times a feather file fails to load is when I've overwritten an existing feather file. If I manually delete the feather file and re-write it with the exact same dataset, the new feather file seems to work.

[gshotwell]

I also received this error when reading in a large (1.6 gb) feather file that was overwritten. When I deleted the original file and re-saved it the problem went away.

[wesm]

We've fixed this with the 0.3.1 release -- pending release to CRAN.

[kylebarron]

I'm getting the same error using read_feather() on a 1.5 Gb file using feather 0.3.1 from CRAN. I've tried deleting the original file and resaving it but I've gotten this error three times in a row.

Error in coldataFeather(x, i) :
embedded nul in string: 'I would like to say that we had an amazing experience, but it was not. When we arrived, there were towels in the dryer and a load in the washer. The housekeeper showed up within 10 minutes of us being there to get the things out '

R version 3.3.2 (2016-10-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] gtrendsR_1.3.5 ivpack_1.2 AER_1.2-4 survival_2.40-1 sandwich_2.3-4 lmtest_0.9-34 zoo_1.7-14 car_2.1-4 stargazer_5.2 lazyeval_0.2.0
[11] feather_0.3.1 gdata_2.17.0 magrittr_1.5 forcats_0.1.1 lubridate_1.6.0 stringr_1.1.0 dplyr_0.5.0 purrr_0.2.2 readr_1.0.0 tidyr_0.6.0
[21] tibble_1.2 ggplot2_2.2.0 tidyverse_1.0.0

[wesm]

any chance to get us access to the source data to try to debug?

[kylebarron]

Sure, I'm working with publicly accessible data. Here's the link to the feather file in my dropbox. Let me know if you need anything else. I still get the same error as yesterday when I try to import that file.

[jameslamb]

An Example:

I was able to reproduce this bug in R tonight. I get this error on files as small as 1.2 GB, even when those files were created from scratch (i.e. not the "overwriting" situation @gshotwell mentioned).

# Set seed for reproducibility
set.seed(1234)

# Load deps
library(data.table)
library(feather)

#===== Test 1: Smallest data.table (~240 MB in memory) =====#

    # Make the data.table
    smallDT <- data.table::data.table(x = rnorm(15*1e6), y = rep("23886fac-4311-41e3-83d2-c9c3e7556af3", 15*1e6))
    object.size(smallDT)
    
    # Write out to feather file
    feather::write_feather(smallDT, "smallDT.feather")
    
    # Clean out env
    rm(list=ls())
    
    # Try to read back in (works with no issue)
    smallDT <- feather::read_feather("smallDT.feather")

#===== Test 2: Medium data.table (~1.2 GB in memory) =====#

    # Make the data.table
    mediumDT <- data.table::data.table(x = rnorm(75*1e6), y = rep("23886fac-4311-41e3-83d2-c9c3e7556af3", 75*1e6))
    object.size(mediumDT)
    
    # Write out to feather file
    feather::write_feather(mediumDT, "mediumDT.feather")
    
    # Clean out env
    rm(list=ls())
    
    # Try to read back in (will throw an encoding error)
    mediumDT <- feather::read_feather("mediumDT.feather")
    
    # Error in coldataFeather(x, i) : 
    #     embedded nul in string: '\024\0\0\0\0\0\016\0\030\0\0\0\f\0\004\0\b\0\0\0\016\0\0\0\024\0\0\0\002\0\0\0\xc0hx\004'

#===== Test 3: Even bigger data.table (~2.5 GB in memory) =====#
    
    # Make a bigger data.table (2.5 GB in memory)
    bigDT <- data.table::data.table(x = rnorm(150*1e6), y = rep("23886fac-4311-41e3-83d2-c9c3e7556af3", 150*1e6))
    object.size(bigDT)
    
    # Write out to feather file
    feather::write_feather(bigDT, "bigDT.feather")
    
    # Clean out env
    rm(list=ls())
    
    # Try to read back in (will throw an encoding error)
    inDT <- feather::read_feather("bigDT.feather")

    # Error in coldataFeather(x, i) : 
    #     embedded nul in string: '23886fac-4311-41e3-83d2-c9c3e755\024\0\0\0'

Reading the files into Python throws similar errors. Hit these issues on CentOS and Mac El Capitan (environment info below)

Is this being worked on? Is it a known limitation of feather that files above a certain size will fail on read in this way?

Mac info:

R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X El Capitan 10.11.6

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets
[6] methods base

other attached packages:
[1] feather_0.3.1 data.table_1.10.4 plyr_1.8.4

loaded via a namespace (and not attached):
[1] assertthat_0.1 hms_0.3 tools_3.3.2
[4] tibble_1.2 Rcpp_0.12.9

CentOS info:

R version 3.3.1 (2016-06-21)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods
[7] base

loaded via a namespace (and not attached):
[1] assertthat_0.1 hms_0.3 tools_3.3.1
[4] tibble_1.2 Rcpp_0.12.9 feather_0.3.1
[7] data.table_1.10.4

[Paxanator]

As a follow up on the above,

> mediumDT <- data.table::data.table( y = rep("23886fac-4311-41e3-83d2-c9c3e7556af3", 75*1e6))
> object.size(mediumDT)
600001160 bytes
> feather::write_feather(mediumDT, "mediumDT.feather")
> rm(list=ls())
> mediumDT <- feather::read_feather("mediumDT.feather")
Error in coldataFeather(x, i) : 
  embedded nul in string: '\024\0\0\0\0\0\016\0\024\0\0\0\f\0\004\0\b\0\0\0\016\0\0\0\020\0\0\0\002\0\0\0\xc0hx\004'
> 

This call fails but this call succeeds, with the only difference being a shorter string within the column.

> mediumDT <- data.table::data.table( y = rep("23886fac-4311-41e3", 75*1e6))
> object.size(mediumDT)
600001144 bytes
> feather::write_feather(mediumDT, "mediumDT.feather")
> rm(list=ls())
> mediumDT <- feather::read_feather("mediumDT.feather")
> 

I'm speculating, but must have to do with the total number of bytes in one column or something to that effect. Ideally feather would need to partition the data correctly prior to dumping it to disk and could reconstitute the large column after read in, is that a good approach?

[kanishkamisra]

Is there an update to this issue? I have the same error while calling read_feather() on a 3 GB feather file (written in feather format from a 18.2 GB csv file after merging a bunch of csvs). No idea how I should reproduce this properly..