[R] Factor type not preserved in Parquet roundtrip

[asfimport]

test_that("Factors are preserved when writing/reading from Parquet", {
  tf <- tempfile()
  on.exit(unlink(tf))
  df <- data.frame(a = factor(c("a", "b")))
  write_parquet(df, tf)
  expect_equivalent(read_parquet(tf), df)
})

Fails:

`object` not equivalent to `expected`.
Component “a”: target is character, current is factor

This has to do with the translation with Parquet and not the R <--> Arrow type mapping (unlike ARROW-7028). If you write_feather and read_feather, the test passes.

Reporter: Neal Richardson / @nealrichardson
Assignee: Hiroaki Yutani / @yutannihilation

PRs and other links:

• GitHub Pull Request #6135

_{Note: This issue was originally created as ARROW-7045. Please see the migration documentation for further details.}

[asfimport]

Hiroaki Yutani / @yutannihilation:
I think I found the cause. This is because ParquetArrowWriterProperties$store_schema() is not called by default, the first if branch in ParquetArrowWriterProperties$create below:

  if (!use_deprecated_int96_timestamps && is.null(coerce_timestamps) && !allow_truncated_timestamps) {
    shared_ptr(ParquetArrowWriterProperties, parquet___default_arrow_writer_properties())
  } else {
    builder <- shared_ptr(ParquetArrowWriterPropertiesBuilder, parquet___ArrowWriterProperties___Builder__create())
    builder$store_schema()
    builder$set_int96_support(use_deprecated_int96_timestamps)
    builder$set_coerce_timestamps(coerce_timestamps)
    builder$set_allow_truncated_timestamps(allow_truncated_timestamps)
    shared_ptr(ParquetArrowWriterProperties, parquet___ArrowWriterProperties___Builder__build(builder))
  }

(

arrow/r/R/parquet.R

Lines 208 to 217 in dd6b17d

	if (!use_deprecated_int96_timestamps && is.null(coerce_timestamps) && !allow_truncated_timestamps) {
	shared_ptr(ParquetArrowWriterProperties, parquet___default_arrow_writer_properties())
	} else {
	builder <- shared_ptr(ParquetArrowWriterPropertiesBuilder, parquet___ArrowWriterProperties___Builder__create())
	builder$store_schema()
	builder$set_int96_support(use_deprecated_int96_timestamps)
	builder$set_coerce_timestamps(coerce_timestamps)
	builder$set_allow_truncated_timestamps(allow_truncated_timestamps)
	shared_ptr(ParquetArrowWriterProperties, parquet___ArrowWriterProperties___Builder__build(builder))
	}

)

Actually, specifying either of the arguments works. For example:

test_that("Factors are preserved when writing/reading from Parquet", {
  tf <- tempfile()
  on.exit(unlink(tf))
  df <- data.frame(a = factor(c("a", "b")))
  write_parquet(df, tf, allow_truncated_timestamps = TRUE)
  expect_equivalent(read_parquet(tf), df)
})

According to #5077 :

Add ArrowWriterProperties::store_schema() option which stores the Arrow schema used to create a Parquet file in a special ARROW:schema key in the metadata, so that we can detect that a column was originally DictionaryArray. This option is off by default, but enabled in the Python bindings. We can always make it the default in the future

I think the R bindings can follow the Python and always use ParquetArrowWriterPropertiesBuilder instead of default_arrow_writer_properties(). If this sounds good, I'm happy to send a PR for this.

[asfimport]

Neal Richardson / @nealrichardson:
Nice work. A PR would be great, thanks!

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 6135
#6135