ARROW-13852: [R] Handle Dataset schema metadata in ExecPlan #11183
Conversation
nealrichardson
force-pushed
the
exec-plan-metadata
branch
from
ea1cd73 to
6f013f6
Compare
jonkeane
left a comment
jonkeane
left a comment
There was a problem hiding this comment.
This looks good — one comment about messaging if we do encounter non-R metadata it would be nice to at least flag that something is getting lost
| if (ncol(tab)) { | ||
| # Apply any column metadata from the original schema, where appropriate | ||
| original_schema <- source_data(.data)$schema | ||
| # TODO: do we care about other (non-R) metadata preservation? |
There was a problem hiding this comment.
Could we detect that it is there and warn/message that we've discarded it? It might be a little bit confusing, but I think that would be preferable to "Arrow lost all my metadata!"
There was a problem hiding this comment.
We already do discard other metadata when you convert from Arrow to R. Check out the example parquet file in inst/, for example: it has "pandas" metadata. I suspect it would be more surprising/alarming if we started messaging about this metadata all of the sudden.
There was a problem hiding this comment.
Hmm, Arrow to R conversion losing the metadata sounds like what I expected. TBH I am a little surprised that the following happens silently, since the tab below does have the pandas metadata there. Though that's been in Arrow for a while and if no one has complained, I guess it's not that important.
> tab <- read_parquet(system.file("v0.7.1.parquet", package = "arrow"), as_data_frame = FALSE)
>
> dplyr::select(tab, carat)$metadata
NULL
There was a problem hiding this comment.
Actually, on 5.0:
new <- tab %>%
select(carat) %>%
compute()
new$metadata
$pandas
[1] "{\"index_columns\": [\"__index_level_0__\"], \"column_indexes\": [{\"name\":
...
I still don't think this is necessarily meaningful--especially when you're dealing with multiple files with potentially different metadata.
Closes apache#11183 from nealrichardson/exec-plan-metadata Authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
2 participants
No description provided.