[C++] Dataset writing can only include projected columns if input columns are also included

[asfimport]

I discovered this while working on #10191. You can project new columns when writing a dataset, but only if they are derived from columns that are included in the output. Here's an R-based example:

# Simple function to write and re-open the new dataset
write_then_open <- function(ds, path, ...) {
  write_dataset(ds, path, ...)
  open_dataset(path)
}

tab <- Table$create(a = 1:5)

tab %>% 
  write_then_open(ds_dir) %>%
  collect()

# # A tibble: 5 x 1
#       a
#   <int>
# 1     1
# 2     2
# 3     3
# 4     4
# 5     5

# If you rename a column, it's all nulls
tab %>%
  select(b = a) %>%
  write_then_open(ds_dir) %>%
  collect()

# # A tibble: 5 x 1
#       b
#   <int>
# 1    NA
# 2    NA
# 3    NA
# 4    NA
# 5    NA

# If you derive a new column and keep the original, it works
tab %>%
  mutate(b = a) %>%
  write_then_open(ds_dir) %>%
  collect()

# # A tibble: 5 x 2
#       a     b
#   <int> <int>
# 1     1     1
# 2     2     2
# 3     3     3
# 4     4     4
# 5     5     5

# transmute() only keeps the added columns, so it also illustrates the failure
tab %>%
  transmute(b = a) %>%
  write_then_open(ds_dir) %>%
  collect()

# # A tibble: 5 x 1
#       b
#   <int>
# 1    NA
# 2    NA
# 3    NA
# 4    NA
# 5    NA

Reporter: Neal Richardson / @nealrichardson
Assignee: David Li / @lidavidm

PRs and other links:

• GitHub Pull Request #10229

_{Note: This issue was originally created as ARROW-12620. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Can confirm this in Python:

import pyarrow.dataset as ds
import pyarrow.parquet as pq

table = pa.table({'a': [1, 2, 3]})
dataset = ds.dataset(table)

projection = {'b': ds.field('a')}
ds.write_dataset(dataset.scanner(columns=projection), "test_scanner.parquet", format="parquet")

In [9]: pq.read_table("test_scanner.parquet").to_pandas()
Out[9]: 
    b
0 NaN
1 NaN
2 NaN

[asfimport]

David Li / @lidavidm:
Ah: the writer executes scan tasks directly, skipping the projection/filtering machinery. So I think you'd also see, for instance, row filters being ignored. spoke too soon

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 10229
#10229

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
@lidavidm I see you added but then removed again the 4.0.1 tag. But if we do a bug fix release, this one might be worth including, so let's tag it to keep track of it?

[asfimport]

David Li / @lidavidm:
@jorisvandenbossche I flip-flopped since I decided the bug should not have been in 4.0 (the affected code wasn't introduced until ARROW-12392).

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Ah, that's a good point. The "Affects Version/s: 4.0.0" is wrong then?

[asfimport]

David Li / @lidavidm:
Yeah, I think so - I just tested with R/Arrow 4 and the R snippet errors with Error: .xmust be a vector, not aTable/ArrowTabular/ArrowObject/R6 object. And of course that Python snippet didn't work until recently either.