[C++] Datasets/Fragment/ScanOptions should be immutable

[asfimport]

A dataset is a collection of fragments, a file format, and some partitioning rules.

A fragment is a description of some location to get data from.

The ScanOptions describe how the user wants the scan to behave.

 

These things don't need mutable state.  Currently, datasets do not have any state and its methods could be converted to const painlessly.

 

The parquet file fragment is a little tricky.  It caches statistics gleamed lazily from the parquet metadata at scan time.  This is a little confusing and feels like it should be a part of a scan (although that would mean we lose out from multiple scans of the same dataset).  Also, though unlikely, what if the files were replaced in the meantime?

 

The scan options currently get populated as the scan runs based on expression simplification and other information figured out at scan time.  Again, this information feels like it should be part of a scan.

 

These different pieces make the code confusing (to me at least) to reason on.  For example, there was some recent discussion in #9802 about whether a fragment or a dataset should be reusable (the conclusion was they should not but I still think maybe they should be).

 

Just to be clear.  I'm not stating these things as self evident and obvious.   This is more of a "for discussion" issue.  I also don't have a concrete idea of implementation.  It would seem there would end up being a "scan context" again of some kind and it might even be persisted to use on future scans to save time (e.g. the cached parquet statistics).

Reporter: Weston Pace / @westonpace

_{Note: This issue was originally created as ARROW-12243. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
@bkietz @lidavidm @jorisvandenbossche

CC people involved in recent discussions on the topic.

[asfimport]

David Li / @lidavidm:
I agree that mostly, we don't need mutable state or could move it to some sort of context object. As with the FragmentScanOptions, though, a separate context object feels like the kind of thing that Python/R users shouldn't have to think about default.

The reusablility part is odd in that yes, I think it's very weird to have a fragment that's only readable once (see ARROW-12231). However, it comes up because of writing, where you may not want to materialize all data in-memory before writing it out. Since both reading and writing use the same machinery, you then end up with a read-once fragment/dataset.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.