[C++][Dataset] Schema evolution for timestamp columns

[asfimport]

In a dataset, one can have timestamp columns with different resolutions. There should be an optional to cast all timestamps to the type mentioned in the schema. A typical example could be that we store a pandas DataFrame with ns precision to Parquet files that only support us resolution in their most widespread from. Then the dataset schema and the actual file content don't match anymore.

Reporter: Uwe Korn / @xhochy

Related issues:

• [C++][Dataset] Schema evolution in Dataset scanning (is related to)

_{Note: This issue was originally created as ARROW-8284. Please see the migration documentation for further details.}

[asfimport]

Uwe Korn / @xhochy:
Tested this with the following code to verify that this is fixed in master:

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds
import pandas as pd
import numpy as np

table1 = pa.table({"timestamp": pa.array([0, 1, 2], type=pa.timestamp("us"))})
pq.write_table(table1, "ts1.parquet")
table2 = pa.table({"timestamp": pa.array([1, 10, 200], type=pa.timestamp("ms"))})
pq.write_table(table2, "ts2.parquet")

dataset = ds.FileSystemDataset.from_paths(
    paths=["ts1.parquet", "ts2.parquet"],
    schema=table1.schema,
    format=pa.dataset.ParquetFileFormat(),
    filesystem=pa.fs.LocalFileSystem()
)
dataset.to_table().to_pandas()