[C++][Dataset] Don't require dictionaries for reading dataset with schema-based Partitioning

[asfimport]

As a follow-up on ARROW-10247 (see also #9130 (comment)). We currently require the user to pass manually specified dictionary values when reading a dataset with a Partitioning based on a schema with dictionary typed fields.

In practice that means that the user for example needs to parse the file paths to get all the possible values the partition field can take, while Arrow will then afterwards again do the same to construct the dataset object.
Naively, it seems that it should be possible to let Arrow infer the dictionary values, even when providing an explicit schema with a dictionary field for the Partitioning (i.e. so when not letting the partitioning schema itself be inferred from the file paths).

An example use case is when you have a Partitioning schema with both dictionary and non-dictionary fields. When discovering the schema, you can only have all or nothing (all dictionary fields or no dictionary fields).

cc @bkietz

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: David Li / @lidavidm

PRs and other links:

• GitHub Pull Request #9677

_{Note: This issue was originally created as ARROW-11260. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Sidenote: this actually also worked for a short time in pyarrow 0.17.0 (and I added a test about that in the PR fixing it (#6641 (comment)), but that got apparently lost in a rebase ;)), but I suppose this was changed in pyarrow 1.0.0 after ensuring that the dictionary-typed partition fields "know" the full dictionary of all possible values the dataset (#7536 (comment)).

The test that worked at that time we forgot to include:

@pytest.mark.pandas
def test_partitioning_dictionary_key(mockfs):
    # ARROW-8088 specifying partition key as dictionary type
    schema = pa.schema([
        pa.field('group', pa.dictionary(pa.int8(), pa.int32())),
        pa.field('key', pa.dictionary(pa.int8(), pa.string()))
    ])
    part = ds.DirectoryPartitioning(schema)

    dataset = ds.dataset(
        "subdir", format="parquet", filesystem=mockfs, partitioning=part
    )
    table = dataset.to_table()

    assert table.column('group').type.equals(schema.types[0])
    assert table.column('group').to_pylist() == [1] * 5 + [2] * 5
    assert table.column('key').type.equals(schema.types[1])
    assert table.column('key').to_pylist() == ['xxx'] * 5 + ['yyy'] * 5

(but note that this doesn't check that each chunk of the ChunkedArray for the partition columns have all dictionary values, which is the feature that was added later)

[asfimport]

Ben Kietzman / @bkietz:
Issue resolved by pull request 9677
#9677