[C++][Python] Variable dictionaries do not survive roundtrip to IPC stream

[asfimport]

Failing test case (from dev@ https://lists.apache.org/thread.html/r338942b4e9f9316b48e87aab41ac49c7ffedd45733d4a6349523b7eb%40%3Cdev.arrow.apache.org%3E)

import pyarrow as pa
from io import BytesIO

pa.__version__

schema = pa.schema([pa.field('foo', pa.int32()), pa.field('bar', pa.dictionary(pa.int32(), pa.string()))] )
r1 = pa.record_batch(
    [
        [1, 2, 3, 4, 5],
        pa.array(["a", "b", "c", "d", "e"]).dictionary_encode()
    ],
    schema
)

r1.validate()
r2 = pa.record_batch(
    [
        [1, 2, 3, 4, 5],
        pa.array(["c", "c", "e", "f", "g"]).dictionary_encode()
    ],
    schema
)

r2.validate()

assert r1.column(1).dictionary != r2.column(1).dictionary


sink =  pa.BufferOutputStream()
writer = pa.RecordBatchStreamWriter(sink, schema)

writer.write(r1)
writer.write(r2)

serialized = BytesIO(sink.getvalue().to_pybytes())
stream = pa.ipc.open_stream(serialized)

deserialized = []

while True:
    try:
        deserialized.append(stream.read_next_batch())
    except StopIteration:
        break

assert deserialized[1][1].to_pylist() == r2[1].to_pylist()

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

PRs and other links:

• GitHub Pull Request #8302

_{Note: This issue was originally created as ARROW-10121. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
AFAICT, this bug was already present in 0.15.1, and perhaps before.
The problem is basically this snippet in IpcFormatWriter::RecordBatch:

    if (!wrote_dictionaries_) {
      RETURN_NOT_OK(WriteDictionaries(batch));
      wrote_dictionaries_ = true;
    }

The solution isn't entirely trivial: presumably we don't want to write dictionaries again when they are identical. We also don't want to compare them every time. Perhaps compare them by pointer?

[asfimport]

Wes McKinney / @wesm:
I would have thought this would have been addressed by delta dictionary work but it seems that it only treated the read side, not the write side.

When encountering an unequal dictionary (based at minimum on the memory address of the dictionary):

• At minimum we need to determine whether it's a permutation, delta, or dictionary prefix.

• If it's a permutation (neither delta not subdictionary) and we are writing a stream (non-file), then we can either permute the indices (and if necessary write a delta dictionary if there are new values) or write a dictionary replacement. Dictionary replacement is the most "preserving"

  In any case, we shouldn't allow the status quo to persist – either we should fix it properly or have it raise an error until we fix it properly.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 8302
#8302