[Flight] Enable integration testing of batches with dictionaries

[asfimport]

Reporter: David Li / @lidavidm

Related issues:

• [Flight] Add application metadata field to DoGet (is superceded by)

_{Note: This issue was originally created as ARROW-5143. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
The Flight integration tests still fail on the dictionary batch, even C+-to-C+.

The gRPC error is "grpc-message: Dictionary with id 0 not found". (Aside: gRPC still doesn't report this to us - it just aborts. May need to send an issue upstream if we can make a minimal reproduction.)

The (relevant part of) the server backtrace is:

  * frame #0: 0x00000001014723a1 libarrow.14.dylib`arrow::ipc::DictionaryMemo::GetDictionary(this=0x00007000020c8ce0, id=0, dictionary=0x00007000020c87b8) const at dictionary.cc:37
    frame #1: 0x000000010152d495 libarrow.14.dylib`arrow::ipc::internal::FieldFromFlatbuffer(field=0x00000001080002e9, dictionary_memo=0x00007000020c8ce0, out=0x0000000104607590) at metadata-internal.cc:673
    frame #2: 0x000000010152c9bc libarrow.14.dylib`arrow::ipc::internal::GetSchema(opaque_schema=0x0000000108000225, dictionary_memo=0x00007000020c8ce0, out=0x00007000020c8ec8) at metadata-internal.cc:1060
    frame #3: 0x0000000101551058 libarrow.14.dylib`arrow::ipc::ReadSchema(message=0x0000000104607190, out=0x00007000020c8ec8) at reader.cc:724
    frame #4: 0x00000001007fdbd8 libarrow_flight.14.dylib`arrow::flight::FlightServiceImpl::DoPut(this=0x0000000104604db0, context=0x000000010501ce20, reader=0x00007000020c9970, response=0x00007000020c9958) at server.cc:252

[asfimport]

Wes McKinney / @wesm:
This might work now, @lidavidm can you give it a try to see?

[asfimport]

David Li / @lidavidm:
@wesm I tried in #4282, the non-nested dictionary case worked with some additional effort, so that PR enables that. I didn't look into why the nested case still fails.

[asfimport]

Neal Richardson / @nealrichardson:
Is this still an issue?

[asfimport]

David Li / @lidavidm:
It's enabled for non-nested dictionaries (C++ and Java) and nested dictionaries (C++ only). So we could close this and use ARROW-7779 to track the remainder.

[asfimport]

Antoine Pitrou / @pitrou:
Ok, closing.