ARROW-6157: [C++] Array data validation

[pitrou]

Add a method ValidateFull() on arrays, batches etc. which does O(N) data validation
for a few types (list, union, dictionary).

Also, fix the assumptions about union arrays to match official semantics.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-6157

[pitrou]

@wesm I was wondering... should we rename these methods to type_codes and raw_type_codes? There is a bit of a mixed terminology here (UnionType uses type_codes, also type_id is used for something different).

[wesm]

I'd be in favor of changing the names to be more clear and where relevant, conforming to the specification

[pitrou]

Well, the specification uses "types" and "type ids" :-/

[pitrou]

cc @wesm For the union changes.

[wesm]

Thanks for doing all this, definitely of high value to provide deeper validation than we had previously.

I left a couple comments about things that stood out but everything else looked in line with expectations

[wesm]

Is this conforming to the specification? My understanding is that we do not place any conditions on the value "underneath" a null

[pitrou]

This code is simply moved around from array.cc.
The spec does not spell it out explicitly but the examples are compliant with this expectation:
https://arrow.apache.org/docs/format/Columnar.html#variable-size-list-layout

[wesm]

Do I have it right that this is adding sizeof(int) * (kMaxTypeId + 1) (so 512 or 1024 bytes depending on the system) to the footprint of every UnionType instance?

[pitrou]

Yes. I think this is simpler than having to lazily-initialize it.

[jorisvandenbossche]

Should the python constructors do a cheap or a full validation? (since full=Falseis the default, I assume they will now do a cheap validation)

So that means that the initial example in the issue (https://issues.apache.org/jira/browse/ARROW-6157) with pa.UnionArray.from_dense can still generate a invalid array that will segfault on certain operations?
That might be fine (it should probably at least be possible to not do the full validation, in case you know you have valid array data), but maybe we should then update some of those docstrings to note that?

[pitrou]

Full validation is essentially O(n), so it sounds undesirable to do it by default in Python constructors. I don't know what @wesm thinks about it.

[pitrou]

Rebased. Will merge when green.