[C++] Potential regression on FieldRef/FieldPath non-flattening Get methods

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

I did not pay attention initially, but it seems #35197 introduced a large regression on the wide-dataframe benchmark.

See benchmark results here:
https://conbench.ursa.dev/compare/runs/9cf73ac83f0a44179e6538b2c1c7babd...3d76cb5ffb8849bf8c3ea9b32d08b3b7/

Note the benchmark is creating a dataframe with 10000 columns.

Component(s)

C++

[pitrou]

@benibus Would you have time to take a look at this?
It would be nice to add benchmarks for the non-flattening methods. They will also help measuring the slowdown before/after that PR.

[raulcd]

@benibus @westonpace I was planning on waiting a couple days to create the next RC (Tuesday?) to see if this was able to make it to 13.0.0. If this sounds totally unreasonable let me know and I can create the new RC sooner as discussed on Zulip we might want to ship 13.0.0 with this known issue and apply the fix targeting 14.0.0.

[raulcd]

@benibus any news here? I plan to create the new RC tomorrow unless there's possibility for this to be solved soon.

[benibus]

Should have a fix for this by the end of the week, provided my diagnosis is actually complete.

At the very least, the PR introduced substantial overhead for calling FieldPath::Get on record batches specifically. It seems this is due to a few instances where usages of RecordBatch::column_data() were replaced with columns(), which is far less trivial (it loads every column atomically on each invocation).

That being said, the two aren't exactly interchangeable in the implementation so there's some restructuring in order.