[FlightRPC] FlightSQL: Expose custom query metadata

[aiguofer]

Describe the enhancement requested

We'd like to be able to expose custom metadata related to a given query through our Arrow Flight SQL service. For example, it'd be great to expose queryId, queryCost, or queryType, which could be useful to clients/users.

It seems like the existing ability to add custom metadata could be leveraged to accomplish this, but the exact mechanics seem a little unclear. The easiest thing would be to expose it via the Schema metadata since every FlightInfo already includes one, making it easy to expose through existing implementations. However, this seems a little hacky conceptually since the metadata could be associated with the query and not the results themselves (although metadata for the results also makes sense).

Another option could be to add a way to expose custom metadata directly through FlightInfo, but this would likely require major changes in every language implementation.

As far as how this gets exposed on the client side, we'd likely need changes to the JDBC and ADBC drivers.

For JDBC, there could be a few methods:

• PreparedStatement.getCustomMetadata()
• ResultSet.getCustomMetadata()
• ResultSetMetaData.getCustomMetadata()

Thoughts?

Component(s)

FlightRPC

[zeroshade]

I can't speak for the Java implementation, but many FlightRPC implementations expose the app_metadata field on the FlightData messages during a stream which tends to be the common method for including this type of metadata (the other method tends to be including it in the schema of the result set, i.e. the "hacky" way you mentioned). Personally I prefer it being part of the FlightData as it ensures that every message is able to contain metadata which may potentially include things like partitionId and so on above and beyond the query level metadata.

@kou @lidavidm what do you think?

[lidavidm]

FlightData isn't relevant here? We're talking about FlightInfo.

I've pushed for app_metadata on FlightInfo before (or possibly FlightEndpoint, or both) and I think we should just do it.

[zeroshade]

That's a fair point, I'd be in favor of it. I can put together a PR and propose it on the mailing list

[lidavidm]

Sounds good, thanks.

I think ADBC can expose this easily now that we have getOptionBytes (and it will also be part of the metadata executePartitions anyways).

[kou]

I'm OK with adding app_metadata to FlightInfo (and FlightEndpoint) too.

[zeroshade]

I'll put a PR together tomorrow for this

[kou]

Vote thread: https://lists.apache.org/thread/113xm107ptn1t76txockp11oc7wlh0ok

[aiguofer]

Awesome, thanks for the action on this!! I missed the vote since I wasn't signed up to the mailing list.

What would need to be done to expose this in Java?

[lidavidm]

A contribution would be welcome. Or, @vibhatha @davisusanibar are one of you able to take this on?