[C++][Python] UDF Integration

[asfimport]

The objective is to list down a set of tasks required to provide UDF support for Apache Arrow streaming execution engine. In the first iteration we will be focusing on providing support for Python-based UDFs which can support Python functions. 

The UDF Integration is going to pan out with a series of sub-tasks associated with the development and PoCs. Note that this is going to be the first iteration of UDF integrations with a limited scope. This ticket will cover the following topics;

1. POC for UDF integration: The objective is to evaluate the existing components in the source and evaluate the required modifications and new building blocks required to integrate UDFs.
2. The language will be limited to C+{}/{}Python users can register Python function as a UDF and use it with an apply method on Arrow Tables or provide a computation API endpoint via arrow::compute API. Note that the C+ API already provides a way to register custom functions via the function registry API. At the moment this is not exposed to Python. 
3. Planned features for this ticket are;
   1. Scalar UDFs : UDFs executed per value (per row)
   2. Vector UDFs : UDFs executed per batch (a full array or partial array)
   3. Aggregate UDFs : UDFs associated with an aggregation operation
4. Integration limitations
   1. Doesn't support custom data types which doesn't support Numpy or Pandas
   2. Complex processing with parallelism within UDFs are not supported
   3. Parallel UDFs are not supported in the initial version of UDFs. Allthough we are documenting what is required and a rough sketch for the next phase. 

Reporter: Vibhatha Lakmal Abeykoon / @vibhatha
Assignee: Vibhatha Lakmal Abeykoon / @vibhatha

Subtasks:

• PoC for UDFs
• [C++][Python] UDF Optimizations
• [Docs] UDF Documentation
• [C++][Python] UDF Scalar Function Implementation
• [Docs] UDFs on Cookbook
• [C++][Python] UDF Aggregate Function Implementation
• [C++][Python] UDF Vector Function Implementation
• [C++][Python] Include UDFOptions
• [C++][Python] Unregister compute functions
• [C++][Python] Register Multiple Kernels for a UDF
• [C++][Python] Generate scalar-argument kernel based on array-argument kernel for UDFs
• [Docs][Python] Scalar UDF Experimental Documentation
• [Python] Allow calling UDF kernels with field/scalar expressions
• [C++][Python] Implement HashAggregate UDF

PRs and other links:

• GitHub Pull Request #13214

_{Note: This issue was originally created as ARROW-15635. Please see the migration documentation for further details.}

[asfimport]

Weston Pace / @westonpace:
Can you add some scope to this? Are you planning on tackling all types of UDFs (e.g. scalar, vector, aggregate, post-aggregate, etc.) in this one JIRA and those will be subtasks? Or is this JIRA to aim for a specific type of UDF first?

[asfimport]

Vibhatha Lakmal Abeykoon / @vibhatha:
Thanks @westonpace  for the note. In this ticket(jira) we should include all and I have missed to document the scope accurately. I will add them as sub tasks. 

[asfimport]

Vibhatha Lakmal Abeykoon / @vibhatha:
@westonpace  I added more content on the scope of the first iteration for UDFs. 
I clearly don't expect to include all the features for UDFs. But we should support all the useful cases for quries we are planning to execute. Any missing pieces? 

[asfimport]

Weston Pace / @westonpace:
Thanks, this is helpful. This might be interesting to @jorisvandenbossche @amol- as well.

[asfimport]

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned per project policy. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.