ARROW-17181: [Docs][Python] Scalar UDF Experimental Documentation #13687

vibhatha · 2022-07-22T08:10:12Z

This PR contains Python Scalar UDF documentation as an experimental version of docs.
At the moment we only support Scalar UDFs and the code snippets include how to use
UDFs with PyArrow.

vibhatha · 2022-07-22T08:12:52Z

cc @westonpace @lidavidm @pitrou

github-actions · 2022-07-22T08:21:06Z

https://issues.apache.org/jira/browse/ARROW-17181

github-actions · 2022-07-22T08:21:08Z

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

vibhatha · 2022-07-22T10:39:04Z

cc @amol- @jorisvandenbossche

pitrou

This is a good start, but see comments below.
Also, can you add API docs for register_scalar_function and ScalarUdfContext?

docs/source/python/compute.rst

pitrou · 2022-07-22T12:06:54Z

docs/source/python/compute.rst

+   >>> func_args = [pc.scalar(5), ds.field("total_amount($)"), pc.scalar(2)]
+   >>> result_table = dataset.to_table(
+   ...             columns={
+   ...                 'total_amount_projected($)': ds.field('')._call(function_name, func_args),


What is function_name here? Perhaps it would be better to spell it explicitly.

Also @jorisvandenbossche do you remember why Expression._call has a leading underscore?

function_name replaced with "affine"

docs/source/python/compute.rst

vibhatha · 2022-07-22T13:41:55Z

Also, can you add API docs for register_scalar_function and ScalarUdfContext?

Do you mean include it in docs for Python? I am not sure how to link it properly. But the text is there for both ScalarUdfContext and register_scalar_function where they are defined.

vibhatha · 2022-07-22T13:42:50Z

@pitrou could you please help me with adding API docs? or explain how to do that?

lidavidm · 2022-07-22T15:56:45Z

@vibhatha add docstrings to the Python/Cython code: https://numpydoc.readthedocs.io/en/latest/format.html (see the example on the bottom, or look through the Arrow source)

pitrou · 2022-07-22T15:59:23Z

@vibhatha add docstrings to the Python/Cython code: https://numpydoc.readthedocs.io/en/latest/format.html (see the example on the bottom, or look through the Arrow source)

And also reference the given symbols in https://github.com/apache/arrow/blob/master/docs/source/python/api/compute.rst

pitrou

@vibhatha Can I ask you to re-read your changes before pushing them? This would help minimize the review cycles. There are several redundancies and oddities here.

cpp/src/arrow/python/udf.h

docs/source/python/compute.rst

pitrou · 2022-07-22T16:16:20Z

docs/source/python/compute.rst

+   >>> def affine_with_python(ctx, m, x, c):
+   ...     m = m[0].as_py()
+   ...     x = x[0].as_py()
+   ...     c = c[0].as_py()
+   ...     return pa.array([m * x + c])


Why is this taking m[0]? I don't understand what this example is supposed to show...

I assumed you asked me to show the all scalar scenario in an example rather than wording. That’s why I added one to showcase that.

I referred to this: #13687 (comment)

DId I misinterpret your statement?

Well, the all-scalar scenario works with the original UDF? no?

@pitrou should we keep this example or just keep a note/warning? I am inclined towards the example, though.

If we are going to keep this example then we should:

Change the above ..note:: so that it isn't a note but a dedicated subsection.

Move the paragraph starting with "More generally," to come after the example.

Add a paragraph motivating the example.

However, I think we should expand this subsection so it isn't "Here is a wierd case that happens when all inputs are scalar" to "Your UDF should generally be capable of handling different combinations of scalar and array shaped inputs"

I re-organized and included content. Please check it.

I'm sorry, but the example still makes no sense to me. Why not use the regular affine function here, instead of this scalar-specific definition?

It is interesting, I think, that a person can define a UDF without using the Arrow compute functions at all, that is the most compelling point of the UDF feature in my mind since compositions of Arrow compute functions could already be done using expressions.

However, it is not clear from the description that this is the purpose of this example (is it?). It's also perhaps not the most motivating example since it can be expressed as an Arrow expression.

docs/source/python/compute.rst

vibhatha · 2022-07-22T23:51:06Z

@vibhatha add docstrings to the Python/Cython code: https://numpydoc.readthedocs.io/en/latest/format.html (see the example on the bottom, or look through the Arrow source)

@lidavidm I have already included the docs in the Cython in the original PR, but that function is in compute.pyx and only referred as an import in the compute.py. I was not sure whether to mark the method in Cython underscore function and call it in the compute.py and add the docs there. That was the confusing part.

vibhatha · 2022-07-23T00:48:40Z

@vibhatha Can I ask you to re-read your changes before pushing them? This would help minimize the review cycles. There are several redundancies and oddities here.

Of course, sorry about the issues. Will make sure to avoid this.

westonpace

Regarding ScalarUdfContext and register_scalar_function: Both seem to have the correct docstrings in the python code (to the best of my knowledge). I think the problem what @pitrou alluded to. You need to include both in some kind of /api/*.rst file (and /pi/compute.rst does make the most sense).

For example: 757ec32

These /api/*.rst files are an index into the API documentation. If a type or method is not referenced by one of these files then API documentation will not be autogenerated for it.

docs/source/python/compute.rst

westonpace · 2022-07-27T20:16:02Z

docs/source/python/compute.rst

+   >>> def affine_with_python(ctx, m, x, c):
+   ...     m = m[0].as_py()
+   ...     x = x[0].as_py()
+   ...     c = c[0].as_py()
+   ...     return pa.array([m * x + c])


If we are going to keep this example then we should:

Change the above ..note:: so that it isn't a note but a dedicated subsection.

Move the paragraph starting with "More generally," to come after the example.

Add a paragraph motivating the example.

However, I think we should expand this subsection so it isn't "Here is a wierd case that happens when all inputs are scalar" to "Your UDF should generally be capable of handling different combinations of scalar and array shaped inputs"

docs/source/python/compute.rst

vibhatha · 2022-07-28T03:19:04Z

@westonpace thanks for the detailed review. I will address these.

pitrou · 2022-08-01T14:22:37Z

docs/source/python/compute.rst

+------------------
+
+PyArrow UDFs accept input types of both scalar and array. Also it can have
+vivid combinations of these types. It is important that the UDF author must make sure,


Is "vivid" the right word here? @westonpace What do you think?

I think any would be better.

pitrou · 2022-08-01T14:23:47Z

docs/source/python/compute.rst

+   >>> def affine_with_python(ctx, m, x, c):
+   ...     m = m[0].as_py()
+   ...     x = x[0].as_py()
+   ...     c = c[0].as_py()
+   ...     return pa.array([m * x + c])


I'm sorry, but the example still makes no sense to me. Why not use the regular affine function here, instead of this scalar-specific definition?

vibhatha · 2022-08-01T14:27:53Z

Re: #13687 (comment)

@pitrou we can use the other affine function
But it beats the purpose right?
I thought we decided to elaborate the a single sentence from an example. May be I misunderstood your point.

pitrou · 2022-08-01T14:43:54Z

The purpose is to show how a regular scalar function can get executed on all-scalar inputs with help from the compute function execution layer.

pitrou · 2022-08-01T14:44:39Z

We can also remove this entire example (the one showing execution on all scalars), because I'm not sure how useful it actually is.

vibhatha · 2022-08-01T14:48:37Z

The purpose is to show how a regular scalar function can get executed on all-scalar inputs with help from the compute function execution layer.

Yes @pitrou that's the most important part. But shouldn't we point out how it could be used with a third-party library function. Let's say the user has already written a custom function which is an old python script, it doesn't use PyArrow compute API. I thought about that angel, that's why added the note. Is this something we don't need to discuss now and may be leave it for a different venue to discuss? Please correct me if I am wrong.

cc @westonpace Any thoughts?

westonpace · 2022-08-22T14:14:47Z

docs/source/python/compute.rst

+------------------
+
+PyArrow UDFs accept input types of both scalar and array. Also it can have
+vivid combinations of these types. It is important that the UDF author must make sure,


I think any would be better.

docs/source/python/compute.rst

westonpace · 2022-08-22T14:30:46Z

docs/source/python/compute.rst

+   >>> def affine_with_python(ctx, m, x, c):
+   ...     m = m[0].as_py()
+   ...     x = x[0].as_py()
+   ...     c = c[0].as_py()
+   ...     return pa.array([m * x + c])


It is interesting, I think, that a person can define a UDF without using the Arrow compute functions at all, that is the most compelling point of the UDF feature in my mind since compositions of Arrow compute functions could already be done using expressions.

However, it is not clear from the description that this is the purpose of this example (is it?). It's also perhaps not the most motivating example since it can be expressed as an Arrow expression.

westonpace · 2022-08-22T14:31:59Z

docs/source/python/compute.rst

+   >>> pc.call_function(function_name, [pa.scalar(10.1), pa.scalar(10.2), pa.scalar(20.2)])
+   <pyarrow.DoubleScalar: 123.22>


The function correctly handles the all-scalar case but it does not handle other cases. Ideally, an example should demonstrate how to write a UDF that can handle all possible cases. For example:

print('10.0 * 5.0 + 1.0 should be 51.0') print(f'Answer={pc.call_function(function_name, [pa.scalar(10.0), pa.scalar(5.0), pa.scalar(1.0)])}') print('[10.0, 10.0] * [5.0, 6.0] + [1.0, 1.0] should be [51.0, 61.0]') print(f'Answer={pc.call_function(function_name, [pa.array([10.0, 10.0]), pa.array([5.0, 6.0]), pa.array([1.0, 1.0])])}') print('10.0 * [5.0, 6.0] + 1.0 should be [51.0, 61.0]') print(f'Answer={pc.call_function(function_name, [pa.scalar(10.0), pa.array([5.0, 6.0]), pa.scalar(1.0)])}')

Right now, the function as it is designed, gives me this output:

10.0 * 5.0 + 1.0 should be 51.0 Answer=51.0 [10.0, 10.0] * [5.0, 6.0] + [1.0, 1.0] should be [51.0, 61.0] Answer=[ 51 ] 10.0 * [5.0, 6.0] + 1.0 should be [51.0, 61.0] Traceback (most recent call last): File "/home/pace/experiments/arrow-17181/repr.py", line 39, in <module> print(f'Answer={pc.call_function(function_name, [pa.scalar(10.0), pa.array([5.0, 6.0]), pa.scalar(1.0)])}') File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/_compute.pyx", line 2506, in pyarrow._compute._scalar_udf_callback File "/home/pace/experiments/arrow-17181/repr.py", line 21, in affine_with_python m = m[0].as_py() TypeError: 'pyarrow.lib.DoubleScalar' object is not subscriptable

westonpace · 2022-08-22T14:32:45Z

docs/source/python/compute.rst

+   >>> pc.call_function(function_name, [pa.scalar(10.1), pa.scalar(10.2), pa.scalar(20.2)])
+   <pyarrow.DoubleScalar: 123.22>
+
+Note that here the the final output is returned as an array. Depending the usage of vivid libraries


Is it an array? I see:

<pyarrow.DoubleScalar: 123.22>

It probably should be an array.

westonpace · 2022-08-22T14:33:28Z

docs/source/python/compute.rst

+   >>> pc.call_function(function_name, [pa.scalar(10.1), pa.scalar(10.2), pa.scalar(20.2)])
+   <pyarrow.DoubleScalar: 123.22>
+
+Note that here the the final output is returned as an array. Depending the usage of vivid libraries


I'm not sure what you are trying to say with the sentence that starts "Depending the..."

westonpace · 2022-08-22T14:36:36Z

docs/source/python/compute.rst

+Here note that the `ds.field('')_call()` returns an expression. The passed arguments
+to this function call are expressions not scalar values 
+(i.e `pc.scalar(5.2), ds.field("value"), pc.scalar(2.1)`). This expression is evaluated
+when the project operator uses this expression.


You say "The passed arguments to this function call are expressions not scalar values".

However, pc.scalar(5.2) and pc.scalar(2.1) look like scalar values. I'm not sure a user will recognize the subtle difference between pc.scalar(5.2) and pa.scalar(5.2) without further explanation.

docs/source/python/compute.rst

westonpace · 2022-08-22T18:02:47Z

I think part of the challenge with this documentation is that implementing affine in pure-python is not a very compelling use case. I think the more interesting case for UDFs is when we want to use some other library that does efficient compute and is capable of working with Arrow data. For example, numpy. Here is an example that exposes numpy's gcd function (greatest common divisor) as an Arrow function:

import numpy as np

import pyarrow as pa
import pyarrow.compute as pc

function_name = "numpy_gcd"
function_docs = {
       "summary": "Calculates the greatest common divisor",
       "description":
           "Given 'x' and 'y' find the greatest number that divides\n"
           "evenly into both x and y."
}

input_types = {
   "x" : pa.int64(),
   "y" : pa.int64()
}

output_type = pa.int64()

def to_np(val):
    if isinstance(val, pa.Scalar):
        return val.as_py()
    else:
        return np.array(val)

def gcd_numpy(ctx, x, y):
    np_x = to_np(x)
    np_y = to_np(y)
    return pa.array(np.gcd(np_x, np_y))

pc.register_scalar_function(gcd_numpy,
                            function_name,
                            function_docs,
                            input_types,
                            output_type)

print('gcd(27, 63) should be 9')
print(f'Answer={pc.call_function(function_name, [pa.scalar(27), pa.scalar(63)])}')
print()
print('gcd([27, 18], [54, 63]) should be [27, 9]')
print(f'Answer={pc.call_function(function_name, [pa.array([27, 18]), pa.array([54, 63])])}')
print()
print('gcd(27, [54, 18]) should be [27, 9]')
print(f'Answer={pc.call_function(function_name, [pa.scalar(27), pa.array([54, 18])])}')

Notice the use of the helper function to_np to convert from inputs of different shapes to ensure that we get something that numpy can work with.

vibhatha · 2022-08-24T04:29:35Z

I think part of the challenge with this documentation is that implementing affine in pure-python is not a very compelling use case. I think the more interesting case for UDFs is when we want to use some other library that does efficient compute and is capable of working with Arrow data. For example, numpy. Here is an example that exposes numpy's gcd function (greatest common divisor) as an Arrow function:
import numpy as np

import pyarrow as pa
import pyarrow.compute as pc

function_name = "numpy_gcd"
function_docs = {
       "summary": "Calculates the greatest common divisor",
       "description":
           "Given 'x' and 'y' find the greatest number that divides\n"
           "evenly into both x and y."
}

input_types = {
   "x" : pa.int64(),
   "y" : pa.int64()
}

output_type = pa.int64()

def to_np(val):
    if isinstance(val, pa.Scalar):
        return val.as_py()
    else:
        return np.array(val)

def gcd_numpy(ctx, x, y):
    np_x = to_np(x)
    np_y = to_np(y)
    return pa.array(np.gcd(np_x, np_y))

pc.register_scalar_function(gcd_numpy,
                            function_name,
                            function_docs,
                            input_types,
                            output_type)

print('gcd(27, 63) should be 9')
print(f'Answer={pc.call_function(function_name, [pa.scalar(27), pa.scalar(63)])}')
print()
print('gcd([27, 18], [54, 63]) should be [27, 9]')
print(f'Answer={pc.call_function(function_name, [pa.array([27, 18]), pa.array([54, 63])])}')
print()
print('gcd(27, [54, 18]) should be [27, 9]')
print(f'Answer={pc.call_function(function_name, [pa.scalar(27), pa.array([54, 18])])}')
Notice the use of the helper function to_np to convert from inputs of different shapes to ensure that we get something that numpy can work with.

I see your point. I will update the example to use this.

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

vibhatha · 2022-09-28T15:56:48Z

@jorisvandenbossche WDYT about this: #13687 (comment)

pitrou

Ok, I think this is good enough now :-)
Thank you @vibhatha !

vibhatha · 2022-09-29T10:43:05Z

Thanks everyone for the reviews 🙂

ursabot · 2022-09-29T14:32:45Z

Benchmark runs are scheduled for baseline = 60c9383 and contender = 902781d. 902781d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.82% ⬆️0.0%] test-mac-arm
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.18% ⬆️0.07%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 902781d1 ec2-t3-xlarge-us-east-2
[Finished] 902781d1 test-mac-arm
[Failed] 902781d1 ursa-i9-9960x
[Finished] 902781d1 ursa-thinkcentre-m75q
[Finished] 60c93833 ec2-t3-xlarge-us-east-2
[Failed] 60c93833 test-mac-arm
[Failed] 60c93833 ursa-i9-9960x
[Finished] 60c93833 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

github-actions bot added the Component: Documentation label Jul 22, 2022

pitrou requested changes Jul 22, 2022

View reviewed changes

vibhatha requested a review from pitrou July 22, 2022 13:43

github-actions bot added the Component: C++ label Jul 22, 2022

pitrou requested changes Jul 22, 2022

View reviewed changes

westonpace self-requested a review July 25, 2022 16:54

vibhatha requested a review from pitrou July 25, 2022 16:58

westonpace requested changes Jul 27, 2022

View reviewed changes

pitrou reviewed Aug 1, 2022

View reviewed changes

vibhatha requested review from pitrou and westonpace August 1, 2022 14:29

westonpace requested changes Aug 22, 2022

View reviewed changes

vibhatha force-pushed the arrow-17181 branch from ce3b282 to b880d29 Compare August 26, 2022 01:55

vibhatha and others added 22 commits September 28, 2022 21:06

updating testing submodule

7c0d11e

revert to uupstream version

25862d8

adding initial scalar udf docs for python

06e06e8

update warnings

79eedd8

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

update udf description

88d46cc

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

grammar fix

0497de0

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

update udf function doc

100ad56

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

update udf func doc description

b0221c9

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

update description

eec2473

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

update the how the udf can be called with datasets.

c48c90a

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

update info on context

60abbc6

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

adding addequate memory

d803a4f

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

adding a new example to show case the scalar limitations'

b4959a7

updating example and rebase

93323d9

adding docs

feb6bb6

addressing review comments

1203d80

fix(reviews): address reviews

6a04e09

fix(reviews): updated write up and example

f702c51

fix(reviews): replaced affine function with gcd and update docs

e86fadd

fix(reviews): updated review comments

4aa2902

fix(reviews): updated the docs and code

7e88faa

fix(submodule)

227db70

vibhatha force-pushed the arrow-17181 branch from 7f47cdf to 227db70 Compare September 28, 2022 15:37

fix(reviews): doc update

a8555bf

pitrou approved these changes Sep 29, 2022

View reviewed changes

pitrou merged commit 902781d into apache:master Sep 29, 2022

		>>> pc.call_function(function_name, [pa.scalar(10.1), pa.scalar(10.2), pa.scalar(20.2)])
		<pyarrow.DoubleScalar: 123.22>

ARROW-17181: [Docs][Python] Scalar UDF Experimental Documentation #13687

ARROW-17181: [Docs][Python] Scalar UDF Experimental Documentation #13687

Uh oh!

Conversation

vibhatha commented Jul 22, 2022

Uh oh!

vibhatha commented Jul 22, 2022

Uh oh!

github-actions bot commented Jul 22, 2022

Uh oh!

github-actions bot commented Jul 22, 2022

Uh oh!

vibhatha commented Jul 22, 2022

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vibhatha commented Jul 22, 2022

Uh oh!

vibhatha commented Jul 22, 2022

Uh oh!

lidavidm commented Jul 22, 2022

Uh oh!

pitrou commented Jul 22, 2022

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vibhatha commented Jul 22, 2022

Uh oh!

vibhatha commented Jul 23, 2022

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!