[Python] Pre-generate pyarrow.compute members

[asfimport]

Static analysis tools (e.g. pylint) don't recognize simple members like pyarrow.compute.equal, they report is as missing. Generating file (well a file imported by this file I assume) https://github.com/apache/arrow/blob/master/python/pyarrow/compute.py instead of runtime wrapping of the functions would improve the developer experience.

Reporter: Adam Lippai / @alippai

Original Issue Attachments:

• compute.py

PRs and other links:

• GitHub Pull Request #13126

_{Note: This issue was originally created as ARROW-12526. Please see the migration documentation for further details.}

[asfimport]

Adam Lippai / @alippai:
A quick question regarding this topic: if we generate code do we push it to the GitHub repo or do we just create it on the fly upon a release? 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
In general, we commit generated code, I think.

Wondering, would something like ARROW-10316 also already help? Although I suppose not for static tools, that probably only makes the interactive use nicer.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
cc @pitrou

[asfimport]

Antoine Pitrou / @pitrou:
I would be fine with generating the file. We would need to commit it to the repository.

Generating the file would also allow generation to be more expensive without fear of growing import times.

[asfimport]

Antoine Pitrou / @pitrou:
cc @amol-

[asfimport]

Adam Lippai / @alippai:
Note: using *.pyi files would help the typing info and autocomplete, but not the import times. I assume we'd prefer full code generation, right?

[asfimport]

Antoine Pitrou / @pitrou:
Probably, yes.

[asfimport]

Antoine Pitrou / @pitrou:
@alippai Can you show an example invocation of pylint or another such utility?

[asfimport]

Kevin Crouse:
I was pointed to this issue because I was similarly interested in allowing pythonic documentation to be written for functions in which docs are autogenerated from the Arrow C++ details. As a result, I've written out the code to do both of these things for the pyarrow.compute module: krcrouse/arrow@master...krcrouse:generate-pyarrow-compute-and-improve-docs 

Major points:

• creates python/docs/additions tree where the reStructrued text docs that include the sections to overwrite. Using raw reSt so that code block examples can be tested using doctest - see the README for more verbose details

• pyarrow.docutils (or maybe should be _docutils) provides functions to processes python/docs/additions and return a data structure of the components per function.

• python/scripts/generate_sources.py uses pyarrow.docutils and writes out the code for the compute functions in pyarrow/generated/compute.py. All of the logic from the release-branch pyarrow.compute module that dynamically generated the compute functions has been moved to this script.

  • I didn't check the generated file into the repo because I generally do not include generated files that would be generated by the build process should be in source control, but I realize there are other perspectives on this

• pyarrow.compute now imports from pyarrow.generated.compute for all of the autogenerated compute bindings. Override and custom functions are still defined here.

• The old pyarrow._compute_docstrings is gone because its purpose is subsumed in the above.

• I've updated the tests so that they work with the above changes.

   

  Below I've attached the pyarrow/generated/compute.py file that is currently created by generate_source.py as of commit 13c2b0e 

   

   

[asfimport]

Antoine Pitrou / @pitrou:
[~krcrouse] Thank you for doing this! This is really interesting. May you perhaps open a draft PR with you changes, to make it easier to add questions and comments?

[asfimport]

Kevin Crouse:
I haven't seen any comments on the PR or here, so maybe they didn't get auto-linked? If so, sorry - I've used Jira in the past with Bitbucket but not Github.

The PR is here:

#13126

[asfimport]

Antoine Pitrou / @pitrou:
Sorry Kevin. The PR was auto-linked above, it's just that noone got around to take a look yet.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.