[Archery][CI][Doc] archery numpydoc fails with latest Cython 3.1.1

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

It seems that the new version of Cython (3.1.1) released yesterday breaks our code to validate docstrings from the Cython generated code. Our code:

arrow/dev/archery/archery/lang/python.py

Lines 164 to 175 in a1707db

	# Is it a Cython-generated method? If so, try to detect
	# whether it only has a implicitly-generated docstring,
	# and no user-defined docstring following it.
	# The generated docstring would lack description of method
	# parameters and therefore fail Numpydoc validation.
	if hasattr(member, '__objclass__'):
	doc = getattr(member, '__doc__', None)
	# The Cython-generated docstring would be a one-liner,
	# such as "ReadOptions.equals(self, ReadOptions other)".
	if (doc and '\n' not in doc and f'.{name}(' in doc):
	continue
	todo.append(member)

I've been able to reproduce locally the failures on https://github.com/apache/arrow/actions/runs/15135088392/job/42544918301?pr=46511 via installing the newer Cython, building pyarrow and running:
archery numpydoc --allow-rule GL10,PR01,PR03,PR04,PR05,PR10,RT03,YD01
with the following failure:

...
pyarrow.lib.FixedSizeBufferWriter.set_memcopy_blocksize
-> pyarrow.lib.FixedSizeBufferWriter.set_memcopy_blocksize(self, int64_t blocksize)
PR01: Parameters {'blocksize'} not documented

pyarrow._json.ReadOptions.equals
-> pyarrow._json.ReadOptions.equals(self, ReadOptions other)
PR01: Parameters {'other'} not documented

pyarrow._json.ParseOptions.equals
-> pyarrow._json.ParseOptions.equals(self, ParseOptions other)
PR01: Parameters {'other'} not documented

Total number of docstring violations: 39

While debugging I can see that our check is not working anymore:

> /home/raulcd/code/arrow/dev/archery/archery/lang/python.py(172)traverse()
-> if hasattr(member, '__objclass__'):
(Pdb) hasattr(member, '__objclass__')
False
(Pdb) module
'pyarrow._csv'
(Pdb) name
'ReadOptions'
(Pdb) member
<class 'pyarrow._csv.ReadOptions'>

Component(s)

Continuous Integration, Documentation, Archery

[raulcd]

@pitrou this seems like a bug on the newest Cython, right? __objclass__ should be there

[raulcd]

The pdb example should have been this:

-> if hasattr(member, '__objclass__'):
(Pdb) member
<cyfunction ReadOptions.equals at 0x71fb825e0040>
(Pdb) name
'equals'
(Pdb) hasattr(member, '__objclass__')
False
``

[raulcd]

After some debugging it seems numpydoc is not correctly generating the parameters from the provided docstring:

(Pdb) self.doc['Parameters']
[]
(Pdb) self.doc['Summary']
['ReadOptions.equals(self, ReadOptions other)', 'Parameters', '----------', 'other : pyarrow.csv.ReadOptions']

This matches with the two different versions of docstring generated on the CPP file (depending on the Cython version used):

- PyDoc_STRVAR(__pyx_doc_7pyarrow_4_csv_11ReadOptions_6equals, "ReadOptions.equals(self, ReadOptions other)\nParameters\n----------\nother : pyarrow.csv.ReadOptions\n\nReturns\n-------\nbool");
+ PyDoc_STRVAR(__pyx_doc_7pyarrow_4_csv_11ReadOptions_6equals, "ReadOptions.equals(self, ReadOptions other)\n\n        Parameters\n        ----------\n        other : pyarrow.csv.ReadOptions\n\n        Returns\n        -------\n        bool\n        ");

[pitrou]

This matches with the two different versions of docstring generated on the CPP file (depending on the Cython version used):

• PyDoc_STRVAR(__pyx_doc_7pyarrow_4_csv_11ReadOptions_6equals, "ReadOptions.equals(self, ReadOptions other)\nParameters\n----------\nother : pyarrow.csv.ReadOptions\n\nReturns\n-------\nbool");

• PyDoc_STRVAR(__pyx_doc_7pyarrow_4_csv_11ReadOptions_6equals, "ReadOptions.equals(self, ReadOptions other)\n\n Parameters\n ----------\n other : pyarrow.csv.ReadOptions\n\n Returns\n -------\n bool\n ");

The one with stripped whitespace is Cython 3.1.1, right?
This is probably because of cython/cython#6241

[raulcd]

The one with stripped whitespace is Cython 3.1.1, right?

Yes, numpydocs expects two line separators (\n) between summary and parameters.

[pitrou]

Yes, numpydocs expects two line separators (\n) between summary and parameters.

Time to report a bug for Cython, then :)

[raulcd]

I've reported: cython/cython#6904

[raulcd]

The Cython issue has been fixed, this should be fixed for Cython 3.1.2 (once is released)