Add PyLong_FromInt64() and PyLong_AsInt64()

[vstinner]

Add 8 new functions to the limited C API to convert C <stdint.h> numbers from/to Python int:

// Return a Python int on success.
// Set an exception and return NULL on error.
PyAPI_FUNC(PyObject*) PyLong_FromInt32(int32_t value);
PyAPI_FUNC(PyObject*) PyLong_FromUInt32(uint32_t value);
PyAPI_FUNC(PyObject*) PyLong_FromInt64(int64_t value);
PyAPI_FUNC(PyObject*) PyLong_FromUInt64(uint64_t value);

// Set *value and return 0 on success.
// Set an exception and return -1 on error.
PyAPI_FUNC(int) PyLong_AsInt32(PyObject *obj, int32_t *value);
PyAPI_FUNC(int) PyLong_AsUInt32(PyObject *obj, uint32_t *value);
PyAPI_FUNC(int) PyLong_AsInt64(PyObject *obj, int64_t *value);
PyAPI_FUNC(int) PyLong_AsUInt64(PyObject *obj, uint64_t *value);

UPDATE 1: I removed PyLong_FromInt32() and PyLong_FromUInt32() functions.

UPDATE 2: I renamed To functions to As functions.

UPDATE 3: I added back PyLong_FromInt32() and PyLong_FromUInt32() functions for 32-bit systems.

• As functions call obj.__index__() if obj is not a Python int.
• As functions converting to unsigned integer raises ValueError if value is negative
• As functions raise OverflowError if the value is too big (doesn't fit into the C integer range)

Compared to PyLong_AsLong(), As function return value is status (success or error, 0 or -1) and the result is set in the value argument (on success). It avoids having to call PyErr_Occurred() to distinguish -1 (success) and -1 (error). See Problem #1 for details.

Links:

• C API WG discussion: Avoid C-specific Types api-evolution#10
• Issue: [C API] Add PyLong_FromInt64() and PyLong_ToInt64() python/cpython#120389
• PR: gh-120389: Add PyLong_FromInt64() and PyLong_AsInt64() python/cpython#120390

[zooba]

There's nothing particularly controversial about this API, right? Other than the question of whether it should be "real" or macros?

I'm quite happy with it being a real API.

[serhiy-storchaka]

Since PyLong_To* functions have signature compatible with the converter signature used with the O& format unit in PyArg_Parse(), I think that it would be better to make the return value compatible with the converter convention: return 1 for a successful conversion and 0 if the conversion has failed. So these functions could be used with PyArg_Parse() (and other functions in this family).

Otherwise we can end with two sets of converters with opposite return conventions. It will be very errorprone.

[vstinner]

I think that it would be better to make the return value compatible with the converter convention: return 1 for a successful conversion and 0 if the conversion has failed.

I don't think that this use case is common enough to not follow the C API Guidelines:
https://devguide.python.org/developer-workflow/c-api/index.html#guidelines-for-expanding-changing-the-public-api

For converters, I would prefer to add support for int32_t and uint64_t to Argument Clinic and reuse these functions.

[zooba]

think that it would be better to make the return value compatible with the converter convention

It's tempting, for sure, but I think we're better to keep converters as the exception. We really want developers to have a single mental model for how to handle return values, and that unfortunately means converters need to be clearly labelled, and not just convenient functions.

Adding some form of sized version of i and u to PyArg_ParseTuple wouldn't be a bad step. That's our public API - Argument Clinic is still internal only, so we don't fix that until we need it, and it doesn't help public users at all when we do.

[encukou]

Other than the question of whether it should be "real" or macros?

Thank you for humoring me on the macros thing; I think it's clear we want real functions :)

keep converters as the exception

Yes. We settled on using -1 (or NULL) for errors. Unfortunately some existing functions -- or even families of functions -- don't do that; we have to live with that.

Eventually, if we get to the point where O& sticks out too much, we might want to add a new PyArg_Parse format code. I don't think we're there yet.

[vstinner]

@serhiy-storchaka has some concerns about calling obj.__index__() which can release the GIL indirectly:

Calling __index__ can release the GIL and make values saved in C variables (borrowed references, the size of a list, etc) invalid. Existing code can rely on atomicity of PyLong_AsUnsignedLong(), so calling __index__ in it is an unsafe change.

IMO it's good that signed and unsigned functions have the same behavior, that both call __index__(). Existing functions are inconsistent: signed functions (ex: PyLong_AsInt()) call __index__(), whereas PyLong_AsUnsignedLong() don't call the method.

The GIL issue can be documented.

[vstinner]

@serhiy-storchaka @mdboom: Do you have any additional comment about these APIs, or can I just open a vote?

[mdboom]

I have nothing to add that hasn't already been said.

[serhiy-storchaka]

My comment was about the existing C API like PyLong_AsUnsignedLong() and why it is not safe to change it.

We need a C API which respects special methods like __index__, and it should be default. On other hand, we need a C API to access the content of the object. This is a contradiction. Creating two sets of functions will cause confusion, especially if there will be more than two sets, with different signatures and different calling conventions. I have an idea how to mitigate this problem, but it is for a different discussion. It does not stay on the way of this set of functions, because they all are just wrappers around PyLong_FromNativeBytes/PyLong_AsNativeBytes.

PyLong_FromInt32 and PyLong_FromUInt32 are redundant, because the integer argument can always be promoted to 64 bits for PyLong_FromInt64/PyLong_FromUInt64. We have both PyLong_FromLong and PyLong_FromLongLong because the latter was optional, not all compilers supported long long.

On other hand, the lack of PyLong_ToInt16 and PyLong_ToInt8 requires adding additional check after calling PyLong_ToInt32, and producing OverflowError with inconsistent error messages (depending on the input value) or writing more verbose code with PyLong_AsNativeBytes.

[zooba]

Right, this is essentially the difference between the abstract and concrete APIs, a difference which has been eroding over the last few years.

If we think we can recover it, then we could argue the caller should PyLong_Check before using these functions, and use a more abstract function if they don't actually have an int already.

Alternatively, if we don't expect to shut the door to concrete APIs operating abstractly, then we could document that if PyLong_Check is true, these functions will not call back into Python code in order to convert to an integer.

I'm +0 on the latter, just out of pragmatism. But it saddens me to lose the clear abstract/concrete separation. (It will make those who want our API to be strongly typed sadder than me, though. I like it being largely abstract, I just think we ought to know which APIs are which.)

[vstinner]

@serhiy-storchaka: It's uneasy for me to understand what do you propose.

Creating two sets of functions will cause confusion, especially if there will be more than two sets, with different signatures and different calling conventions.

So are you ok with PyLong_ToInt64() calling __index__() or not? What do you suggest?

PyLong_FromInt32 and PyLong_FromUInt32 are redundant

Do you suggest to not add them and use PyLong_FromInt64() and PyLong_FromUInt64()? That sounds reasonable.

On other hand, the lack of PyLong_ToInt16 and PyLong_ToInt8 requires adding additional check after calling PyLong_ToInt32 (...)

I didn't want to add too many functions at the beginning, so I started with the two most common sizes: 32 and 64 bits. We can add more functions later, once we agree on the design.

[vstinner]

If we think we can recover it, then we could argue the caller should PyLong_Check before using these functions, and use a more abstract function if they don't actually have an int already.

I agree that PyLong_Check(), or even PyLong_CheckExact(), is a good way to check if an argument is an integer.

In general, users don't care much if it's exactly an integer or not (if the object type is Python int). For example, I'm used to pass a boolean when an integer is expected (bool inherits from int), even if a boolean is a different concept than an integer (to me at least).