gh-110850: Add PyTime C API (PyTime_t type)

[vstinner]

API:

• type PyTime_t
• constant PyTime_MIN (PyTime_t type)
• constant PyTime_MAX (PyTime_t type)
• double PyTime_AsSecondsDouble(PyTime_t t): Convert a timestamp to a number of seconds.
• PyTime_t PyTime_Monotonic(void): similar to time.monotonic_ns()
• PyTime_t PyTime_PerfCounter(void): similar to time.perf_counter_ns()
• PyTime_t PyTime_Time(void): similar to time.time_ns()
• PyTime_Monotonic(), PyTime_PerfCounter() and PyTime_Time() return 0 on error (and ignore silently the error), and clamp the clock to the [PyTime_MIN; PyTime_MAX] range on integer overflow.

These functions are being used internally in Python since around Python 3.5 to avoid rounding issues (floating point <=> integer) at nanosecond resolution. PyTime_t is just a 64-bit signed integer.

The "nanosecond" unit is not explicit in the API, the unit is "arbitrary" even if it's nanosecond in practice. The internal C API has functions to create PyTime_t values from seconds and from nanoseconds. I didn't add them in this inital public C API.

Cython started to use this API whereas the private API was removed in Python 3.13 alpha 1. Cython needs:

• PyTime_t type
• PyTime_Time()
• PyTime_AsSecondsDouble()

The Python API reports errors as regular exceptions, whereas the C API silently ignores errors. When I designed and implemented PEP 418: Add monotonic time, performance counter, and process time functions, I was worried about errors while reading time. I added code at Python startup to read the 3 clocks (time, perf_counter, monotonic) and fail with a fatal error if any failed. Many years later, I removed the check since it never failed.

The C API is designed to be convenient to use, not to be "perfect" (report unlikely error). Over 10 years, I saw a single failure in a custom sandbox which blocked syscalls to read time. It was a single user on a very specific issue, and it was an issue in the sandbox config, not in Python. IMO returning 0 in the C API is the sane behavior in this case. Bothering all users to have to check for errors just for that would be overkill.

Example of usage:

double benchmark(void)
{
    PyTime_t t1 = PyTime_PerfCounter();
    // ... code ...
    PyTime_t t2 = PyTime_PerfCounter();
    return PyTime_AsSecondsDouble(t2 - t1)
}

The PyTime internal C API is way more complete, but I chose to start with the bare minimum for the public C API.

Pull request: python/cpython#112135

[encukou]

I removed the check since it never failed

My concern is that this ties the API to current platforms, with currently common settings.
For example, I expect that sandboxting will be more common on Wasm.

As I said elsewhere, if an API could fail in the future, it should have a way to report failure. Error-checking is unpleasant, but it's part of using a C API.

[vstinner]

Clock functions like Windows GetTickCount64() used by time.monotonic() and gettimeofday() used by time.time() simply cannot fail. The strange thing is that time() can fail according to its manual page, whereas gettimeofday() cannot :-) Nowadays, Python tries first to call clock_gettime(), but then have multiple fallbacks if it fails.

Error-checking is unpleasant, but it's part of using a C API.

My point is more that if there is a corner case where reading clock can fail, returning 0 and ignoring error is an acceptable trade-off. The program is unlikely to run normally if all clocks are blocked by a sandbox, other things will fail anyway. And for me, it's more the responsibility of the user to fix their setup, than Python to report errors in such case.

For example, I expect that sandboxting will be more common on Wasm.

Do you have a concrete example? Or is it a theorical concern?

[encukou]

Well, I guess we disagree.
I think that API functions should be able to report failure. Exceptions to that should be very, very rare (and if they do happen I'd prefer adding an additional variant without error-checking.)

Bothering all users to have to check for errors just for that would be overkill.

As a code reviewer, seeing a Py* API call without an error check is a red flag; for functions that don't need this I need to check the docs (unless it's very common, like an incref). This bothers me more than having to write a few extra lines of error-checking.

The C API is designed to be convenient to use, not to be "perfect" (report unlikely error).

If we're OK with imperfect API, let's bring back _PyTime_t &co.
If we -- as a group -- believe that its biggest problem is the leading underscore, then I vote for not removing it.

[vstinner]

Python has around 34 calls to _PyTime_GetMonotonicClock(), _PyTime_GetPerfCounter() and _PyTime_GetSystemClock() functions which cannot report errors. It also has around 35 calls _PyDeadline_Init() and _PyDeadline_Get() functions which cannot report errors: wrapper on top of _PyTime_GetMonotonicClock() to implement a "deadline".

I would prefer to have not modify this code just for a theorical issue, or at least "unlikely" case.

_PyTime_GetSystemClock() implementation

• Windows: GetSystemTimeAsFileTime() cannot fail
• clock_gettime(CLOCK_REALTIME) can fail, or: gettimeofday(): cannot fail
• In short: It cannot fail

_PyTime_GetMonotonicClock() implementation

• GetTickCount64() cannot fail, but convertion from millisecond to nanosecond (x 1_000_000) can overflow when we are getting closer to _PyTime_MAX (in year 2262: so in 238 years).

• macOS: mach_absolute_time(), mach_timebase_info() cannot fail. Same here, Python can report an error if mach_timebase_info() returns (0, 0), but it would be a macOS bug and most programs would be affected by that.

• HP-UX: gethrtime() can fail according its manual page, but the manual page example doesn't check for error. I'm not sure why an error case is documented. Can it really happen in practice?

• Solaris: clock_gettime(CLOCK_HIGHRES) can fail. I never fail any of such failure.

• Unix: clock_gettime(CLOCK_MONOTONIC) can fail. I never fail any of such failure.

_PyTime_GetPerfCounter() implementation

• Windows: QueryPerformanceFrequency(), QueryPerformanceCounter(): cannot fail. Python can report an error if QueryPerformanceFrequency() return 0 but that would be a big bug in Windows affecting most programs!

• Otherwise: call _PyTime_GetMonotonicClock().

Other programming languages.

• C time() cannot fail.
• C gettimeofday() cannot fail.
• Rust std::time::SystemTime::now() cannot fail.
• Rust std::time::Instant::now() cannot fail.
• PHP https://www.php.net/manual/en/function.time.php cannot fail.
• PHP https://www.php.net/manual/en/function.gettimeofday.php cannot fail.
• PHP https://www.php.net/manual/en/function.microtime.php cannot fail.
• PHP https://www.php.net/manual/en/function.hrtime.php can fail (return "false" on error).

[vstinner]

@gvanrossum @iritkatriel @zooba: So, what do you think about these APIs?

[vstinner]

This bothers me more than having to write a few extra lines of error-checking.

See how these functions are used. There are cases where errors cannot be reported, such as PyThread_acquire_lock_timed().

I think that API functions should be able to report failure. Exceptions to that should be very, very rare (and if they do happen I'd prefer adding an additional variant without error-checking.)

Python is using these APIs for around 10 years without any issue. It seems like Cython is also using them. We cannot get any issue about them. So I'm confused on why they should now be changed.

[encukou]

Python has around 34 calls to _PyTime_GetMonotonicClock(), _PyTime_GetPerfCounter() and _PyTime_GetSystemClock() functions which cannot report errors. It also has around 35 calls _PyDeadline_Init() and _PyDeadline_Get() functions which cannot report errors: wrapper on top of _PyTime_GetMonotonicClock() to implement a "deadline".

But there's no need for these to use the public API. Is there?
We can adjust these calls any time we want, if we need to port to a platform without a good clock. We can't quite do that for public API.

Other programming languages

• C time() can fail. When it does, it returns -1 and should set errno.
• C gettimeofday() can fail. When it does, it returns -1 and should set errno.
• Rust std::time::*::now() cannot return an error, but it may panic.
• PHP has exceptions; all its functions can fail.

So I'm confused on why they should now be changed.

We are now exposing them as supported public API, rather than just-good-enough internal helpers.
That is what removing the underscore means. We need to do API design now.

---

And to return to a question i omitted:

Do you have a concrete example? Or is it a theorical concern?

You've provided an example yourself:

Over 10 years, I saw a single failure in a custom sandbox which blocked syscalls to read time. It was a single user on a very specific issue, and it was an issue in the sandbox config, not in Python.

Python has a mechanism for reporting rare issues like a misconfigured environment: raising an exception.

[vstinner]

But there's no need for these to use the public API. Is there?

See the rationale of my first message: Cython uses this API.

We are now exposing them as supported public API, rather than just-good-enough internal helpers.
That is what removing the underscore means. We need to do API design now.

Cython uses the current private API.

We can adjust these calls any time we want, if we need to port to a platform without a good clock. We can't quite do that for public API.

Last time I checked, Hurd has no support for a monotonic clock. Python doesn't support Hurd. If someone wants to run Python on Hurd and there is no CLOCK_MONOTONIC support, just use CLOCK_REALTIME and cross fingers :-)

C gettimeofday() can fail. When it does, it returns -1 and should set errno.

I used some shortcuts: py_get_system_clock() pass NULL as the second argument. The function cannot fail with EINVAL. Other errors are about settimeofday().

C time() can fail. When it does, it returns -1 and should set errno.

Oh sorry, I read the wrong manual page. Python doesn't call time() anymore.

Rust std::time::*::now() cannot return an error, but it may panic.

The documentation says "Note: mathematical operations like add may panic if the underlying structure cannot represent the new point in time." But I didn't see anything about now() triggering a panic. I suppose that yes, it can trigger a panic. I just didn't see any clear mention in the doc.

[vstinner]

Also, one alternative is to call Py_FatalError() if a clock fails. I hate this function, since it kills the process. It's really bad when Python is embedded. But it's a tradeoff for things which "must not happen".

[encukou]

I think the discussion is starting to go in circles.

I fail to see how Cython is related to _PyTime_GetMonotonicClock(), _PyTime_GetPerfCounter(), _PyTime_GetSystemClock(), _PyDeadline_Init(), _PyDeadline_Get(), and _PyTime_GetMonotonicClock(). Cython's use of the time API looks trivally portable to fallible functions. Any projects that use this more heavily can use a wrapper that catches the exception and returns 0, kills the process, or does any other kind of error handling.

Again, if we're concerned that porting to a better API is too hard, then I vote for keeping the existing imperfect API, with a leading underscore, while (and after) we add API without known issues.

[vstinner]

Cython's use of the time API looks trivally portable to fallible functions

What do you mean by "trivally portable to fallible functions"? Cython calls _PyTime_GetSystemClock() and _PyTime_AsSecondsDouble() in Cython/Includes/cpython/time.pxd, in the function:

cdef inline double time() nogil:

source: https://github.com/cython/cython/blob/0b6cab31738870e60b4ae552dd1c4335529c71a0/Cython/Includes/cpython/time.pxd#L20-L25

This function doesn't hold the GIL and always return a C double.

[vstinner]

Again, if we're concerned that porting to a better API is too hard, then I vote for keeping the existing imperfect API, with a leading underscore, while (and after) we add API without known issues.

Why do you say "too hard"?

I'm not sure that there is an advantage of having a C API which behaves exactly like time.monotonic_ns(): need to hold the GIL and can raise an exception. If you hold the GIL and can report errors, well, just call time.monotonic_ns() directly. The only advantage would be to avoid having to import the time module and to get the time.monotonic_ns attribute.

The proposed API is different, it has less constraints, it can be used without the GIL and it doesn't require to report errors. So it's more efficient and can be used in functions which cannot or don't want to report errors.

PHP has exceptions; all its functions can fail.

PHP cannot raise exceptions when reading time. Example with gettimeofday() and microtime(). That's the code which is running Wikipedia :-)

	if (gettimeofday(&tp, NULL)) {
		ZEND_ASSERT(0 && "gettimeofday() can't fail");
	}

code: https://github.com/php/php-src/blob/b3bdc23656116f6f281385c05ead1a5d0b9aea6e/ext/standard/microtime.c#L56-L58

So PHP users don't have to bother with errors on reading time.

Rust std::time::*::now() cannot return an error, but it may panic.

Right. It means of API, it means that the user doesn't have to both with this case which make the API more convenient to use. Apparently, errors on reading time is considered too unlikely to be reported in Rust as a regular std::io::Result to report errors, but panic which exits the process.

[encukou]

Now we're definitely running in circles, repeating what's already been said. So I'll be brief:

[Cython's time] doesn't hold the GIL and always return a C double.

Right; its nogil needs to be removed.
The same is true with the proposed API: in general, you need to hold the GIL before calling C-API.
Or did you want to propose a guarantee that the GIL doesn't need to be held?

The proposed API is different, it has less constraints, it can be used without the GIL and it doesn't require to report errors. So it's more efficient and can be used in functions which cannot or don't want to report errors.

I see no demand for such functions to be public API, as far as I can see.

I'm not sure that there is an advantage of having a C API which behaves exactly like time.monotonic_ns()

Good point. Let's not add it, then.