Python 3.15.0: test_sys.test_getallocatedblocks() fails if run after test_collections.test_odd_sizes()

[Fearless-Badger]

Bug report

Bug description:

test_sys fails while running the test suite of python 3.15.0 when --with-pydebug and --enable-optimizations are both enabled.

I get consistent failure of the test suite, but when I run the test individually, it passes. I do not know if this is expected or not, as I understand that enabling the debugger and optimizations is kind of contradictory. I included the test header, as well as the tests summary for additional info.

== CPython 3.15.0a0 (heads/main-dirty:9983c7d4416, May 19 2025, 09:41:00) [GCC 13.3.0]
== Linux-6.11.0-25-generic-x86_64-with-glibc2.39 little-endian
== Python build: debug PGO
== cwd: /home/badger/oss/cpython/build/test_python_worker_177894æ
== CPU count: 8
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 3038718328


...


0:32:13 load avg: 1.39 [396/491] test_sys
test test_sys failed -- Traceback (most recent call last):
  File "/home/badger/oss/cpython/Lib/test/test_sys.py", line 1156, in test_getallocatedblocks
    self.assertLess(a, sys.gettotalrefcount())
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 698068 not less than 663517

0:32:18 load avg: 1.36 [396/491/1] test_sys failed (1 failure)


...


== Tests result: FAILURE ==

22 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test_android test_apple test_dbm_gnu test_dbm_ndbm test_devpoll
    test_free_threading test_kqueue test_launcher test_msvcrt
    test_startfile test_winapi test_winconsoleio test_winreg test_wmi
    test_zstd

11 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_smtpnet
    test_socketserver test_tkinter test_ttk test_urllib2net
    test_urllibnet test_winsound test_zipfile64

1 test failed:
    test_sys

457 tests OK.

Total duration: 39 min 17 sec
Total tests: run=46,135 failures=1 skipped=2,164
Total test files: run=480/491 failed=1 skipped=22 resource_denied=11
Result: FAILURE

I don't suspect my local environment to be the problem, since the test suite passes when I remove the "--enable-optimizations" flag, and just enable the debugger. The variable "a" seems to change each time I run the suite, so I included the random seed.

Build setup:

Tested on Python 3.15.0.

Distributor ID:	Ubuntu
Description:	Ubuntu 24.04.2 LTS
Release:	24.04
Codename:	noble

CPU: product: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz

Configuration I used

./configure --with-pydebug --enable-optimizations 

CPython versions tested on:

3.15, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-134248 test_getallocatedblocks pre-check to ignore immortalized strings #134871
• [3.14] gh-134248 test_getallocatedblocks pre-check to ignore immortalized strings (GH-134871) #135095

[Fearless-Badger]

After investigating further, I have found that when running the tests with ./python -m test -j3 the test suite will pass. It is only when I run the test suite with ./python -m test that "test_sys" will fail.

Basically, "test_sys.test_getallocatedblocks() " only fails when run sequentially, and as part of the full suite.

[tpburns]

I had experienced the same error with a slightly different configuration, and have been experimenting to reproduce and maybe isolate. I have now observed the bug with a configure using only the --with-pydebug flag, though similarly only in the full test suite. See below (I cleaned up the outputs a bit)

tpburns@tpb-M-linux:~/dev/cpython$ make clean -s
tpburns@tpb-M-linux:~/dev/cpython$ ./configure -q --with-pydebug
tpburns@tpb-M-linux:~/dev/cpython$ make -s
Checked 114 modules (35 built-in, 78 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
tpburns@tpb-M-linux:~/dev/cpython$ ./python -m unittest -v test.test_sys.SysModuleTest.test_getallocatedblocks -q
----------------------------------------------------------------------
Ran 1 test in 0.009s

OK

and

tpburns@tpb-M-linux:~/dev/cpython$ make clean -s
tpburns@tpb-M-linux:~/dev/cpython$ ./configure -q --with-pydebug
tpburns@tpb-M-linux:~/dev/cpython$ make -s
Checked 114 modules (35 built-in, 78 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
tpburns@tpb-M-linux:~/dev/cpython$ ./python -m test -q
Using random seed: 4087959006
0:00:00 load avg: 0.81 Run 491 tests sequentially in a single process

test test_sys failed -- Traceback (most recent call last):
  File "/home/tpburns/dev/cpython/Lib/test/test_sys.py", line 1156, in test_getallocatedblocks
    self.assertLess(a, sys.gettotalrefcount())
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 702836 not less than 669601

== Tests result: FAILURE ==

1 test failed:
    test_sys

Total duration: 28 min 21 sec
Total tests: run=46,559 failures=1 skipped=2,055
Total test files: run=480/491 failed=1 skipped=14 resource_denied=11
Result: FAILURE

This was on Ubuntu 24.04.2 LTS, Core™ i7-1195G7 × 8 processor, Python 3.15.0a0 (main),

[vstinner]

I can reproduce the issue with:

./configure --with-pydebug
make
./python -m test

[vstinner]

The test_sys.test_getallocatedblocks() failure is related to test_collections.test_odd_sizes():

$ ./python -m test test_collections test_sys -m test_odd_sizes -m test_getallocatedblocks -v
== CPython 3.15.0a0 (heads/main:27bd08273ce, May 19 2025, 22:38:56) [GCC 15.1.1 20250425 (Red Hat 15.1.1-1)]
== Linux-6.14.4-300.fc42.x86_64-x86_64-with-glibc2.41 little-endian
== Python build: debug
== cwd: /home/vstinner/python/main/build/test_python_worker_177002æ
== CPU count: 10 (process) / 12 (system)
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 3050664783
0:00:00 load avg: 0.61 Run 2 tests sequentially in a single process
0:00:00 load avg: 0.61 [1/2] test_collections
test_odd_sizes (test.test_collections.TestNamedTuple.test_odd_sizes) ... ok

----------------------------------------------------------------------
Ran 1 test in 9.453s

OK
0:00:09 load avg: 0.75 [1/2] test_collections passed
0:00:09 load avg: 0.75 [2/2] test_sys
test_getallocatedblocks (test.test_sys.SysModuleTest.test_getallocatedblocks) ... FAIL

======================================================================
FAIL: test_getallocatedblocks (test.test_sys.SysModuleTest.test_getallocatedblocks)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/main/Lib/test/test_sys.py", line 1156, in test_getallocatedblocks
    self.assertLess(a, sys.gettotalrefcount())
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 259399 not less than 130596

----------------------------------------------------------------------
Ran 1 test in 0.048s

FAILED (failures=1)
test test_sys failed
0:00:10 load avg: 0.75 [2/2/1] test_sys failed (1 failure)

== Tests result: FAILURE ==

1 test failed:
    test_sys

1 test OK.

Total duration: 10.3 sec
Total tests: run=2 (filtered) failures=1
Total test files: run=2/2 (filtered) failed=1
Result: FAILURE

[vstinner]

According to git bisect, it started to fail at commit 2498c22.

commit 2498c22fa0a2b560491bc503fa676585c1a603d0
Author: Mark Shannon <mark@hotpy.org>
Date:   Wed Feb 19 11:44:57 2025 +0000

    GH-91079: Implement C stack limits using addresses, not counters. (GH-130007)

[vstinner]

The following change introduced the regression:

diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py
index f31d98bf731..6a2b316e3f6 100644
--- a/Lib/test/support/__init__.py
+++ b/Lib/test/support/__init__.py
@@ -2634,7 +2634,7 @@ def get_c_recursion_limit():
 
 def exceeds_recursion_limit():
     """For recursion tests, easily exceeds default recursion limit."""
-    return get_c_recursion_limit() * 3
+    return 100_000
 
 
 # Windows doesn't have os.uname() but it doesn't support s390x.

[tpburns]

Another bisect showed that it was 6f1d448 that first had the dependency of a large n in test_odd_sizes (now test_large_size after a recent refactor) causes the assert to fail in test_getallocatedblocks. Here's the story as far as I can tell:

1. The namedtuples __new__ method is implemented by passing a lambda function to eval with the input string including all of the names. During parsing each name is interned.

2. When immortalization was implemented in ea2c001, all interned strings were immortalized, but two references were kept for inclusion in the total reference count.

cpython/Objects/unicodeobject.c

Lines 14638 to 14644 in ea2c001

	/* The reference count value excluding the 2 references from the
	interned dictionary should be excluded from the RefTotal. The
	decrements to these objects will not be registered so they
	need to be accounted for in here. */
	for (Py_ssize_t i = 0; i < Py_REFCNT(s) - 2; i++) {
	_Py_DecRefTotal(_PyInterpreterState_GET());
	}

3. In response to Interned strings are immortal, despite what the documentation says #113993 6f1d448 reworked the implementation and corrected the book-keeping so those extra two references were excluded from the total count. This also made explicit that parsing will immortalize identifiers.

cpython/Objects/unicodeobject.c

Lines 15272 to 15277 in 6f1d448

	/* The reference count value should be excluded from the RefTotal.
	The decrements to these objects will not be registered so they
	need to be accounted for in here. */
	for (Py_ssize_t i = 0; i < Py_REFCNT(s); i++) {
	_Py_DecRefTotal(_PyThreadState_GET());
	}

4. The stack limit is bumped by an order of magnitude in GH-91079: Implement C stack limits using addresses, not counters. #130007, increasing the number of names used to construct the namedtuple.

Therefore, the large n number of names input to the namedtuple are immortalized and getallocatedblocks will include those blocks, but the current implementation doesn't show any references for them in gettotalrefcount.

I'm arranging a PR which brings the check in test_getallocatedblocks more in line with its original intent by removing the size of the blocks used on immortalized strings. This same technique is already used by refleak.py (see #122420) so it seems appropriate here as well.

There are other ways this could be separately or simultaneously be addressed. I want to specifically link the discussion in #130384 to adjust how sys_getallocatedblocks_impl works. It's my understanding that immortalization is per interpreter unless ThreadState is explicitly shared, so such a change would open up the possibility of isolating either of the two tests in its own interpreter, giving a more thorough decoupling of these tests.

I'd very much welcome insight from anyone who worked on these tests or immortalization previously!

Edited to add a correction to (3), the linked code above is accounting for all of the references prior to interning, but in intern_common is where the two associated with the dict are handled both in the total and for the count on the string object:

cpython/Objects/unicodeobject.c

Lines 15518 to 15527 in 046670c

	if (!_Py_IsImmortal(s)) {
	/* The two references in interned dict (key and value) are not counted.
	unicode_dealloc() and _PyUnicode_ClearInterned() take care of this. */
	Py_SET_REFCNT(s, Py_REFCNT(s) - 2);
	#ifdef Py_REF_DEBUG
	/* let's be pedantic with the ref total */
	_Py_DecRefTotal(_PyThreadState_GET());
	_Py_DecRefTotal(_PyThreadState_GET());
	#endif
	}

The call to immortalize_interned (where the loop is) happens afterwards, which is why its accounting changed in that commit, but the behavior difference was introduced by the two decrements in intern_common.