From 9fcccce4fe3199dab5d6cdaab0b52f4e332e3a06 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Mon, 19 Feb 2024 13:57:41 +0000 Subject: [PATCH 1/7] gh-89480: Document the gdb helpers --- Doc/howto/gdb_helpers.rst | 418 ++++++++++++++++++++++++++++++++++++++ Doc/howto/index.rst | 1 + 2 files changed, 419 insertions(+) create mode 100644 Doc/howto/gdb_helpers.rst diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst new file mode 100644 index 00000000000000..b02992f08a8181 --- /dev/null +++ b/Doc/howto/gdb_helpers.rst @@ -0,0 +1,418 @@ +.. _gdb: + +=========== +GDB support +=========== + +.. highlight:: none + +If you experience low-level problems such as crashes or deadlocks +(e.g. when tinkering with parts of CPython which are written in C), +it can be convenient to use a low-level debugger such as gdb in +order to diagnose and fix the issue. By default, however, gdb (or any +of its front-ends) doesn't know about high-level information specific to the +CPython interpreter, such as which Python function is currently executing, +or what type or value has a given Python object represented by a standard +``PyObject *`` pointer. We hereafter present two ways to overcome this +limitation. + +Prerequisites +=============== + +You need to have gdb on your system and Python debugging extensions. Extensions +package includes debugging symbols and adds Python-specific commands into gdb. +On a modern Linux system, you can easily install these with: + +- Fedora: + + .. code-block:: shell + + sudo yum install gdb python-debuginfo + +- Ubuntu: + + .. code-block:: shell + + sudo apt-get install gdb python3.10-dbg + +- Centos*: + + .. code-block:: shell + + sudo yum install yum-utils + sudo debuginfo-install glibc + sudo yum install gdb python-debuginfo + +GDB 7 and later +=============== + +In gdb 7, support for `extending gdb with Python +`_ was +added. When CPython is built you will notice a ``python-gdb.py`` file in the +root directory of your checkout. Read the module docstring for details on how +to use the file to enhance gdb for easier debugging of a CPython process. + +To activate support, you must add the directory containing ``python-gdb.py`` +to GDB's "auto-load-safe-path". Put this in your ``~/.gdbinit`` file:: + + add-auto-load-safe-path /path/to/checkout + +You can also add multiple paths, separated by ``:``. + +This is what a backtrace looks like (truncated) when this extension is +enabled:: + + #0 0x000000000041a6b1 in PyObject_Malloc (nbytes=Cannot access memory at address 0x7fffff7fefe8 + ) at Objects/obmalloc.c:748 + #1 0x000000000041b7c0 in _PyObject_DebugMallocApi (id=111 'o', nbytes=24) at Objects/obmalloc.c:1445 + #2 0x000000000041b717 in _PyObject_DebugMalloc (nbytes=24) at Objects/obmalloc.c:1412 + #3 0x000000000044060a in _PyUnicode_New (length=11) at Objects/unicodeobject.c:346 + #4 0x00000000004466aa in PyUnicodeUCS2_DecodeUTF8Stateful (s=0x5c2b8d "__lltrace__", size=11, errors=0x0, consumed= + 0x0) at Objects/unicodeobject.c:2531 + #5 0x0000000000446647 in PyUnicodeUCS2_DecodeUTF8 (s=0x5c2b8d "__lltrace__", size=11, errors=0x0) + at Objects/unicodeobject.c:2495 + #6 0x0000000000440d1b in PyUnicodeUCS2_FromStringAndSize (u=0x5c2b8d "__lltrace__", size=11) + at Objects/unicodeobject.c:551 + #7 0x0000000000440d94 in PyUnicodeUCS2_FromString (u=0x5c2b8d "__lltrace__") at Objects/unicodeobject.c:569 + #8 0x0000000000584abd in PyDict_GetItemString (v= + {'Yuck': , '__builtins__': , '__file__': 'Lib/test/crashers/nasty_eq_vs_dict.py', '__package__': None, 'y': , 'dict': {0: 0, 1: 1, 2: 2, 3: 3}, '__cached__': None, '__name__': '__main__', 'z': , '__doc__': None}, key= + 0x5c2b8d "__lltrace__") at Objects/dictobject.c:2171 + +(Notice how the dictionary argument to ``PyDict_GetItemString`` is displayed +as its ``repr()``, rather than an opaque ``PyObject *`` pointer.) + +The extension works by supplying a custom printing routine for values of type +``PyObject *``. If you need to access lower-level details of an object, then +cast the value to a pointer of the appropriate type. For example:: + + (gdb) p globals + $1 = {'__builtins__': , '__name__': + '__main__', 'ctypes': , '__doc__': None, + '__package__': None} + + (gdb) p *(PyDictObject*)globals + $2 = {ob_refcnt = 3, ob_type = 0x3dbdf85820, ma_fill = 5, ma_used = 5, + ma_mask = 7, ma_table = 0x63d0f8, ma_lookup = 0x3dbdc7ea70 + , ma_smalltable = {{me_hash = 7065186196740147912, + me_key = '__builtins__', me_value = }, + {me_hash = -368181376027291943, me_key = '__name__', + me_value ='__main__'}, {me_hash = 0, me_key = 0x0, me_value = 0x0}, + {me_hash = 0, me_key = 0x0, me_value = 0x0}, + {me_hash = -9177857982131165996, me_key = 'ctypes', + me_value = }, + {me_hash = -8518757509529533123, me_key = '__doc__', me_value = None}, + {me_hash = 0, me_key = 0x0, me_value = 0x0}, { + me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}} + +The pretty-printers try to closely match the ``repr()`` implementation of the +underlying implementation of Python, and thus vary somewhat between Python 2 +and Python 3. + +An area that can be confusing is that the custom printer for some types look a +lot like gdb's built-in printer for standard types. For example, the +pretty-printer for a Python 3 ``int`` gives a ``repr()`` that is not +distinguishable from a printing of a regular machine-level integer:: + + (gdb) p some_machine_integer + $3 = 42 + + (gdb) p some_python_integer + $4 = 42 + + (gdb) p *(PyLongObject*)some_python_integer + $5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1}, + ob_digit = {42}} + +A similar confusion can arise with the ``str`` type, where the output looks a +lot like gdb's built-in printer for ``char *``:: + + (gdb) p ptr_to_python_str + $6 = '__builtins__' + +The pretty-printer for ``str`` instances defaults to using single-quotes (as +does Python's ``repr`` for strings) whereas the standard printer for ``char *`` +values uses double-quotes and contains a hexadecimal address:: + + (gdb) p ptr_to_char_star + $7 = 0x6d72c0 "hello world" + +Here's how to see the implementation details of a ``str`` instance (for Python +3, where a ``str`` is a ``PyUnicodeObject *``):: + + (gdb) p *(PyUnicodeObject*)$6 + $8 = {ob_base = {ob_refcnt = 33, ob_type = 0x3dad3a95a0}, length = 12, + str = 0x7ffff2128500, hash = 7065186196740147912, state = 1, defenc = 0x0} + +As well as adding pretty-printing support for ``PyObject *``, +the extension adds a number of commands to gdb: + +``py-list`` +----------- + + List the Python source code (if any) for the current frame in the selected + thread. The current line is marked with a ">":: + + (gdb) py-list + 901 if options.profile: + 902 options.profile = False + 903 profile_me() + 904 return + 905 + >906 u = UI() + 907 if not u.quit: + 908 try: + 909 gtk.main() + 910 except KeyboardInterrupt: + 911 # properly quit on a keyboard interrupt... + + Use ``py-list START`` to list at a different line number within the python + source, and ``py-list START,END`` to list a specific range of lines within + the python source. + +``py-up`` and ``py-down`` +------------------------- + + .. note:: + + Versions older than Python 3.12 have a 1 to 1 mapping between the C stack + and the Python stack. This means than using ``py-up`` and ``py-down`` will + move the C stack exactly one frame up or down. This is not the case for + older versions of Python as the same C stack frame can be used for multiple + Python stack frames. This means that when using ``py-up`` and ``py-down`` + there may be multiple Python frames that will be moved at the same time. + + + The ``py-up`` and ``py-down`` commands are analogous to gdb's regular ``up`` + and ``down`` commands, but try to move at the level of CPython frames, rather + than C frames. + + gdb is not always able to read the relevant frame information, depending on + the optimization level with which CPython was compiled. Internally, the + commands look for C frames that are executing ``PyEval_EvalFrameEx`` (which + implements the core bytecode interpreter loop within CPython) and look up + the value of the related ``PyFrameObject *``. + + They emit the frame number (at the C level) within the thread. + + For example:: + + (gdb) py-up + #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/ + gnome_sudoku/main.py, line 906, in start_game () + u = UI() + (gdb) py-up + #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/ + gnome_sudoku/gnome_sudoku.py, line 22, in start_game(main=) + main.start_game() + (gdb) py-up + Unable to find an older python frame + + so we're at the top of the python stack. Going back down:: + + (gdb) py-down + #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game () + u = UI() + (gdb) py-down + #34 (unable to read python frame information) + (gdb) py-down + #23 (unable to read python frame information) + (gdb) py-down + #19 (unable to read python frame information) + (gdb) py-down + #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': , 'gsd.hints': 0, 'timer.active_time': , 'timer.total_time': }], dialog=, saved_game_model=, sudoku_maker=, main_page=0) at remote 0x98fa6e4>, d=) + gtk.main() + (gdb) py-down + #8 (unable to read python frame information) + (gdb) py-down + Unable to find a newer python frame + + and we're at the bottom of the python stack. + +``py-bt`` +--------- + + The ``py-bt`` command attempts to display a Python-level backtrace of the + current thread. + + For example:: + + (gdb) py-bt + #8 (unable to read python frame information) + #11 Frame 0x9aead74, for file /usr/lib/python2.6/site-packages/gnome_sudoku/dialog_swallower.py, line 48, in run_dialog (self=, main_page=0) at remote 0x98fa6e4>, d=) + gtk.main() + #14 Frame 0x99262ac, for file /usr/lib/python2.6/site-packages/gnome_sudoku/game_selector.py, line 201, in run_swallowed_dialog (self=, puzzle=None, saved_games=[{'gsd.auto_fills': 0, 'tracking': {}, 'trackers': {}, 'notes': [], 'saved_at': 1270084485, 'game': '7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 0 0 0 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5\n7 8 0 0 0 0 0 5 6 0 0 9 0 8 0 1 0 0 0 4 6 0 0 0 0 7 0 6 5 1 8 3 4 7 9 2 0 0 0 9 0 1 0 0 0 3 9 7 6 0 0 0 1 8 0 6 0 0 0 0 2 8 0 0 0 5 0 4 0 6 0 0 2 1 0 0 0 0 0 4 5', 'gsd.impossible_hints': 0, 'timer.__absolute_start_time__': , 'gsd.hints': 0, 'timer.active_time': , 'timer.total_time': }], dialog=, saved_game_model=, sudoku_maker=) + main.start_game() + + The frame numbers correspond to those displayed by gdb's standard + ``backtrace`` command. + +``py-print`` +------------ + The ``py-print`` command looks up a Python name and tries to print it. + It looks in locals within the current thread, then globals, then finally + builtins:: + + (gdb) py-print self + local 'self' = , + main_page=0) at remote 0x98fa6e4> + (gdb) py-print __name__ + global '__name__' = 'gnome_sudoku.dialog_swallower' + (gdb) py-print len + builtin 'len' = + (gdb) py-print scarlet_pimpernel + 'scarlet_pimpernel' not found + +``py-locals`` +------------- + + The ``py-locals`` command looks up all Python locals within the current + Python frame in the selected thread, and prints their representations:: + + (gdb) py-locals + self = , + main_page=0) at remote 0x98fa6e4> + d = + +Other commands +-------------- + +You can of course use other gdb commands. For example, the ``frame`` command +takes you directly to a particular frame within the selected thread. +We can use it to go a specific frame shown by ``py-bt`` like this:: + + (gdb) py-bt + (output snipped) + #68 Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in () + main() + (gdb) frame 68 + #68 0x00000000004cd1e6 in PyEval_EvalFrameEx (f=Frame 0xaa4560, for file Lib/test/regrtest.py, line 1548, in (), throwflag=0) at Python/ceval.c:2665 + 2665 x = call_function(&sp, oparg); + (gdb) py-list + 1543 # Run the tests in a context manager that temporary changes the CWD to a + 1544 # temporary and writable directory. If it's not possible to create or + 1545 # change the CWD, the original CWD will be used. The original CWD is + 1546 # available from test_support.SAVEDCWD. + 1547 with test_support.temp_cwd(TESTCWD, quiet=True): + >1548 main() + +The ``info threads`` command will give you a list of the threads within the +process, and you can use the ``thread`` command to select a different one:: + + (gdb) info threads + 105 Thread 0x7fffefa18710 (LWP 10260) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 + 104 Thread 0x7fffdf5fe710 (LWP 10259) sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86 + * 1 Thread 0x7ffff7fe2700 (LWP 10145) 0x00000038e46d73e3 in select () at ../sysdeps/unix/syscall-template.S:82 + +You can use ``thread apply all COMMAND`` or (``t a a COMMAND`` for short) to run +a command on all threads. You can use this with ``py-bt`` to see what every +thread is doing at the Python level:: + + (gdb) t a a py-bt + + Thread 105 (Thread 0x7fffefa18710 (LWP 10260)): + #5 Frame 0x7fffd00019d0, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140737213728528), count=1, owner=140737213728528) + self.__block.acquire() + #8 Frame 0x7fffac001640, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, acquire=, _is_owned=, _release_save=, release=, _acquire_restore=, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=, saved_state=(1, 140737213728528)) + self._acquire_restore(saved_state) + #12 Frame 0x7fffb8001a10, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f () + cond.wait() + #16 Frame 0x7fffb8001c40, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140737213728528) + f() + + Thread 104 (Thread 0x7fffdf5fe710 (LWP 10259)): + #5 Frame 0x7fffe4001580, for file /home/david/coding/python-svn/Lib/threading.py, line 155, in _acquire_restore (self=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, count_owner=(1, 140736940992272), count=1, owner=140736940992272) + self.__block.acquire() + #8 Frame 0x7fffc8002090, for file /home/david/coding/python-svn/Lib/threading.py, line 269, in wait (self=<_Condition(_Condition__lock=<_RLock(_Verbose__verbose=False, _RLock__owner=140737354016512, _RLock__block=, _RLock__count=1) at remote 0xd7ff40>, acquire=, _is_owned=, _release_save=, release=, _acquire_restore=, _Verbose__verbose=False, _Condition__waiters=[]) at remote 0xd7fd10>, timeout=None, waiter=, saved_state=(1, 140736940992272)) + self._acquire_restore(saved_state) + #12 Frame 0x7fffac001c90, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 348, in f () + cond.wait() + #16 Frame 0x7fffac0011c0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 37, in task (tid=140736940992272) + f() + + Thread 1 (Thread 0x7ffff7fe2700 (LWP 10145)): + #5 Frame 0xcb5380, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 16, in _wait () + time.sleep(0.01) + #8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=, _stdout_buffer=, _stderr_buffer=, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={: , : , : , : , `_, +you can break at C labels, such as those generated for computed gotos. +If you are debugging an interpreter compiled with computed goto support +(generally true, certainly when using GCC), each instruction will be +prefaced with a label named ``TARGET_``, e.g., +``TARGET_LOAD_CONST``. You can then set a breakpoint with a command +like:: + + (gdb) break ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST + +Add commands, save to a file, then reload in future sessions without +worrying that the starting line number of individual instructions +change over time. \ No newline at end of file diff --git a/Doc/howto/index.rst b/Doc/howto/index.rst index bb507953582639..8b334555ab6463 100644 --- a/Doc/howto/index.rst +++ b/Doc/howto/index.rst @@ -16,6 +16,7 @@ Currently, the HOWTOs are: cporting.rst curses.rst descriptor.rst + gdb_helpers.rst enum.rst functional.rst logging.rst From bb74f8c7ce162e29d321927796c89479f66a89d3 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Mon, 19 Feb 2024 17:49:37 +0100 Subject: [PATCH 2/7] Update info; remove stuff specific to CPython internals --- Doc/howto/gdb_helpers.rst | 240 ++++++++++++++++++-------------------- 1 file changed, 116 insertions(+), 124 deletions(-) diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst index b02992f08a8181..b26fe2b905b851 100644 --- a/Doc/howto/gdb_helpers.rst +++ b/Doc/howto/gdb_helpers.rst @@ -7,57 +7,82 @@ GDB support .. highlight:: none If you experience low-level problems such as crashes or deadlocks -(e.g. when tinkering with parts of CPython which are written in C), -it can be convenient to use a low-level debugger such as gdb in -order to diagnose and fix the issue. By default, however, gdb (or any -of its front-ends) doesn't know about high-level information specific to the -CPython interpreter, such as which Python function is currently executing, -or what type or value has a given Python object represented by a standard -``PyObject *`` pointer. We hereafter present two ways to overcome this -limitation. +(e.g. when debugging extensions or tinkering with parts of CPython +which are written in C), it can be convenient to use a low-level +debugger such as GDB in order to diagnose and fix the issue. +By default, however, GDB (or any of its front-ends) doesn't know about +high-level information specific to the CPython interpreter, such as which +Python function is currently executing, or what type or value has a given +Python object represented by a standard ``PyObject *`` pointer. +We hereafter present two ways to overcome this limitation. + Prerequisites =============== -You need to have gdb on your system and Python debugging extensions. Extensions -package includes debugging symbols and adds Python-specific commands into gdb. -On a modern Linux system, you can easily install these with: +You need to have GDB 7 or later, Python debugging information, and the +``python-gdb.py`` extension. +The extension is build with Python, but might be distributed separately or +not at all. Below, we include tips for a few common systems as examples. +Note that even if they match your system, they might be outdated. -- Fedora: +For earlier versions of GDB, see ``Misc/gdbinit`` in the sources of Python 3.11 +or earlier. - .. code-block:: shell - sudo yum install gdb python-debuginfo +Python built from source +------------------------ -- Ubuntu: +When you build CPython from source, debugging information should be available, +and the build should add a ``python-gdb.py`` file to the root directory of +your repository. - .. code-block:: shell +To activate support, you must add the directory containing ``python-gdb.py`` +to GDB's "auto-load-safe-path". +If you haven't done this, recent versions of GDB will print out a warning +with instructions on how to do this. - sudo apt-get install gdb python3.10-dbg +.. note:: -- Centos*: + If you do not seee instructionns for your version of GDB, put this in your + configuration file (``~/.gdbinit`` or ``~/.config/gdb/gdbinit``): - .. code-block:: shell + add-auto-load-safe-path /path/to/cpython - sudo yum install yum-utils - sudo debuginfo-install glibc - sudo yum install gdb python-debuginfo + You can also add multiple paths, separated by ``:``. -GDB 7 and later -=============== -In gdb 7, support for `extending gdb with Python -`_ was -added. When CPython is built you will notice a ``python-gdb.py`` file in the -root directory of your checkout. Read the module docstring for details on how -to use the file to enhance gdb for easier debugging of a CPython process. +Linux distros +------------- -To activate support, you must add the directory containing ``python-gdb.py`` -to GDB's "auto-load-safe-path". Put this in your ``~/.gdbinit`` file:: +Most Linux systems provide debug information for the system Python +in a package called ``python-debuginfo``, ``python-dbg`` or similar. +For example: + +- Fedora:: + + .. code-block:: shell + + sudo dnf install gdb + sudo dnf debuginfo-install python3 + +- Ubuntu: - add-auto-load-safe-path /path/to/checkout + .. code-block:: shell -You can also add multiple paths, separated by ``:``. + sudo apt install gdb python3-dbg + +On several recent Linux systems, GDB can download debugging symbols +automatically using *debuginfod*. +However, this will not install the ``python-gdb.py`` extension; +you generally do need to install the debuginfo package separately. + + +Using the ``python-gdb`` extension +================================== + +Pretty-printers +--------------- This is what a backtrace looks like (truncated) when this extension is enabled:: @@ -104,9 +129,8 @@ cast the value to a pointer of the appropriate type. For example:: {me_hash = 0, me_key = 0x0, me_value = 0x0}, { me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}} -The pretty-printers try to closely match the ``repr()`` implementation of the -underlying implementation of Python, and thus vary somewhat between Python 2 -and Python 3. +Note that the pretty-printers do not actually call ``repr()``. +They merely try to match it closely, and they only work on basic core types. An area that can be confusing is that the custom printer for some types look a lot like gdb's built-in printer for standard types. For example, the @@ -165,22 +189,13 @@ the extension adds a number of commands to gdb: 910 except KeyboardInterrupt: 911 # properly quit on a keyboard interrupt... - Use ``py-list START`` to list at a different line number within the python + Use ``py-list START`` to list at a different line number within the Python source, and ``py-list START,END`` to list a specific range of lines within - the python source. + the Python source. ``py-up`` and ``py-down`` ------------------------- - .. note:: - - Versions older than Python 3.12 have a 1 to 1 mapping between the C stack - and the Python stack. This means than using ``py-up`` and ``py-down`` will - move the C stack exactly one frame up or down. This is not the case for - older versions of Python as the same C stack frame can be used for multiple - Python stack frames. This means that when using ``py-up`` and ``py-down`` - there may be multiple Python frames that will be moved at the same time. - The ``py-up`` and ``py-down`` commands are analogous to gdb's regular ``up`` and ``down`` commands, but try to move at the level of CPython frames, rather @@ -188,9 +203,9 @@ the extension adds a number of commands to gdb: gdb is not always able to read the relevant frame information, depending on the optimization level with which CPython was compiled. Internally, the - commands look for C frames that are executing ``PyEval_EvalFrameEx`` (which - implements the core bytecode interpreter loop within CPython) and look up - the value of the related ``PyFrameObject *``. + commands look for C frames that are executing the default frame evaluation + function (that is, the core bytecode interpreter loop within CPython) and + look up the value of the related ``PyFrameObject *``. They emit the frame number (at the C level) within the thread. @@ -207,7 +222,13 @@ the extension adds a number of commands to gdb: (gdb) py-up Unable to find an older python frame - so we're at the top of the python stack. Going back down:: + so we're at the top of the Python stack. + + The frame numbers correspond to those displayed by gdb's standard + ``backtrace`` command. The skipped frames correspond to C functions. + which were not found to be executing Python code. + + Going back down:: (gdb) py-down #37 Frame 0x9420b04, for file /usr/lib/python2.6/site-packages/gnome_sudoku/main.py, line 906, in start_game () @@ -229,7 +250,30 @@ the extension adds a number of commands to gdb: (gdb) py-down Unable to find a newer python frame - and we're at the bottom of the python stack. + and we're at the bottom of the Python stack. + + Note that in Python 3.12 and newer, the same C stack frame can be used for + multiple Python stack frames. This means that ``py-up`` and ``py-down`` + may move multiple Python frames at once. For example:: + + (gdb) py-up + #6 Frame 0x7ffff7fb62b0, for file /tmp/rec.py, line 5, in recursive_function (n=0) + time.sleep(5) + #6 Frame 0x7ffff7fb6240, for file /tmp/rec.py, line 7, in recursive_function (n=1) + recursive_function(n-1) + #6 Frame 0x7ffff7fb61d0, for file /tmp/rec.py, line 7, in recursive_function (n=2) + recursive_function(n-1) + #6 Frame 0x7ffff7fb6160, for file /tmp/rec.py, line 7, in recursive_function (n=3) + recursive_function(n-1) + #6 Frame 0x7ffff7fb60f0, for file /tmp/rec.py, line 7, in recursive_function (n=4) + recursive_function(n-1) + #6 Frame 0x7ffff7fb6080, for file /tmp/rec.py, line 7, in recursive_function (n=5) + recursive_function(n-1) + #6 Frame 0x7ffff7fb6020, for file /tmp/rec.py, line 9, in () + recursive_function(5) + (gdb) py-up + Unable to find an older python frame + ``py-bt`` --------- @@ -258,6 +302,7 @@ the extension adds a number of commands to gdb: ``py-print`` ------------ + The ``py-print`` command looks up a Python name and tries to print it. It looks in locals within the current thread, then globals, then finally builtins:: @@ -283,6 +328,25 @@ the extension adds a number of commands to gdb: main_page=0) at remote 0x98fa6e4> d = + If the current C frame corrresponds to multiple Python frames, locals from + all of them will be shown:: + + (gdb) py-locals + Locals for recursive_function + n = 0 + Locals for recursive_function + n = 1 + Locals for recursive_function + n = 2 + Locals for recursive_function + n = 3 + Locals for recursive_function + n = 4 + Locals for recursive_function + n = 5 + Locals for + + Other commands -------------- @@ -344,75 +408,3 @@ thread is doing at the Python level:: time.sleep(0.01) #8 Frame 0x7fffd00024a0, for file /home/david/coding/python-svn/Lib/test/lock_tests.py, line 378, in _check_notify (self=, skipped=[], _mirrorOutput=False, testsRun=39, buffer=False, _original_stderr=, _stdout_buffer=, _stderr_buffer=, _moduleSetUpFailed=False, expectedFailures=[], errors=[], _previousTestClass=, unexpectedSuccesses=[], failures=[], shouldStop=False, failfast=False) at remote 0xc185a0>, _threads=(0,), _cleanups=[], _type_equality_funcs={: , : , : , : , `_, -you can break at C labels, such as those generated for computed gotos. -If you are debugging an interpreter compiled with computed goto support -(generally true, certainly when using GCC), each instruction will be -prefaced with a label named ``TARGET_``, e.g., -``TARGET_LOAD_CONST``. You can then set a breakpoint with a command -like:: - - (gdb) break ceval.c:_PyEval_EvalFrameDefault:TARGET_LOAD_CONST - -Add commands, save to a file, then reload in future sessions without -worrying that the starting line number of individual instructions -change over time. \ No newline at end of file From 4639810b9b0b348abd20b1e60c88742296be8178 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 20 Feb 2024 13:17:46 +0100 Subject: [PATCH 3/7] Another editing pass --- Doc/howto/gdb_helpers.rst | 98 ++++++++++++++++++++++++--------------- 1 file changed, 61 insertions(+), 37 deletions(-) diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst index b26fe2b905b851..e042bcbde5b01a 100644 --- a/Doc/howto/gdb_helpers.rst +++ b/Doc/howto/gdb_helpers.rst @@ -16,18 +16,24 @@ Python function is currently executing, or what type or value has a given Python object represented by a standard ``PyObject *`` pointer. We hereafter present two ways to overcome this limitation. +This guide assumes you are familiar with GDB, and focuses on Python-specific +details. + Prerequisites -=============== +============= -You need to have GDB 7 or later, Python debugging information, and the -``python-gdb.py`` extension. -The extension is build with Python, but might be distributed separately or -not at all. Below, we include tips for a few common systems as examples. -Note that even if they match your system, they might be outdated. +You need to have: -For earlier versions of GDB, see ``Misc/gdbinit`` in the sources of Python 3.11 -or earlier. +- GDB 7 or later. (For earlier versions of GDB, see ``Misc/gdbinit`` in the + sources of Python 3.11 or earlier.) +- GDB-compatible debugging information for Python and any extension you are + debugging. +- The ``python-gdb.py`` extension. + +The extension is built with Python, but might be distributed separately or +not at all. Below, we include tips for a few common systems as examples. +Note that even if the instructions match your system, they might be outdated. Python built from source @@ -45,7 +51,7 @@ with instructions on how to do this. .. note:: If you do not seee instructionns for your version of GDB, put this in your - configuration file (``~/.gdbinit`` or ``~/.config/gdb/gdbinit``): + configuration file (``~/.gdbinit`` or ``~/.config/gdb/gdbinit``):: add-auto-load-safe-path /path/to/cpython @@ -59,7 +65,7 @@ Most Linux systems provide debug information for the system Python in a package called ``python-debuginfo``, ``python-dbg`` or similar. For example: -- Fedora:: +- Fedora: .. code-block:: shell @@ -75,7 +81,22 @@ For example: On several recent Linux systems, GDB can download debugging symbols automatically using *debuginfod*. However, this will not install the ``python-gdb.py`` extension; -you generally do need to install the debuginfo package separately. +you generally do need to install the debug info package separately. + + +Debug build and Development mode +================================ + +For easier debugging, you might want to: + +- Use a :ref:`debug build ` of Python. (When building from source, + use ``configure --with-pydebug``. On Linux distros, install and run a package + like ``python-debug`` or ``python-dbg``, if available.) +- Use the runtime :ref:`development mode ` (``-X dev``). + +Both enable extra assertions and disable some optimizations. +Sometimes this hides the bug you are trying to find, but in most cases they +make the process easier. Using the ``python-gdb`` extension @@ -84,7 +105,7 @@ Using the ``python-gdb`` extension Pretty-printers --------------- -This is what a backtrace looks like (truncated) when this extension is +This is what a GDB backtrace looks like (truncated) when this extension is enabled:: #0 0x000000000041a6b1 in PyObject_Malloc (nbytes=Cannot access memory at address 0x7fffff7fefe8 @@ -103,8 +124,8 @@ enabled:: {'Yuck': , '__builtins__': , '__file__': 'Lib/test/crashers/nasty_eq_vs_dict.py', '__package__': None, 'y': , 'dict': {0: 0, 1: 1, 2: 2, 3: 3}, '__cached__': None, '__name__': '__main__', 'z': , '__doc__': None}, key= 0x5c2b8d "__lltrace__") at Objects/dictobject.c:2171 -(Notice how the dictionary argument to ``PyDict_GetItemString`` is displayed -as its ``repr()``, rather than an opaque ``PyObject *`` pointer.) +Notice how the dictionary argument to ``PyDict_GetItemString`` is displayed +as its ``repr()``, rather than an opaque ``PyObject *`` pointer. The extension works by supplying a custom printing routine for values of type ``PyObject *``. If you need to access lower-level details of an object, then @@ -130,12 +151,13 @@ cast the value to a pointer of the appropriate type. For example:: me_hash = 6614918939584953775, me_key = '__package__', me_value = None}}} Note that the pretty-printers do not actually call ``repr()``. -They merely try to match it closely, and they only work on basic core types. +For basic types, they try to match its result closely. An area that can be confusing is that the custom printer for some types look a -lot like gdb's built-in printer for standard types. For example, the -pretty-printer for a Python 3 ``int`` gives a ``repr()`` that is not -distinguishable from a printing of a regular machine-level integer:: +lot like GDB's built-in printer for standard types. For example, the +pretty-printer for a Python ``int`` (:c:expr:`PyLongObject *`) +gives a representation that is not distinguishable from one of a +regular machine-level integer:: (gdb) p some_machine_integer $3 = 42 @@ -143,6 +165,8 @@ distinguishable from a printing of a regular machine-level integer:: (gdb) p some_python_integer $4 = 42 +The internal structure can be revealed with a cast to :c:expr:`PyLongObject *`: + (gdb) p *(PyLongObject*)some_python_integer $5 = {ob_base = {ob_base = {ob_refcnt = 8, ob_type = 0x3dad39f5e0}, ob_size = 1}, ob_digit = {42}} @@ -160,20 +184,18 @@ values uses double-quotes and contains a hexadecimal address:: (gdb) p ptr_to_char_star $7 = 0x6d72c0 "hello world" -Here's how to see the implementation details of a ``str`` instance (for Python -3, where a ``str`` is a ``PyUnicodeObject *``):: +Again, the implementation details can be revealed with a cast to +:c:expr:`PyUnicodeObject *`:: (gdb) p *(PyUnicodeObject*)$6 $8 = {ob_base = {ob_refcnt = 33, ob_type = 0x3dad3a95a0}, length = 12, str = 0x7ffff2128500, hash = 7065186196740147912, state = 1, defenc = 0x0} -As well as adding pretty-printing support for ``PyObject *``, -the extension adds a number of commands to gdb: - ``py-list`` ----------- - List the Python source code (if any) for the current frame in the selected + The extension adds a ``py-list`` command, which + lists the Python source code (if any) for the current frame in the selected thread. The current line is marked with a ">":: (gdb) py-list @@ -196,12 +218,11 @@ the extension adds a number of commands to gdb: ``py-up`` and ``py-down`` ------------------------- - - The ``py-up`` and ``py-down`` commands are analogous to gdb's regular ``up`` + The ``py-up`` and ``py-down`` commands are analogous to GDB's regular ``up`` and ``down`` commands, but try to move at the level of CPython frames, rather than C frames. - gdb is not always able to read the relevant frame information, depending on + GDB is not always able to read the relevant frame information, depending on the optimization level with which CPython was compiled. Internally, the commands look for C frames that are executing the default frame evaluation function (that is, the core bytecode interpreter loop within CPython) and @@ -224,9 +245,9 @@ the extension adds a number of commands to gdb: so we're at the top of the Python stack. - The frame numbers correspond to those displayed by gdb's standard - ``backtrace`` command. The skipped frames correspond to C functions. - which were not found to be executing Python code. + The frame numbers correspond to those displayed by GDB's standard + ``backtrace`` command. + The command skips C frames which are not executing Python code. Going back down:: @@ -297,7 +318,7 @@ the extension adds a number of commands to gdb: #40 Frame 0x948e82c, for file /usr/lib/python2.6/site-packages/gnome_sudoku/gnome_sudoku.py, line 22, in start_game (main=) main.start_game() - The frame numbers correspond to those displayed by gdb's standard + The frame numbers correspond to those displayed by GDB's standard ``backtrace`` command. ``py-print`` @@ -317,6 +338,9 @@ the extension adds a number of commands to gdb: (gdb) py-print scarlet_pimpernel 'scarlet_pimpernel' not found + If the current C frame corrresponds to multiple Python frames, ``py-print`` + only considers the first one. + ``py-locals`` ------------- @@ -347,12 +371,12 @@ the extension adds a number of commands to gdb: Locals for -Other commands --------------- +Use with GDB commands +=====================· -You can of course use other gdb commands. For example, the ``frame`` command -takes you directly to a particular frame within the selected thread. -We can use it to go a specific frame shown by ``py-bt`` like this:: +The extension comands complement GDB's built-in commands. +For example, you can use a frame numbers shown by ``py-bt`` with the ``frame`` +command to go a specific frame within the selected thread, like this:: (gdb) py-bt (output snipped) @@ -378,7 +402,7 @@ process, and you can use the ``thread`` command to select a different one:: * 1 Thread 0x7ffff7fe2700 (LWP 10145) 0x00000038e46d73e3 in select () at ../sysdeps/unix/syscall-template.S:82 You can use ``thread apply all COMMAND`` or (``t a a COMMAND`` for short) to run -a command on all threads. You can use this with ``py-bt`` to see what every +a command on all threads. With ``py-bt``, this lets you see what every thread is doing at the Python level:: (gdb) t a a py-bt From 8acd7a39ad994a20b5c20f061b112ce652523a34 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 20 Feb 2024 13:50:47 +0100 Subject: [PATCH 4/7] Apply suggestions from code review Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- Doc/howto/gdb_helpers.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst index e042bcbde5b01a..dd785e9718061f 100644 --- a/Doc/howto/gdb_helpers.rst +++ b/Doc/howto/gdb_helpers.rst @@ -338,7 +338,7 @@ Again, the implementation details can be revealed with a cast to (gdb) py-print scarlet_pimpernel 'scarlet_pimpernel' not found - If the current C frame corrresponds to multiple Python frames, ``py-print`` + If the current C frame corresponds to multiple Python frames, ``py-print`` only considers the first one. ``py-locals`` @@ -352,7 +352,7 @@ Again, the implementation details can be revealed with a cast to main_page=0) at remote 0x98fa6e4> d = - If the current C frame corrresponds to multiple Python frames, locals from + If the current C frame corresponds to multiple Python frames, locals from all of them will be shown:: (gdb) py-locals @@ -374,7 +374,7 @@ Again, the implementation details can be revealed with a cast to Use with GDB commands =====================· -The extension comands complement GDB's built-in commands. +The extension commands complement GDB's built-in commands. For example, you can use a frame numbers shown by ``py-bt`` with the ``frame`` command to go a specific frame within the selected thread, like this:: From deb10f1a9f6ac57b79fc7fe096173299c7a735ec Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 21 Feb 2024 13:22:37 +0100 Subject: [PATCH 5/7] Apply suggestions from code review Co-authored-by: Ezio Melotti --- Doc/howto/gdb_helpers.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst index dd785e9718061f..9aab0c7bb37b91 100644 --- a/Doc/howto/gdb_helpers.rst +++ b/Doc/howto/gdb_helpers.rst @@ -50,7 +50,7 @@ with instructions on how to do this. .. note:: - If you do not seee instructionns for your version of GDB, put this in your + If you do not see instructions for your version of GDB, put this in your configuration file (``~/.gdbinit`` or ``~/.config/gdb/gdbinit``):: add-auto-load-safe-path /path/to/cpython @@ -372,7 +372,7 @@ Again, the implementation details can be revealed with a cast to Use with GDB commands -=====================· +===================== The extension commands complement GDB's built-in commands. For example, you can use a frame numbers shown by ``py-bt`` with the ``frame`` From c0fc1ac61d2336a551f4a4c59825bd315b7644fb Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 21 Feb 2024 13:26:05 +0100 Subject: [PATCH 6/7] Rewordings, heading fixes --- Doc/howto/gdb_helpers.rst | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst index 9aab0c7bb37b91..a0b5cc685c62f6 100644 --- a/Doc/howto/gdb_helpers.rst +++ b/Doc/howto/gdb_helpers.rst @@ -1,23 +1,26 @@ .. _gdb: -=========== -GDB support -=========== +========================================================= +Debugging C API extensions and CPython Internals with GDB +========================================================= .. highlight:: none -If you experience low-level problems such as crashes or deadlocks -(e.g. when debugging extensions or tinkering with parts of CPython -which are written in C), it can be convenient to use a low-level +If you experience low-level problems such as crashes or deadlocks, +especially when debugging extensions or tinkering with parts of CPython +which are written in C, it can be convenient to use a low-level debugger such as GDB in order to diagnose and fix the issue. By default, however, GDB (or any of its front-ends) doesn't know about high-level information specific to the CPython interpreter, such as which Python function is currently executing, or what type or value has a given Python object represented by a standard ``PyObject *`` pointer. -We hereafter present two ways to overcome this limitation. +This guide introduces an extenstion that provides this information to GDB. -This guide assumes you are familiar with GDB, and focuses on Python-specific +The guide assumes you are familiar with GDB, and focuses on Python-specific details. +It is a consolidation of guidance from the CPython +`devguide `_ and the +`Python wiki `_ Prerequisites @@ -36,8 +39,8 @@ not at all. Below, we include tips for a few common systems as examples. Note that even if the instructions match your system, they might be outdated. -Python built from source ------------------------- +Setup with Python built from source +----------------------------------- When you build CPython from source, debugging information should be available, and the build should add a ``python-gdb.py`` file to the root directory of @@ -58,8 +61,8 @@ with instructions on how to do this. You can also add multiple paths, separated by ``:``. -Linux distros -------------- +Setup for Python from a Linux distro +------------------------------------ Most Linux systems provide debug information for the system Python in a package called ``python-debuginfo``, ``python-dbg`` or similar. @@ -84,8 +87,8 @@ However, this will not install the ``python-gdb.py`` extension; you generally do need to install the debug info package separately. -Debug build and Development mode -================================ +Using the Debug build and Development mode +========================================== For easier debugging, you might want to: @@ -102,6 +105,9 @@ make the process easier. Using the ``python-gdb`` extension ================================== +When the extension is loaded, it provides two main features: +pretty printers for Python values, and additional commands. + Pretty-printers --------------- From e646760b73487e9bfc4fd1c8a8df061a63df1b8a Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Fri, 23 Feb 2024 10:27:32 +0100 Subject: [PATCH 7/7] Rewrite the introduction based on Carol's comment Co-Authored-By: Carol Willing --- Doc/howto/gdb_helpers.rst | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/Doc/howto/gdb_helpers.rst b/Doc/howto/gdb_helpers.rst index a0b5cc685c62f6..53bbf7ddaa2ab9 100644 --- a/Doc/howto/gdb_helpers.rst +++ b/Doc/howto/gdb_helpers.rst @@ -6,21 +6,30 @@ Debugging C API extensions and CPython Internals with GDB .. highlight:: none -If you experience low-level problems such as crashes or deadlocks, -especially when debugging extensions or tinkering with parts of CPython -which are written in C, it can be convenient to use a low-level -debugger such as GDB in order to diagnose and fix the issue. -By default, however, GDB (or any of its front-ends) doesn't know about -high-level information specific to the CPython interpreter, such as which -Python function is currently executing, or what type or value has a given -Python object represented by a standard ``PyObject *`` pointer. -This guide introduces an extenstion that provides this information to GDB. - -The guide assumes you are familiar with GDB, and focuses on Python-specific -details. -It is a consolidation of guidance from the CPython -`devguide `_ and the -`Python wiki `_ +This document explains how the Python GDB extension, ``python-gdb.py``, can +be used with the GDB debugger to debug CPython extensions and the +CPython interpreter itself. + +When debugging low-level problems such as crashes or deadlocks, a low-level +debugger, such as GDB, is useful to diagnose and correct the issue. +By default, GDB (or any of its front-ends) doesn't support high-level +information specific to the CPython interpreter. + +The ``python-gdb.py`` extension adds CPython interpreter information to GDB. +The extension helps introspect the stack of currently executing Python functions. +Given a Python object represented by a :c:expr:`PyObject *` pointer, +the extension surfaces the type and value of the object. + +Developers who are working on CPython extensions or tinkering with parts +of CPython that are written in C can use this document to learn how to use the +``python-gdb.py`` extension with GDB. + +.. note:: + + This document assumes that you are familiar with the basics of GDB and the + CPython C API. It consolidates guidance from the + `devguide `_ and the + `Python wiki `_. Prerequisites