cupsd 100% CPU load

[phvogtrbi]

Before you write the report
Read the REPORTING_ISSUES.md file in the main repository and prepare data mentioned there which looks relevant to you issue.

Describe the bug
The cupsd process has 100% CPU load until killed / restarted.
The web UI until restart is unresponsive, i.e. connecting with the browser shows the page loads forever.
Last log entry on DEBUG level: "[Client 12345] Connection now encrypted." and no log lines after that until restart.

To Reproduce
Steps to reproduce the behavior:
Could find no way to reproduce. We see the problem to occur about once a week.

Expected behavior
The cupsd process does not cause a CPU load of 100% and connections to the web UI are possible.

Screenshots

top - 16:24:23 up 14 days, 11:23,  1 user,  load average: 1.13, 1.18, 1.12
Tasks: 149 total,   2 running, 146 sleeping,   0 stopped,   1 zombie
%Cpu(s): 48.6 us,  0.0 sy,  0.0 ni, 48.6 id,  0.0 wa,  0.0 hi,  0.0 si,  2.8 st
MiB Mem :   3916.1 total,    294.0 free,    405.1 used,   3217.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   3267.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                        
  53082 root      20   0  345.5m  24.5m  13.7m R 100.0   0.6 120:26.50 /usr/sbin/cupsd -l    

System Information:

• OS and its version: AmazonLinux 2023
• CUPS version 1:2.4.14-1.amzn2023.0.2

Additional context
We had no such problems with cups until about 2026-02-06.
The DNF logs show, that the upgrade of cups to 1:2.4.14-1.amzn2023.0.2 was done on 2026-01-12.
The Amazon Linux 2023 version 2023.10.20260105 release notes showed, that cups was to be updated and ALAS2023-2025-1320 shows, that CVE-2025-58436 was part of the update. This fixes something in the client connection. Also the patch file shows, that the connection handling was changed.

This is the commit referenced in the CVE: 40008d7

[ValdikSS]

Attach the debugger:
gdb --pid $(pidof cupsd)
execute bt there, and provide the output.

Make sure to press yes on debuginfod question.

[phvogtrbi]

@ValdikSS, thanks a lot for the feedback.

I will update our internal documentation for handling the incident, once it happens again with your proposal as well.

As soon as it happens again, we will:

• provide you with the information
• downgrade to the previous version to prevent further impact to our customers

As I am no C programmer, I tried to use AI to find the root cause and if the problem is really related to the patch mentioned above. The AI was partially successful in finding something, that at least increases the CPU load to 12%, but the log messages are not fitting our problem. But at least on a test system the version before the patch didn't show any higher CPU load.
And I got some scripts out of it, which will capture a lot of information and do a core dump we will use, when the problem occurs again.

[phvogtrbi]

The problem occurred again and I collected the information:

#0  0x00007fdfb269daa0 in __errno_location@plt () from /lib64/libcups.so.2
#1  0x00007fdfb26c76c5 in _httpTLSRead () from /lib64/libcups.so.2
#2  0x00007fdfb26b3570 in http_read () from /lib64/libcups.so.2
#3  0x00007fdfb26b3723 in httpGets () from /lib64/libcups.so.2
#4  0x00007fdfb26b4661 in httpReadRequest () from /lib64/libcups.so.2
#5  0x0000559a0428c9c8 in cupsdReadClient (con=0x559a099570b0) at /usr/src/debug/cups-2.4.14-1.amzn2023.0.2.x86_64/scheduler/client.c:748
#6  0x0000559a04285351 in cupsdDoSelect (timeout=<optimized out>) at /usr/src/debug/cups-2.4.14-1.amzn2023.0.2.x86_64/scheduler/select.c:484
#7  main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/cups-2.4.14-1.amzn2023.0.2.x86_64/scheduler/main.c:878

[phvogtrbi]

I analyzed the problem with an AI tool. And it could find a possible reason for the problem and provide a tool, which caused the same problem of 100% CPU load on a test system.

Here the files to reproduce (remove the .txt extension):
cups_spin_reproducer.c.txt
Makefile.txt

Here the analysis:

cupsd 100% CPU spin after TLS 1.3 client disconnect

Summary

When a TLS 1.3 client sends a KeyUpdate handshake message and then
abruptly closes the TCP connection, cupsd enters an infinite tight loop
inside http_read() consuming 100% CPU on one core. The CUPS web UI
becomes unreachable and the daemon must be restarted to recover.

Affected versions: CUPS 2.4.14 with GnuTLS (confirmed on Amazon Linux
2023, GnuTLS 3.8.3). Likely affects all CUPS versions using GnuTLS with
TLS 1.3.

Problem Description

The spin is caused by two cooperating bugs in cups/http.c:

1. Dead code in http_read() (line 4224)

if (errno == EWOULDBLOCK || errno == EAGAIN)
{
    if (http->timeout_cb && !(*http->timeout_cb)(http, http->timeout_data))
    {
        http->error = errno;
        return (-1);
    }
    else if (!http->timeout_cb && errno != EAGAIN)  // ← BUG: always FALSE here
    {
        http->error = errno;
        return (-1);
    }
}

The condition errno != EAGAIN is always FALSE inside a block that is
only entered when errno == EAGAIN. When timeout_cb is NULL (which is
always the case for cupsd server-side connections), neither branch
returns — the EAGAIN error is silently swallowed and the do { ... } while (bytes < 0) loop runs forever.

2. Unconditional continue in httpGets() (line 1247)

else if (!http->timeout_cb && errno == EAGAIN)
    continue;

Even if bug 1 is fixed (so http_read returns -1 on EAGAIN), this line
causes httpGets() to loop back without any timeout or exit condition.

How the spin is triggered

1. A TLS 1.3 client connects to the CUPS HTTPS port.
2. The client sends a partial HTTP request (no complete line), keeping
   the server in httpGets() → http_read() → gnutls_record_recv().
3. The client sends a TLS 1.3 KeyUpdate message. GnuTLS on the server
   processes it asynchronously inside gnutls_record_recv() and returns
   GNUTLS_E_AGAIN (-28). CUPS' _httpTLSRead() translates this to
   errno = EAGAIN and returns -1.
4. The client abruptly closes the TCP connection (RST).
5. On the next gnutls_record_recv(), GnuTLS reads EOF, invalidates the
   session, and returns GNUTLS_E_PREMATURE_TERMINATION (-110). However,
   _httpTLSRead() only maps GnuTLS errors to errno when !errno
   (errno is zero). Since errno is still EAGAIN from step 3, the
   mapping is skipped — errno stays EAGAIN and the raw GnuTLS
   error code is returned.
6. http_read() sees errno == EAGAIN, hits the dead-code bug, and loops.
7. All subsequent gnutls_record_recv() calls return
   GNUTLS_E_INVALID_SESSION (-10) immediately — a simple struct field
   check with no I/O. The loop runs entirely in userspace with zero
   syscalls at ~100% CPU.

Proposed fix

http_read() line 4224 — remove the impossible errno != EAGAIN condition:

-    else if (!http->timeout_cb && errno != EAGAIN)
+    else if (!http->timeout_cb)

httpGets() lines 1247–1248 — remove the unconditional continue:

     if (http->timeout_cb && (*http->timeout_cb)(http, http->timeout_data))
       continue;
-    else if (!http->timeout_cb && errno == EAGAIN)
-      continue;

     http->error = errno;
     return (NULL);

Both changes are one-liners and together ensure that an EAGAIN on a
connection without a timeout callback propagates as an error, causing
httpGets() to return NULL → httpReadRequest() returns
HTTP_STATE_ERROR → cupsdCloseClient() closes the connection cleanly.

Reproducer

A self-contained C test program is provided: cups_spin_reproducer.c.

Prerequisites

• GnuTLS development headers
• pkg-config, gcc

# AlmaLinux / RHEL / Amazon Linux
sudo dnf install gnutls-devel pkg-config gcc

# Debian / Ubuntu
sudo apt install libgnutls28-dev pkg-config gcc

Build

make

Or manually:

gcc -o cups_spin_reproducer cups_spin_reproducer.c \
    $(pkg-config --cflags --libs gnutls) -Wall -O2

Usage

./cups_spin_reproducer <hostname> [port] [attempts]

Argument	Description	Default
hostname	CUPS server hostname or IP	—
port	CUPS HTTPS port	631
attempts	Number of trigger attempts	5

Example:

./cups_spin_reproducer print.internal 631

The program will:

1. Open a TLS 1.3 connection to the CUPS server
2. Send a partial HTTP request (no terminating newline)
3. Send a TLS 1.3 KeyUpdate message
4. Abruptly close the TCP connection (RST)

Multiple attempts are made because the trigger depends on the KeyUpdate
and the RST arriving in the server's socket buffer close together.

Verifying the spin

After running the reproducer, check on the server:

top -b -n1 -p $(pidof cupsd)

If cupsd shows ~100% CPU, the spin has been triggered.

Recovery

sudo systemctl restart cups

⚠ WARNING: This reproducer will make the target CUPS server spin at
100% CPU and become unresponsive. Only use on test servers. The
server must be restarted to recover.

Environment

• OS: Amazon Linux 2023
• CUPS: 2.4.14
• TLS backend: GnuTLS 3.8.3
• Protocol: TLS 1.3

[jsmeix]

@phvogtrbi
could you please also have a look at
#827

Both issues look very similar so I guess
both might have one same root cause
or both could have similar root causes.

[phvogtrbi]

@jsmeix
It may be, but as I didn't really deep dive into the CUPS C code, I cannot say for sure. As I wrote above I am not really a C programmer and used an AI tool to do the analysis for me. At least it produced a tool, that causes 100% CPU load. Not even sure, if it is the same problem or it on accident discovered another problem.

I checked with the tool the previous version of CUPS offered on AWS AmazonLinux 2023 and it leads to the same behaviour. So a downgrade of CUPS does not help.
And as far as I could see on the blame log for the code lines, which are according to the analysis the root cause for the problem, they weren't changed in years. So I am not sure why we see in our production environment since February the problem about once a week. Maybe another issue is related to this problem and started to cause it.

It is annoying for us, because we need to restart CUPS manually right now. Most likely we need to add some tool to at least automatically restart CUPS to not impact our customers too much.

I hope my bug report helps to fix the problem and maybe even the other one as well.