2-5x slowdown, long-standing Event performance regression

[xloem]

Bug report

The below example completes in 6s for me with python2.7, but takes 21s for me in python3 . As of yet, I have not found any workaround to speed it up in python3.

import os, threading
lock = threading.Lock()
event = threading.Event()
def worker():
    for ct in range(4000):
        while not event.is_set():
            event.wait(1)
        with lock:
            event.clear()
def test():
    thread = threading.Thread(target=worker)
    thread.start()
    while thread.is_alive():
        with lock:
            event.set()
test()

Your environment

• CPython versions tested on:
  • 2.7.5
  • 3.10.11
• Operating system and architecture:
  • Red Hat Enterprise Linux 7 x86_64

[itamaro]

I think this is an artifact of the main thread running a tight busy loop while the worker thread blocks on event.wait.

$ time python2 mt_repro.py
python2 mt_repro.py  5.93s user 0.60s system 106% cpu 6.133 total

$ time python3 mt_repro.py
python3 mt_repro.py  45.24s user 0.52s system 100% cpu 45.684 total

note CPU utilization at 100+% and most time in "user"

forcing the main thread to yield after setting the event and releasing the lock: (only difference is added time.sleep)

import os, threading, time

lock = threading.Lock()
event = threading.Event()
def worker():
    for ct in range(4000):
        while not event.is_set():
            event.wait(1)
        with lock:
            event.clear()

def test():
    thread = threading.Thread(target=worker)
    thread.start()
    while thread.is_alive():
        with lock:
            event.set()
        time.sleep(0.001)

test()

total times are similar (with python3 a little faster)

$ time python2 mt_repro.py
python2 mt_repro.py  0.38s user 0.21s system 10% cpu 5.621 total

$ time python3 mt_repro.py
python3 mt_repro.py  0.25s user 0.11s system 7% cpu 5.150 total

note low CPU utilization and most time spent idling.

I'm not sure why there's such a big difference in the busy loop case - it could be a result of preemption time slices being much longer in python3 (are they? I don't know)

[pochmann]

Try sys.setswitchinterval(0.001).

Default is 0.005 seconds, and 4000 × 0.005 seconds is 20 seconds.

Documentation
Announcement
Details

[itamaro]

sys.setswitchinterval

TIL!