Can't run python-libjuju async loop on background thread in py3.8, py3.9 and py3.10? due to "set_wakeup_fd() only works in main thread"

[ajkavanagh]

Description

For zaza, for various reasons, we need/have to run python-libjuju's async loop in a background thread. This is because the zaza is written as a non-blocking test framework on top of unitest. However, the async thread needs to keep running to get status updates and so to make it all work, python-libjuju runs on the background thread.

However, due to python/cpython#87173 and a recent-ish change this now blows up on py3.8, py3.9 and py3.10. This is a major problem for our CI as it runs py3.8 test runners (a limitation we do eventually need to solve).

The traceback is here:

   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/controller.py", line 108, in connect
     await self._connector.connect_controller(controller_name, **kwargs)
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connector.py", line 127, in connect_controller
     await self.connect(
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connector.py", line 86, in connect
     self._connection = await Connection.connect(**kwargs)
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connection.py", line 356, in connect
     await self._connect_with_redirect([_ep])
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connection.py", line 908, in _connect_with_redirect
     login_result = await self._connect_with_login(endpoints)
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connection.py", line 875, in _connect_with_login
     await self._connect(endpoints)
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connection.py", line 824, in _connect
     result = await task
   File "/usr/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
     return f.result()  # May raise f.exception().
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connection.py", line 813, in _try_endpoint
     return await self._open(endpoint, cacert)
   File "/home/runner/work/zaza/zaza/.tox/func-target/lib/python3.10/site-packages/juju/client/connection.py", line 430, in _open
     loop.add_signal_handler(sig, _exit_tasks)
   File "/usr/lib/python3.10/asyncio/unix_events.py", line 107, in add_signal_handler
     raise RuntimeError(str(exc))
 RuntimeError: set_wakeup_fd only works in main thread of the main interpreter
 func-target: exit 1 (0.70 seconds) /home/runner/work/zaza/zaza> functest-run-suite --keep-model --bundle first pid=11317
.pkg: _exit> python /home/runner/.local/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
 func-target: FAIL code 1 (26.05=setup[24.57]+cmd[0.78,0.70] seconds)
 evaluation failed :( (26.13 seconds)
Error: Process completed with exit code 1.

The easiest fix is probably to wrap the code in the same way as in juju/jasyncio.py:

    added = False
    try:
        loop.add_signal_handler(signal.SIGINT, abort)
        added = True
    except (ValueError, OSError, RuntimeError) as e:
        # add_signal_handler doesn't work in a thread
        if 'main thread' not in str(e):
            raise

Urgency

Annoying bug in our test suite

Python-libjuju version

=3.2.2

Juju version

3.1

Reproduce / Test

A bit tricky for a reproducer; zaza runs libjuju in a background thread here: https://github.com/openstack-charmers/zaza/blob/master/zaza/__init__.py