BUG?: faulthandler seems to only give its output with -s

[larsoner]

I've noticed with some segfaulty tests that if I don't have the -s switch, it seems like I don't get the faulthandler output. I'm not 100% sure if this is the case because the segfaults tend to occur on CI services like Travis and Azure and be sporadic.

• pytest and operating system versions
• output of pip list from the virtual environment you are using

(base) bunker:mne-python larsoner$ pip list | grep pytest
pytest                        5.4.3
pytest-cov                    2.9.0
pytest-forked                 1.1.3
pytest-mock                   3.1.0
pytest-notebook               0.6.0
pytest-timeout                1.3.4
pytest-xdist                  1.32.0
(base) bunker:mne-python larsoner$ python --version # macOS catalina
Python 3.8.3

I can also remove pytest dependencies and try master if it's expected to be fixed there.

• minimal example if possible

pytest mne/viz/_brain/tests/test_brain.py -vvk timeviewer on a specific branch with specific data, etc. If it seems reasonable that this behavior might be occurring I might be able to replicate it on one of my machines and test code changes or fixes that might help. But the failing code involves VTK and Qt and event processing so it's going to take some time to whittle it down to a minimal example...

If it does seem reasonable that this behavior might be occuring, I actually think it might be faster for me to try to fix pytest's use of faulthandler to pump into sys.__stdout__ or sys.__stderr__ instead of sys.stdout/sys.stderr, assuming that is the problem (a possibly naive idea, but would explain this behavior).

[bluetech]

The faulthandler's plugin code does try handle this issue, but perhaps it doesn't work?

I'm curious if this happens also with --capture=sys (instead of -s; the default is --capture=fd).

[The-Compiler]

Always worked fine for me FWIW. Probably not much we can do without having a way to reproduce it.

[larsoner]

Indeed with --capture=sys I get the full traceback:

Current thread 0x000000010a86edc0 (most recent call first):
  File "/Users/larsoner/python/mne-python/mne/viz/backends/_pyvista.py", line 746 in _process_events
  File "/Users/larsoner/python/mne-python/mne/viz/backends/_pyvista.py", line 531 in show
  File "/Users/larsoner/python/mne-python/mne/viz/_brain/_brain.py", line 1135 in show
  File "/Users/larsoner/python/mne-python/mne/viz/_brain/_timeviewer.py", line 415 in __init__
  File "/Users/larsoner/python/mne-python/mne/viz/_brain/tests/test_brain.py", line 251 in test_brain_timeviewer_traces
  File "/Users/larsoner/opt/miniconda3/lib/python3.8/site-packages/_pytest/python.py", line 182 in pytest_pyfunc_call
...

I'll work on whittling this down to a minimal example.

[larsoner]

Okay, the segfault is actually easy to reproduce via a Qt abort trap. Hooray!

The tricky part was recognizing that the problem is when some other module does faulthandler.enable() (effectively within a test) -- it seems to wipe out whatever magic pytest does. On Qt 5.13.2 on macOS this snippet:

import warnings
from PyQt5.QtWidgets import QApplication, QMainWindow, QWidget


class BadWidget(QWidget):
    def resizeEvent(self, event):
        raise RuntimeError('cause an abort trap')


def test_crash():
    import faulthandler
    try:
        faulthandler.enable()
    except Exception as e:
        warnings.warn(str(e))
    app = QApplication([])
    window = QMainWindow()
    bad = BadWidget()
    window.layout().addWidget(bad)
    window.show()

causes

(base) bunker:Desktop larsoner$ pytest test/test_crash.py
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.8.3, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /Users/larsoner/Desktop
plugins: cov-2.9.0
collected 1 item                                                                                                                                               

test/test_crash.py Abort trap: 6
(base) bunker:Desktop larsoner$ pytest test/test_crash.py --capture=sys
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.8.3, pytest-5.4.3, py-1.8.1, pluggy-0.13.1
rootdir: /Users/larsoner/Desktop
plugins: cov-2.9.0
collected 1 item                                                                                                                                               

test/test_crash.py QMainWindowLayout::addItem: Please use the public QMainWindow API instead
Traceback (most recent call last):
  File "/Users/larsoner/Desktop/test/test_crash.py", line 7, in resizeEvent
    raise RuntimeError('cause an abort trap')
RuntimeError: cause an abort trap
Fatal Python error: Aborted

Current thread 0x000000010e657dc0 (most recent call first):
  File "/Users/larsoner/Desktop/test/test_crash.py", line 20 in test_crash
  File "/Users/larsoner/opt/miniconda3/lib/python3.8/site-packages/_pytest/python.py", line 182 in pytest_pyfunc_call
...

The try/except is necessary because faulthandler emits this warning when you call faulthandler.enable:

>       faulthandler.enable()
E       io.UnsupportedOperation: fileno

[nicoddemus]

The try/except is necessary

Hmm sorry, which try/except?

I can reproduce the problem on Windows too.

[larsoner]

Sorry, example updated. It was missing the faulthandler business that's necessary to make the capture behave as I reported above (forgot I had changed the example a bit).

[larsoner]

The naive workaround/fix here is just to monkey-patch faulthandler such that faulthandler.enable() is a no-op.

[The-Compiler]

I'm not sure pytest can (or should) protect against users re-enabling faulthandler.

[larsoner]

Fair enough, the sys is a sufficient workaround for me in any case. Thanks for the help!