Exception when using capturer inside pytest with capsys

[clara-avnet]

Using capturer inside a pytest test that uses the pytest fixture capsys leads to the following error in the call to CaptureOutput():

=================================== FAILURES ===================================
________________________________ test_capturer _________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f07bed1f4a8>

    def test_capturer(capsys):
>       with capturemod.CaptureOutput() as capturer:

test_file.py:5: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/scherercl/msc/0000/libMscBoostPython.git/venv/lib/python3.6/site-packages/capturer/__init__.py:238: in __init__
    self.stdout_stream = self.initialize_stream(sys.stdout, STDOUT_FD)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <capturer.CaptureOutput object at 0x7f07bed1f080>
file_obj = <_io.TextIOWrapper encoding='UTF-8'>, expected_fd = 1

    def initialize_stream(self, file_obj, expected_fd):
        """
        Initialize one or more :class:`Stream` objects to capture a standard stream.
    
        :param file_obj: A file-like object with a ``fileno()`` method.
        :param expected_fd: The expected file descriptor of the file-like object.
        :returns: The :class:`Stream` connected to the file descriptor of the
                  file-like object.
    
        By default this method just initializes a :class:`Stream` object
        connected to the given file-like object and its underlying file
        descriptor (a simple one-liner).
    
        If however the file descriptor of the file-like object doesn't have the
        expected value (``expected_fd``) two :class:`Stream` objects will be
        created instead: One of the stream objects will be connected to the
        file descriptor of the file-like object and the other stream object
        will be connected to the file descriptor that was expected
        (``expected_fd``).
    
        This approach is intended to make sure that "nested" output capturing
        works as expected: Output from the current Python process is captured
        from the file descriptor of the file-like object while output from
        subprocesses is captured from the file descriptor given by
        ``expected_fd`` (because the operating system defines special semantics
        for the file descriptors with the numbers one and two that we can't
        just ignore).
    
        For more details refer to `issue 2 on GitHub
        <https://github.com/xolox/python-capturer/issues/2>`_.
        """
>       real_fd = file_obj.fileno()
E       io.UnsupportedOperation: fileno

I suspect, this has to do with nested capturing, as the function's documentation helpfully states (#2).

Reproduction

python 3.7.3
pytest 5.3.5
capturer 2.4

File test_file.py, called with pytest test_file.py:

import subprocess
from capturer import CaptureOutput

def test_capturer(capsys):
    with CaptureOutput() as capturer:
        # Generate some output from Python.
        print("Output from Python")
        # Generate output from a subprocess.
        subprocess.run(["echo", "Output from a subprocess"])
        # Get the output in each of the supported formats.
        assert capturer.get_bytes() == b'Output from Python\r\nOutput from a subprocess\r\n'
        assert capturer.get_lines() == [u'Output from Python', u'Output from a subprocess']
        assert capturer.get_text() == u'Output from Python\nOutput from a subprocess'

[xolox]

Hi @clara-avnet and thanks for the feedback. The exception you reported is indeed reproducible. This prompted me to play around a bit with the interactions between pytest's capturing behavior and capturer to see if it can be improved. The good news is that the exception can be avoided:

diff --git a/capturer/__init__.py b/capturer/__init__.py
index 407ba26..442ca86 100644
--- a/capturer/__init__.py
+++ b/capturer/__init__.py
@@ -7,6 +7,7 @@
 """Easily capture stdout/stderr of the current process and subprocesses."""
 
 # Standard library modules.
+import io
 import multiprocessing
 import os
 import pty
@@ -264,12 +265,19 @@ class CaptureOutput(MultiProcessHelper):
         For more details refer to `issue 2 on GitHub
         <https://github.com/xolox/python-capturer/issues/2>`_.
         """
-        real_fd = file_obj.fileno()
-        stream_obj = Stream(real_fd)
-        self.streams.append((expected_fd, stream_obj))
+        try:
+            real_fd = file_obj.fileno()
+            stream_obj = Stream(real_fd)
+            self.streams.append((expected_fd, stream_obj))
+        except io.UnsupportedOperation:
+            # This happens when the pytest capture fixture is being used.
+            real_fd = None
+        # Regardless of how the above went we're interested in what child
+        # processes have to report on the OS defined file descriptor.
         if real_fd != expected_fd:
             self.streams.append((expected_fd, Stream(expected_fd)))
-        return stream_obj
+        # Get the first stream available (we know there will be at least one).
+        return next(stream for fd, stream in self.streams if fd == expected_fd)
 
     def __enter__(self):
         """Automatically call :func:`start_capture()` when entering a :keyword:`with` block."""

Unfortunately this means output capturing by capturer is only done on the level of file descriptors and not inside Python, which is likely not the expected behavior (as your test nicely illustrates). This means I don't see a realistic way to make the test as you wrote it succeed, it would end up like this:

def test_pytest_capture_fixture(capsys):
    with CaptureOutput() as capturer:
        # Generate some output from Python.
        print("Output from Python")
        # Generate output from a subprocess.
        subprocess.call(["echo", "Output from a subprocess"])
        # Get the output in each of the supported formats.
        assert capturer.get_bytes() == b'Output from a subprocess\r\n'
        assert capturer.get_lines() == [u'Output from a subprocess']
        assert capturer.get_text() == u'Output from a subprocess'

However this seems like counterintuitive behavior to me, so I'm not convinced about enabling this. Maybe instead the documentation should just explicitly state not to mix capturer and the pytest output capturing fixtures? After all they serve very similar purposes. I'm open to discussion about whether there's a right choice here 🙂.

[clara-avnet]

I think your suggestion to simply warn users not mix usages of capturer and the capsys fixture, sounds like the best option.
That way, you have expected behavior for most use cases and in the one case where it doesn't work you get an exception and know that something will not work as expected. From a user perspective it would also help if the io.UnsupportedOperation was caught and another exception was raised, explaining that capturer cannot be used together with pytest's capsys fixture.