subprocess PIPE error

[SiggiGue]

Using PySoundFile with a subprocess PIPE file-like-object leads to an error:
Example raising the error:

sp = subprocess.Popen(
    ['ffmpeg', '-i', 'test.mp3', '-f', 'wav', '-'],
    stdout=subprocess.PIPE)

sp.stdout.read(8)
Out[69]: b'RIFF\xff\xff\xff\xff'

sp.stdout.seek(0)
Out[70]: 0

dir(sp.stdout)
Out[74]: 
['__class__',
 '__del__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__enter__',
 '__eq__',
 '__exit__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getstate__',
 '__gt__',
 '__hash__',
 '__init__',
 '__iter__',
 '__le__',
 '__lt__',
 '__ne__',
 '__new__',
 '__next__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '_checkClosed',
 '_checkReadable',
 '_checkSeekable',
 '_checkWritable',
 '_dealloc_warn',
 '_finalizing',
 'close',
 'closed',
 'detach',
 'fileno',
 'flush',
 'isatty',
 'mode',
 'name',
 'peek',
 'raw',
 'read',
 'read1',
 'readable',
 'readinto',
 'readline',
 'readlines',
 'seek',
 'seekable',
 'tell',
 'truncate',
 'writable',
 'write',
 'writelines']

# Try reading stdout with soundfile
sf = soundfile.SoundFile(sp.stdout)
From cffi callback <function SoundFile._init_virtual_io.<locals>.vio_get_filelen at 0x0000000005DF3730>:
Traceback (most recent call last):
  File "C:\Miniconda3\envs\anaconda\lib\site-packages\soundfile.py", line 1027, in vio_get_filelen
    file.seek(curr, SEEK_SET)
OSError: [Errno 22] Invalid argument
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-73-38dc580aa6b1> in <module>()
----> 1 sf = soundfile.SoundFile(sp.stdout)

C:\Miniconda3\envs\anaconda\lib\site-packages\soundfile.py in __init__(self, file, mode, samplerate, channels, subtype, endian, format, closefd)
    654         self._info = _create_info_struct(file, mode, samplerate, channels,
    655                                          format, subtype, endian)
--> 656         self._file = self._open(file, mode_int, closefd)
    657         if set(mode).issuperset('r+') and self.seekable():
    658             # Move write position to 0 (like in Python file objects)

C:\Miniconda3\envs\anaconda\lib\site-packages\soundfile.py in _open(self, file, mode_int, closefd)
   1008             raise TypeError("Invalid file: {0!r}".format(self.name))
   1009         _error_check(_snd.sf_error(file_ptr),
-> 1010                      "Error opening {0!r}: ".format(self.name))
   1011         if mode_int == _snd.SFM_WRITE:
   1012             # Due to a bug in libsndfile version <= 1.0.25, frames != 0

C:\Miniconda3\envs\anaconda\lib\site-packages\soundfile.py in _error_check(err, prefix)
   1167     if err != 0:
   1168         err_str = _snd.sf_error_number(err)
-> 1169         raise RuntimeError(prefix + _ffi.string(err_str).decode())
   1170 
   1171 

RuntimeError: Error opening <_io.BufferedReader name=7>: File contains data in an unknown format.

I'd be very happy getting some advice to fix it without reading the whole stdout in cache befor loading a SoundFile instance.

Thanks!

[bastibe]

The problem is that the pipe is not seekable, and soundfile expects something seekable.

We guard against this quite thoroughly by checking each object whether it has a seek method, and refrain from trying to call it if there is none. However, the pipe object does have a seek method, but throws an error if you try to use it. This is unfortunate.

This means that any use of seek and tell should probably be regarded as unsafe, and should be wrapped in a try/catch. Interestingly enough, the same goes for write. Pipes do have a write method, but throw an error when the write method is actually used. This will require some re-architecting.

[SiggiGue]

Ok, i tried too seek and it worked without errors. But the seek method tricket me by returning the buffer size...

sp.stdout.seek(0, 2)
Out[]: 4686

Ok what would be a good way fixing it? Does libsndfile need to know the exact file length?

Maybe it turns out not to bee solvable withoud reading all data to cache if i want to have a normal seekable soundfile object?

[bastibe]

seek(0, 2) is expected to return the buffer size, as you are seeking from SEEK_END.

Interestingly though, my sp.stdout.seek simply throws an error. What operating system are you running? This should be solvable within PySoundFile.

[mgeier]

As far as work-arounds are concerned, you can try to use the file descriptor corresponding to your "pipe" file object:

sf = soundfile.SoundFile(sp.stdout.fileno())

But note that this will not work on Windows if you use our packaged DLL (nor with the DLL from the official libsndfile installer), you'd have to use a DLL that's compiled with the same compiler as the Python interpreter itself (see http://www.mega-nerd.com/libsndfile/api.html#open_fd).

Another work-around would be to use a named pipe (e.g. with os.mkfifo()).
Sadly, this won't work on Windows either, because this is only supported for Unix-like systems.

[mgeier]

Another thing you can try is to call ffmpeg outside of Python:

ffmpeg -i test.mp3 -f wav - | python myscript.py

Within myscript.py, you can then use '-' as file name which tells libsndfile to use the standard input.

As a slight variation, you can pass the "file name" to your script:

ffmpeg -i test.mp3 -f wav - | python myscript.py -

I tried a similar thing once (see https://github.com/bastibe/PySoundFile/pull/68#issuecomment-69583755) and it works on Linux, but I have no idea if this works on Windows.

[SiggiGue]

Thanks for the replies!
Actually i'm running a windows system but have the demand running on linux as well. In case of running the software on a linux server.
So the .fileno() and named pipe is not a workaround.
The abstraction layer piping in the system console is not the way i want to design the software.
My temporary workaround is like following snippet:

import subprocess
from io import BytesIO
from soundfile import SoundFile


sp = subprocess.Popen(
    ['ffmpeg', '-i', 'test.mp3', '-f', 'wav', '-'],
    stdout=subprocess.PIPE)
bio = BytesIO(sp.stdout.read())
sf = SoundFile(bio)

The caveats of this method are:

• Simmilar fast to using temp-file on windows (BytesIO is quite slow).
• All data needs do be converted and read.

But it works and the harddisk is used more economical...

[mgeier]

@bastibe wrote:

This should be solvable within PySoundFile.

Is there something we can do?

If not, I think we can close this issue.

[bastibe]

I believe we can not seek if the file is not seekable. This limits us to sequential reads, but that might be a fair tradeoff in many circumstances.

[SiggiGue]

That would be great! Because my RAM is overflowing permanently with the BytesIO solution :-)
if i can help, just say me what to do...

[tgarc]

I'm seconding this issue. Basically I'm just trying to pipe audio data from sox:

import subprocess, soundfile as sf

p = subprocess.Popen('sox -n -t au -c 1 - synth 5 sine 440'.split(), stdout=subprocess.PIPE)
audiofh = sf.SoundFile(p.stdout)

# Read input incrementally
audiofh.read(100, dtype='int32')
# and so on...

The approach mentioned above about asserting whether a file is seekable by looking for a seek method seems wrong:

The problem is that the pipe is not seekable, and soundfile expects something seekable.

Why? Is that really required for reading input?

We guard against this quite thoroughly by checking each object whether it has a seek method, and refrain from trying to call it if there is none. However, the pipe object does have a seek method, but throws an error if you try to use it. This is unfortunate.

I wouldn't see it as unfortunate - this is by design! All io objects in Python implement a base interface (IOBase) that includes a seek() and a seekable() method. Why wouldn't you use seekable?

But note that this will not work on Windows if you use our packaged DLL (nor with the DLL from the official libsndfile installer), you'd have to use a DLL that's compiled with the same compiler as the Python interpreter itself

Yikes! Thanks for that important note. Indeed, it doesn't work. I wonder if using the cibuildwheel would address this since it uses the recommended python build environment when generating wheels. Just a thought.

I hope to have some time soon to commit to working on this; personally I find it useful to be able to read from a non-seekable file object.

[mgeier]

@tgarc

Thanks for the nicely reproducible code example!

I still don't think there is anything the soundfile module can do about that.

The problem is, if I'm not mistaken, that libsndfile insists on seeking when using the "virtual file I/O" mode (http://www.mega-nerd.com/libsndfile/api.html#open_virtual), which is what the soundfile module uses when you pass it a file-like object.

Libsndfile expects us to pass a pointer to some seek() function and it seems to unconditionally call this function (even when opening a RAW file). This causes an error when reading from a pipe file object (OSError: [Errno 29] Illegal seek), which I guess is what you experienced.

  typedef struct
  {    sf_vio_get_filelen  get_filelen ;
       sf_vio_seek         seek ;
       sf_vio_read         read ;
       sf_vio_write        write ;
       sf_vio_tell         tell ;
  } SF_VIRTUAL_IO ;

I even went out on a limb and tried to pass a NULL pointer for sf_vio_seek, but this resulted in an error (Error : bad pointer on SF_VIRTUAL_IO struct.).

So the problem isn't the soundfile module checking for a .seek() method or certain Python file objects having a .seek() method that throws. The problem is that libsndfile stubbornly insists on using seek() even if it wouldn't actually need to.
You should probably open an issue there?

When changing your example to use the file descriptor of the pipe, it works fine!

>>> audiofh = sf.SoundFile(p.stdout.fileno())
>>> audiofh.read(100, dtype='int32')
array([          0,   123617807,   246825651,   369214931,   490379759,
         609918308,   727434144,   842537542,   954846776,  1063989388,
...
       -1913421940, -1854127826, -1788684744, -1717309727, -1640239481,
       -1557729600, -1470053716, -1377502594, -1280383169, -1179017523], dtype=int32)

Except probably on Windows ...

Yikes!

Yikes indeed.

What you can try, however, is to persuade Christoph Gohlke to provide a wheel for the soundfile module on his page http://www.lfd.uci.edu/~gohlke/pythonlibs/.
Currently it's listed under "Other useful packages and applications not currently available on this page".
The Gohlke libraries are typically compiled with a Microsoft compiler, so if you are lucky, file descriptors work with it.

Or you could try to build a libsndfile DLL on Appveyor, this might work, too?