Reading from Stream

[upsonp]

I'm currently working on a web based interface for reading BTL files. One of the challenges this creates is that the web browser passes a "Stream" rather than a file to the server. I'd like to see if I can create a from_btl_stream(file_stream) function that will basically be the same as the from_btl(fname) function after the _read_file(fname) function has been called.

The Django web server gets an InMemoryUploadedFile, which can easily get opened as an io.StringIO stream. So I just need a hook to by-pass _read_file()

[ocefpaf]

In theory this should short-circuit should work then:

def _read_file(fname):
    """Read file contents."""
    if instance(fname, InMemoryUploadedFile):
        return StringIO(InMemoryUploadedFile)

    if not isinstance(fname, Path):
        fname = Path(fname).resolve()

    extension = fname.suffix.casefold()
    if extension in [".gzip", ".gz", ".bz2", ".zip"]:
        contents = _open_compressed(fname)
    elif extension in [".cnv", ".edf", ".txt", ".ros", ".btl"]:
        contents = fname.read_bytes()
    else:
        raise ValueError(
            f"Unrecognized file extension. Expected .cnv, .edf, .txt, .ros, or .btl got {extension}",
        )
    # Read as bytes but we need to return strings for the parsers.
    encoding = chardet.detect(contents)["encoding"]
    text = contents.decode(encoding=encoding, errors="replace")
    return StringIO(text)

Want to test it and try a PR?

[ocefpaf]

Actually thinking more about this, we should short-circuit it in the StringIO(text) to avoid adding an extra dependency. Something like:

def _read_file(fname):
    """Read file contents."""
    if instance(fname, StringIO):
        text = fname.seek(0)
        return text

    if not isinstance(fname, Path):
        fname = Path(fname).resolve()

    extension = fname.suffix.casefold()
    if extension in [".gzip", ".gz", ".bz2", ".zip"]:
        contents = _open_compressed(fname)
    elif extension in [".cnv", ".edf", ".txt", ".ros", ".btl"]:
        contents = fname.read_bytes()
    else:
        raise ValueError(
            f"Unrecognized file extension. Expected .cnv, .edf, .txt, .ros, or .btl got {extension}",
        )
    # Read as bytes but we need to return strings for the parsers.
    encoding = chardet.detect(contents)["encoding"]
    text = contents.decode(encoding=encoding, errors="replace")
    return StringIO(text)

[upsonp]

I like this solution. I do see one more place where this might cause an issue.

So the user calls from_btl(fname) where fname is already a StringIO object.

The problem is further down in from_btl when _basename(fname) is called, the file name is then added to the metadata dictionary. But StringIO can't be broken down into file name parts.

Ideally, fname should be renamed to file then fname could actually be used to mean "File Name", but I think the easiest solution would to allow the user to pass in the file name as name. It only changes two lines of code:

def from_btl(fname, name=None):

    ...

    if name is None:
        name = _basename(fname)[1]

    ...

    metadata["name"] = str(name)
    setattr(df, "_metadata", metadata)
    return df

[ocefpaf]

In that case the file name is "unknown" and adding a keyword for it increases the complexity. Maybe just another check if fname is stream and, if so, hardcode a name = "stream". What do you think?

PS: let's move this discussion to the PR.

[ocefpaf]

Done in #146.

Thanks @upsonp!

[upsonp]

Thank you @ocefpaf. I appreciate your patients.

[ocefpaf]

Thank you for the PRs! I made a new release, v.1.4.0, with your changes BTW.