Latest typeshed changes break a couple of tests in pandas-stubs

[erictraut]

Just a heads up that tomorrow's release of pyright 1.1.314 will include the latest typeshed changes. The most significant of these changes are prompted by PEP 688. These changes break a couple of tests in pandas-stubs. You should see the same behaviors with mypy once it ships the next release with an updated typeshed.

The errors are as follows:

• In test_types_to_csv (within test_frame.py), passing an instance of io.BytesIO as the first argument to df.to_csv no longer works. This is because io.BytesIO no longer satisfies the WriteBuffer[bytes] | WriteBuffer[str] types defined in the pandas stubs.
• In test_orc_buffer (within test_io.py), the first check statement fails because file_w is an instance of io.BufferedWriter, and this is no longer compatible with the WriteBuffer[bytes] type defined in the pandas stubs.

@JelleZijlstra, for your visibility. I'm guessing that these showed up in the mypy_primer output when the typeshed changes went in?

[Dr-Irv]

@twoertwein This might affect internal type checks for the pandas source.

[twoertwein]

Thanks @erictraut! As far as I follow, the issue is that str and bytes (in the read/write methods) are not compatible with Buffer (pandas/pandas-stubs uses generic protocols with str/bytes).

Is there still an option to write tailored protocols that differentiate between str/bytes except for typing.IO (which requires way too many methods)?

[JelleZijlstra]

I'll take a look at this later today; on first impression I don't understand why my change would break things here. The relevant change should be python/typeshed#10225.

[JelleZijlstra]

@erictraut I'm looking at this again now and I think there is a bug in pyright.

Consider this example, simplified from the first error you cite above:

from typing import overload, Protocol, TypeVar, Any

AnyStr_contra = TypeVar("AnyStr_contra", str, bytes, contravariant=True)


class Buffer(Protocol):
    def __bytes__(self) -> bytes:
        ...


class BytesIO:
    def write(self, __b: Buffer) -> None:
        pass


class WriteBuffer(Protocol[AnyStr_contra]):
    def write(self, __b: AnyStr_contra) -> Any:
        ...


class NDFrame:
    @overload
    def to_csv(self, path_or_buf: WriteBuffer[bytes]) -> None:
        ...

    @overload
    def to_csv(self, path_or_buf: None = ...) -> str:
        ...

    def to_csv(self, path_or_buf: Any = None) -> Any:
        ...


def test_types_to_csv() -> None:
    df = NDFrame()
    df.to_csv(BytesIO())

With pyright 1.1.314 this gives:

  /Users/jelle/py/pandas-stubs/tests/test_frame.py:36:15 - error: Argument of type "BytesIO" cannot be assigned to parameter "path_or_buf" of type "None" in function "to_csv"
    Type cannot be assigned to type "None" (reportGeneralTypeIssues)

mypy --python-version 3.11 passes, correctly as far as I can tell.

Interestingly, if I remove the second overload for to_csv and leave only the first one (that takes WriteBuffer[bytes]), pyright passes. It seems pyright has some bug with the combination of overloads and generic protocols.

[erictraut]

Thanks for the analysis and simplified rerpo case. This does look like a pyright bug. I've filed a new pyright bug report for tracking purposes.

[JelleZijlstra]

Thanks! I haven't looked at the second case you mentioned (test_orc_buffer) in as much detail, but it appears to be a similar issue, as the to_orc method is defined with two overloads similar to those for to_csv.

[erictraut]

I've addressed the issue in pyright, and it will be included in next week's release (1.1.315). It addresses both of the issues I reported in this thread. @JelleZijlstra, apologies for misattributing the problem to the recent typeshed changes.

I'll mark this issue closed since there's nothing remaining to do in the pandas-stubs project.

[Dr-Irv]

I've addressed the issue in pyright, and it will be included in next week's release (1.1.315)

Can you confirm that the bug is in pyright 1.1.314 ? If so, and someone does a PR prior to release of 1.1.315, we'll have to pin pyright to 1.1.313 until 1.1.315 is released so that the CI works.

[erictraut]

Yes, this bug is in pyright 1.1.314, which also contains the updated typeshed stubs. Pinning to 1.1.313 is a fine temporary workaround.