The type of tokenize.generate_tokens() is stated as a callable returning bytes (same as tokenize.tokenize()), but actually the (only) difference between the two is that generate_tokens() works on str, not bytes. (UPDATE: The return type is the same either way -- TokenInfo contains str values.)
The type of
tokenize.generate_tokens()is stated as a callable returningbytes(same astokenize.tokenize()), but actually the (only) difference between the two is thatgenerate_tokens()works onstr, notbytes. (UPDATE: The return type is the same either way --TokenInfocontainsstrvalues.)