Special characters in the changelog break installation on Python 3.X

[jezdez]

See http://hudson.testrun.org/view/pytest/job/tox/TOXENV=py31,label=linux/412/console

This is probably related to an ö in the changelog.

[hpk42]

Maybe it's just on machines where the default locale is ascii.

[kumar303]

default locale is ascii for any EN-US ubuntu install afaik :(

[carljm]

Since I wasted 45 minutes on this, may as well record what I found for posterity, in case anyone else has a better solution.

This only happens on systems where the locale (and thus Python's default encoding) are ASCII. (To reproduce on Linux, export LANG=C).

The initial error is on reading from the file containing non-ASCII (docs/news.txt in our case). This part is easily fixable by switching from open(...) to codecs.open(..., encoding="UTF-8").

But then you get an encoding error on the output side. The usual way to fix this would be to explicitly encode the string to bytes in the desired encoding before output, e.g. long_description = long_description.encode("UTF-8"). We can't do that for long-description in setup.py, though, because distutils rfc822-escapes it internally, and that requires string-only methods (split). So long-description must be a string (unicode), it can't be encoded bytes.

Attempts to change the default encoding of sys.stdout are pointless, as in the installation case long-description isn't written to stdout, it's written to a file.

So I can't see any other fix here than to mangle the spelling of people's names in our changelog. Which sucks.

"Fixed" in 25ab511