bpo-40260: Allow compile() to handle the module's source decoding

[barry-scott]

bpo-40260: Allow compile() to handle the module's source decoding
by passing in an fp that is opened in "rb".

This fixes problems seen on Windows where the source code is not
decoded as utf-8 but as the code page default for the user, cp1252
for example.

Add three new tests to test_modulefinder.py to excercise
the decoding of source code.

Always write the test source as bytes to avoid using
locale.getpreferredencoding() as the encoding.

test_coding_default_utf8 - designed to fail if decoded with cp1252
test_coding_explicit_utf8 - designed to fail if decoded with cp1252
test_coding_explicit_cp1252 - designed to fail if decoded with utf-8

https://bugs.python.org/issue40260

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@barry-scott

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

[merwok]

To add a NEWS item, you can install and run blurb on your machine, or use a web app (the link is Details next to the bedevere/news check line).

Please refer to the devguide for more information about the process!

[barry-scott]

Change made. Tests passed on macOS 3.8 and 3.9a5, Windows 3.8 and 3.9a5.

[zooba]

Thanks! I updated the NEWS entry to help people find this later on, so once the retest passes I'll merge.

[merwok]

I would suggest to edit the PR title (and thus the future commit message) to mention io.open_code, so that this can be found by a search later with all the other commits related to that function.

[zooba]

I was going to update the commit message when merging. Updating the issue title is probably more important for future searches - there's far more useful information on the tracker than here.

[miss-islington]

Thanks @barry-scott for the PR, and @zooba for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

[bedevere-bot]

GH-19522 is a backport of this pull request to the 3.8 branch.