pip 8.0

[mattrobenolt]

https://pip.pypa.io/en/stable/news/#release-notes

pip 8.0 introduces some backwards incompatibilities, so not even entirely sure if this is a great idea.

But on the otherhand, we have no way of versioning pip independently from python in these builds.

So we may end up with someone tagging to a very specific python:2.7.11 which had pip 7.1.2, then tomorrow, gets new build with 8.0.0 and breaks their container.

Thoughts?

[mattrobenolt]

Also cc'ing @dstufft here since he may have an opinion.

[alex]

I think it's reasonable to just move to the latest version of pip.

[mattrobenolt]

I'm also not sure why the tests fail here.

/cc @tianon @yosifkit

[mattrobenolt]

8.0.2 is going to be cut soon, so we'll just wait and I'll update this PR: pypa/pip#3415 (comment)

[mattrobenolt]

Updated for 8.0.2.

[yosifkit]

Same strange failure: https://travis-ci.org/docker-library/python/jobs/103982805#L8818-L8821. 😢 But those lines look the same to me!

[mattrobenolt]

It's a difference in trailing whitespace. Lemme see what this test is doing.

[yosifkit]

(https://github.com/docker-library/official-images/tree/master/test/tests/python-hy)

[mattrobenolt]

Yeah, I have literally no idea. :) I'm guessing this is just broken even if we didn't change pip here. I can't see how updating pip would affect that.

[yosifkit]

I just ran the test on current python:3 and it works fine. I built a new image with pip 8.0.2 (literally just update pip).

diff --git a/3.5/Dockerfile b/3.5/Dockerfile
index aa32199..51c3340 100644
--- a/3.5/Dockerfile
+++ b/3.5/Dockerfile
@@ -13,7 +13,7 @@
 ENV PYTHON_VERSION 3.5.1

 # if this is called "PIP_VERSION", pip explodes with "ValueError: invalid truth value '<VERSION>'"
-ENV PYTHON_PIP_VERSION 7.1.2
+ENV PYTHON_PIP_VERSION 8.0.2

 RUN set -ex \
        && gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEY" \

And I get the same error:

$ testing python:3
    'python-hy' [1/1]...failed; unexpected output:
--- /home/sauron/dicker/stackbrew/test/tests/python-hy/expected-std-out.txt 2015-03-10 16:42:26.130754729 -0700
+++ -   2016-01-21 16:26:37.942146623 -0800
@@ -1,2 +1,2 @@
-4
-42
+4
+42
$ # piped to a file
$ hexdump --canonical _u temp-file
00000000  74 65 73 74 69 6e 67 20  70 79 74 68 6f 6e 3a 33  |testing python:3|
00000010  0a 09 27 70 79 74 68 6f  6e 2d 68 79 27 20 5b 31  |..'python-hy' [1|
00000020  2f 31 5d 2e 2e 2e 66 61  69 6c 65 64 3b 20 75 6e  |/1]...failed; un|
00000030  65 78 70 65 63 74 65 64  20 6f 75 74 70 75 74 3a  |expected output:|
00000040  0a 2d 2d 2d 20 2f 68 6f  6d 65 2f 73 61 75 72 6f  |.--- /home/sauro|
00000050  6e 2f 64 69 63 6b 65 72  2f 73 74 61 63 6b 62 72  |n/dicker/stackbr|
00000060  65 77 2f 74 65 73 74 2f  74 65 73 74 73 2f 70 79  |ew/test/tests/py|
00000070  74 68 6f 6e 2d 68 79 2f  65 78 70 65 63 74 65 64  |thon-hy/expected|
00000080  2d 73 74 64 2d 6f 75 74  2e 74 78 74 09 32 30 31  |-std-out.txt.201|
00000090  35 2d 30 33 2d 31 30 20  31 36 3a 34 32 3a 32 36  |5-03-10 16:42:26|
000000a0  2e 31 33 30 37 35 34 37  32 39 20 2d 30 37 30 30  |.130754729 -0700|
000000b0  0a 2b 2b 2b 20 2d 09 32  30 31 36 2d 30 31 2d 32  |.+++ -.2016-01-2|
000000c0  31 20 31 36 3a 32 36 3a  33 37 2e 39 34 32 31 34  |1 16:26:37.94214|
000000d0  36 36 32 33 20 2d 30 38  30 30 0a 40 40 20 2d 31  |6623 -0800.@@ -1|
000000e0  2c 32 20 2b 31 2c 32 20  40 40 0a 2d 34 0a 2d 34  |,2 +1,2 @@.-4.-4|
000000f0  32 0a 2b 1b 5b 3f 32 35  6c 1b 5b 3f 32 35 68 1b  |2.+.[?25l.[?25h.|
00000100  5b 3f 32 35 6c 1b 5b 3f  32 35 68 1b 5b 3f 32 35  |[?25l.[?25h.[?25|
00000110  6c 1b 5b 3f 32 35 68 34  0a 2b 1b 5b 3f 32 35 6c  |l.[?25h4.+.[?25l|
00000120  1b 5b 3f 32 35 68 34 32  0a                       |.[?25h42.|
00000129

upgrading pip changed the output?

[mattrobenolt]

Oh, I see. This test is actually calling pip.

It looks like it's outputting color escape sequences now?

[mattrobenolt]

Actually, looking at this more, I'm more confused. This isn't output from pip, it'd be output from the sh library, is my best guess, which is a wrapper for executing shell commands.

This hy script translated to python would be equivalent to:

print(2 + 2)

import pip
pip.main(['install', '-q', 'sh'])

from sh import echo

print(int(echo('-n', 21)) + int(echo('-n', 21)))

So the fact that we're getting garbage before both 4 and 42 doesn't make sense since we're not even invoking pip at those times. The print(2 + 2) is the first thing in the script. So something is adding some junk to the output.

I assume versions of hy and sh are identical on both builds for you? $ pip freeze | egrep -i 'hy|sh' would tell.

I'm not sure what else to do here except point my finger at a weird test that appears to be unrelated.

[mattrobenolt]

So this is even weirder.

I get this from a python prompt:

>>> h
'2b1b5b3f32356c1b5b3f3235681b5b3f32356c1b5b3f3235681b5b3f32356c1b5b3f323568340a2b1b5b3f32356c1b5b3f32356834320a'
>>> print(h.decode('hex'))
+4
+42

>>> h.decode('hex')
'+\x1b[?25l\x1b[?25h\x1b[?25l\x1b[?25h\x1b[?25l\x1b[?25h4\n+\x1b[?25l\x1b[?25h42\n'

This sheds a bit more light:

\x1b is the ESC key. \x1b[ starts a control sequence, followed by ?25h (and ?25l)

?25h means "show the cursor", and ?25l means "hide the cursor".

See: https://en.wikipedia.org/wiki/ANSI_escape_code#CSI_codes and look down for CSI ?25l

Why this is happening? Literally no idea, but these are clearly invisible characters that don't change the output visually.

[tianon]

Turns out that pip install -q silences pip's normal output, but it still prints out these spurious control codes. 😢 Does that constitute a pip bug, or considered WAD?

[dstufft]

Sounds like a bug.

[tianon]

❤️ @dstufft ok, I'll file an issue 😄

[tianon]

pypa/pip#3418 🤘

[frewsxcv]

Update: pypa/pip#3418 has been addressed, but a new version has not been pushed yet

[xblitz]

pip 8.0.3 has been released :)

[mattrobenolt]

yay, I will update when I actually wake up.

[mattrobenolt]

@yosifkit should be fixed here, and I've removed the hack that was previously needed for 3.2.

[yosifkit]

🎉 🎉 🍰 🎉 🎉 LGTM

[frewsxcv]

Nice work everyone! 🎊