Fixed `wheel unpack` not setting the executable bit by agronholm · Pull Request #514 · pypa/wheel

agronholm · 2023-03-13T16:29:09Z

Fixes #505.

…les in the archive Fixes #505.

codecov · 2023-03-13T16:31:44Z

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.07 🎉

Comparison is base (26d2e0d) 72.76% compared to head (9d0249e) 72.83%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #514      +/-   ##
==========================================
+ Coverage   72.76%   72.83%   +0.07%     
==========================================
  Files          13       13              
  Lines        1061     1064       +3     
==========================================
+ Hits          772      775       +3     
  Misses        289      289

Impacted Files	Coverage Δ
src/wheel/cli/unpack.py	`100.00% <100.00%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

henryiii · 2023-03-13T17:30:24Z

This seems to have been a zipfile bug since at least 2012, python/cpython#59999.

agronholm · 2023-03-13T18:05:58Z

I copied this technique from pip. Would you say this is good for merging?

henryiii · 2023-03-13T18:23:51Z

I think so, looks okay, but planning to check it in fully about an hour. I think this code path might get hit in the retagging command.

henryiii · 2023-03-13T19:19:36Z

Ah, never mind, that was an older version of the tags PR. The merged one doesn't go through this code.

Something weird is happening, though. I tried this on something I thought had a script (based on a GitHub code search), and it seems spyder-5.4.2/spyder-5.4.2.data/data/share/applications/spyder.desktop is now unpacking to spyder-5.4.2/spyder-5.4.2.data/data/share/applications/spyder.desktop/spyder-5.4.2.data/data/share/applications/spyder.desktop.

agronholm · 2023-03-13T19:21:23Z

Can you tell me how to repro?

henryiii · 2023-03-13T19:21:33Z

        print(f"Unpacking to: {destination}...", end="", flush=True)
-        wf.extractall(destination)
+        for zinfo in wf.filelist:
+            extracted_path = destination / zinfo.filename


zinfo already has the filename path, so this is duplicating the path, I think. a/b/c.txt -> a/b/a/b/c.txt.

No, that makes no sense, as destination remains the same, only zinfo.filename keeps changing. I tested extracting Django-3.2.16-py3-none-any.whl (which happened to be lying around) with both this branch and main, and I didn't see any difference.

You are setting destination with the filename, so now it has both. I think you want wf.extract(zinfo, destination) here. Extract extracts "a/b.txt" from argument 1 to argument 2, keeping the original relative path. Flatting this requires workarounds, see https://stackoverflow.com/a/47632134 for example. Are you sure you were using this branch?

tree Django-4.1.7/Django-4.1.7.dist-info Django-4.1.7/Django-4.1.7.dist-info ├── AUTHORS │ └── Django-4.1.7.dist-info │ └── AUTHORS ├── LICENSE │ └── Django-4.1.7.dist-info │ └── LICENSE ├── LICENSE.python │ └── Django-4.1.7.dist-info │ └── LICENSE.python ├── METADATA │ └── Django-4.1.7.dist-info │ └── METADATA ├── RECORD │ └── Django-4.1.7.dist-info │ └── RECORD ├── WHEEL │ └── Django-4.1.7.dist-info │ └── WHEEL ├── entry_points.txt │ └── Django-4.1.7.dist-info │ └── entry_points.txt └── top_level.txt └── Django-4.1.7.dist-info └── top_level.txt

~~However, this only affects the non-package files, like .dist-info. The package files (for both packages) seem fine.~~ Nevermind, happening to both.

On a deeper look, I saw the problem.

I basically pushed almost the exact same fix right as you posted that comment 😄

henryiii · 2023-03-13T20:00:10Z

Repro:

diff --git a/tests/cli/test_unpack.py b/tests/cli/test_unpack.py
index e614aa9..9cd4a94 100644
--- a/tests/cli/test_unpack.py
+++ b/tests/cli/test_unpack.py
@@ -27,9 +27,9 @@ def test_unpack_executable_bit(tmp_path):
     script_path.write_bytes(b"test script")
     script_path.chmod(0o777)
     with WheelFile(wheel_path, "w") as wf:
-        wf.write(str(script_path), "script")
+        wf.write(str(script_path), "nested/script")

     script_path.unlink()
-    script_path = script_path.parent / "test-1.0" / "script"
+    script_path = tmp_path / "test-1.0" / "nested" / "script"
     unpack(str(wheel_path), str(tmp_path))
     assert stat.S_IMODE(script_path.stat().st_mode) & 0o111

Fix:

diff --git a/src/wheel/cli/unpack.py b/src/wheel/cli/unpack.py
index 17ef0fa..fc3ffa4 100644
--- a/src/wheel/cli/unpack.py
+++ b/src/wheel/cli/unpack.py
@@ -23,11 +23,11 @@ def unpack(path: str, dest: str = ".") -> None:
         destination = Path(dest) / namever
         print(f"Unpacking to: {destination}...", end="", flush=True)
         for zinfo in wf.filelist:
-            extracted_path = destination / zinfo.filename
-            wf.extract(zinfo, extracted_path)
+            wf.extract(zinfo, destination)

             # Set the executable bit if it was set in the archive
             if stat.S_IMODE(zinfo.external_attr >> 16 & 0o111):
+                extracted_path = destination / zinfo.filename
                 extracted_path.chmod(0o777 & ~umask | 0o111)

     print("OK")

henryiii · 2023-03-13T20:03:42Z

:D I had the fix almost immediately, getting the repro was the hard part. For some reason I was convinced wf.write(str(script_path), "script") was writing the word "script" to script_path inside the wheel, I think...

I'd recommend adding the patch for the tests, though, just to cover this case. Or a second test / parametrization with a nested file. I pulled your fix, passes my test.

agronholm · 2023-03-13T20:10:37Z

I tried that repro patch against 933d556, but it didn't fail.

henryiii · 2023-03-13T20:14:10Z

Checked it on pygal (from https://cs.github.com/?scopeName=All+repos&scope=&q=path%3Asetup.py+%22scripts%3D%22):

$ pip download --no-deps pygal
$ wheel unpack pygal-3.0.0-py2.py3-none-any.whl
$ ls -l pygal-3.0.0/pygal-3.0.0.data/scripts/pygal_gen.py
-rw-r--r--  1 henryschreiner  staff  2316 Mar 13 16:09 pygal-3.0.0/pygal-3.0.0.data/scripts/pygal_gen.py
$ gh pr checkout 514
$ wheel unpack pygal-3.0.0-py2.py3-none-any.whl
$ ls -l pygal-3.0.0/pygal-3.0.0.data/scripts/pygal_gen.py
-rwxr-xr-x  1 henryschreiner  staff  2316 Mar 13 16:08 pygal-3.0.0/pygal-3.0.0.data/scripts/pygal_gen.py*

Looks better! What about if all x's aren't set, though? This only checks for 111?

agronholm · 2023-03-13T20:17:41Z

Looks better! What about if all x's aren't set, though? This only checks for 111?

No, it checks for any x (otherwise it would check zinfo.external_attr >> 16 & 0o111 == 0o111 and not what it does now (zinfo.external_attr >> 16 & 0o111 ).

agronholm · 2023-03-13T20:18:38Z

What about the repro though? Can you get it to fail against 933d556?

henryiii · 2023-03-13T20:22:59Z

No, it's not failing. Not sure why, but I had a mistake when I first wrote the test, then didn't go back and check the original after I fixed the test after writing the patch. Not sure why it's passing, though.

I know it checks for any x, but it looks like it sets all of them if one of them was set? Shouldn't it try to keep the original setting? (Pip's extraction probably doesn't care, but that's not trying to support round trips, just installing)

agronholm · 2023-03-13T20:29:06Z

Pip doesn't respect the original setting, but I guess wheel unpack is used for a different purpose, so I suppose I could just forego that umask stuff and set the exact same executable pattern on output.

henryiii · 2023-03-13T20:32:12Z

Got it: add

assert not script_path.is_dir()

(I think maybe to the original, tested with the nesting). Directories are "executable". :)

henryiii · 2023-03-13T20:37:24Z

Yes, pip's goal is different, see my (maybe too slow) edit above. :) Unpack should replicate the exact original permission bits. It's just a bug in CPython that it doesn't work via extractall.

henryiii

I'm happy with it now. :)

agronholm · 2023-03-13T20:52:55Z

Thanks for the review! It was really helpful.

Fixed wheel unpack not setting the executable bit for executable fi…

42e342c

…les in the archive Fixes #505.

Fixed EncodingWarning

933d556

agronholm requested a review from henryiii March 13, 2023 16:52

henryiii reviewed Mar 13, 2023

View reviewed changes

Fixed unpack() unpacking files to wrong paths

aa24fc4

Set the same permissions on output as in the archive

ee19d11

henryiii approved these changes Mar 13, 2023

View reviewed changes

henryiii reviewed Mar 13, 2023

View reviewed changes

Comment thread src/wheel/cli/unpack.py

Link to the CPython bug

9d0249e

agronholm merged commit 934fe17 into main Mar 13, 2023

agronholm deleted the fix-505 branch March 13, 2023 20:52

henryiii mentioned this pull request Mar 13, 2023

docs: add a couple of missing changelog entries #516

Merged

gilfree mentioned this pull request Apr 2, 2024

WheelFile extract does not preserve permissions #608

Closed

Conversation

agronholm commented Mar 13, 2023

Uh oh!

codecov Bot commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

henryiii commented Mar 13, 2023

Uh oh!

agronholm commented Mar 13, 2023

Uh oh!

henryiii commented Mar 13, 2023

Uh oh!

henryiii commented Mar 13, 2023

Uh oh!

agronholm commented Mar 13, 2023

Uh oh!

henryiii Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agronholm Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

henryiii Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

henryiii Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agronholm Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

agronholm Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

henryiii commented Mar 13, 2023

Uh oh!

henryiii commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agronholm commented Mar 13, 2023

Uh oh!

henryiii commented Mar 13, 2023

Uh oh!

agronholm commented Mar 13, 2023

Uh oh!

agronholm commented Mar 13, 2023

Uh oh!

henryiii commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

agronholm commented Mar 13, 2023

Uh oh!

henryiii commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

henryiii commented Mar 13, 2023

Uh oh!

henryiii left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

agronholm commented Mar 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Mar 13, 2023 •

edited

Loading

henryiii Mar 13, 2023 •

edited

Loading

henryiii Mar 13, 2023 •

edited

Loading

henryiii Mar 13, 2023 •

edited

Loading

henryiii commented Mar 13, 2023 •

edited

Loading

henryiii commented Mar 13, 2023 •

edited

Loading

henryiii commented Mar 13, 2023 •

edited

Loading