{Packaging} Optimize Linux package and docker image by removing py file#25801
Open
{Packaging} Optimize Linux package and docker image by removing py file#25801
Conversation
️✔️acr
️✔️acs
️✔️advisor
️✔️ams
️✔️apim
️✔️appconfig
️✔️appservice
️✔️aro
️✔️backup
️✔️batch
️✔️batchai
️✔️billing
️✔️botservice
️✔️cdn
️✔️cloud
️✔️cognitiveservices
️✔️config
️✔️configure
️✔️consumption
️✔️container
️✔️core
️✔️cosmosdb
️✔️databoxedge
️✔️dla
️✔️dls
️✔️dms
️✔️eventgrid
️✔️eventhubs
️✔️feedback
️✔️find
️✔️hdinsight
️✔️identity
️✔️iot
️✔️keyvault
️✔️kusto
️✔️lab
️✔️managedservices
️✔️maps
️✔️marketplaceordering
️✔️monitor
️✔️netappfiles
️✔️network
️✔️policyinsights
️✔️privatedns
️✔️profile
️✔️rdbms
️✔️redis
️✔️relay
️✔️resource
️✔️role
️✔️search
️✔️security
️✔️servicebus
️✔️serviceconnector
️✔️servicefabric
️✔️signalr
️✔️sql
️✔️sqlvm
️✔️storage
️✔️synapse
️✔️telemetry
️✔️util
️✔️vm
|
ghost
requested review from
ghost
added
the
Collaborator
|
Packaging |
3 tasks
jiasli
reviewed
Apr 17, 2023
| # Python image has build-in env $PYTHON_VERSION=3.10.10. | ||
| # `ARG PYTHON_VERSION="3.10"` works on ARM64, but it can't override the default value on AMD64. | ||
| RUN ./scripts/install_full.sh && python ./scripts/trim_sdk.py \ | ||
| && python ./scripts/use_pyc.py /usr/local/lib/python${PYTHON_VERSION:0:4}/site-packages/ \ |
Member
There was a problem hiding this comment.
I would prefer to make the pyc change in a separate PR, just in case something goes wrong and we need to roll back.
3 tasks
bebound
changed the title
bebound
changed the title
# Conflicts: # Dockerfile # scripts/release/debian/build.sh
bebound
modified the milestones:
April 2023 (2023-04-25),
May 2023 (2023-05-23) - For Build (Breaking change release)
bebound
modified the milestones:
May 2023 (2023-05-23) - For Build (Breaking change release),
Backlog
3 tasks
bebound
commented
Feb 24, 2025
| # If pip's py files are also removed, the error is raised when installing some packages. | ||
| # See https://github.com/Azure/azure-cli/pull/25801 for details. | ||
| if os.path.join('site-packages', 'pip') in file: | ||
| continue |
Contributor
Author
There was a problem hiding this comment.
Removing py file has side effects.
For example: /opt/az/bin/python3 -m pip install --no-cache pgcli==3.4.1
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> [1 lines of output]
/opt/az/bin/python3: can't open file '/opt/az/lib/python3.10/site-packages/pip/__pip-runner__.py': [Errno 2] No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
The MSI also keeps py files under pip folder.
bebound
commented
Feb 24, 2025
|
|
||
| for f in glob.glob(f'{folder}/**/__pycache__', recursive=True): | ||
| # Remove emtpy __pycache__ folder | ||
| shutil.rmtree(f) |
Contributor
Author
There was a problem hiding this comment.
Need to add -B in Linux entry point to prevent generating pyc in pip folder.
|
This work is interesting but I would argue that this effort is a hack around the poor design choices for the Azure SDK releases. It does not solve the core problem which is that the SDKs have multiple copies per year since 2015 and that it is NOT sustainable and shows poor respect for Microsoft customers that have been struggling with this design choice for years. Who is responsible for refusing to do releases with a single SDK version, which is a strong ask from many azure SDK users? They should spend some time having a direct conversation with the users and hear the complaints. I can see that this removes 25 to 57% of the disk space but when over 90% of the SDK versions are useless, this would achieve much better results. Especially considering more versions are going to be added over time and the sizes will keep on growing at an increasing rate even after deleting the source files. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We have used
trim_sdk.pyand only keep pyc file on Windows. Apply them on Linux and docker to reduce package size.Trimming SKD is #26172
This is the second part, removing py file on Linux.
Final result:
Ubuntu 22.04 installed size 576 MB -> 326 MB, become 56% of original size.
RHEL 9 installed size 403 MB -> 172 MB, become 43% of original size.
Docker image size 956 MB -> 714 MB, become 75% of original size.
History Notes
[Component Name 1] BREAKING CHANGE:
az command a: Make some customer-facing breaking change[Component Name 2]
az command b: Add some customer-facing featureThis checklist is used to make sure that common guidelines for a pull request are followed.
The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.