{Packaging} Optimize Linux package and docker image by removing py file #25801
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
️✔️acr
️✔️acs
️✔️advisor
️✔️ams
️✔️apim
️✔️appconfig
️✔️appservice
️✔️aro
️✔️backup
️✔️batch
️✔️batchai
️✔️billing
️✔️botservice
️✔️cdn
️✔️cloud
️✔️cognitiveservices
️✔️config
️✔️configure
️✔️consumption
️✔️container
️✔️core
️✔️cosmosdb
️✔️databoxedge
️✔️dla
️✔️dls
️✔️dms
️✔️eventgrid
️✔️eventhubs
️✔️feedback
️✔️find
️✔️hdinsight
️✔️identity
️✔️iot
️✔️keyvault
️✔️kusto
️✔️lab
️✔️managedservices
️✔️maps
️✔️marketplaceordering
️✔️monitor
️✔️netappfiles
️✔️network
️✔️policyinsights
️✔️privatedns
️✔️profile
️✔️rdbms
️✔️redis
️✔️relay
️✔️resource
️✔️role
️✔️search
️✔️security
️✔️servicebus
️✔️serviceconnector
️✔️servicefabric
️✔️signalr
️✔️sql
️✔️sqlvm
️✔️storage
️✔️synapse
️✔️telemetry
️✔️util
️✔️vm
|
ghost
requested review from
ghost
added
the
|
Packaging |
3 tasks
bebound
commented
Mar 21, 2023
| if exist %%d rmdir /s /q "%%d" | ||
| ) | ||
| REM replace .py with .pyc and remove __pycache__ dir to save space | ||
| %BUILDING_DIR%\python.exe %REPO_ROOT%\scripts\use_pyc.py %BUILDING_DIR%\Lib\site-packages\ %PYTHON_VERSION% |
There was a problem hiding this comment.
I use use_pyc.py to replace cmd scripts not only for readability, but also for bug fix.
invoke==1.2.0 has a weird file: invoke/completion/__pycache__/__init__.cpython-36.pyc, and cmd script saves it as __init_.pyc in final build.
There was a problem hiding this comment.
Won't it be pretty hard to debug anything when there's no source code left?
There was a problem hiding this comment.
That's true, but I think most of the users never edit CLI source code for debug.
As evidence of this claim, we've removed the source code on Windows for years and no one complains about this.
But users do care about package size, we get lots of feedback in #7387.
For those who want to change source code, they need to install from pypi.
jiasli
reviewed
Apr 17, 2023
| # Python image has build-in env $PYTHON_VERSION=3.10.10. | ||
| # `ARG PYTHON_VERSION="3.10"` works on ARM64, but it can't override the default value on AMD64. | ||
| RUN ./scripts/install_full.sh && python ./scripts/trim_sdk.py \ | ||
| && python ./scripts/use_pyc.py /usr/local/lib/python${PYTHON_VERSION:0:4}/site-packages/ \ |
There was a problem hiding this comment.
I would prefer to make the pyc change in a separate PR, just in case something goes wrong and we need to roll back.
3 tasks
bebound
changed the title
bebound
changed the title
# Conflicts: # Dockerfile # scripts/release/debian/build.sh
bebound
modified the milestones:
April 2023 (2023-04-25),
May 2023 (2023-05-23) - For Build (Breaking change release)
bebound
modified the milestones:
May 2023 (2023-05-23) - For Build (Breaking change release),
Backlog
3 tasks
bebound
commented
Nov 28, 2023
|
|
||
| for f in glob.glob(f'{folder}/**/__pycache__', recursive=True): | ||
| # Remove pip __pycache__ folder for Windows only to save more space | ||
| if 'site-packages/pip' in f and not platform.system() == 'Windows': |
There was a problem hiding this comment.
Removing py file has side effects.
For example: /opt/az/bin/python3 -m pip install --no-cache pgcli==3.4.1
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 2
╰─> [1 lines of output]
/opt/az/bin/python3: can't open file '/opt/az/lib/python3.10/site-packages/pip/__pip-runner__.py': [Errno 2] No such file or directory
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
The MSI also keeps py files under pip folder.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We have used
trim_sdk.pyand only keep pyc file on Windows. Apply them on Linux and docker to reduce package size.Trimming SKD is #26172
This is the second part, removing py file on Linux.
Final result:
Ubuntu 22.04 installed size 576 MB -> 326 MB, become 56% of original size.
RHEL 9 installed size 403 MB -> 172 MB, become 43% of original size.
Docker image size 956 MB -> 714 MB, become 75% of original size.
History Notes
[Component Name 1] BREAKING CHANGE:
az command a: Make some customer-facing breaking change[Component Name 2]
az command b: Add some customer-facing featureThis checklist is used to make sure that common guidelines for a pull request are followed.
The PR title and description has followed the guideline in Submitting Pull Requests.
I adhere to the Command Guidelines.
I adhere to the Error Handling Guidelines.