Maintenance: Shrink size of Powertools Layer

[mploski]

Summary

Our overall, current size of pypi package is 74MB:
➜ test du -h -d1 | sort -h
8.0K ./bin
24K ./six-1.16.0.dist-info
28K ./pycache
28K ./fastjsonschema-2.16.1.dist-info
28K ./jmespath-1.0.1.dist-info
28K ./wrapt-1.14.1.dist-info
36K ./python_dateutil-2.8.2.dist-info
36K ./s3transfer-0.6.0.dist-info
44K ./boto3-1.24.35.dist-info
48K ./aws_lambda_powertools-1.26.5.dist-info
68K ./aws_xray_sdk-2.10.0.dist-info
72K ./urllib3-1.26.10.dist-info
148K ./jmespath
164K ./botocore-1.27.35.dist-info
184K ./fastjsonschema
208K ./wrapt
560K ./s3transfer
716K ./dateutil
824K ./urllib3
908K ./aws_xray_sdk
1.3M ./boto3
1.6M ./aws_lambda_powertools
67M ./botocore

As it can be seen on the listing 92.2% of the size is due to botocore + boto3 package. We should find a way to make it smaller to limit overall size of our project

Why is this needed?

Less time needed for downloading powertools package via pip or lambda with powertools layer.

Which area does this relate to?

No response

Solution

No response

Acknowledgment

• This request meets Lambda Powertools Tenets
• Should this be considered in other Lambda Powertools languages? i.e. Java, TypeScript

[mploski]

with botocore models compressed fully and pydantic not included, here is the overall size of unzipped packages
➜ python git:(master) ✗ du -h -d 1 | sort -h
12K ./bin
24K ./six-1.16.0.dist-info
28K ./fastjsonschema-2.16.1.dist-info
28K ./jmespath-1.0.1.dist-info
28K ./wrapt-1.14.1.dist-info
32K ./idna-3.3.dist-info
36K ./python_dateutil-2.8.2.dist-info
36K ./s3transfer-0.6.0.dist-info
36K ./typing_extensions-4.3.0.dist-info
40K ./dnspython-2.2.1.dist-info
44K ./boto3-1.24.62.dist-info
48K ./email_validator
52K ./aws_lambda_powertools-1.26.6.dist-info
52K ./email_validator-1.2.1.dist-info
68K ./aws_xray_sdk-2.10.0.dist-info
72K ./urllib3-1.26.12.dist-info
92K ./pycache
148K ./jmespath
164K ./botocore-1.27.62.dist-info
164K ./pydantic-1.10.0.dist-info
184K ./fastjsonschema
324K ./wrapt
488K ./idna
560K ./s3transfer
716K ./dateutil
828K ./urllib3
912K ./aws_xray_sdk
1.3M ./boto3
1.9M ./dns
7.1M ./aws_lambda_powertools
8.1M ./botocore
24M .

compressed python layer
python-layer.tar.gz -> 5.5MB

with removed all dist-info + pyc files
8.0K ./bin
32K ./email_validator
80K ./jmespath
100K ./fastjsonschema
268K ./wrapt
280K ./idna
300K ./s3transfer
444K ./aws_xray_sdk
460K ./urllib3
488K ./dateutil
1.0M ./boto3
1.1M ./dns
5.4M ./aws_lambda_powertools
7.1M ./botocore
17M .

Compressed files

python-layer2.tar.gz -> 4MB

[heitorlessa]

What’s the cold start overhead with models compressed? They’ll need to be unzipped at runtime by boto loader. Don’t remove the dist file from Powertools package as that will raise a runtime error when we expose a version variable to customers

…

On Tue, 6 Sep 2022 at 11:01, mploski ***@***.***> wrote: with botocore models compressed fully and pydantic not included, here is the overall size of unzipped packages ➜ python git:(master) ✗ du -h -d 1 | sort -h 12K ./bin 24K ./six-1.16.0.dist-info 28K ./fastjsonschema-2.16.1.dist-info 28K ./jmespath-1.0.1.dist-info 28K ./wrapt-1.14.1.dist-info 32K ./idna-3.3.dist-info 36K ./python_dateutil-2.8.2.dist-info 36K ./s3transfer-0.6.0.dist-info 36K ./typing_extensions-4.3.0.dist-info 40K ./dnspython-2.2.1.dist-info 44K ./boto3-1.24.62.dist-info 48K ./email_validator 52K ./aws_lambda_powertools-1.26.6.dist-info 52K ./email_validator-1.2.1.dist-info 68K ./aws_xray_sdk-2.10.0.dist-info 72K ./urllib3-1.26.12.dist-info 92K ./*pycache* 148K ./jmespath 164K ./botocore-1.27.62.dist-info 164K ./pydantic-1.10.0.dist-info 184K ./fastjsonschema 324K ./wrapt 488K ./idna 560K ./s3transfer 716K ./dateutil 828K ./urllib3 912K ./aws_xray_sdk 1.3M ./boto3 1.9M ./dns 7.1M ./aws_lambda_powertools 8.1M ./botocore 24M . compressed python layer python-layer.tar.gz -> 5.5MB with removed all dist-info + pyc files 8.0K ./bin 32K ./email_validator 80K ./jmespath 100K ./fastjsonschema 268K ./wrapt 280K ./idna 300K ./s3transfer 444K ./aws_xray_sdk 460K ./urllib3 488K ./dateutil 1.0M ./boto3 1.1M ./dns 5.4M ./aws_lambda_powertools 7.1M ./botocore 17M . Compressed files python-layer2.tar.gz -> 4MB — Reply to this email directly, view it on GitHub <#1348 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBAAKRINUCNCVT4VK7DV44B7ZANCNFSM54LCXDAA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.*** com>

[dverheijden]

Something you're probably already aware of: The Python 3.8 runtime for Lambda already ships with boto3 (see also here).

Since layers can only be used with lambda functions, you may want to consider making a layer with Python 3.8+ compatibility without boto3 included.

[heitorlessa]

Something you're probably already aware of: The Python 3.8 runtime for Lambda already ships with boto3 (see also here).

Since layers can only be used with lambda functions, you may want to consider making a layer with Python 3.8+ compatibility without boto3 included.

Yes @dverheijden ; - ) We're following that + other optimizations to drop the final Lambda Layer to ~1M including Pydantic.

Upstream work: aws-powertools/powertools-lambda-layer-cdk#4

[heitorlessa]

Closing as we're wrap to launch V2

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.