feat(layers): add support for publishing v2 layer #1558

rubenfonseca · 2022-09-30T10:37:48Z

Issue number: #1379

Summary

This PR adds support for publishing the v2 layer of Lambda Powertools for Python

Changes

Please provide a summary of what's being changed

Created separate workflows for v2 (so we can keep v1 running)
Moved all additional dependencies to a new "all" group
Marked boto3 as dev dependency
Changed Layer CDK code to produce and canary test 2 layers (regular + arm64)
Configured github actions to be able to produce arm64 (via qemu emulation)

User experience

Please share what the user experience looks like before and after this change

After merging this we should be able to start testing the v2 layer on the beta branch.

In order for it to work, we need a new major release of the CDK Layer Construct

Checklist

If your change doesn't seem to apply, please leave them unchecked.

Meet tenets criteria
I have performed a self-review of this change
Changes have been tested
Changes are documented
PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

Migration process documented
Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

pyproject.toml

heitorlessa

as per peer review session

.github/workflows/publish_v2_layer.yml

.github/workflows/reusable_deploy_v2_layer_stack.yml

layer/layer/layer_stack.py

layer/app.py

pyproject.toml

codecov-commenter · 2022-10-03T19:16:59Z

Codecov Report

Base: 99.42% // Head: 99.42% // No change to project coverage 👍

Coverage data is based on head (9be0000) compared to base (4099d42).
Patch has no changes to coverable lines.

Additional details and impacted files

@@           Coverage Diff           @@
##               v2    #1558   +/-   ##
=======================================
  Coverage   99.42%   99.42%           
=======================================
  Files         125      125           
  Lines        5726     5726           
  Branches      357      357           
=======================================
  Hits         5693     5693           
  Misses         18       18           
  Partials       15       15

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Co-authored-by: Heitor Lessa <[email protected]>

heitorlessa

Some quick suggestions, praises (my god, amazing work), and two issues spotted: 1/ Pip is bringing a wheel for X86 despite building for ARM (works but slower), 2/ Sin from the past on how we mistakenly accepted a PR with email-validator where we're downloading all available versions before choosing one.

.github/workflows/publish_v2_layer.yml

.github/workflows/reusable_deploy_v2_layer_stack.yml

layer/layer/canary/app.py

tests/e2e/utils/lambda_layer/powertools_layer.py

pyproject.toml

heitorlessa · 2022-10-05T17:43:24Z

layer/layer/canary_stack.py

        )

        execution_role.add_to_policy(
-            PolicyStatement(effect=Effect.ALLOW, actions=["lambda:GetFunction"], resources=["*"])
+            PolicyStatement(
+                effect=Effect.ALLOW, actions=["lambda:GetFunction"], resources=["*"]


@am29d please remind me again why do we need this permission in the Lambda itself? leftover?

Looking at the new canary I don't see an API call that would require this permission.

heitorlessa · 2022-10-06T05:32:52Z

Disregard my comment on JMESPath - we need to cater for PyPi customers too!

We just need to define the JMESPath as an optional dependency for extra to work

Co-authored-by: Heitor Lessa <[email protected]>

rubenfonseca · 2022-10-06T11:43:49Z

@heitorlessa ready to go again, incorporated your feedback, and added an code path to run our e2e tests using arm64 (no CLI flag was created yet)

heitorlessa

last changes and it's good to go! (pinky promise)

pyproject.toml

tests/e2e/utils/infrastructure.py

tests/e2e/utils/lambda_layer/powertools_layer.py

This will help customers prototyping locally without SAM CLI

This is due to jmespath and boto in general not being available locally, if a customer doesn't depend on boto3 as a dep (uses runtime)

Co-authored-by: Heitor Lessa <[email protected]>

boring-cyborg bot added dependencies Pull requests that update a dependency file github-actions Pull requests that update Github_actions code internal Maintenance changes labels Sep 30, 2022

pull-request-size bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Sep 30, 2022

github-actions bot added the feature New feature or functionality label Sep 30, 2022

rubenfonseca marked this pull request as ready for review September 30, 2022 10:40

rubenfonseca requested a review from a team as a code owner September 30, 2022 10:40

rubenfonseca requested review from heitorlessa and removed request for a team September 30, 2022 10:40

heitorlessa reviewed Oct 3, 2022

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

heitorlessa suggested changes Oct 3, 2022

View reviewed changes

heitorlessa linked an issue Oct 3, 2022 that may be closed by this pull request

Maintenance: Shrink size of Powertools Layer #1348

Closed

2 tasks

boring-cyborg bot added the tests label Oct 5, 2022

rubenfonseca and others added 16 commits October 5, 2022 12:02

feat(layers): add initial support for publishing v2 beta layer

741b87a

chore(layer): rename workflow

b850cf1

chore(deps): moved extra dependencies into an extra poetry group

3294d88

feat(layers): add regular and arm64 flavours

57be7ee

chore(deps): move fastjsonschema to optinal dependency group

c141cdc

feat(layers): use qmeu for arm64 builds

0ba4d38

fix(layers): pinned github actions

55e7e2d

chore: create extra dependency groups per utility

ec4f5e6

chore: apply suggestions from code review

43bc98f

Co-authored-by: Heitor Lessa <[email protected]>

chore: use latest cdk layer construct

0d19a80

chore: add jmespath and idempotency deps groups

04f44b9

chore: lock deps

bef044e

chore: use new cdk construct name

273be4c

chore: update call to CDK architecture

c5d2669

chore: fix canary deployment

c5b1df8

feat: use canary to test for presence of optional dependencies

2c3b64c

chore: remove badly escaped strings

84eb4b5

rubenfonseca force-pushed the feat/layer-v2 branch from 5b516c2 to 84eb4b5 Compare October 5, 2022 10:04

rubenfonseca requested a review from heitorlessa October 5, 2022 12:17

heitorlessa suggested changes Oct 5, 2022

View reviewed changes

rubenfonseca and others added 8 commits October 6, 2022 11:38

chore: apply suggestions from code review

f0093a4

Co-authored-by: Heitor Lessa <[email protected]>

chore: apply suggestions from code review

7a45c12

Co-authored-by: Heitor Lessa <[email protected]>

chore: lock dependencies

5dd961a

chore: apply suggestions from code review

4ad4b5e

Co-authored-by: Heitor Lessa <[email protected]>

chore: remove unecessary code

2c057a4

feat: allow e2e tests to run on arm64

36fff2f

chore: add jmespath as an optional dependency

4eef6bd

chore: bumped layer construct version

c8ecdb4

rubenfonseca requested a review from heitorlessa October 6, 2022 11:43

heitorlessa suggested changes Oct 7, 2022

View reviewed changes

heitorlessa added 6 commits October 7, 2022 13:54

chore(dep): add boto as optional dep

683784f

This will help customers prototyping locally without SAM CLI

chore(dep): add aws-sdk extras to ease local dev

b7c0780

This is due to jmespath and boto in general not being available locally, if a customer doesn't depend on boto3 as a dep (uses runtime)

chore(typing): remove unnecessary optional type

f61f03a

chore(typing): remove unnecessary optional type

8b4447c

refactor: use more explicit name for intent

978cba4

chore: use more explicit name

3286573

heitorlessa approved these changes Oct 7, 2022

View reviewed changes

rubenfonseca added 2 commits October 7, 2022 13:58

chore: lock dependencies

9be0000

chore: update cdk construct dependency

2c56344

rubenfonseca merged commit 2e5d0cb into v2 Oct 7, 2022

rubenfonseca deleted the feat/layer-v2 branch October 7, 2022 12:47

heitorlessa added a commit to heitorlessa/aws-lambda-powertools-python that referenced this pull request Oct 7, 2022

feat(layers): add support for publishing v2 layer (aws-powertools#1558)

157eacc

Co-authored-by: Heitor Lessa <[email protected]>

rubenfonseca added a commit that referenced this pull request Oct 14, 2022

feat(layers): add support for publishing v2 layer (#1558)

bbc1c14

Co-authored-by: Heitor Lessa <[email protected]>

lmmx mentioned this pull request Aug 9, 2023

Feature request: Pydantic v2 managed layer #2939

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(layers): add support for publishing v2 layer #1558

feat(layers): add support for publishing v2 layer #1558

rubenfonseca commented Sep 30, 2022 •

edited by heitorlessa

Loading

heitorlessa left a comment

codecov-commenter commented Oct 3, 2022 •

edited

Loading

heitorlessa left a comment

heitorlessa Oct 5, 2022

heitorlessa commented Oct 6, 2022

rubenfonseca commented Oct 6, 2022

heitorlessa left a comment

feat(layers): add support for publishing v2 layer #1558

feat(layers): add support for publishing v2 layer #1558

Conversation

rubenfonseca commented Sep 30, 2022 • edited by heitorlessa Loading

Summary

Changes

User experience

Checklist

Acknowledgment

heitorlessa left a comment

Choose a reason for hiding this comment

codecov-commenter commented Oct 3, 2022 • edited Loading

Codecov Report

heitorlessa left a comment

Choose a reason for hiding this comment

heitorlessa Oct 5, 2022

Choose a reason for hiding this comment

heitorlessa commented Oct 6, 2022

rubenfonseca commented Oct 6, 2022

heitorlessa left a comment

Choose a reason for hiding this comment

rubenfonseca commented Sep 30, 2022 •

edited by heitorlessa

Loading

codecov-commenter commented Oct 3, 2022 •

edited

Loading