Maintenance: publish layers & increment version during main release process

[dreamorosi]

Bug description

At the moment the release process for the utilities and layers loosely works as follows:

graph LR
    A[Make Release] --> B(Version)
    B --> C(Publish)
    C -->|Triggers| D[Publish Layers]
    D --> E(Layer Published)
    E -->|Manual commit| H(Update layer version in docs)
    C -->|Manual trigger| F[GitHub Release]
    F --> G(Docs Published using tag)
    G -->|Tag from here| C

Loading

Looking at the diagram above we can observe that the Layer publishing workflow and the release one diverge from the publishing one. This is a problem because the following happens:

• During versioning & publishing (1st workflow) we increment the packages version & push a tag (i.e. v.1.3.0)
• Subsequently, the layers are published (2nd workflow) and we manually have to increment the Layer version (i.e. `3) in the docs
• Finally, we make a release using the GitHub UI (3rd workflow) which uses the tag created at step 1

This means that by the time we make the GitHub release and release the docs, the tag that was created at step 1 is outdated and that the commit that updates the Layer version in the docs won't be taken in account.

This is what happened in the last release (v1.3.0) which should correspond to Layer version 3 but instead shows version 2 (from before the process above) that corresponds to v1.2.1 of the library.

Expected Behavior

The tag that is used to release a version & publish docs should already include the commit that updates the Layer version in the docs.

Current Behavior

Layer version in the docs is not synced with latest & with release version.

Possible Solution

There are three options that I could think of.

Option 1

Publish workflow optimistically increments by one the Layer version in the docs at some point BEFORE versioning.

graph LR
    A[Make Release] --> B(Version)
    B --> L(Opimistically update docs with new version)
    L --> M(Push new tag)
    M --> C(Publish)
    C -->|Triggers| K[Publish layers]
    K --> E(Layer Published)
    C -->|Manual trigger| F[GitHub Release]
    F --> G(Docs Published using tag)
    G -->|Tag from here| C

Loading

In this scenario the layers are published at the same step as now, but the Make Release workflow optimistically bumps the layer version in the docs before making the release. This way the commit that increments the layer version in the docs will have been made before the tag is pushed.

This option is the simpler to implement and hinges on the fact that Layers are always and exclusively published as part of a full release train.

Option 2

Layers + docs update are published AFTER versioning has happened BUT BEFORE the tags is pushed.

graph LR
    A[Make Release] --> B(Version)
    B --> K[Publish layers]
    K --> L(Update docs with new version)
    L --> M(Push new tag)
    M --> C(Publish)
    C -->|Manual trigger| F[GitHub Release]
    F --> G(Docs Published using tag)
    G -->|Tag from here| C

Loading

In this scenario, the Publish Layers action is triggered as intermediate step between npm run version & npm run publish (here). This way the layer publishing process

This option will require us to run the commands currently run in the postversion command in each package.json file (git push && git push --tags) after the docs have already been updated with the new Layers version.

This will work as long as the Publish Layers workflow is able to use the same context as the Make Release one & be able to see the that the version in the package.json files has been bumped.

Option 3

Leave all workflow as they are and manually update the version in the docs BEFORE starting the release process.

This step requires no changes in the pipelines but revolves around us maintainers following the process & making no errors.

If we are to follow this, we'll have to document the release SOP and make sure it's followed.

Steps to Reproduce

See above.

Environment

• Powertools version used: N/A
• Packaging format (Layers, npm): N/A
• AWS Lambda function runtime: N/A
• Debugging logs: N/A

Related issues, RFCs

#826

[heitorlessa]

Chat with @rubenfonseca as he's just finished integrating our release with Layer version bump in the docs + ARM64 too, since we're now making ~168 Layer deployments per release

[flochaz]

Need to check what has been done on python side (aws-powertools/powertools-lambda-python#1610) but thinking of this , the script could use https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html#listLayerVersions-property API in the github action to bump based on true version published

[dreamorosi]

Seems that one of the workflows introduced in the PR that closed this issue has a circular dependency: https://github.com/awslabs/aws-lambda-powertools-typescript/actions/runs/4271035677

[github-actions]

⚠️ COMMENT VISIBILITY WARNING ⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.