chore(tests): enable end-to-end test workflow (#1470)

aws-powertools · Aug 23, 2022 · a45a792 · a45a792
1 parent 4dcfb71
commit a45a792
Show file tree

Hide file tree

Showing 8 changed files with 286 additions and 817 deletions.
diff --git a/.github/workflows/run-e2e-tests.yml b/.github/workflows/run-e2e-tests.yml
@@ -1,9 +1,17 @@
-name: run-e2e-tests
+name: Run end-to-end tests
+
 on:
   workflow_dispatch:
+
+  push:
+    branches: [develop]
+
+# Maintenance: Add support for triggering on `run-e2e` label
+# and enforce repo origin to prevent abuse
+
 env:
   AWS_DEFAULT_REGION: us-east-1
-  E2E_TESTS_PATH: tests/e2e/
+
 jobs:
   run:
     runs-on: ubuntu-latest

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,4 +1,20 @@
-[![Gitpod Ready-to-Code](https://img.shields.io/badge/Gitpod-Ready--to--Code-blue?logo=gitpod)](https://gitpod.io/from-referrer/)
+<!-- markdownlint-disable MD043 MD041 -->
+# Table of contents <!-- omit in toc -->
+
+- [Contributing Guidelines](#contributing-guidelines)
+    - [Reporting Bugs/Feature Requests](#reporting-bugsfeature-requests)
+    - [Contributing via Pull Requests](#contributing-via-pull-requests)
+        - [Dev setup](#dev-setup)
+        - [Local documentation](#local-documentation)
+    - [Conventions](#conventions)
+        - [General terminology and practices](#general-terminology-and-practices)
+        - [Testing definition](#testing-definition)
+    - [Finding contributions to work on](#finding-contributions-to-work-on)
+    - [Code of Conduct](#code-of-conduct)
+    - [Security issue notifications](#security-issue-notifications)
+    - [Troubleshooting](#troubleshooting)
+        - [API reference documentation](#api-reference-documentation)
+    - [Licensing](#licensing)
 
 # Contributing Guidelines
 
@@ -25,9 +41,11 @@ Contributions via pull requests are much appreciated. Before sending us a pull r
 
 ### Dev setup
 
+[![Gitpod Ready-to-Code](https://img.shields.io/badge/Gitpod-Ready--to--Code-blue?logo=gitpod)](https://gitpod.io/from-referrer/)
+
 Firstly, [fork the repository](https://github.com/awslabs/aws-lambda-powertools-python/fork).
 
-To setup your development environment, we recommend using our pre-configured Cloud environment: https://gitpod.io/#https://github.com/YOUR_USERNAME/aws-lambda-powertools-python. Replace YOUR_USERNAME with your GitHub username or organization so the Cloud environment can target your fork accordingly.
+To setup your development environment, we recommend using our pre-configured Cloud environment: <https://gitpod.io/#https://github.com/YOUR_USERNAME/aws-lambda-powertools-python>. Replace YOUR_USERNAME with your GitHub username or organization so the Cloud environment can target your fork accordingly.
 
 Alternatively, you can use `make dev` within your local virtual environment.
 
@@ -47,22 +65,38 @@ GitHub provides additional document on [forking a repository](https://help.githu
 
 You might find useful to run both the documentation website and the API reference locally while contributing:
 
-* **API reference**: `make docs-api-local`
-* **Docs website**: `make docs-local`
-	- If you prefer using Docker: `make docs-local-docker`
+- **API reference**: `make docs-api-local`
+- **Docs website**: `make docs-local`
+    - If you prefer using Docker: `make docs-local-docker`
+
+## Conventions
+
+### General terminology and practices
 
-### Conventions
+| Category              | Convention                                                                                                                                                                                                                                                                  |
+| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Docstring**         | We use a slight variation of Numpy convention with markdown to help generate more readable API references.                                                                                                                                                                  |
+| **Style guide**       | We use black as well as flake8 extensions to enforce beyond good practices [PEP8](https://pep8.org/). We use type annotations and enforce static type checking at CI (mypy).                                                                                                |
+| **Core utilities**    | Core utilities use a Class, always accept `service` as a constructor parameter, can work in isolation, and are also available in other languages implementation.                                                                                                            |
+| **Utilities**         | Utilities are not as strict as core and focus on solving a developer experience problem while following the project [Tenets](https://awslabs.github.io/aws-lambda-powertools-python/#tenets).                                                                               |
+| **Exceptions**        | Specific exceptions live within utilities themselves and use `Error` suffix e.g. `MetricUnitError`.                                                                                                                                                                         |
+| **Git commits**       | We follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). We do not enforce conventional commits on contributors to lower the entry bar. Instead, we enforce a conventional PR title so our label automation and changelog are generated correctly. |
+| **API documentation** | API reference docs are generated from docstrings which should have Examples section to allow developers to have what they need within their own IDE. Documentation website covers the wider usage, tips, and strive to be concise.                                          |
+| **Documentation**     | We treat it like a product. We sub-divide content aimed at getting started (80% of customers) vs advanced usage (20%). We also ensure customers know how to unit test their code when using our features.                                                                   |
 
-Category | Convention
-------------------------------------------------- | ---------------------------------------------------------------------------------
-**Docstring** |  We use a slight variation of Numpy convention with markdown to help generate more readable API references.
-**Style guide** | We use black as well as flake8 extensions to enforce beyond good practices [PEP8](https://pep8.org/). We use type annotations and enforce static type checking at CI (mypy).
-**Core utilities** | Core utilities use a Class, always accept `service` as a constructor parameter, can work in isolation, and are also available in other languages implementation.
-**Utilities** | Utilities are not as strict as core and focus on solving a developer experience problem while following the project [Tenets](https://awslabs.github.io/aws-lambda-powertools-python/#tenets).
-**Exceptions** | Specific exceptions live within utilities themselves and use `Error` suffix e.g. `MetricUnitError`.
-**Git commits** | We follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/). We do not enforce conventional commits on contributors to lower the entry bar. Instead, we enforce a conventional PR title so our label automation and changelog are generated correctly.
-**API documentation** | API reference docs are generated from docstrings which should have Examples section to allow developers to have what they need within their own IDE. Documentation website covers the wider usage, tips, and strive to be concise.
-**Documentation** | We treat it like a product. We sub-divide content aimed at getting started (80% of customers) vs advanced usage (20%). We also ensure customers know how to unit test their code when using our features.
+### Testing definition
+
+We group tests in different categories
+
+| Test              | When to write                                                                                         | Notes                                                                                                                           | Speed                                             |
+| ----------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
+| Unit tests        | Verify the smallest possible unit works.                                                              | Networking access is prohibited. Prefer Functional tests given our complexity.                                                  | Lightning fast (nsec to ms)                       |
+| Functional tests  | Guarantee functionality works as expected. It's a subset of integration test covering multiple units. | No external dependency. Prefer Fake implementations (in-memory) over Mocks and Stubs.                                           | Fast (ms to few seconds at worst)                 |
+| Integration tests | Gain confidence that code works with one or more external dependencies.                               | No need for a Lambda function. Use our code base against an external dependency _e.g., fetch an existing SSM parameter_.        | Moderate to slow (a few minutes)                  |
+| End-to-end tests  | Gain confidence that a Lambda function with our code operates as expected.                            | It simulates how customers configure, deploy, and run their Lambda function - Event Source configuration, IAM permissions, etc. | Slow (minutes)                                    |
+| Performance tests | Ensure critical operations won't increase latency and costs to customers.                             | CI arbitrary hardware can make it flaky. We'll resume writing perf test after our new Integ/End have significant coverage.      | Fast to moderate (a few seconds to a few minutes) |
+
+**NOTE**: Functional tests are mandatory. We have plans to create a guide on how to create these different tests. Maintainers will help indicate whether additional tests are necessary and provide assistance as required.
 
 ## Finding contributions to work on
 
@@ -75,6 +109,7 @@ For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of
 [email protected] with any additional questions or comments.
 
 ## Security issue notifications
+
 If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our [vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do **not** create a public github issue.
 
 ## Troubleshooting
@@ -85,9 +120,9 @@ When you are working on the codebase and you use the local API reference documen
 
 This happens when:
 
-* You did not install the local dev environment yet
+- You did not install the local dev environment yet
     - You can install dev deps with `make dev` command
-* The code in the repository is raising an exception while the `pdoc` is scanning the codebase
+- The code in the repository is raising an exception while the `pdoc` is scanning the codebase
     - Unfortunately, this exception is not shown to you, but if you run, `poetry run pdoc --pdf aws_lambda_powertools`, the exception is shown and you can prevent the exception from being raised
     - Once resolved the documentation should load correctly again
 

diff --git a/MAINTAINERS.md b/MAINTAINERS.md
@@ -15,6 +15,8 @@
     - [Releasing a new version](#releasing-a-new-version)
         - [Drafting release notes](#drafting-release-notes)
     - [Run end to end tests](#run-end-to-end-tests)
+        - [Structure](#structure)
+        - [Workflow](#workflow)
     - [Releasing a documentation hotfix](#releasing-a-documentation-hotfix)
     - [Maintain Overall Health of the Repo](#maintain-overall-health-of-the-repo)
     - [Manage Roadmap](#manage-roadmap)
@@ -212,11 +214,85 @@ This will kick off the [Publishing workflow](https://github.com/awslabs/aws-lamb
 
 ### Run end to end tests
 
-In order to run end to end tests you need to install CDK CLI first and bootstrap your account with `cdk bootstrap` command. For additional details follow [documentation](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html).
-
-To run locally, export `AWS_PROFILE` environment variable and run `make e2e tests`. To run from GitHub Actions, use [run-e2e-tests workflow](https://github.com/awslabs/aws-lambda-powertools-python/actions/workflows/run-e2e-tests.yml) and pick the branch you want to run tests against.
-
-**NOTE**: E2E tests are run as part of each merge to `develop` branch.
+E2E tests are run on every push to `develop` or manually via [run-e2e-tests workflow](https://github.com/awslabs/aws-lambda-powertools-python/actions/workflows/run-e2e-tests.yml).
+
+To run locally, you need [AWS CDK CLI](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites) and an [account bootstrapped](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html) (`cdk bootstrap`). With a default AWS CLI profile configured, or `AWS_PROFILE` environment variable set, run `make e2e tests`.
+
+#### Structure
+
+Our E2E framework relies on pytest fixtures to coordinate infrastructure and test parallelization (see [Workflow](#workflow)). You'll notice multiple `conftest.py`, `infrastructure.py`, and `handlers`.
+
+- **`infrastructure`**. Uses CDK to define what a Stack for a given feature should look like. It inherits from `BaseInfrastructure` to handle all boilerplate and deployment logic necessary.
+- **`conftest.py`**. Imports and deploys a given feature Infrastructure. Hierarchy matters. Top-level `conftest` deploys stacks only once and blocks I/O across all CPUs. Feature-level `conftest` deploys stacks in parallel, and once complete run all tests in parallel.
+- **`handlers`**. Lambda function handlers that will be automatically deployed and exported as PascalCase for later use.
+
+```shell
+.
+├── __init__.py
+├── conftest.py # deploys Lambda Layer stack
+├── logger
+│   ├── __init__.py
+│   ├── conftest.py  # deploys LoggerStack
+│   ├── handlers
+│   │   └── basic_handler.py
+│   ├── infrastructure.py # LoggerStack definition
+│   └── test_logger.py
+├── metrics
+│   ├── __init__.py
+│   ├── conftest.py  # deploys MetricsStack
+│   ├── handlers
+│   │   ├── basic_handler.py
+│   │   └── cold_start.py
+│   ├── infrastructure.py # MetricsStack definition
+│   └── test_metrics.py
+├── tracer
+│   ├── __init__.py
+│   ├── conftest.py  # deploys TracerStack
+│   ├── handlers
+│   │   ├── async_capture.py
+│   │   └── basic_handler.py
+│   ├── infrastructure.py  # TracerStack definition
+│   └── test_tracer.py
+└── utils
+    ├── Dockerfile
+    ├── __init__.py
+    ├── data_builder  # build_service_name(), build_add_dimensions_input, etc.
+    ├── data_fetcher  # get_traces(), get_logs(), get_lambda_response(), etc.
+    ├── infrastructure.py # base infrastructure like deploy logic, Layer Stack, etc.
+```
+
+#### Workflow
+
+We parallelize our end-to-end tests to benefit from speed and isolate Lambda functions to ease assessing side effects (e.g., traces, logs, etc.). The following diagram demonstrates the process we take every time you use `make e2e`:
+
+```mermaid
+graph TD
+    A[make e2e test] -->Spawn{"Split and group tests <br>by feature and CPU"}
+
+    Spawn -->|Worker0| Worker0_Start["Load tests"]
+    Spawn -->|Worker1| Worker1_Start["Load tests"]
+    Spawn -->|WorkerN| WorkerN_Start["Load tests"]
+
+    Worker0_Start -->|Wait| LambdaLayerStack["Lambda Layer Stack Deployment"]
+    Worker1_Start -->|Wait| LambdaLayerStack["Lambda Layer Stack Deployment"]
+    WorkerN_Start -->|Wait| LambdaLayerStack["Lambda Layer Stack Deployment"]
+
+    LambdaLayerStack -->|Worker0| Worker0_Deploy["Launch feature stack"]
+    LambdaLayerStack -->|Worker1| Worker1_Deploy["Launch feature stack"]
+    LambdaLayerStack -->|WorkerN| WorkerN_Deploy["Launch feature stack"]
+
+    Worker0_Deploy -->|Worker0| Worker0_Tests["Run tests"]
+    Worker1_Deploy -->|Worker1| Worker1_Tests["Run tests"]
+    WorkerN_Deploy -->|WorkerN| WorkerN_Tests["Run tests"]
+
+    Worker0_Tests --> ResultCollection
+    Worker1_Tests --> ResultCollection
+    WorkerN_Tests --> ResultCollection
+
+    ResultCollection{"Wait for workers<br/>Collect test results"}
+    ResultCollection --> TestEnd["Report results"]
+    ResultCollection --> DeployEnd["Delete Stacks"]
+```
 
 ### Releasing a documentation hotfix
 

diff --git a/aws_lambda_powertools/event_handler/appsync.py b/aws_lambda_powertools/event_handler/appsync.py
@@ -1,5 +1,4 @@
 import logging
-from abc import ABC
 from typing import Any, Callable, Optional, Type, TypeVar
 
 from aws_lambda_powertools.utilities.data_classes import AppSyncResolverEvent
@@ -10,7 +9,7 @@
 AppSyncResolverEventT = TypeVar("AppSyncResolverEventT", bound=AppSyncResolverEvent)
 
 
-class BaseRouter(ABC):
+class BaseRouter:
     current_event: AppSyncResolverEventT  # type: ignore[valid-type]
     lambda_context: LambdaContext