Merge branch 'master' into zappa-settings-add-exclude

.github/ISSUE_TEMPLATE.md

@@ -1,7 +1,7 @@
 <!--- Provide a general summary of the issue in the Title above -->
 ## Context
 <!--- Provide a more detailed introduction to the issue itself, and why you consider it to be a bug -->
-<!--- Also, please make sure that you are running Zappa _from a virtual environment_ and are using Python 3.7/3.8/3.9/3.10 -->
+<!--- Also, please make sure that you are running Zappa _from a virtual environment_ and are using Python 3.7/3.8/3.9/3.10/3.11 -->
 
 ## Expected Behavior
 <!--- Tell us what should happen -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -20,7 +20,7 @@ Before you submit this PR, please make sure that you meet these criteria:
 
 * Did you **make sure this code actually works on Lambda**, as well as locally?
 
-* Did you test this code with all of **Python 3.7**, **Python 3.8**, **Python 3.9** and **Python 3.10** ?
+* Did you test this code with all of **Python 3.7**, **Python 3.8**, **Python 3.9**, **Python 3.10** and **Python 3.11** ?
 
 * Does this commit ONLY relate to the issue at hand and have your linter shit all over the code?
 

.github/workflows/cd.yaml

@@ -15,10 +15,10 @@ jobs:
     steps:
       - name: Checkout Code Repository
         uses: actions/checkout@v3
-      - name: Set up Python 3.10
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: "3.10"
+          python-version: "3.11"
       - name: Install `pypa/build`
         run: python -m pip install build
       - name: Build sdist and wheel

.github/workflows/ci.yaml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-20.04
     strategy:
       matrix:
-        python: [3.7, 3.8, 3.9, "3.10"]
+        python: [3.7, 3.8, 3.9, "3.10", "3.11"]
     steps:
       - name: Checkout Code Repository
         uses: actions/checkout@v3

CHANGELOG.md

@@ -1,5 +1,15 @@
 # Zappa Changelog
 
+## 0.58.0
+
+* Add Python 3.11 support (#1262)
+* support new ephemeral storage feature in zappa_settings.json (#1120)
+* Update permissions (PR #1119)
+* Outdated manylinux wheels download logic (#1249)
+* cryptography>=35.0, plus pip>=20.3 - downloads wrong cryptography anywheel package (GLIBC_2.18 error) (#1063)
+* fix response time improperly configured for micro-seconds. (#1265)
+* fix unquote issue with querystring handling. (#1264)
+
 ## 0.57.0
 
 * Python 3.10 support (#1124, #1160)

Makefile

@@ -27,7 +27,9 @@ clean:
 	coverage erase
 
 requirements:
-	pip install pipenv>2021.11.15
+	pip install pip --upgrade
+	pip install "pipenv>2021.11.15"
+
 	pipenv lock
 	pipenv sync --dev
 

Pipfile

@@ -14,7 +14,6 @@ Flask = "*"
 isort = "*"
 mock = "*"
 mypy = "*"
-pipenv = ">2021.11.15"
 packaging = "*"
 pytest = "*"
 pytest-cov = "*"

README.md

@@ -133,7 +133,7 @@ __Awesome!__
 
 ## Installation and Configuration
 
-_Before you begin, make sure you are running Python 3.7/3.8/3.9/3.10 and you have a valid AWS account and your [AWS credentials file](https://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs) is properly installed._
+_Before you begin, make sure you are running Python 3.7/3.8/3.9/3.10/3.11 and you have a valid AWS account and your [AWS credentials file](https://blogs.aws.amazon.com/security/post/Tx3D6U6WSFGOK2H/A-New-and-Standardized-Way-to-Manage-Credentials-in-the-AWS-SDKs) is properly installed._
 
 **Zappa** can easily be installed through pip, like so:
 
@@ -443,7 +443,7 @@ For instance, suppose you have a basic application in a file called "my_app.py",
 
 Any remote print statements made and the value the function returned will then be printed to your local console. **Nifty!**
 
-You can also invoke interpretable Python 3.7/3.8/3.9/3.10 strings directly by using `--raw`, like so:
+You can also invoke interpretable Python 3.7/3.8/3.9/3.10/3.11 strings directly by using `--raw`, like so:
 
     $ zappa invoke production "print(1 + 2 + 3)" --raw
 
@@ -974,6 +974,7 @@ to change Zappa's behavior. Use these at your own risk!
         "log_level": "DEBUG", // Set the Zappa log level. Can be one of CRITICAL, ERROR, WARNING, INFO and DEBUG. Default: DEBUG
         "manage_roles": true, // Have Zappa automatically create and define IAM execution roles and policies. Default true. If false, you must define your own IAM Role and role_name setting.
         "memory_size": 512, // Lambda function memory in MB. Default 512.
+        "ephemeral_storage": { "Size": 512 }, // Lambda function ephemeral_storage size in MB, Default 512, Max 10240
         "num_retained_versions":null, // Indicates the number of old versions to retain for the lambda. If absent, keeps all the versions of the function.
         "payload_compression": true, // Whether or not to enable API gateway payload compression (default: true)
         "payload_minimum_compression_size": 0, // The threshold size (in bytes) below which payload compression will not be applied (default: 0)
@@ -984,7 +985,7 @@ to change Zappa's behavior. Use these at your own risk!
         "role_name": "MyLambdaRole", // Name of Zappa execution role. Default <project_name>-<env>-ZappaExecutionRole. To use a different, pre-existing policy, you must also set manage_roles to false.
         "role_arn": "arn:aws:iam::12345:role/app-ZappaLambdaExecutionRole", // ARN of Zappa execution role. Default to None. To use a different, pre-existing policy, you must also set manage_roles to false. This overrides role_name. Use with temporary credentials via GetFederationToken.
         "route53_enabled": true, // Have Zappa update your Route53 Hosted Zones when certifying with a custom domain. Default true.
-        "runtime": "python3.10", // Python runtime to use on Lambda. Can be one of "python3.7", "python3.8", "python3.9", or "python3.10". Defaults to whatever the current Python being used is.
+        "runtime": "python3.11", // Python runtime to use on Lambda. Can be one of "python3.7", "python3.8", "python3.9", or "python3.10", or "python3.11". Defaults to whatever the current Python being used is.
         "s3_bucket": "dev-bucket", // Zappa zip bucket,
         "slim_handler": false, // Useful if project >50M. Set true to just upload a small handler to Lambda and load actual project from S3 at runtime. Default false.
         "settings_file": "~/Projects/MyApp/settings/dev_settings.py", // Server side settings file location,
@@ -1060,7 +1061,7 @@ You can also simply handle CORS directly in your application. Your web framework
 
 ### Large Projects
 
-AWS currently limits Lambda zip sizes to 50 megabytes. If your project is larger than that, set `slim_handler: true` in your `zappa_settings.json`. In this case, your fat application package will be replaced with a small handler-only package. The handler file then pulls the rest of the large project down from S3 at run time! The initial load of the large project may add to startup overhead, but the difference should be minimal on a warm lambda function. Note that this will also eat into the storage space of your application function. Note that AWS currently [limits](https://docs.aws.amazon.com/lambda/latest/dg/limits.html) the `/tmp` directory storage to 512 MB, so your project must still be smaller than that.
+AWS currently limits Lambda zip sizes to 50 megabytes. If your project is larger than that, set `slim_handler: true` in your `zappa_settings.json`. In this case, your fat application package will be replaced with a small handler-only package. The handler file then pulls the rest of the large project down from S3 at run time! The initial load of the large project may add to startup overhead, but the difference should be minimal on a warm lambda function. Note that this will also eat into the storage space of your application function. Note that AWS [supports](https://aws.amazon.com/blogs/compute/using-larger-ephemeral-storage-for-aws-lambda/) custom `/tmp` directory storage size in a range of 512 - 10240 MB. Use `ephemeral_storage` in `zappa_settings.json` to adjust to your needs if your project is larger than default 512 MB.
 
 ### Enabling Bash Completion
 

example/policy/deploy.json

@@ -31,6 +31,8 @@
                 "lambda:AddPermission",
                 "lambda:CreateFunction",
                 "lambda:DeleteFunction",
+                "lambda:DeleteFunctionConcurrency",
+                "lambda:GetAlias",
                 "lambda:GetFunction",
                 "lambda:GetFunctionConfiguration",
                 "lambda:GetPolicy",

setup.py

@@ -48,6 +48,7 @@
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
         "Framework :: Django",
         "Framework :: Django :: 1.11",
         "Framework :: Django :: 2.0",

test_settings.json

@@ -10,6 +10,9 @@
            "zip": "test_settings.callback"
        },
        "delete_local_zip": true,
+       "ephemeral_storage": {
+           "Size": 1024
+       },
        "debug": true,
        "parameter_depth": 2,
        "prebuild_script": "tests.test_app.prebuild_me",
@@ -107,6 +110,18 @@
           "EXTENDO": "You bet"
        }
     },
+    "invalid_ephemeral_storage_out_of_range": {
+        "extends": "ttt888",
+        "ephemeral_storage": {
+            "Size": 99999
+        }
+    },
+    "invalid_ephemeral_storage_missing_key": {
+        "extends": "ttt888",
+        "ephemeral_storage": {
+            "BadKey": 1024
+        }
+    },
     "build_package_only_delete_local_zip_false": {
       "delete_local_zip": false,
       "use_precompiled_packages": false,

tests/tests.py

@@ -198,6 +198,94 @@ def test_get_manylinux_python310(self):
             self.assertTrue(os.path.isfile(path))
             os.remove(path)
 
+    def test_verify_python37_does_not_download_2_24_manylinux_wheel(self):
+        z = Zappa(runtime="python3.7")
+        cached_wheels_dir = os.path.join(tempfile.gettempdir(), "cached_wheels")
+        expected_wheel_path = os.path.join(
+            cached_wheels_dir, "cryptography-35.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl"
+        )
+
+        # Check with known manylinux wheel package
+        actual_wheel_path = z.get_cached_manylinux_wheel("cryptography", "35.0.0")
+        self.assertEqual(actual_wheel_path, expected_wheel_path)
+        os.remove(actual_wheel_path)
+
+    def test_verify_downloaded_manylinux_wheel(self):
+        z = Zappa(runtime="python3.10")
+        cached_wheels_dir = os.path.join(tempfile.gettempdir(), "cached_wheels")
+        expected_wheel_path = os.path.join(
+            cached_wheels_dir,
+            "pycryptodome-3.16.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl",
+        )
+
+        # check with a known manylinux wheel package
+        actual_wheel_path = z.get_cached_manylinux_wheel("pycryptodome", "3.16.0")
+        self.assertEqual(actual_wheel_path, expected_wheel_path)
+        os.remove(actual_wheel_path)
+
+    def test_verify_manylinux_filename_is_lowered(self):
+        z = Zappa(runtime="python3.10")
+        expected_filename = "markupsafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
+
+        mock_package_data = {
+            "releases": {
+                "2.1.3": [
+                    {
+                        "url": "https://files.pythonhosted.org/packages/a6/56/f1d4ee39e898a9e63470cbb7fae1c58cce6874f25f54220b89213a47f273/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
+                        "filename": "MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
+                    },
+                    {
+                        "url": "https://files.pythonhosted.org/packages/12/b3/d9ed2c0971e1435b8a62354b18d3060b66c8cb1d368399ec0b9baa7c0ee5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
+                        "filename": "MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
+                    },
+                    {
+                        "url": "https://files.pythonhosted.org/packages/bf/b7/c5ba9b7ad9ad21fc4a60df226615cf43ead185d328b77b0327d603d00cc5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
+                        "filename": "MarkupSafe-2.1.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
+                    },
+                ]
+            }
+        }
+
+        with mock.patch("zappa.core.requests.get") as mock_get:
+            mock_get.return_value.json.return_value = mock_package_data
+            wheel_url, file_name = z.get_manylinux_wheel_url("markupsafe", "2.1.3", ignore_cache=True)
+
+            self.assertEqual(file_name, expected_filename)
+            mock_get.assert_called_once_with(
+                "https://pypi.python.org/pypi/markupsafe/json", timeout=float(os.environ.get("PIP_TIMEOUT", 1.5))
+            )
+
+        # Clean the generated files
+        cached_pypi_info_dir = os.path.join(tempfile.gettempdir(), "cached_pypi_info")
+        os.remove(os.path.join(cached_pypi_info_dir, "markupsafe-2.1.3.json"))
+
+    def test_get_manylinux_python311(self):
+        z = Zappa(runtime="python3.11")
+        self.assertIsNotNone(z.get_cached_manylinux_wheel("psycopg2-binary", "2.9.7"))
+        self.assertIsNone(z.get_cached_manylinux_wheel("derp_no_such_thing", "0.0"))
+
+        # mock with a known manylinux wheel package so that code for downloading them gets invoked
+        mock_installed_packages = {"psycopg2-binary": "2.9.7"}
+        with mock.patch(
+            "zappa.core.Zappa.get_installed_packages",
+            return_value=mock_installed_packages,
+        ):
+            z = Zappa(runtime="python3.11")
+            path = z.create_lambda_zip(handler_file=os.path.realpath(__file__))
+            self.assertTrue(os.path.isfile(path))
+            os.remove(path)
+
+        # same, but with an ABI3 package
+        mock_installed_packages = {"cryptography": "2.8"}
+        with mock.patch(
+            "zappa.core.Zappa.get_installed_packages",
+            return_value=mock_installed_packages,
+        ):
+            z = Zappa(runtime="python3.11")
+            path = z.create_lambda_zip(handler_file=os.path.realpath(__file__))
+            self.assertTrue(os.path.isfile(path))
+            os.remove(path)
+
     def test_getting_installed_packages(self, *args):
         z = Zappa(runtime="python3.7")
 
@@ -1154,6 +1242,24 @@ def test_load_settings(self):
         zappa_cli.load_settings("test_settings.json")
         self.assertEqual(False, zappa_cli.stage_config["touch"])
 
+    def test_load_settings_ephemeral_storage_overwrite(self):
+        zappa_cli = ZappaCLI()
+        zappa_cli.api_stage = "ttt888"
+        zappa_cli.load_settings("test_settings.json")
+        self.assertEqual(zappa_cli.stage_config["ephemeral_storage"]["Size"], 1024)
+
+    def test_load_settings_ephemeral_storage_out_of_range(self):
+        zappa_cli = ZappaCLI()
+        zappa_cli.api_stage = "invalid_ephemeral_storage_out_of_range"
+        with self.assertRaises(ClickException) as err:
+            zappa_cli.load_settings("test_settings.json")
+
+    def test_load_settings_ephemeral_storage_missing_key(self):
+        zappa_cli = ZappaCLI()
+        zappa_cli.api_stage = "invalid_ephemeral_storage_missing_key"
+        with self.assertRaises(ClickException) as err:
+            zappa_cli.load_settings("test_settings.json")
+
     def test_load_extended_settings(self):
         zappa_cli = ZappaCLI()
         zappa_cli.api_stage = "extendo"
@@ -2684,7 +2790,37 @@ def test_wsgi_query_string_unquoted(self):
             "requestContext": {},
         }
         request = create_wsgi_request(event)
-        self.assertEqual(request["QUERY_STRING"], "a=A,B&b=C#D")
+        expected = "a=A%2CB&b=C%23D"  # unencoded result: "a=A,B&b=C#D"
+        self.assertEqual(request["QUERY_STRING"], expected)
+
+    def test_wsgi_query_string_ampersand_unencoded(self):
+        event = {
+            "body": None,
+            "headers": {},
+            "pathParameters": {},
+            "path": "/path/path1",
+            "httpMethod": "GET",
+            "queryStringParameters": {
+                "test": "M&M",
+            },
+            "requestContext": {},
+        }
+        request = create_wsgi_request(event)
+        self.assertEqual(request["QUERY_STRING"], "test=M%26M")
+
+    def test_wsgi_query_string_with_encodechars(self):
+        event = {
+            "body": None,
+            "headers": {},
+            "pathParameters": {},
+            "path": "/path/path1",
+            "httpMethod": "GET",
+            "queryStringParameters": {"query": "Jane&John", "otherquery": "B", "test": "hello+m.te&how&are&you"},
+            "requestContext": {},
+        }
+        request = create_wsgi_request(event)
+        expected = "query=Jane%26John&otherquery=B&test=hello%2Bm.te%26how%26are%26you"
+        self.assertEqual(request["QUERY_STRING"], expected)
 
 
 if __name__ == "__main__":

zappa/__init__.py

@@ -12,7 +12,7 @@ def running_in_docker() -> bool:
     return running_in_docker_flag
 
 
-SUPPORTED_VERSIONS = [(3, 7), (3, 8), (3, 9), (3, 10)]
+SUPPORTED_VERSIONS = [(3, 7), (3, 8), (3, 9), (3, 10), (3, 11)]
 MINIMUM_SUPPORTED_MINOR_VERSION = 7
 
 if not running_in_docker() and sys.version_info[:2] not in SUPPORTED_VERSIONS:
@@ -30,4 +30,4 @@ def running_in_docker() -> bool:
     )
     raise RuntimeError(err_msg)
 
-__version__ = "0.57.0"
+__version__ = "0.58.0"

zappa/cli.py

@@ -108,6 +108,7 @@ class ZappaCLI:
     handler_path = None
     vpc_config = None
     memory_size = None
+    ephemeral_storage = None
     use_apigateway = None
     lambda_handler = None
     django_settings = None
@@ -810,6 +811,7 @@ def deploy(self, source_zip=None, docker_image_uri=None):
                 dead_letter_config=self.dead_letter_config,
                 timeout=self.timeout_seconds,
                 memory_size=self.memory_size,
+                ephemeral_storage=self.ephemeral_storage,
                 runtime=self.runtime,
                 aws_environment_variables=self.aws_environment_variables,
                 aws_kms_key_arn=self.aws_kms_key_arn,
@@ -1050,6 +1052,7 @@ def update(self, source_zip=None, no_upload=False, docker_image_uri=None):
             vpc_config=self.vpc_config,
             timeout=self.timeout_seconds,
             memory_size=self.memory_size,
+            ephemeral_storage=self.ephemeral_storage,
             runtime=self.runtime,
             aws_environment_variables=self.aws_environment_variables,
             aws_kms_key_arn=self.aws_kms_key_arn,
@@ -2229,6 +2232,14 @@ def load_settings(self, settings_file=None, session=None):
         )
         self.vpc_config = self.stage_config.get("vpc_config", {})
         self.memory_size = self.stage_config.get("memory_size", 512)
+        self.ephemeral_storage = self.stage_config.get("ephemeral_storage", {"Size": 512})
+
+        # Validate ephemeral storage structure and size
+        if "Size" not in self.ephemeral_storage:
+            raise ClickException("Please provide a valid Size for ephemeral_storage in your Zappa settings.")
+        elif not 512 <= self.ephemeral_storage["Size"] <= 10240:
+            raise ClickException("Please provide a valid ephemeral_storage size between 512 - 10240 in your Zappa settings.")
+
         self.app_function = self.stage_config.get("app_function", None)
         self.exception_handler = self.stage_config.get("exception_handler", None)
         self.aws_region = self.stage_config.get("aws_region", None)