-
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Figure out how to deploy Datasette to AWS Lambda using function URLs and Mangum #6
Comments
YouTube video from three months ago about deploying fast api on lambda https://youtu.be/RGIM4JfsSk0 Wow that video genuinely has everything I need to know - including how to make the zip file and how to deploy (via the console) and how to add a function URL https://github.com/pixegami/fastapi-tutorial from mangum import Mangum
app = FastAPI()
handler = Mangum(app)
Looks like you need cloudfront if you want a custom domain or subdomain though: https://medium.com/@walid.karray/configuring-a-custom-domain-for-aws-lambda-function-url-with-serverless-framework-c0d78abdc253 (I bet cloudflare is easier) Limit is still 50 MB (zipped, for direct upload) 250 MB (unzipped) I want "datasette publish lambda" I'm really tempted to have the datasette-publish-lambda GitHub repo build a release which is a zip file that has everything it needs - then the plugin itself downloads (and caches) the latest releases zip file, adds the SQLite DB and metadata and maybe some templates and plugins and static files to that zip file, adds any --install plugins to it and then ships the result But the first release can build the zip file from scratch every time. That's probably simpler! So the plugin mainly creates a working zip file for you and deploys it. It could even have a --test option which creates the zip file, then unzips it into a temp virtual environment to test it before deploying it Trickiest code here is the code that deploys the function. I'm VERY tempted to build a separate asgi-lambda-publish python package here that this depends on, because I'm so angry about how hard this is Start with a TIL though. |
Could I start by doing a full lambda deployment entirely using the AWS CLI tool? |
https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-awscli.html is VERY useful.
Then exports.handler = async function(event, context) {
console.log("ENVIRONMENT VARIABLES\n" + JSON.stringify(process.env, null, 2))
console.log("EVENT\n" + JSON.stringify(event, null, 2))
return context.logStreamName
} Create
Next step needs account ID:
Outputs:
To create function:
Then invoke the function like this:
|
https://docs.aws.amazon.com/lambda/latest/dg/urls-configuration.html#create-url-cli shows how to add a function URL afterwards. |
Tried out those steps, got to:
This didn't work though:
Ran that through
|
That's because I broke my
Trying again with a fixed one. |
I recreated the
|
That seems to work!
{
"StatusCode": 200,
"LogResult": "U1RBUlQgUmVxdWVzdElkOiBmYmMxMGJmMC0yZWZkLTRkYTctOTA0Yy1hNGE4MTAyYzA1MzMgVmVyc2lvbjogJExBVEVTVAoyMDIyLTA5LTE4VDE3OjQ4OjQ2LjA5N1oJZmJjMTBiZjAtMmVmZC00ZGE3LTkwNGMtYTRhODEwMmMwNTMzCUlORk8JRU5WSVJPTk1FTlQgVkFSSUFCTEVTCnsKICAiQVdTX0xBTUJEQV9GVU5DVElPTl9WRVJTSU9OIjogIiRMQVRFU1QiLAogICJBV1NfU0VTU0lPTl9UT0tFTiI6ICJJUW9KYjNKcFoybHVYMlZqRUlyLy8vLy8vLy8vL3dFYUNYVnpMV1ZoYzNRdE1TSkhNRVVDSUNkQmV5OTFYcnNvTjdWNmtoRkgwTkpsVXhuVy9pN0N2cDlGNitCS0NEUjlBaUVBc1k3VUMzZnF1ZXpwam93QmswRE5TTUlEY2pTeXY3Y1RUMkJYVFhCbjZqZ3FrUU1JTXhBREdndzBOakl3T1RJM09EQTBOallpREFzYXFlcE41ZG5Rb0N5aklTcnVBbEVtQVJIRTVzVkpKWTdkcGNmL2VhcXd3TGZDMUc4cmJ2OHdkY250dS9ZUVVnWjAreGtLbGZRSG9wU1JaV3NhQ3hyaFBqRitCVDRvUGg5Y0xYU2R2ZFhUS2VVOGM1c0k2NmNWVUtzT3drS2NXK2poNmp6THpvQXpJdkFyYlB4b3duL3BmSDBBQ2dxNGFsU0RVQWdyVWJiV3dSNk1Vb1FsOUhMUmd2VnFEWjlSbnR6OXpOcHcwU0pnS0lSWTRYZloxeFNQTGlnVGRmblRLcTVja1RBaFFDWEgrbFN4WW12aWlsRGJNUXdlYzJSczYvdzNJZ0I5MkFJcDJYdkVNL00wRnZuK01QbjRCMTQ5Q2piaFRCTmVGdlFlWkY3Wmd4b1Q1eWFveFRLOGFBL0YyTG9mQkt4RzF0akxyeDc5OXFwQ0JWZ2d4bkFPcVErT1dJTUFkVjZjYTFkNUFtUm5ybHdISUMzV3FkL1FwTGNBTTdTL3pvSU41Rk5VQm1WOXlMcG5FR2pVVVJLYU1Fa05xZld1MFVvZVZoaHVPY1hwdVZSR01zZENTMXR1amc2QmJYcXJWS0U5K05uZHd3UVVRa0h3R0tKcGlOa2VXWWIyYjJENlp4WmZ5VGpKMHRqc21jZGZmcTlNL05DRlJqRDZzNTJaQmpxZEFYR1VVTnBabWFQSWMvNk4yZ21BNWZlTHpyZml2UEhaVEM0K3VqMDI2VFNLRUJsa2RPTTk3N2RPbzBMTGJPL3grdWUzRXZOWTZVa1N1ek9kQktRNWVuNTJwVXBQUmd6RTVvWUI1WFlHMVJZc0xaVkx6di9lNzNmaHROemtvVFpkdGdiK2FSWUdXbzczZHoxUkM3NnJ1NTlvdlAwSE9YNkU1bXRtTm1lWnhRU1BvRDVQamRxUzhYQWQ5K1RwMDBBRGdncjV3M1M5MkFWVFFqL2RncHc9IiwKICAiQVdTX0xBTUJEQV9MT0dfR1JPVVBfTkFNRSI6ICIvYXdzL2xhbWJkYS9teS10ZXN0LWxhbWJkYS1ub2RlanMtZnVuY3Rpb24iLAogICJMRF9MSUJSQVJZX1BBVEgiOiAiL3Zhci9sYW5nL2xpYjovbGliNjQ6L3Vzci9saWI2NDovdmFyL3J1bnRpbWU6L3Zhci9ydW50aW1lL2xpYjovdmFyL3Rhc2s6L3Zhci90YXNrL2xpYjovb3B0L2xpYiIsCiAgIkxBTUJEQV9UQVNLX1JPT1QiOiAiL3Zhci90YXNrIiwKICAiQVdTX0xBTUJEQV9SVU5USU1FX0FQSSI6ICIxMjcuMC4wLjE6OTAwMSIsCiAgIkFXU19MQU1CREFfTE9HX1NUUkVBTV9OQU1FIjogIjIwMjIvMDkvMTgvWyRMQVRFU1RdNzNkZWI3NGRkOGI1NDczNzhkNzQxOGQwMjVkZGE2NzAiLAogICJBV1NfRVhFQ1VUSU9OX0VOViI6ICJBV1NfTGFtYmRhX25vZGVqczE2LngiLAogICJBV1NfTEFNQkRBX0ZVTkNUSU9OX05BTUUiOiAibXktdGVzdC1sYW1iZGEtbm9kZWpzLWZ1bmN0aW9uIiwKICAiQVdTX1hSQVlfREFFTU9OX0FERFJFU1MiOiAiMTY5LjI1NC43OS4xMjk6MjAwMCIsCiAgIlBBVEgiOiAiL3Zhci9sYW5nL2JpbjovdXNyL2xvY2FsL2JpbjovdXNyL2Jpbi86L2Jpbjovb3B0L2JpbiIsCiAgIkFXU19ERUZBVUxUX1JFR0lPTiI6ICJ1cy1lYXN0LTEiLAogICJQV0QiOiAiL3Zhci90YXNrIiwKICAiQVdTX1NFQ1JFVF9BQ0NFU1NfS0VZIjogIlBXOXRrQWlibFNjQXJpUG14aHRNVWlrWVFvUmtLRDdGdmtLSG5pdGoiLAogICJMQU1CREFfUlVOVElNRV9ESVIiOiAiL3Zhci9ydW50aW1lIiwKICAiTEFORyI6ICJlbl9VUy5VVEYtOCIsCiAgIkFXU19MQU1CREFfSU5JVElBTElaQVRJT05fVFlQRSI6ICJvbi1kZW1hbmQiLAogICJOT0RFX1BBVEgiOiAiL29wdC9ub2RlanMvbm9kZTE2L25vZGVfbW9kdWxlczovb3B0L25vZGVqcy9ub2RlX21vZHVsZXM6L3Zhci9ydW50aW1lL25vZGVfbW9kdWxlczovdmFyL3J1bnRpbWU6L3Zhci90YXNrIiwKICAiVFoiOiAiOlVUQyIsCiAgIkFXU19SRUdJT04iOiAidXMtZWFzdC0xIiwKICAiQVdTX0FDQ0VTU19LRVlfSUQiOiAiQVNJQVdYRlhBSU9aTVlUVDZBTUciLAogICJTSExWTCI6ICIwIiwKICAiX0FXU19YUkFZX0RBRU1PTl9BRERSRVNTIjogIjE2OS4yNTQuNzkuMTI5IiwKICAiX0FXU19YUkFZX0RBRU1PTl9QT1JUIjogIjIwMDAiLAogICJBV1NfWFJBWV9DT05URVhUX01JU1NJTkciOiAiTE9HX0VSUk9SIiwKICAiX0hBTkRMRVIiOiAiaW5kZXguaGFuZGxlciIsCiAgIkFXU19MQU1CREFfRlVOQ1RJT05fTUVNT1JZX1NJWkUiOiAiMTI4IiwKICAiTk9ERV9FWFRSQV9DQV9DRVJUUyI6ICIvZXRjL3BraS90bHMvY2VydHMvY2EtYnVuZGxlLmNydCIsCiAgIl9YX0FNWk5fVFJBQ0VfSUQiOiAiUm9vdD0xLTYzMjc1OWZlLTA3MTdkNTc0MTVhM2NlNjMzNjljN2I0MztQYXJlbnQ9MTM4NGI3NzMxNmU3MTcyZDtTYW1wbGVkPTAiCn0KMjAyMi0wOS0xOFQxNzo0ODo0Ni4wOTdaCWZiYzEwYmYwLTJlZmQtNGRhNy05MDRjLWE0YTgxMDJjMDUzMwlJTkZPCUVWRU5UCnt9CkVORCBSZXF1ZXN0SWQ6IGZiYzEwYmYwLTJlZmQtNGRhNy05MDRjLWE0YTgxMDJjMDUzMwpSRVBPUlQgUmVxdWVzdElkOiBmYmMxMGJmMC0yZWZkLTRkYTctOTA0Yy1hNGE4MTAyYzA1MzMJRHVyYXRpb246IDEuNDggbXMJQmlsbGVkIER1cmF0aW9uOiAyIG1zCU1lbW9yeSBTaXplOiAxMjggTUIJTWF4IE1lbW9yeSBVc2VkOiA1NyBNQgkK",
"ExecutedVersion": "$LATEST"
} Or:
|
I had to upgrade
Then the upgrade with:
Then I ran this:
{
"FunctionUrl": "https://wtqdq25qylflndenvxzlssyhmi0zksbf.lambda-url.us-east-1.on.aws/",
"FunctionArn": "arn:aws:lambda:us-east-1:462092780466:function:my-test-lambda-nodejs-function",
"AuthType": "NONE",
"CreationTime": "2022-09-18T17:55:55.359524Z"
} |
https://wtqdq25qylflndenvxzlssyhmi0zksbf.lambda-url.us-east-1.on.aws/ returns: {
"Message": "Forbidden"
} Probably because I need to deploy a function that actually handles HTTP requests. That's the next step - I'll likely switch to Python at this point too. |
Generated this with GPT-3:
|
|
Still getting that Forbidden error from https://wtqdq25qylflndenvxzlssyhmi0zksbf.lambda-url.us-east-1.on.aws/ - spotted this HTTP header:
I thought this would fix that:
|
On https://us-east-1.console.aws.amazon.com/lambda/home?region=us-east-1#/functions/my-test-lambda-nodejs-function?tab=configure found this:
|
Another tutorial: https://docs.aws.amazon.com/lambda/latest/dg/urls-tutorial.html That tutorial includes this step (I added
Response: {
"Statement": "{\"Sid\":\"url\",\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"lambda:InvokeFunctionUrl\",\"Resource\":\"arn:aws:lambda:us-east-1:462092780466:function:my-test-lambda-nodejs-function\",\"Condition\":{\"StringEquals\":{\"lambda:FunctionUrlAuthType\":\"NONE\"}}}"
} Decoded:
{
"Sid": "url",
"Effect": "Allow",
"Principal": "*",
"Action": "lambda:InvokeFunctionUrl",
"Resource": "arn:aws:lambda:us-east-1:462092780466:function:my-test-lambda-nodejs-function",
"Condition": {
"StringEquals": {
"lambda:FunctionUrlAuthType": "NONE"
}
}
} And now this works! https://wtqdq25qylflndenvxzlssyhmi0zksbf.lambda-url.us-east-1.on.aws/ |
OK let's try a Python one instead. I'll call that Trying this as def lambda_handler(event, context):
return {
"statusCode": 200,
"headers": {
"Content-Type": "text/html"
},
"body": "<h1>Hello World</h1>"
}
|
Reply: {
"FunctionName": "my-test-lambda-python-function",
"FunctionArn": "arn:aws:lambda:us-east-1:462092780466:function:my-test-lambda-python-function",
"Runtime": "python3.9",
"Role": "arn:aws:iam::462092780466:role/lambda-ex",
"Handler": "lambda_function.lambda_handler",
"CodeSize": 323,
"Description": "",
"Timeout": 3,
"MemorySize": 128,
"LastModified": "2022-09-18T19:11:15.900+0000",
"CodeSha256": "TfPlhn/OBnr7Pn3zb3rm8FTMKWUD92XG1YsRGfvkjvU=",
"Version": "$LATEST",
"TracingConfig": {
"Mode": "PassThrough"
},
"RevisionId": "afb3a3e0-5187-4a2b-ab52-a2e0232f27c5",
"State": "Pending",
"StateReason": "The function is being created.",
"StateReasonCode": "Creating",
"PackageType": "Zip",
"Architectures": [
"x86_64"
],
"EphemeralStorage": {
"Size": 512
}
} I tried removing
I got |
Next:
https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/ seems to work now - but it's giving me the same response as the JavaScript one did. |
Update that to: def lambda_handler(event, context):
return {
"statusCode": 200,
"headers": {
"Content-Type": "text/html"
},
"body": "<h1>Hello World from Python</h1>"
} Then re-deploy with:
That worked! https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/ now returns "Hello World from Python". |
OK, I'm going to try deploying Datasette(!). |
OMG that worked! https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/
#!/bin/bash
python3 -m pip install -t lib -r requirements.txt
rm -f lambda_function.zip
(cd lib; zip ../lambda_function.zip -r .)
# Now add my code
zip lambda_function.zip lambda_function.py Then deploy like this:
|
I bundled
Then modified my import mangum
from datasette.app import Datasette
ds = Datasette(["fixtures.db"])
lambda_handler = mangum.Mangum(ds.app()) Then this to over-write the zipped version of that file with the new one:
Then deploy:
And now: https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/fixtures/sortable |
Note that I'm not calling |
Mangum class constructor is useful: https://github.com/jordaneremieff/mangum/blob/c58e85745d6bd7c9cf8734398750e179751b716d/mangum/adapter.py#L30-L47 It tries these handlers in turn: HANDLERS: List[Type[LambdaHandler]] = [
ALB,
HTTPGateway,
APIGateway,
LambdaAtEdge,
] |
It looks like it sets the original AWS event and context inside the ASGI scope: https://github.com/jordaneremieff/mangum/blob/c58e85745d6bd7c9cf8734398750e179751b716d/mangum/handlers/api_gateway.py#L195-L196 "aws.event": self.event,
"aws.context": self.context, |
I'm going to build |
Another clue in https://github.com/aws-samples/aws-lambda-layer-create-script #!/bin/bash
if [ "$1" != "" ] || [$# -gt 1]; then
echo "Creating layer compatible with python version $1"
docker run -v "$PWD":/var/task "lambci/lambda:build-python$1" /bin/sh -c "pip install -r requirements.txt -t python/lib/python$1/site-packages/; exit"
zip -r layer.zip python > /dev/null
rm -r python
echo "Done creating layer!"
ls -lah layer.zip
else
echo "Enter python version as argument - ./createlayer.sh 3.6"
fi |
This issue:
|
I like the look of https://github.com/hmrc/grant-ssh-access/blob/b7515aa1933723855c269c381f97794fbcdc7738/Jenkinsfile#L20
Where
|
So could I do this?
Trying that now - I deleted my |
That might have worked! |
Ran this:
And |
https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/-/versions now shows: {
"python": {
"version": "3.9.13",
"full": "3.9.13 (main, Jun 10 2022, 16:49:31) \n[GCC 7.3.1 20180712 (Red Hat 7.3.1-15)]"
},
"datasette": {
"version": "0.62"
},
"asgi": "3.0",
"uvicorn": "0.18.3",
"sqlite": {
"version": "3.39.2",
"fts_versions": [
"FTS5",
"FTS4",
"FTS3"
],
"extensions": {
"json1": null
},
"compile_options": [
"ATOMIC_INTRINSICS=1",
"COMPILER=gcc-6.3.0 20170516",
"DEFAULT_AUTOVACUUM",
"DEFAULT_CACHE_SIZE=-2000",
"DEFAULT_FILE_FORMAT=4",
"DEFAULT_JOURNAL_SIZE_LIMIT=-1",
"DEFAULT_MMAP_SIZE=0",
"DEFAULT_PAGE_SIZE=4096",
"DEFAULT_PCACHE_INITSZ=20",
"DEFAULT_RECURSIVE_TRIGGERS",
"DEFAULT_SECTOR_SIZE=4096",
"DEFAULT_SYNCHRONOUS=2",
"DEFAULT_WAL_AUTOCHECKPOINT=1000",
"DEFAULT_WAL_SYNCHRONOUS=2",
"DEFAULT_WORKER_THREADS=0",
"ENABLE_FTS3",
"ENABLE_FTS3_PARENTHESIS",
"ENABLE_FTS4",
"ENABLE_FTS5",
"ENABLE_LOAD_EXTENSION",
"ENABLE_MATH_FUNCTIONS",
"ENABLE_RTREE",
"ENABLE_STAT4",
"ENABLE_UPDATE_DELETE_LIMIT",
"MALLOC_SOFT_LIMIT=1024",
"MAX_ATTACHED=10",
"MAX_COLUMN=2000",
"MAX_COMPOUND_SELECT=500",
"MAX_DEFAULT_PAGE_SIZE=8192",
"MAX_EXPR_DEPTH=1000",
"MAX_FUNCTION_ARG=127",
"MAX_LENGTH=1000000000",
"MAX_LIKE_PATTERN_LENGTH=50000",
"MAX_MMAP_SIZE=1099511627776",
"MAX_PAGE_COUNT=1073741823",
"MAX_PAGE_SIZE=65536",
"MAX_SQL_LENGTH=1000000000",
"MAX_TRIGGER_DEPTH=1000",
"MAX_VARIABLE_NUMBER=250000",
"MAX_VDBE_OP=250000000",
"MAX_WORKER_THREADS=0",
"MUTEX_PTHREADS",
"SOUNDEX",
"SYSTEM_MALLOC",
"TEMP_STORE=3",
"THREADSAFE=1",
"USE_URI"
]
},
"pysqlite3": "0.4.7.post7"
} And I'm not getting that parameter error any more, which I think was caused by the old SQLite version. |
https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/fixtures/compound_three_primary_keys is returning "Internal Server Error" though. |
Logs: Maybe there's a 3s timeout that breaks for some reason? |
|
Trying this (guessed by Copilot):
After that first hit to https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/fixtures/compound_three_primary_keys seemed to take about 7s - subsequent hits were faster. |
I wonder if giving it more RAM would result in faster responses, so I wouldn't need the 10s timeout? |
Relevant info from
I'm going to try |
https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/fixtures/facetable feels a bit happier.
Now it feels positively snappy - might be imagining it a bit though. I wonder how much this really does affect the cost? |
The output of {
"FunctionName": "my-test-lambda-python-function",
"FunctionArn": "arn:aws:lambda:us-east-1:462092780466:function:my-test-lambda-python-function",
"Runtime": "python3.9",
"Role": "arn:aws:iam::462092780466:role/lambda-ex",
"Handler": "lambda_function.lambda_handler",
"CodeSize": 9316796,
"Description": "",
"Timeout": 3,
"MemorySize": 512,
"LastModified": "2022-09-18T20:36:25.000+0000",
"CodeSha256": "84Vc6tqX6VTWBdSh5zWiV7RgFTD98/vWf7ByykRKQG0=",
"Version": "$LATEST",
"TracingConfig": {
"Mode": "PassThrough"
},
"RevisionId": "d8a6d6c0-3cda-4c4e-ab56-e8e7a05027c0",
"State": "Active",
"LastUpdateStatus": "InProgress",
"LastUpdateStatusReason": "The function is being created.",
"LastUpdateStatusReasonCode": "Creating",
"PackageType": "Zip",
"Architectures": [
"x86_64"
],
"EphemeralStorage": {
"Size": 512
}
} The |
Also interesting:
Might be worth researching the option of publishing a Datasette layer somewhere. |
Datasette serving off EFS remains very interesting indeed. Not sure how I'd ensure multiple lambdas didn't try to write at the same time, but could be good for scaling reads against large DBs. |
So you can request up to 10GB of ephemeral storage. |
You can pass |
You can pass environment variables to
|
I think that's everything I need to know in order to build a |
Initial research and proof of concept for the
datasette-publish-lambda
plugin.The text was updated successfully, but these errors were encountered: