-
Notifications
You must be signed in to change notification settings - Fork 560
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SAM local building response properly, but Internal Server Error when deployed #264
Comments
Thanks @rickbhardwaj, I'll try to replicate with your code and let you know. To be clear, you don't see any exception, the code just hangs until the Lambda function times out? |
The Lambda logs in the issue only cover 2seconds of an execution. Can you attach a more complete log? Which version of Jersey are you using? |
Hi @sapessi , I think this is the complete log. But it's the same takeaway- some startup logging, run until time out, and ends execution.
|
@sapessi I'm using jersey version 2.26 |
Looks like you are using Spring. Why use it inside Jersey and not go straight for the Spring implementation of this framework? |
Some extra context: here is my sam.yaml
and I am invoking the API Gateway with https://******.execute-api.us-west-2.amazonaws.com/Stage/configurations |
@sapessi I'm using Spring mostly just for dependency injection and caching. I'm using jersey for the actual REST/JAX-RS implementation. |
I've never tried the two together. I can take a look into it. Out of curiosity, why Spring instead of the built-in hk2? My recommendation for Lambda normally is Dagger2 for performance reasons, although I know it doesn't integrate nicely with Jersey. Spring is really slow to initialize inside Lambda. |
Yeah I saw your comments here. Mostly I'm trying to write a library for my team to easily port our existing services to Lambda, and this is our setup. Asking them to rewrite their applications to switch from spring to dagger will mean nobody will use my library haha. |
Fair, is there any code you can share with me so that I can try and replicate with your code? |
Sure, I'm using this on a toy service built with our setup, I can share most of it. You won't be able to build it because it depends on some libraries in our maven repo, but I think you can see how we're approaching it. Just to confirm, have other users reported any issues with the jersey library since the execution environment upgrade? |
No, this is the first time I see it. some code is better than nothing. |
Sorry for the delay @sapessi. After seeing this thread, I realized I hadn't tried turning on the lazyInit flag on my Spring @ComponentScans. I deployed the function and it started working very quickly. I thought I had solved it, but then I made another minor change and deployed again- back to not working. I'll share the code in a second. |
Sounds good. My guess is that you are running into the 10 seconds limit for the init time. Lambda allows a max of 10 seconds for your handler class to instantiate, if you take longer than that, it will assume the sandbox is dead and discard it. It is possible that behind the scenes spring is taking a long time to initialize without logging anything useful for you. I had planned to address this in 1.4 by creating an async initializer that runs in a separate thread for a max of 10 seconds, then releases the latch for the constructor to complete. Another latch then holds the handler execution until the initialization is actually complete. |
That sounds very likely, and would explain why it runs on sam local and suddenly started working after a lazyInit flag was enabled. The first execution took 500 ms, but the subsequent ones took ~5ms before my next deployment. |
Here, this is basically what I'm building. Important classes are: StreamLambdaHandler, RoutingConfigurationApi, RoutingConfigurationActivity |
I'm going to try to get rid of the component scan call. |
I tried using the spring-context-indexer and the deployed lambda function began working. However, I'm not sure if this is just luck on Spring init and will revert back at some point. |
It is, you'll be safer by slightly increasing the memory allocated to the function - however, still unpredictable. I would say move the init code to the handler method (lazy load) so that Lambda will continue no matter what (although your first invocation will be even slower). I will keep this issue open for the implementation of the async init in version 1.4 of the framework. |
The first successful invocation took 880 ms. If we're right about this being part of the ten second limit on init, that is close. I'll try increasing the memory and see if I can move the init to the handler (not completely sure, there might be circular dependency since I need spring initialized in the handler.) |
Hi @rickbhardwaj, better late than never we have a fix for this about to go out in the next release. See my comment in this other issue. |
Release 1.4 is out on maven central. The release notes include a list of all the changes. Closing this issue. |
Your response should be like |
Scenario
I built a REST service using Jersey serverless container. Previously, sam local and the deployed lambda function had identical behavior.
Now, however, execution works against sam local (version 0.17.0,) albeit quite a bit slower than before, but does not execute when deployed to Lambda- it simply initializes the container, runs until timeout, and terminates the function. (perhaps related to the execution env upgrade?)
Logs attached:
Expected behavior
Invocation on sam local:
Actual behavior
Invocation on deployed Lambda function:
{"message": "Internal server error"}
Full log output
From CloudWatch logs of deployed Lambda function:
Steps to reproduce
My StreamLambdaHandler class:
The text was updated successfully, but these errors were encountered: