The incoming event in an AWS Lambda Function is always null when the first invocation runs (cold start), but is populated in subsequent invocations (warm start)

[tristann9]

Describe the bug

The issue is present when running the function in a native build only when a new lambda container instance is started (e.g. A new output to the CloudWatch logs), but goes away when more events come in and the same lambda container instance is re-used for subsequent invocations. The issue is NOT present when deploying using the Java11 runtime.

The issue has been noticed when subscribing a Lambda function to an SNS topic and when subscribing to an SQSQueue, which in turn is subscribed to an SNS topic.

I haven't crossed checked, but I believe this issue hasn't been experienced in previous deployments of Quarkus (1.11.3.Final and 2.1.3.Final) since a similar setup was used when deploying in native mode.

Expected behavior

The incoming event to be populated on all invocations without having to have a warmed up lambda instance.

Actual behavior

Cloud watch logs:

In native, when cold starting a lambda request

INFO  [io.quarkus] bot-notification 1.0.0-SNAPSHOT native (powered by Quarkus 2.3.0.Final) started in 0.252s.
INFO  [io.quarkus] Profile prod activated.
INFO  [io.quarkus] Installed features: [amazon-lambda, amazon-sqs, cdi, resteasy-jsonb, smallrye-context-propagation, vertx]
INFO  [BotNotificationHandler] Processing request: null

In native, when warm starting a lambda request. Also cold/warm start when using the Java11 runtime.

INFO  [io.quarkus] bot-notification 1.0.0-SNAPSHOT on JVM (powered by Quarkus 2.3.0.Final) started in 2.870s.
INFO  [io.quarkus] Profile prod activated.
INFO  [io.quarkus] Installed features: [amazon-lambda, amazon-sqs, cdi, resteasy-jsonb, smallrye-context-propagation, vertx]
INFO  [BotNotificationHandler] Processing request: {"records":[{"attribut......

How to Reproduce?

The source code is in a private repository so I can't provide a link for it, but I've included relevant snippets in the additional info section. Any more details are required let me know and I'll be glad to get them for you.

Steps to reproduce:

1. Build AWS Lambda function in native mode and deploy
2. Subscribe a lambda function to an SNS topic or SQS queue
3. Submit a topic/message
4. Watch the CloudWatch logs, the incoming request event is null.
5. Wait a couple of minutes for the Lambda container to shut-off to force a cold start in the next invocation
6. Submit a topic/message
7. Watch the CloudWatch logs, the incoming request event is null.
8. Submit a topic/message shortly after looking at the logs, the previous lambda container should still be accepting new requests
9. Watch the CloudWatch logs, the incoming request event is populated

Output of uname -a or ver

x86_64 GNU/Linux (AWS Lambda)

Output of java -version

java 11.0.11 2021-04-20 LTS

GraalVM version (if different from Java)

GraalVM EE 21.1.0 (build 11.0.11+9-LTS-jvmci-21.1-b05)

Quarkus version or git rev

2.3.0.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.1

Additional information

Sample Function

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.SQSEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.inject.Inject;
import javax.inject.Named;
import javax.json.bind.Jsonb;
import java.util.Map;

@Named("bot-notification")
public class BotNotificationHandler implements RequestHandler<SQSEvent, String> {

    final static Logger log = LoggerFactory.getLogger(BotNotificationHandler.class.getSimpleName());

    @Inject
    Jsonb jsonb;

    @Override
    public String handleRequest(SQSEvent event, Context context) {
        log.info("Processing request: {}", jsonb.toJson(event));
        return jsonb.toJson(Map.of("statusCode", 200, "body", "Complete"));
    }
}

Maven Snippet

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-amazon-sqs</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-amazon-lambda</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-amazon-lambda-xray</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-arc</artifactId>
        </dependency>
        <dependency>
            <groupId>org.jboss.resteasy</groupId>
            <artifactId>resteasy-multipart-provider</artifactId>
        </dependency>
        <dependency>
            <groupId>org.jboss.logging</groupId>
            <artifactId>commons-logging-jboss-logging</artifactId>
            <version>1.0.0.Final</version>
        </dependency>

        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>url-connection-client</artifactId>
        </dependency>

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-jsonb</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-vertx</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-resteasy-jsonb</artifactId>
        </dependency>

AWS Cloud Formation Snippet

  BotNotificationFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../functions/bot-notification/target/function.zip
      MemorySize: 1024
      Timeout: 25
      #Tracing: Active
      Environment:
        Variables:
          DISABLE_SIGNAL_HANDLERS: true
      Events:
        BotNotificationTopicEvent:
          Type: SNS
          Properties:
            Topic: !Ref BotNotificationTopic
            SqsSubscription:
              QueueArn: !GetAtt BotNotificationQueue.Arn
              QueueUrl: !Ref BotNotificationQueue
              BatchSize: 10
      Policies:
        - AWSXrayWriteOnlyAccess
        - Statement:
            - Effect: 'Allow'
              Resource: 'arn:aws:logs:*:*:*'
              Action:
                - 'logs:*'

[quarkus-bot]

/cc @matejvasek, @patriot1burke

[geoand]

This is a duplicate of #20665