Disabled devservices + quarkus-amazon-lambda breaks @QuarkusTest & Dev mode

[dagrammy]

Describe the bug

Disabled devservices (quarkus.devservices.enabled=false) and usage of 'io.quarkus:quarkus-amazon-lambda' dependency leads to failing @QuarkusTests and failing dev mode.

It seems to be related to the Lambda mock event server because of the following log statement in dev mode & @QuarkusTest
Aborting lambda poll loop: ending dev/test mode

Test execution is sometimes successful. This could also be a timing problem. Dev mode always fails.

Expected behavior

@QuarkusTests are always executed and application is successfully started in dev mode.

Actual behavior

Dev mode: Aborting lambda poll loop: ending dev/test mode

2023-07-17 20:57:25,410 INFO  [io.quarkus] (Quarkus Main Thread) Profile dev activated. Live Coding activated.
2023-07-17 20:57:25,411 INFO  [io.quarkus] (Quarkus Main Thread) Installed features: [amazon-lambda, cdi]
2023-07-17 20:57:25,417 WARN  [io.qua.ama.lam.run.AbstractLambdaPollLoop] (Lambda Thread (DEVELOPMENT)) Aborting lambda poll loop: ending dev/test mode
2023-07-17 20:57:25,421 INFO  [io.quarkus] (Lambda Thread (DEVELOPMENT)) devservices-reproducer stopped in 0.003s

Exception during test execution:

java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.bean(String)" because "<local3>" is null
	at org.acme.MyService_ClientProxy.<init>(Unknown Source)
	at org.acme.MyService_Bean.proxy(Unknown Source)
	at org.acme.MyService_Bean.get(Unknown Source)
	at org.acme.MyService_Bean.get(Unknown Source)
	at org.acme.LambdaHandler_Bean.doCreate(Unknown Source)
	at org.acme.LambdaHandler_Bean.create(Unknown Source)
	at org.acme.LambdaHandler_Bean.create(Unknown Source)
	at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
	at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
	at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
	at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
	at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
	at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
	at org.acme.LambdaHandler_ClientProxy.arc$delegate(Unknown Source)
	at org.acme.LambdaHandler_ClientProxy.handleRequest(Unknown Source)
	at org.acme.LambdaHandlerTest.testHandleRequest(LambdaHandlerTest.java:16)
.....

Click me for full stack trace

java.lang.NullPointerException: Cannot invoke "io.quarkus.arc.ArcContainer.bean(String)" because "" is null at org.acme.MyService_ClientProxy.(Unknown Source) at org.acme.MyService_Bean.proxy(Unknown Source) at org.acme.MyService_Bean.get(Unknown Source) at org.acme.MyService_Bean.get(Unknown Source) at org.acme.LambdaHandler_Bean.doCreate(Unknown Source) at org.acme.LambdaHandler_Bean.create(Unknown Source) at org.acme.LambdaHandler_Bean.create(Unknown Source) at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37) at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34) at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26) at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69) at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34) at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21) at org.acme.LambdaHandler_ClientProxy.arc$delegate(Unknown Source) at org.acme.LambdaHandler_ClientProxy.handleRequest(Unknown Source) at org.acme.LambdaHandlerTest.testHandleRequest(LambdaHandlerTest.java:16) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:1015) at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:829) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:110) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:90) at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:85) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:568) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at jdk.proxy1/jdk.proxy1.$Proxy2.stop(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193) at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65) at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69) at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

How to Reproduce?

see https://github.com/dagrammy/devservices-reproducer to reproduce

for example:

• clone project
• run quarkus dev
• app will terminate immediately (Aborting lambda poll loop: ending dev/test mode)

Output of uname -a or ver

Darwinlocal 22.3.0 Darwin Kernel Version 22.3.0: Thu Jan 5 20:49:43 PST 2023; root:xnu-8792.81.2~2/RELEASE_ARM64_T8103 arm64

Output of java -version

openjdk version "17.0.5" 2022-10-18 OpenJDK Runtime Environment GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08) OpenJDK 64-Bit Server VM GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.2.0.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 8.1.1

Additional information

No response

[quarkus-bot]

/cc @geoand (devservices), @matejvasek (amazon-lambda), @patriot1burke (amazon-lambda), @stuartwdouglas (devservices)

[andrezimmermann]

I was also affected by this, I worked around enabling the "global dev services" and manually disabling all individual dev services.

From my findings, this is the cause of the behavior change. #33836

It ties the test mode lambda poll server feature with the "global dev services"

[dagrammy]

I was also affected by this, I worked around enabling the "global dev services" and manually disabling all individual dev services.

That's what we did too. It's just a bit "noisy".

From my findings, this is the cause of the behavior change. #33836

It ties the test mode lambda poll server feature with the "global dev services"

In my opinion, the mock event server should not be tied to the devservices. I am curious what the community and maintainers think. 🙂

[patriot1burke]

mock event server is a dev service. There's currently no other way to launch it during integration tests. Would have to have something called "Test Services".

[dagrammy]

However, the mock server only became a devservice with version 3.2.0.Final. See also the commit linked by @andrezimmermann above.

I see the mock event server as an integral part of testing a Quarkus Lambda or running it in dev mode. Without the mock event server, no Lambda will work in a non-AWS environment.

Other services, such as AWS services or a keycloak, can be configured and added locally or in a CI pipeline relatively easily.

For example, we disabled devservices in a gitlab CI, as the environment does not allow containers to be started using testcontainer. Therefore, we configure so-called services that are used for integration tests.

[gsmet]

We had another person complaining about this here: #34822 .

[parasjain27031994]

Any updates on this issue or is this the way going forward and we would need to disable dev services at a granular level?

[evc-erik]

I just ran into this issue when I bumped up to 3.2.3.Final. In my case, it's exacerbated by the fact that the cognito-user-pools dev services cannot be selectively disabled. It looks like the corresponding build step doesn't pay attention to the quarkus.cognito-user-pools.devservices.enabled property.

Should be an easy fix and I'll take a look when I have some time.

Maybe what we really need is a better way to do fine-grained enablement of dev services? I imagine in many cases it's preferable to explicitly enable the services you want instead of disabling all the ones you don't. Especially if existing features decide to unexpectedly opt-in to the dev services umbrella like the lambda poller 😉

[madocx]

I think that's an excellent suggestion. I just ran into this as well when updating to 3.2.3.Final. The current granular disabling of services is extremely tedious. A whitelist is always easier to maintain than a blacklist.

Edit - After implementing the work around I noticed that the implementation already ignores any devservices config values not on the classpath by means of an extension. So I'm not sure how important this is. Probably just need to fix the root cause that breaks dev mode when the global devservices are disabled.

[slinstaedt]

I was also running into this issue while upgrading to 3.2.x. As far as I can tell, this is probably originated by quarkus-internal.aws-lambda.test-api not being set, when quarkus.devservices.enabled=false and environment variable AWS_LAMBDA_RUNTIME_API not being defined, causing an UrlConnection trying to resolve a "null" hostname with the thrown UnknownHostException being disregarded as a "graceful" shutdown of the lambda poll loop.

Manually configuring either quarkus-internal.aws-lambda.test-api=localhost:8080/_lambda_ or export AWS_LAMBDA_RUNTIME_API=localhost:8080/_lambda_ works around this issue.

[slinstaedt]

Correcting myself... this works around the hostname being resolvable again, but still no http server being started at that port.

[patriot1burke]

Sorry everybody...Been on vacation and working on other things. Easy fix incoming. See pull request.

[winks]

Is there any reason why this fix will not be in 3.3 which is not final yet? Because of the workaround?

[geoand]

It will be part of 3.3.1

[evc-erik]

@geoand did this make it into 3.3.1? I don't see it in the release notes...

[geoand]

It should be included