Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenshiftProcessor throws StackOverflowError on 3.0.0.Alpha6 #31872

Closed
yrodiere opened this issue Mar 15, 2023 · 20 comments · Fixed by #31986
Closed

OpenshiftProcessor throws StackOverflowError on 3.0.0.Alpha6 #31872

yrodiere opened this issue Mar 15, 2023 · 20 comments · Fixed by #31986

Comments

@yrodiere
Copy link
Member

Describe the bug

Since 3.0.0.Alpha6, using quarkus.kubernetes.deploy=true leads to a StackOverflowError, then IllegalStateException: Build:quarkus-github-lottery-2 failed! Failed to fetch the input source.

3.0.0.Alpha5 worked fine.

Expected behavior

No StackOverflowError.

Actual behavior

See https://github.com/quarkusio/quarkus-github-lottery/actions/runs/4428450881/jobs/7767890656

[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Starting (in-cluster) container image build for jar using: BINARY on server: ***/ in namespace:prod-quarkus-github-lottery.
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: ImageStream ubi-quarkus-native-binary-s2i
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: ImageStream quarkus-github-lottery
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: BuildConfig quarkus-github-lottery
Error:  Unhandled exception
java.lang.StackOverflowError
    at java.lang.ClassLoader.defineClass1 (Native Method)
    at java.lang.ClassLoader.defineClass (ClassLoader.java:1012)
    at java.security.SecureClassLoader.defineClass (SecureClassLoader.java:150)
    at java.net.URLClassLoader.defineClass (URLClassLoader.java:524)
    at java.net.URLClassLoader$1.run (URLClassLoader.java:427)
    at java.net.URLClassLoader$1.run (URLClassLoader.java:421)
    at java.security.AccessController.doPrivileged (AccessController.java:712)
    at java.net.URLClassLoader.findClass (URLClassLoader.java:420)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClassFromSelf (ClassRealm.java:425)
    at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass (SelfFirstStrategy.java:42)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass (ClassRealm.java:271)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:247)
    at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass (ClassRealm.java:239)
    at org.slf4j.impl.SimpleLogger.formatAndLog (SimpleLogger.java:394)
    at org.slf4j.impl.SimpleLogger.error (SimpleLogger.java:612)
    at io.vertx.core.logging.SLF4JLogDelegate.log (SLF4JLogDelegate.java:205)
    at io.vertx.core.logging.SLF4JLogDelegate.log (SLF4JLogDelegate.java:160)
    at io.vertx.core.logging.SLF4JLogDelegate.error (SLF4JLogDelegate.java:77)
    at io.vertx.core.impl.logging.LoggerAdapter.error (LoggerAdapter.java:94)
    at io.vertx.core.impl.ContextBase.reportException (ContextBase.java:201)
    at io.vertx.core.impl.future.FutureImpl$3.onSuccess (FutureImpl.java:144)
    at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:60)
    at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:196)
    at io.vertx.core.impl.future.PromiseImpl.addListener (PromiseImpl.java:23)
    at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:164)
    at io.vertx.core.impl.future.PromiseImpl.onComplete (PromiseImpl.java:23)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.readChunk (InputStreamReadStream.java:101)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.lambda$readChunk$3 (InputStreamReadStream.java:107)
    at io.vertx.core.impl.future.FutureImpl$3.onSuccess (FutureImpl.java:141)
    at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:60)
    at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:196)
    at io.vertx.core.impl.future.PromiseImpl.addListener (PromiseImpl.java:23)
    at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:164)
    at io.vertx.core.impl.future.PromiseImpl.onComplete (PromiseImpl.java:23)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.readChunk (InputStreamReadStream.java:101)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.lambda$readChunk$3 (InputStreamReadStream.java:107)
    at io.vertx.core.impl.future.FutureImpl$3.onSuccess (FutureImpl.java:141)
    at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:60)
    at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:196)
    at io.vertx.core.impl.future.PromiseImpl.addListener (PromiseImpl.java:23)
    at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:164)
    at io.vertx.core.impl.future.PromiseImpl.onComplete (PromiseImpl.java:23)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.readChunk (InputStreamReadStream.java:101)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.lambda$readChunk$3 (InputStreamReadStream.java:107)
    at io.vertx.core.impl.future.FutureImpl$3.onSuccess (FutureImpl.java:141)
    at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:60)
    at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:196)
    at io.vertx.core.impl.future.PromiseImpl.addListener (PromiseImpl.java:23)
    at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:164)
    at io.vertx.core.impl.future.PromiseImpl.onComplete (PromiseImpl.java:23)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.readChunk (InputStreamReadStream.java:101)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.lambda$readChunk$3 (InputStreamReadStream.java:107)
[... removed lots of stack elements, but I think you got the idea...]
    at io.vertx.core.impl.future.FutureImpl$3.onSuccess (FutureImpl.java:141)
    at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:60)
    at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:196)
    at io.vertx.core.impl.future.PromiseImpl.addListener (PromiseImpl.java:23)
    at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:164)
    at io.vertx.core.impl.future.PromiseImpl.onComplete (PromiseImpl.java:23)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.readChunk (InputStreamReadStream.java:101)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.lambda$readChunk$3 (InputStreamReadStream.java:107)
    at io.vertx.core.impl.future.FutureImpl$3.onSuccess (FutureImpl.java:141)
    at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:60)
    at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:196)
    at io.vertx.core.impl.future.PromiseImpl.addListener (PromiseImpl.java:23)
    at io.vertx.core.impl.future.FutureImpl.onComplete (FutureImpl.java:164)
    at io.vertx.core.impl.future.PromiseImpl.onComplete (PromiseImpl.java:23)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.readChunk (InputStreamReadStream.java:101)
    at io.fabric8.kubernetes.client.vertx.InputStreamReadStream.lambda$readChunk$3 (InputStreamReadStream.java:107)
    at io.vertx.core.impl.future.FutureImpl$3.onSuccess (FutureImpl.java:141)
    at io.vertx.core.impl.future.FutureBase.emitSuccess (FutureBase.java:60)
    at io.vertx.core.impl.future.FutureImpl.addListener (FutureImpl.java:196)
    at io.vertx.core.impl.future.PromiseImpl.addListener (PromiseImpl.java:23)
Error:  Failed to upload archive file for the build: quarkus-github-lottery
Error:  Please check cluster events via `oc get events` to see what could have possibly gone wrong
Error:  Connection was closed
Warning:  [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] An exception: 'Can't instantiate binary build, due to error reading/writing stream. Can be caused if the output stream was closed by the server.See if something's wrong in recent events in Cluster = Scheduled quarkus-github-lottery-2-build.174ca432388bb828 Successfully assigned prod-quarkus-github-lottery/quarkus-github-lottery-2-build to ip-10-0-142-232.us-west-2.compute.internal by ip-10-0-182-138
BuildStarted quarkus-github-lottery-2.174ca4330fc17990 Build prod-quarkus-github-lottery/quarkus-github-lottery-2 is now running
AddedInterface quarkus-github-lottery-2-build.174ca432d17684bb Add eth0 [10.128.2.221/23] from openshift-sdn
Pulled quarkus-github-lottery-2-build.174ca432d3162e2c Container image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e7936be00b487aac810f6c519a0cec1341b3f8065a23b969730f96c1a5c5dd0" already present on machine
Created quarkus-github-lottery-2-build.174ca432dfefdb34 Created container git-clone
Started quarkus-github-lottery-2-build.174ca432e1860a9e Started container git-clone
 ' occurred while instantiating the build, however the build has been started.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  13:42 min
[INFO] Finished at: 2023-03-15T16:26:40Z
[INFO] ------------------------------------------------------------------------
Error:  Failed to execute goal io.quarkus:quarkus-maven-plugin:3.0.0.Alpha6:build (default) on project quarkus-github-lottery: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
Error: [ERROR] 	[error]: Build step io.quarkus.container.image.openshift.deployment.OpenshiftProcessor#openshiftBuildFromNative threw an exception: java.lang.IllegalStateException: Build:quarkus-github-lottery-2 failed! Failed to fetch the input source.
Error:  	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuild(OpenshiftProcessor.java:484)
Error:  	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.lambda$openshiftBuild$10(OpenshiftProcessor.java:424)
Error:  	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
Error:  	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
Error:  	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
Error:  	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
Error:  	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
Error:  	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
Error:  	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
Error:  	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
Error:  	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
Error:  	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
Error:  	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuild(OpenshiftProcessor.java:424)
Error:  	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.createContainerImage(OpenshiftProcessor.java:388)
Error:  	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuildFromNative(OpenshiftProcessor.java:352)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Error:  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
Error:  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Error:  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
Error:  	at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909)
Error:  	at io.quarkus.builder.BuildContext.run(BuildContext.java:282)
Error:  	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
Error:  	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
Error:  	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
Error:  	at java.base/java.lang.Thread.run(Thread.java:833)
Error:  	at org.jboss.threads.JBossThread.run(JBossThread.java:501)
Error:  -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error: Process completed with exit code 1.

How to Reproduce?

I don't have a reproducer but here is the project: https://github.com/quarkusio/quarkus-github-lottery/

Command:

mvn clean package -Dquarkus.kubernetes.deploy=true -Dquarkus.native.container-build=true -Dnative -Drevision=${{ github.sha }}

See https://github.com/quarkusio/quarkus-github-lottery/blob/4a0f8d6655025fffa099396854de89d2fc3c1d73/.github/workflows/deploy.yml#L36-L43

Configuration:

quarkus.application.name=quarkus-github-lottery
quarkus.application.version=${buildNumber:999-SNAPSHOT}

quarkus.qute.suffixes=md
quarkus.qute.content-types."md"=text/markdown

%test.quarkus.scheduler.enabled=false
%test.quarkus.log.category."io.quarkus.github.lottery".level=DEBUG

%dev.quarkus.scheduler.enabled=false
%dev.quarkus.log.category."io.quarkus.github.lottery".level=DEBUG

%prod.quarkus.openshift.labels."app"=quarkus-github-lottery
# renew the SSL certificate automatically
%prod.quarkus.openshift.annotations."kubernetes.io/tls-acme"=true
%prod.quarkus.openshift.env.configmaps=quarkus-github-lottery-config
%prod.quarkus.openshift.env.secrets=quarkus-github-lottery-secrets

See https://github.com/quarkusio/quarkus-github-lottery/blob/4a0f8d6655025fffa099396854de89d2fc3c1d73/src/main/resources/application.properties

Output of uname -a or ver

No response

Output of java -version

No response

GraalVM version (if different from Java)

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

I was affected by #31476 on Alpha4, not sure if this is related.

@yrodiere yrodiere added the kind/bug Something isn't working label Mar 15, 2023
@quarkus-bot
Copy link

quarkus-bot bot commented Mar 15, 2023

/cc @Sgitario (kubernetes), @geoand (kubernetes,openshift), @iocanel (kubernetes,openshift)

@gsmet
Copy link
Member

gsmet commented Mar 15, 2023

/cc @manusa @metacosm as I wonder if it could be a bug in the new version of the Kubernetes Client.

@metacosm
Copy link
Contributor

Looks like a bug in the vert.x http client indeed.

@gsmet
Copy link
Member

gsmet commented Mar 16, 2023

@Sgitario @metacosm @manusa is someone working on this? Because it looks like a blocker to me if we can't deploy simple apps to OpenShift clusters.

@Sgitario
Copy link
Contributor

I tried to reproduce this issue and to me, it's consistently working fine (either using 3.0.0.Alpha6 or 999-SNAPSHOT).
However, I think this change fabric8io/kubernetes-client@2216980 might be related, so maybe @vietj can help here (sorry if it's totally unrelated tho).

@manusa
Copy link
Contributor

manusa commented Mar 16, 2023 via email

@gsmet
Copy link
Member

gsmet commented Mar 16, 2023

@manusa no worries, you shouldn't even have read this thread, go back to your PTO!

@yrodiere
Copy link
Member Author

I tried to reproduce this issue and to me, it's consistently working fine (either using 3.0.0.Alpha6 or 999-SNAPSHOT).

You made me doubt, so I restarted the job, and I'm happy (?) to report that it fails consistently, it's not a random fluke: https://github.com/quarkusio/quarkus-github-lottery/actions/runs/4428450881/jobs/7786953713

@manusa no worries, you shouldn't even have read this thread, go back to your PTO!

+1

@vietj
Copy link

vietj commented Mar 16, 2023

@gsmet here is a patch you can test vietj/kubernetes-client@3b8ee08

@gsmet
Copy link
Member

gsmet commented Mar 16, 2023

@vietj unfortunately I wasn't able to reproduce it when deploying locally. There must be something specific to the GitHub Actions network environment.

We will try to deploy the patch to GitHub Actions somehow and report back.

@shawkins
Copy link
Contributor

@manusa the end warning is similar to errors we see in the e2e tests with vertx. I was also able to reproduce the stack overflow by using a larger POST stream and some artificial wait in the readChunk method - adding System.out.println(new Exception().getStackTrace().length); - was enough to trigger see the growth of the stack and trigger the overflow.

It turns out this is what is happening in the e2e tests - https://pipelines.actions.githubusercontent.com/serviceHosts/15be8e7a-97a9-402c-ad30-bbe66fddae95/_apis/pipelines/1/runs/42329/signedlogcontent/11?urlExpires=2023-03-16T21%3A17%3A01.5488581Z&urlSigningMethod=HMACV1&urlSignature=Uhcd9ka3y1xupK44fdHtaa%2BDa8S%2F8C5CKWYlMcdASXc%3D - it's just that the build exception handling isn't making it obvious to the test class.

So we'll need a fix to prevent the possibility of ever increasing recursion with large streams.

@vietj
Copy link

vietj commented Mar 17, 2023

you can try this patch @gsmet vietj/kubernetes-client@3b8ee08

@gsmet
Copy link
Member

gsmet commented Mar 17, 2023

@shawkins could you also try the patch ^. I'm currently working on pushing the patch to GitHub Actions and see how it goes.

@shawkins
Copy link
Contributor

@gsmet it works for me.

@gsmet
Copy link
Member

gsmet commented Mar 20, 2023

@vietj it also works for us AFAICS. Could you open a PR against the Kubernetes Client?

@manusa @shawkins @metacosm we will need a new release with this fix very soon.

@manusa
Copy link
Contributor

manusa commented Mar 20, 2023

Nice work.

The only changes added to the client's master branch (6.6) are bug-fixes, so as soon as we merge @vietj patch we can release a 6.5.1 version.

@gsmet
Copy link
Member

gsmet commented Mar 20, 2023

@vietj @metacosm @manusa @shawkins if we could merge the Kubernetes Client update in Quarkus by tomorrow evening, that would be ideal as I will build the Beta1.

Let me know how things are going on your side.

Thanks!

@manusa
Copy link
Contributor

manusa commented Mar 20, 2023

Let me try to create the PR by cherry-picking Julien's commit

@vietj
Copy link

vietj commented Mar 20, 2023

@manusa @gsmet the commit should be trivial to cherry-pick indeed

@metacosm
Copy link
Contributor

Thank you, @vietj!

@quarkus-bot quarkus-bot bot added this to the 3.0 - main milestone Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

7 participants