Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deploying on OpenShift leads to a Quay error for the first deployment #15415

Closed
gsmet opened this issue Mar 2, 2021 · 19 comments
Closed

Deploying on OpenShift leads to a Quay error for the first deployment #15415

gsmet opened this issue Mar 2, 2021 · 19 comments
Assignees
@iocanel
Labels
area/kubernetes kind/bug Something isn't working

Comments

@gsmet
Copy link
Member

gsmet commented Mar 2, 2021 •
edited
Loading

Nearly every time I deploy my Quarkus application to an OpenShift cluster using the OpenShift extension, I get the following error on the OpenShift side for my first attempt:

Receiving source from STDIN as archive ...
Pulling image "quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:d9a59f5d84de0d01a10aa77a0f3ef03533e2651a496854bfbaf86797d7694577" ...
pulling image error : manifest for quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:d9a59f5d84de0d01a10aa77a0f3ef03533e2651a496854bfbaf86797d7694577 not found
error: build error: unable to get quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:d9a59f5d84de0d01a10aa77a0f3ef03533e2651a496854bfbaf86797d7694577

The second attempt succeeds with:

Receiving source from STDIN as archive ...
Pulling image "quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:2d00427fc97d217a279e54d610c2e35da5a2c1ec9c410dce41e5203f35a05281" ...
Using quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:2d00427fc97d217a279e54d610c2e35da5a2c1ec9c410dce41e5203f35a05281 as the s2i builder image

As you can see, the hash is not the same. As the Quarkus images are rebuilt often, and I don't deploy that often, maybe that could explain things.

I deploy my application with:

mvn clean package -Dquarkus.kubernetes.deploy=true -Dquarkus.native.container-build=true -Dnative

and I use the quarkus-openshift extension.

Another issue slightly related, I have absolutely no feedback of the error: the Maven build is green:

[INFO] Checking for existing resources in: /data/home/gsmet/git/quarkus-status/src/main/kubernetes.
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Performing openshift binary build with native image on server: https://api.pro-us-east-1.openshift.com:443/ in namespace:quarkusio.
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: ImageStream quarkus-status
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Found: ImageStream ubi-quarkus-native-binary-s2i repository: quay.io/quarkus/ubi-quarkus-native-binary-s2i
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Applied: BuildConfig quarkus-status
[INFO] [io.quarkus.container.image.openshift.deployment.OpenshiftProcessor] Receiving source from STDIN as archive ...
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Deploying to openshift server: https://api.pro-us-east-1.openshift.com:443/ in namespace: quarkusio.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: Service quarkus-status.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: ImageStream quarkus-status.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: ImageStream ubi-quarkus-native-binary-s2i.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: BuildConfig quarkus-status.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: DeploymentConfig quarkus-status.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] Applied: Route quarkus-status.
[INFO] [io.quarkus.kubernetes.deployment.KubernetesDeployer] The deployed application can be accessed at: http://status.quarkus.io
[INFO] [io.quarkus.deployment.QuarkusAugmentor] Quarkus augmentation completed in 118186ms

The error is only visible in the OpenShift UI.

@iocanel this is the issue we discussed on Zulip the other day. You wanted a GH issue, here we go :).
@gsmet gsmet added the kind/bug Something isn't working label Mar 2, 2021
@quarkus-bot
Copy link

quarkus-bot bot commented Mar 2, 2021

/cc @geoand

@snowdrop-bot snowdrop-bot mentioned this issue Mar 2, 2021
Open
@iocanel
Copy link
Contributor

iocanel commented Mar 3, 2021

@iocanel this is the issue we discussed on Zulip the other day. You wanted a GH issue, here we go :).
Mercy Monsieur!

@maxandersen
Copy link
Member

@gsmet what version of OpenShift are you doing this on ?

@gsmet
Copy link
Member Author

gsmet commented Apr 12, 2021

So this was the old error with OpenShift 3. The problem with 4 is slightly different: it complains about manifest.

Unfortunately, for my last deployment, I didn't get the error in the console (dunno why) and the build pod in OpenShift is gone. I'll take note of the error next Sunday.

@maxandersen
Copy link
Member

gotcha - just to check you are using quarkus mvn command to do the deploy,right ?

i have a hunch what could be causing it. let me check.

@maxandersen
Copy link
Member

okey so just for my own sanity to keep track of what happens:

git clone https://github.com/quarkusio/status.quarkus.io
mvn clean package -Dquarkus.native.container-build=true -Dnative

will generate a target/kubernets/openshift.yml that refer to https://quay.io/repository/quarkus/ubi-quarkus-native-binary-s2i?tab=tags using

strategy:
    sourceStrategy:
      from:
        kind: ImageStreamTag
        name: ubi-quarkus-native-binary-s2i:1.0

which definitely gets bumped every week.

i'm a bit confused why that gives an error since the image stream policy should be be IfNotPresent by default meaning it should just stay on the old unless one explicitly updates.

@maxandersen
Copy link
Member

its as if openshift does actually use the cached sha but then remotely fetches the manifest which of course wont exist anymore ...kinda beating the whole purpose of imagestreams.

I must be missing something here.

@gsmet
Copy link
Member Author

gsmet commented Apr 12, 2021

Yes I use the OpenShift extension and a simple mvn clean package -Dquarkus.kubernetes.deploy=true -Dquarkus.native.container-build=true -Dnative.

You might be on the right path as IIRC the message complains about the manifest not being found.

In this very case, I think we would want to upgrade to the latest version as we are using floating tags to push security fixes.

@maxandersen
Copy link
Member

I wonder if a "fix" is to do as documented in https://docs.openshift.com/container-platform/4.7/openshift_images/image-streams-manage.html#images-imagestreams-update-tag_image-streams-managing

oc tag quay.io/quarkus/ubi-quarkus-native-binary-s2i:1.0 ubi-quarkus-native-binary-s2i

to force an update or even have it scheduled:

oc tag quay.io/quarkus/ubi-quarkus-native-binary-s2i:1.0 ubi-quarkus-native-binary-s2i --scheduled

note: these are workarounds to the problem and I don't grok why there is a pull of images that the cluster already knows...that should not happen.

@maxandersen
Copy link
Member

You might be on the right path as IIRC the message complains about the manifest not being found.

yes, but if you used the same cluster to push previously why is it fetching manifest to check...it will quite often fail on a floating tag...so there is either some bug in how openshift does this or we configured it wrong :)

@maxandersen
Copy link
Member

long shot: could there be a policy on the cluster pruning the images somehow ? causing the cache to be partially fullfilled -i.e. sha1 is there but image gone, so it tries pulling the manifest using the sha rather than the tag?

@gsmet
Copy link
Member Author

gsmet commented Apr 19, 2021 •
edited
Loading

@maxandersen so I can confirm I get the error every time the image is refreshed.

Here is the error I get:

Caching blobs under "/var/cache/blobs".
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
Warning: Pull failed, retrying in 5s ...
error: build error: After retrying 2 times, Pull image still failed due to error: while pulling "docker://quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:f89fdf909b7378c1a69802a8b15539ee71da517b79918b21cac71085754a0c63" as "quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:f89fdf909b7378c1a69802a8b15539ee71da517b79918b21cac71085754a0c63": Error initializing source docker://quay.io/quarkus/ubi-quarkus-native-binary-s2i@sha256:f89fdf909b7378c1a69802a8b15539ee71da517b79918b21cac71085754a0c63: Error reading manifest sha256:f89fdf909b7378c1a69802a8b15539ee71da517b79918b21cac71085754a0c63 in quay.io/quarkus/ubi-quarkus-native-binary-s2i: manifest unknown: manifest unknown

I fix the issue by executing the following command:

oc delete is ubi-quarkus-native-binary-s2i

and then restart the deployment.

BTW @iocanel , I used to have the error in the terminal and now I have to go to the OpenShift console to get the error.

I get this in the terminal:

[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.13.2.Final:build (default) on project quarkus-bot: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] 	[error]: Build step io.quarkus.container.image.openshift.deployment.OpenshiftProcessor#openshiftBuildFromNative threw an exception: java.lang.IllegalStateException: Build:quarkus-bot-1 failed! Generic Build failure - check logs for details.
[ERROR] 	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuild(OpenshiftProcessor.java:465)
[ERROR] 	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.lambda$openshiftBuild$10(OpenshiftProcessor.java:410)
[ERROR] 	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
[ERROR] 	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
[ERROR] 	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
[ERROR] 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
[ERROR] 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
[ERROR] 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
[ERROR] 	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
[ERROR] 	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
[ERROR] 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[ERROR] 	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
[ERROR] 	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuild(OpenshiftProcessor.java:410)
[ERROR] 	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.createContainerImage(OpenshiftProcessor.java:352)
[ERROR] 	at io.quarkus.container.image.openshift.deployment.OpenshiftProcessor.openshiftBuildFromNative(OpenshiftProcessor.java:317)

@geoand
Copy link
Contributor

geoand commented Jul 26, 2021

Is this still an issue or can we close it?

@maxandersen
Copy link
Member

i haven't verified it is gone - last i checked it still happened but been a while ;/

@geoand
Copy link
Contributor

geoand commented Sep 30, 2021

@iocanel one of your latest fixes should take care of this, right?

@iocanel
Copy link
Contributor

iocanel commented Sep 30, 2021

@iocanel one of your latest fixes should take care of this, right?

Not intentionally no!

@geoand
Copy link
Contributor

geoand commented Sep 30, 2021

Okay then, let's leave it open

@geoand
Copy link
Contributor

geoand commented Feb 29, 2024

Is this still an issue with Quarkus 3.8?

@geoand
Copy link
Contributor

geoand commented Sep 13, 2024

Let's close for now

@geoand geoand closed this as completed Sep 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iocanel iocanel

Labels
area/kubernetes kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@maxandersen @iocanel @gsmet @geoand