Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] When upgrading from crc 2.33 -> 2.34 OperatorHub shows no items #4109

Open
ruivieira opened this issue Apr 8, 2024 · 6 comments
Open

[BUG] When upgrading from crc 2.33 -> 2.34 OperatorHub shows no items #4109

ruivieira opened this issue Apr 8, 2024 · 6 comments
Labels
kind/bug Something isn't working known-issue Known issue preset/openshift priority/critical

Comments

@ruivieira
Copy link

ruivieira commented Apr 8, 2024 •
edited
Loading

General information

  • OS: Linux
  • Hypervisor: KVM
  • Did you run crc setup before starting it (Yes/No)? Yes
  • Running CRC on: Laptop / Baremetal-Server / VM: Server

CRC version

CRC version: 2.34.1+b470b5
OpenShift version: 4.15.3
Podman version: 4.4.4

CRC status

DEBU CRC version: 2.34.1+b470b5
DEBU OpenShift version: 4.15.3
DEBU Podman version: 4.4.4
DEBU Running 'crc status'
crc does not seem to be setup correctly, have you run 'crc setup'?

NOTE: despite this status, crc seems to be working fine.

CRC config

- consent-telemetry                     : no
- cpus                                  : 8
- disk-size                             : 200
- enable-shared-dirs                    : false
- kubeadmin-password                    : <redacted>
- memory                                : 32000
- pull-secret-file                      : /home/rui/crc/pull-secret.txt

Host Operating System

NAME="Fedora Linux"
VERSION="39 (Workstation Edition)"
ID=fedora
VERSION_ID=39
VERSION_CODENAME=""
PLATFORM_ID="platform:f39"
PRETTY_NAME="Fedora Linux 39 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:39"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=39
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=39
SUPPORT_END=2024-11-12
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

  1. Having an existing crc 2.33, upgrade to 2.34
  2. crc stop ; crc delete --cache
  3. crc setup
  4. crc start

Expected

OperatorHub to be available

Actual

OperatorHub empty with no items.
community-operators, redhat-operators, redhat-marketplace, certified-operators pods crashing with incorrect SHA value errors.

Logs

No logs collected. Errors detect in the cluster, not crc directly.

This was fixed by disabling and renabling the sources, eg.

oc patch operatorhubs/cluster --type merge --patch "'{"spec":{"sources":[{"disabled": true,"name": <operatorname>}]}}'"
# wait
oc patch operatorhubs/cluster --type merge --patch "'{"spec":{"sources":[{"disabled": false,"name": <operatorname>}]}}'"
@ruivieira ruivieira added kind/bug Something isn't working status/need triage labels Apr 8, 2024
@praveenkumar praveenkumar pinned this issue Apr 8, 2024
@praveenkumar
Copy link
Member

Looks like marketplace operator is not able to resync the pull secret because if you check the pod description you will find auth error. Easy fix for it delete all the pods from openshift-marketplace which will allow to have new pods which have updated pull-secret.

$ oc delete pods --all -n openshift-marketplace

@praveenkumar
Copy link
Member

Looks like there was issue on OLM side https://issues.redhat.com/browse/OCPBUGS-31645 have more info and operator-framework/operator-lifecycle-manager#3197 upstream PR which is merged. So hopefully when we have latest version of OLM it will be fixed automatic.

@ruivieira
Copy link
Author

Thanks for the update @praveenkumar!

@marcelomrwin
Copy link

I faced the same problem. After updating the version, Operator Hub disappeared. My environment is macOS 14.4 Intel based.

@gbraad gbraad moved this to Hold in Project planning: crc Apr 10, 2024
@lavocatt lavocatt mentioned this issue Apr 15, 2024
Merged
@cfergeau
Copy link
Contributor

I faced the same problem. After updating the version, Operator Hub disappeared. My environment is macOS 14.4 Intel based.

You will need tihs workaround #4109 (comment) until at least the next crc release.

@karelyatin karelyatin mentioned this issue Apr 17, 2024
Merged
1 task
raukadah added a commit to openstack-k8s-operators/ci-framework that referenced this issue Apr 17, 2024
@raukadah
[cert_manager]Delete/Recreate openshift marketplace
8ea3496 
On crc Zuul reproducer job, cert-manager operator installation is
failing with following error.
```
the cert-manager CRDs are not yet installed on the Kubernetes API server
```

The cert-manager operator get installed from openshift marketplace.
After digging deep, we found that pods under openshift-marketplace
namespace are hitting CrashLoopBackOff due to following error.
```
failed to populate resolver cache from source redhat-operators/openshift-marketplace:
failed to list bundles: rpc error: code = Unavailable desc = connection error: desc =
```

Based on crc-org/crc#4109 (comment),
Delete and recreating openshift-marketplace pods fixes the issue.

Since OCP is deployed after pre_infra hook and cert_manager role is
called before post_infra. There is no way to run this workaround
as a hook.

It would be best to include under openshift_setup role.

Signed-off-by: Chandan Kumar <[email protected]>
raukadah added a commit to openstack-k8s-operators/ci-framework that referenced this issue Apr 17, 2024
@raukadah
[cert_manager]Delete/Recreate openshift marketplace
2b60416 
On crc Zuul reproducer job, cert-manager operator installation is
failing with following error.
```
the cert-manager CRDs are not yet installed on the Kubernetes API server
```

The cert-manager operator get installed from openshift marketplace.
After digging deep, we found that pods under openshift-marketplace
namespace are hitting CrashLoopBackOff due to following error.
```
failed to populate resolver cache from source redhat-operators/openshift-marketplace:
failed to list bundles: rpc error: code = Unavailable desc = connection error: desc =
```

Based on crc-org/crc#4109 (comment),
Delete and recreating openshift-marketplace pods fixes the issue.

Since OCP is deployed after pre_infra hook and cert_manager role is
called before post_infra. There is no way to run this workaround
as a hook.

It would be best to include under openshift_setup role.

Signed-off-by: Chandan Kumar <[email protected]>
openshift-merge-bot bot pushed a commit to openstack-k8s-operators/ci-framework that referenced this issue Apr 17, 2024
@raukadah @openshift-merge-bot
[cert_manager]Delete/Recreate openshift marketplace
a113e1a 
On crc Zuul reproducer job, cert-manager operator installation is
failing with following error.
```
the cert-manager CRDs are not yet installed on the Kubernetes API server
```

The cert-manager operator get installed from openshift marketplace.
After digging deep, we found that pods under openshift-marketplace
namespace are hitting CrashLoopBackOff due to following error.
```
failed to populate resolver cache from source redhat-operators/openshift-marketplace:
failed to list bundles: rpc error: code = Unavailable desc = connection error: desc =
```

Based on crc-org/crc#4109 (comment),
Delete and recreating openshift-marketplace pods fixes the issue.

Since OCP is deployed after pre_infra hook and cert_manager role is
called before post_infra. There is no way to run this workaround
as a hook.

It would be best to include under openshift_setup role.

Signed-off-by: Chandan Kumar <[email protected]>
@nazcardev
Copy link

Also faced this on Windows. I deleted the Pods that were saying crashloopback and reload/saved YAML in openshift-marketplace project to get OperatorHub working again.

@adrianriobo adrianriobo mentioned this issue Apr 18, 2024
Closed
@praveenkumar praveenkumar mentioned this issue Apr 23, 2024
Closed
@praveenkumar praveenkumar mentioned this issue May 13, 2024
Closed
rdoproject pushed a commit to rdo-infra/review.rdoproject.org-config that referenced this issue Jul 20, 2024
@danpawlik
Add workaround for openshift-marketplace ImagePullBackOff error
6f7710e 
It seems that on the CI, we have many times same issue with the
OpenShift Marketplace ImagePullBackOff error.
This commit is applying solution proposed by the CRC community [1].

[1] crc-org/crc#4109 (comment)

Change-Id: I1572332044dad4787df685e0e6a1e6204aac2187
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working known-issue Known issue preset/openshift priority/critical
Projects
Project planning: crc
Status: Hold
Milestone
No milestone
Development

No branches or pull requests
6 participants
@gbraad @ruivieira @praveenkumar @cfergeau @marcelomrwin @nazcardev