Add Barbican #562

vakwetu · 2023-11-17T20:02:42Z

This commit adds the barbican operator to the openstack-operator.

softwarefactory-project-zuul · 2023-11-17T20:11:31Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/96f5fb72a9f441278f1b294d30bce329

❌ openstack-k8s-operators-content-provider FAILURE in 5m 56s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

vakwetu · 2023-11-17T20:55:59Z

/retest

softwarefactory-project-zuul · 2023-11-17T21:09:39Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/737d77657c894f67b344e5f3aa9a5ab5

❌ openstack-k8s-operators-content-provider NODE_FAILURE Node request 200-0006650721 failed in 0s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

vakwetu · 2023-11-17T21:11:09Z

/retest

vakwetu · 2023-11-17T21:11:52Z

recheck

softwarefactory-project-zuul · 2023-11-17T22:38:00Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/61b5da1db41b4818a27c4ad7a7058186

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 24m 50s
❌ podified-multinode-edpm-deployment-crc FAILURE in 1h 05m 14s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 05m 15s
❌ openstack-operator-tempest-multinode FAILURE in 1h 09m 47s

abays · 2023-11-20T10:11:12Z

config/samples/core_v1beta1_openstackcontrolplane.yaml

One question here and for all the other samples: do we intend to have Barbican enabled by default? I'm just asking because I don't know one way or the other. Right now it's enabled.

abays · 2023-11-20T10:38:19Z

One thing I've noticed while trying this out is that the rabbitMqClusterName field is empty in the Barbican section of the OpenStackControlPlane CR. This results in the control plane never finishing deployment, because the TransportURL created by the Barbican controller then uses that empty string as the RabbitMQ cluster name (which the Infra operator considers an error and thus it goes no further).

I see that other operators have the rabbitMqClusterName in their base spec struct as opposed to a template struct that is in-lined [1]. Perhaps using that same approach would fix the problem here. I think the webhooks and the kubebuilder defaulting are interfering here and are setting the rabbitMqClusterName to the empty string, which is somehow tricking the validation into allowing such a zero-value as if the user had explicitly provided it. We've seen similar problems in other operators.

I think this is the cause of the CI failure too:

TASK [edpm_prepare : Wait for OpenStack controlplane to be deployed _raw_params=oc wait OpenStackControlPlane {{ cr_name }} --namespace={{ cifmw_install_yamls_defaults['NAMESPACE'] }} --for=condition=ready --timeout={{ cifmw_edpm_prepare_timeout }}m] ***
Friday 17 November 2023  16:49:55 -0500 (0:00:00.724)       0:10:14.725 ******* 
fatal: [localhost]: FAILED! => ...

[1] https://github.com/openstack-k8s-operators/cinder-operator/blob/3c5c40e6cc3aba6e287b94d26770519f76ae9528/api/v1beta1/cinder_types.go#L55-L59

abays · 2023-11-20T13:54:25Z

One thing I've noticed while trying this out is that the rabbitMqClusterName field is empty in the Barbican section of the OpenStackControlPlane CR. This results in the control plane never finishing deployment, because the TransportURL created by the Barbican controller then uses that empty string as the RabbitMQ cluster name (which the Infra operator considers an error and thus it goes no further).

I see that other operators have the rabbitMqClusterName in their base spec struct as opposed to a template struct that is in-lined [1]. Perhaps using that same approach would fix the problem here. I think the webhooks and the kubebuilder defaulting are interfering here and are setting the rabbitMqClusterName to the empty string, which is somehow tricking the validation into allowing such a zero-value as if the user had explicitly provided it. We've seen similar problems in other operators.

I think this is the cause of the CI failure too:
TASK [edpm_prepare : Wait for OpenStack controlplane to be deployed _raw_params=oc wait OpenStackControlPlane {{ cr_name }} --namespace={{ cifmw_install_yamls_defaults['NAMESPACE'] }} --for=condition=ready --timeout={{ cifmw_edpm_prepare_timeout }}m] ***
Friday 17 November 2023  16:49:55 -0500 (0:00:00.724)       0:10:14.725 ******* 
fatal: [localhost]: FAILED! => ...
[1] https://github.com/openstack-k8s-operators/cinder-operator/blob/3c5c40e6cc3aba6e287b94d26770519f76ae9528/api/v1beta1/cinder_types.go#L55-L59

I did some more testing. It seems like the struct definition as-is is actually fine. Something else must be causing CI to fail.

vakwetu · 2023-11-20T22:47:00Z

recheck

softwarefactory-project-zuul · 2023-11-21T01:00:45Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/463c81acca384963926bf21b7d4e3de2

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 05m 31s
❌ podified-multinode-edpm-deployment-crc FAILURE in 1h 06m 44s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 24m 25s
❌ openstack-operator-tempest-multinode RETRY_LIMIT in 1h 52m 07s

softwarefactory-project-zuul · 2023-11-21T02:29:17Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/9f9194d5630b407886782401fd61288f

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 24m 53s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 03m 00s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 23m 06s
❌ openstack-operator-tempest-multinode FAILURE in 1h 10m 26s

abays · 2023-11-21T11:40:15Z

tests/kuttl/common/assert-sample-deployment.yaml

@@ -159,6 +159,17 @@ spec:
        replicas: 0 # backend needs to be configured
      designateBackendbind9:
        replicas: 0 # backend needs to be configured
+  barbican:
+    enabled: false


The current samples have Barbican enabled by default, since they don't have enabled: false in the YAML. So we need to reflect that here. Also, since Barbican is enabled by default, its condition entry needs to be added to the status.conditions list after the first entry in the list (the one with type: Ready):

- message: OpenStackControlPlane Barbican completed reason: Ready status: "True" type: OpenStackControlPlaneBarbicanReady

It also needs an entry for the Barbican "expose" condition:

- message: OpenStackControlPlane barbican service exposed reason: Ready status: "True" type: OpenStackControlPlaneExposeBarbicanReady

That should go after the OpenStackControlPlaneClientReady condition.

Yes, per JP Jung, the current plan is to always enable Barbican. This also means other projects that integrate with Barbican will be set up to use it by default. e.g. encrypted volumes will store keys there, etc.

dmendiza · 2023-11-21T20:01:00Z

apis/core/v1beta1/openstackcontrolplane_webhook.go

@@ -155,6 +155,10 @@ func (r *OpenStackControlPlane) checkDepsEnabled(name string) string {
 		if !((r.Spec.Mariadb.Enabled || r.Spec.Galera.Enabled) && r.Spec.Memcached.Enabled && r.Spec.Keystone.Enabled) {
 			reqs = "MariaDB or Galera, Memcached, Keystone"
 		}
+	case "Barbican":
+		if !((r.Spec.Mariadb.Enabled || r.Spec.Galera.Enabled) && r.Spec.Memcached.Enabled && r.Spec.Keystone.Enabled) {


IIRC, Barbican does not depend or use Memcached in any way.

vakwetu · 2023-11-21T23:52:14Z

/retest

softwarefactory-project-zuul · 2023-11-22T00:55:45Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/f2cd794f02964a72aec1d93f7c3a4913

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 48m 05s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 08m 31s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 1h 05m 06s
❌ openstack-operator-tempest-multinode FAILURE in 1h 31m 54s

softwarefactory-project-zuul · 2023-11-27T18:28:08Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/70dd054cee1446b1862c2aac14a77820

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 36m 09s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 03m 02s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 12m 43s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 21m 21s

vakwetu · 2023-11-27T20:59:58Z

recheck

softwarefactory-project-zuul · 2023-11-27T22:42:25Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/eb80594def0445c5a9c0c03889d478a8

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 40m 50s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 07m 04s
❌ cifmw-crc-podified-edpm-baremetal FAILURE in 14m 25s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 25m 23s

abays · 2023-11-28T09:32:17Z

recheck

error: Failed to attach interface\nerror: Cannot get interface MTU on 'default': No such device

softwarefactory-project-zuul · 2023-11-28T12:39:20Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/ac14f2e0cac9465498c8aeb7338339d3

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 04m 55s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 03m 37s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 58m 09s
❌ openstack-operator-tempest-multinode FAILURE in 1h 33m 15s

abays · 2023-11-28T12:42:17Z

Between the last two Zuul attempts, we have full success for the 4 jobs there. Let's try one more recheck and then we can override if it flakes-out again.

vakwetu · 2023-11-28T13:08:19Z

recheck

softwarefactory-project-zuul · 2023-11-28T15:37:55Z

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/a277a3a34c62429fbbe1f69e60f8714c

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 41m 15s
❌ podified-multinode-edpm-deployment-crc RETRY_LIMIT in 52m 53s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 03m 20s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 19m 45s

abays

/lgtm

openshift-ci · 2023-11-28T15:46:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, vakwetu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abays]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

abays · 2023-11-28T15:47:56Z

The Zuul jobs have all passed independently over several attempts, so I think we're good there despite the hiccups. I will override by the end of the day unless another reviewer requests changes to the PR.

This adds the barbican section to the nova.conf template so nova can access the barbican. Depends-On: openstack-k8s-operators/openstack-operator#562

abays · 2023-11-29T09:24:41Z

/override rdoproject.org/github-check

gibizer · 2023-11-29T09:24:45Z

I try to deploy this from a local build and I see that for me barbican is disabled by default even though the code suggests it should be enabled by default.

openshift-ci · 2023-11-29T09:24:50Z

@abays: Overrode contexts on behalf of abays: rdoproject.org/github-check

In response to this:

/override rdoproject.org/github-check

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

abays · 2023-11-29T09:24:59Z

/hold

gibizer · 2023-11-29T09:30:53Z

I try to deploy this from a local build and I see that for me barbican is disabled by default even though the code suggests it should be enabled by default.

I used the wrong sample and updating it from the sample from this PR did not work. But recreating the OpenStackControlPlane from the sample from this PR works. Sorry for the noise

abays · 2023-11-29T09:54:42Z

/unhold

This adds the barbican section to the nova.conf template so nova can access the barbican. Depends-On: openstack-k8s-operators/openstack-operator#562

vakwetu requested review from xek, dmendiza, abays and afaranha November 17, 2023 20:02

openshift-ci bot requested review from lewisdenny and viroel November 17, 2023 20:02

vakwetu force-pushed the add_barbican_2 branch from 0390de0 to 26761a2 Compare November 17, 2023 21:04

abays reviewed Nov 20, 2023

View reviewed changes

vakwetu force-pushed the add_barbican_2 branch from 26761a2 to 6e5600f Compare November 20, 2023 22:53

vakwetu force-pushed the add_barbican_2 branch from 6e5600f to ebc98c8 Compare November 21, 2023 01:02

abays reviewed Nov 21, 2023

View reviewed changes

dmendiza reviewed Nov 21, 2023

View reviewed changes

vakwetu force-pushed the add_barbican_2 branch 5 times, most recently from b02981d to 83ceac6 Compare November 21, 2023 23:05

vakwetu force-pushed the add_barbican_2 branch from 1e95c37 to 3d630a3 Compare November 27, 2023 16:50

abays approved these changes Nov 28, 2023

View reviewed changes

openshift-ci bot assigned abays Nov 28, 2023

openshift-ci bot added the lgtm label Nov 28, 2023

openshift-ci bot added the approved label Nov 28, 2023

gibizer added a commit to gibizer/nova-operator that referenced this pull request Nov 28, 2023

Configure barbican access

4717a09

This adds the barbican section to the nova.conf template so nova can access the barbican. Depends-On: openstack-k8s-operators/openstack-operator#562

gibizer mentioned this pull request Nov 28, 2023

Configure barbican access openstack-k8s-operators/nova-operator#609

Merged

openshift-ci bot added the do-not-merge/hold label Nov 29, 2023

openshift-ci bot removed the do-not-merge/hold label Nov 29, 2023

openshift-merge-bot bot merged commit b296006 into openstack-k8s-operators:main Nov 29, 2023
1 check passed

gibizer added a commit to gibizer/nova-operator that referenced this pull request Nov 30, 2023

Configure barbican access

51e57e3

This adds the barbican section to the nova.conf template so nova can access the barbican. Depends-On: openstack-k8s-operators/openstack-operator#562

fmount mentioned this pull request Dec 1, 2023

Barbican integration openstack-k8s-operators/glance-operator#378

Merged

gibizer added a commit to gibizer/nova-operator that referenced this pull request Dec 5, 2023

Configure barbican access

1025756

This adds the barbican section to the nova.conf template so nova can access the barbican. Depends-On: openstack-k8s-operators/openstack-operator#562

Add Barbican #562

Add Barbican #562

Conversation

vakwetu commented Nov 17, 2023

softwarefactory-project-zuul bot commented Nov 17, 2023

vakwetu commented Nov 17, 2023

softwarefactory-project-zuul bot commented Nov 17, 2023

vakwetu commented Nov 17, 2023

vakwetu commented Nov 17, 2023

softwarefactory-project-zuul bot commented Nov 17, 2023

abays Nov 20, 2023

Choose a reason for hiding this comment

abays commented Nov 20, 2023

abays commented Nov 20, 2023

vakwetu commented Nov 20, 2023

softwarefactory-project-zuul bot commented Nov 21, 2023

softwarefactory-project-zuul bot commented Nov 21, 2023

abays Nov 21, 2023 • edited Loading

Choose a reason for hiding this comment

abays Nov 21, 2023

Choose a reason for hiding this comment

dmendiza Nov 21, 2023

Choose a reason for hiding this comment

dmendiza Nov 21, 2023

Choose a reason for hiding this comment

vakwetu Nov 21, 2023

Choose a reason for hiding this comment

vakwetu commented Nov 21, 2023

softwarefactory-project-zuul bot commented Nov 22, 2023

softwarefactory-project-zuul bot commented Nov 27, 2023

vakwetu commented Nov 27, 2023

softwarefactory-project-zuul bot commented Nov 27, 2023

abays commented Nov 28, 2023

softwarefactory-project-zuul bot commented Nov 28, 2023

abays commented Nov 28, 2023

vakwetu commented Nov 28, 2023

softwarefactory-project-zuul bot commented Nov 28, 2023

abays left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Nov 28, 2023

abays commented Nov 28, 2023

abays commented Nov 29, 2023

gibizer commented Nov 29, 2023

openshift-ci bot commented Nov 29, 2023

abays commented Nov 29, 2023

gibizer commented Nov 29, 2023

abays commented Nov 29, 2023

abays Nov 21, 2023 •

edited

Loading