Ceph: workloads adoption with cinder volume

Enable back the cinder volume commands on the source cloud, and resume testing of the ceph-backed volume attached to the test VM. Extend volume/backup/snapshot/attachment commands to wait for the previous step results. Follow the EDPM Post Ceph steps of HCI VA to prepare adopted workloads for using Ceph backend on EDPM. Add Nova discover host command (step 5 of the HCI VA). Add Nova Ceph custom configs to properly configure ceph vms pool for libvirt. Combine nova-ceph related configurations and nova FFU related ones into a single nova-compute-extraconfig service (by design, having two dataplane services for Nova in the same node set is not supported). Note about available choises for libvirt storage backends for Nova Add nova_libvirt_backend to control either to deploy with the local or ceph storage EDPM backends Signed-off-by: Bohdan Dobrelia <[email protected]>
openstack-k8s-operators · Apr 19, 2024 · 140aa59 · 140aa59
1 parent ad6fc25
commit 140aa59
Show file tree

Hide file tree

Showing 12 changed files with 263 additions and 36 deletions.
diff --git a/docs_dev/assemblies/development_environment.adoc b/docs_dev/assemblies/development_environment.adoc
@@ -108,13 +108,16 @@ export EDPM_COMPUTE_ADDITIONAL_NETWORKS=$(jq -c . /tmp/addtional_nets.json)
 export STANDALONE_COMPUTE_DRIVER=ironic
 export NTP_SERVER=pool.ntp.org  # Only necessary if not on the RedHat network ...
 export EDPM_COMPUTE_CEPH_ENABLED=false  # Optional
+export EDPM_COMPUTE_CEPH_NOVA=false # Optional
 export EDPM_COMPUTE_SRIOV_ENABLED=false # Without this the standalone deploy fails when compute driver is ironic.
 ----
 
 [Note]
 ===
 If `EDPM_COMPUTE_CEPH_ENABLED=false` is set, TripleO configures `Glance` with
 `Swift` as a backend.
+If `EDPM_COMPUTE_CEPH_NOVA=false` is set, TripleO configures `Nova/Libvirt` with
+a local storage backend.
 ===
 '''
 
@@ -301,19 +304,20 @@ ping -c 4 $(openstack server show baremetal-test -f json -c addresses | jq -r .a
 
 ==== Virtual Machine Steps
 
+Create a test VM instance with a test volume attachement:
+
 [,bash]
 ----
 cd ~/data-plane-adoption
 bash tests/roles/development_environment/files/pre_launch.bash
 ----
 
+This also creates a test Cinder volume, a backup from it, and a snapshot of it.
+
 '''
 
 ==== Ceph Storage Steps
 
-Make sure a cinder-volume backend is properly configured, or skip below steps
-to create a test workload without volume attachments.
-
 Confirm the image UUID can be seen in Ceph's images pool.
 
 [,bash]
@@ -326,22 +330,6 @@ Create a Barbican secret
 openstack secret store --name testSecret --payload 'TestPayload'
 ```
 
-Create a Cinder volume, a backup from it, and snapshot it.
-
-[,bash]
-----
-openstack volume create --image cirros --bootable --size 1 disk
-openstack volume backup create --name backup disk
-openstack volume snapshot create --volume disk snapshot
-----
-
-Add volume to the test VM
-
-[,bash]
-----
-openstack server add volume test disk
-----
-
 == Performing the Data Plane Adoption
 
 The development environment is now set up, you can go to the https://openstack-k8s-operators.github.io/data-plane-adoption/[Adoption

diff --git a/docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc b/docs_user/modules/proc_adopting-compute-services-to-the-data-plane.adoc
@@ -7,8 +7,9 @@
 .Prerequisites
 
 * Remaining source cloud xref:stopping-infrastructure-management-and-compute-services_{context}[Stopping infrastructure management and Compute services] on Compute hosts.
-* Make sure the IPAM is configured:
-+
+* Ceph backend for Nova/Libvirt is configured xref:configuring-a-ceph-backend_{context}[Configuring a Ceph backend].
+* Make sure the IPAM is configured
+
 ----
 oc apply -f - <<EOF
 apiVersion: network.openstack.org/v1beta1
@@ -77,9 +78,10 @@ EOF
 ----
 +
 * When `neutron-sriov-nic-agent` is running on the existing Compute nodes, check the physical device mappings and ensure that they match the values that are defined in the `OpenStackDataPlaneNodeSet` custom resource (CR). For more information, see xref:reviewing-the-openstack-control-plane-configuration_adopt-control-plane[Reviewing the {rhos_prev_long} control plane configuration].
-* Define the shell variables necessary to run the script that runs the fast-forward upgrade:
+* Define the shell variables necessary to run the script that runs the fast-forward upgrade. Omit setting `CEPH_FSID`, if the local storage backend is going to be configured by Nova for Libvirt. The storage backend cannot be changed during adoption, and must match the one used on the source cloud:
 ----
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //'
 
 alias openstack="oc exec -t openstackclient -- openstack"
 FIP=192.168.122.20
@@ -156,7 +158,7 @@ rm -f id*
 cd -
 ----
 
-. Create a `nova-compute-extra-config` service:
+. Create a `nova-compute-extra-config` service (with local storage backend for lbivrt):
 +
 . If TLS Everywhere is enabled, append the following to the OpenStackDataPlaneService spec:
 +
@@ -204,6 +206,63 @@ EOF
 The secret `nova-cell<X>-compute-config` is auto-generated for each
 `cell<X>`. You must specify `nova-cell<X>-compute-config` and `nova-migration-ssh-key` for each custom `OpenStackDataPlaneService` related to the Compute service.
 
+That service removes pre-FFU workarounds and configures Nova compute
+services for local storage backend.
+
+* Or, create a `nova-compute-extra-config` service (with Ceph backend for libvirt):
++
+[source,yaml]
+----
+oc apply -f - <<EOF
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nova-compute-extraconfig
+  namespace: openstack
+data:
+  19-nova-compute-cell1-workarounds.conf: |
+    [workarounds]
+    disable_compute_service_check_for_ffu=true
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ceph-nova
+  namespace: openstack
+data:
+  03-ceph-nova.conf: |
+    [libvirt]
+    images_type=rbd
+    images_rbd_pool=vms
+    images_rbd_ceph_conf=/etc/ceph/ceph.conf
+    images_rbd_glance_store_name=default_backend
+    images_rbd_glance_copy_poll_interval=15
+    images_rbd_glance_copy_timeout=600
+    rbd_user=openstack
+    rbd_secret_uuid=$CEPH_FSID
+---
+apiVersion: dataplane.openstack.org/v1beta1
+kind: OpenStackDataPlaneService
+metadata:
+  name: nova-compute-extraconfig
+  namespace: openstack
+spec:
+  label: nova.compute.extraconfig
+  configMaps:
+    - nova-compute-extraconfig
+    - ceph-nova
+  secrets:
+    - nova-cell1-compute-config
+    - nova-migration-ssh-key
+  playbook: osp.edpm.nova
+EOF
+----
++
+That service removes pre-FFU workarounds and configures Nova compute
+services for Ceph storage backend.
+Provided above resources should contain a cell-specific configurations.
+For multi-cell, config maps and OpenStack dataplane services should be named like `nova-custom-ceph-cellX` and `nova-compute-extraconfig-cellX`.
+
 ifeval::["{build}" == "downstream"]
 . Create a secret for the subscription manager and a secret for the Red Hat registry:
 +
@@ -409,6 +468,41 @@ endif::[]
 EOF
 ----
 +
+* Prepare adopted EDPM workloads to use Ceph backend for Cinder, if configured so
++
+[source,yaml]
+----
+oc patch osdpns/openstack --type=merge --patch "
+spec:
+  services:
+    - repo-setup
+    - download-cache
+    - bootstrap
+    - configure-network
+    - validate-network
+    - install-os
+    - configure-os
+    - run-os
+    - install-certs
+    - ceph-client
+    - libvirt
+    - nova-compute-extraconfig
+    - ovn
+    - neutron-metadata
+  nodeTemplate:
+    extraMounts:
+    - extraVolType: Ceph
+      volumes:
+      - name: ceph
+        secret:
+          secretName: ceph-conf-files
+      mounts:
+      - name: ceph
+        mountPath: "/etc/ceph"
+        readOnly: true
+"
+----
++
 * Replace `<bridge_mappings>` with the value of the bridge mappings in your configuration, for example, `"datacentre:br-ctlplane"`.
 
 . Ensure that the `ovn-controller` settings that are configured in the `OpenStackDataPlaneNodeSet` CR are the same as were set in the Compute nodes before adoption. This configuration is stored in the `external_ids`` column in the `Open_vSwitch` table in the Open vSwitch database:
@@ -419,6 +513,10 @@ ovs-vsctl list Open .
 external_ids        : {hostname=standalone.localdomain, ovn-bridge=br-int, ovn-bridge-mappings=<bridge_mappings>, ovn-chassis-mac-mappings="datacentre:1e:0a:bb:e6:7c:ad", ovn-encap-ip="172.19.0.100", ovn-encap-tos="0", ovn-encap-type=geneve, ovn-match-northd-version=False, ovn-monitor-all=True, ovn-ofctrl-wait-before-clear="8000", ovn-openflow-probe-interval="60", ovn-remote="tcp:ovsdbserver-sb.openstack.svc:6642", ovn-remote-probe-interval="60000", rundir="/var/run/openvswitch", system-id="2eec68e6-aa21-4c95-a868-31aeafc11736"}
 ...
 ----
+
+Note that you should retain the original `OpenStackDataPlaneNodeSet` services
+composition, except the inserted `ceph-client` service.
+
 +
 * Replace `<bridge_mappings>` with the value of the bridge mappings in your configuration, for example, `"datacentre:br-ctlplane"`.
 

diff --git a/docs_user/modules/proc_performing-a-fast-forward-upgrade-on-compute-services.adoc b/docs_user/modules/proc_performing-a-fast-forward-upgrade-on-compute-services.adoc
@@ -109,6 +109,12 @@ oc exec -it nova-cell0-conductor-0 -- nova-manage db online_data_migrations
 oc exec -it nova-cell1-conductor-0 -- nova-manage db online_data_migrations
 ----
 
+. Discover Nova compute hosts in the cell:
++
+----
+oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
+----
+
 . Verify if Compute services can stop the existing test VM instance:
 +
 ----

diff --git a/tests/playbooks/test_rollback_with_ceph.yaml b/tests/playbooks/test_rollback_with_ceph.yaml
@@ -9,6 +9,7 @@
     manila_backend: cephfs
     cinder_volume_backend: ceph
     cinder_backup_backend: ceph
+    nova_libvirt_backend: ceph
   module_defaults:
     ansible.builtin.shell:
       executable: /bin/bash

diff --git a/tests/playbooks/test_with_ceph.yaml b/tests/playbooks/test_with_ceph.yaml
@@ -9,6 +9,7 @@
     manila_backend: cephfs
     cinder_volume_backend: ceph
     cinder_backup_backend: ceph
+    nova_libvirt_backend: ceph
   module_defaults:
     ansible.builtin.shell:
       executable: /bin/bash

diff --git a/tests/playbooks/test_with_ironic.yaml b/tests/playbooks/test_with_ironic.yaml
@@ -25,6 +25,7 @@
       executable: /bin/bash
   vars:
     ironic_adoption: true
+    nova_libvirt_backend: local
   roles:
     - role: development_environment
     - role: backend_services

diff --git a/tests/roles/dataplane_adoption/defaults/main.yaml b/tests/roles/dataplane_adoption/defaults/main.yaml
@@ -123,3 +123,5 @@ neutron_public_interface_name: "{{ dataplane_public_iface | default('eth0') }}"
 edpm_sshd_allowed_ranges: "{{ ['192.168.122.0/24'] if dataplane_os_net_config_set_route|default(true)|bool else ['0.0.0.0/0'] }}"
 edpm_neutron_sriov_agent_enabled: true
 edpm_neutron_dhcp_agent_enabled: true
+
+nova_libvirt_backend: local
diff --git a/tests/roles/dataplane_adoption/tasks/main.yaml b/tests/roles/dataplane_adoption/tasks/main.yaml
@@ -1,3 +1,11 @@
+- name: set shell vars for edpm adoption
+  no_log: "{{ use_no_log }}"
+  when:
+    - ('ceph' in [nova_libvirt_backend])
+  ansible.builtin.set_fact:
+    ceph_backend_configuration_fsid_shell_vars: |
+      CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //')
+
 - name: Save ansibleee-operator pod name to be able to wait for the rollout of the new pod
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |
@@ -125,7 +133,9 @@
     rm -f id*
     cd -
 
-- name: create a Nova Compute Extra Config service
+- name: create a Nova Compute Extra Config service (no ceph backend in use)
+  when:
+    - ('ceph' not in [nova_libvirt_backend])
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |
     {{ shell_header }}
@@ -166,6 +176,58 @@
     {% endif %}
     EOF
 
+- name: create a Nova Compute Extra Config service (with ceph backend)
+  when:
+    - ('ceph' in [nova_libvirt_backend])
+  no_log: "{{ use_no_log }}"
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    {{ ceph_backend_configuration_fsid_shell_vars }}
+    oc apply -f - <<EOF
+    apiVersion: v1
+    kind: ConfigMap
+    metadata:
+      name: nova-compute-extraconfig
+      namespace: openstack
+    data:
+      19-nova-compute-cell1-workarounds.conf: |
+        [workarounds]
+        disable_compute_service_check_for_ffu=true
+    ---
+    apiVersion: v1
+    kind: ConfigMap
+    metadata:
+      name: ceph-nova
+      namespace: openstack
+    data:
+      03-ceph-nova.conf: |
+        [libvirt]
+        images_type=rbd
+        images_rbd_pool=vms
+        images_rbd_ceph_conf=/etc/ceph/ceph.conf
+        images_rbd_glance_store_name=default_backend
+        images_rbd_glance_copy_poll_interval=15
+        images_rbd_glance_copy_timeout=600
+        rbd_user=openstack
+        rbd_secret_uuid=$CEPH_FSID
+    ---
+    apiVersion: dataplane.openstack.org/v1beta1
+    kind: OpenStackDataPlaneService
+    metadata:
+      name: nova-compute-extraconfig
+      namespace: openstack
+    spec:
+      label: nova.compute.extraconfig
+      configMaps:
+        - nova-compute-extraconfig
+        - ceph-nova
+      secrets:
+        - nova-cell1-compute-config
+        - nova-migration-ssh-key
+      playbook: osp.edpm.nova
+    EOF
+
 - name: Create OpenStackDataPlaneNodeSet
   ansible.builtin.shell: |
     {{ shell_header }}
@@ -279,6 +341,42 @@
             edpm_ovn_ofctrl_wait_before_clear: 8000
     EOF
 
+# TODO: Apply the ceph backend config for Cinder in the original openstack CR, via kustomize
+- name: prepare adopted EDPM workloads to use Ceph backend for Cinder, if configured so
+  no_log: "{{ use_no_log }}"
+  when: cinder_volume_backend == "ceph" or cinder_backup_backend == "ceph"
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    oc patch osdpns/openstack --type=merge --patch "
+    spec:
+      services:
+        - bootstrap
+        - download-cache
+        - configure-network
+        - validate-network
+        - install-os
+        - configure-os
+        - run-os
+        - install-certs
+        - ceph-client
+        - libvirt
+        - nova-compute-extraconfig
+        - ovn
+        - neutron-metadata
+      nodeTemplate:
+        extraMounts:
+        - extraVolType: Ceph
+          volumes:
+          - name: ceph
+            secret:
+              secretName: ceph-conf-files
+          mounts:
+          - name: ceph
+            mountPath: "/etc/ceph"
+            readOnly: true
+    "
+
 - name: set neutron-sriov-nic-agent configuration in the OpenStackDataPlaneNodeSet CR
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |