Ceph: workloads adoption with cinder volume

Enable back the cinder volume commands on the source cloud, and resume testing of the ceph-backed volume attached to the test VM. Extend volume/backup/snapshot/attachment commands to wait for the previous step results. Follow the EDPM Post Ceph steps of HCI VA to prepare adopted workloads for using Ceph backend on EDPM. Add Nova discover host command (step 5 of the HCI VA). Add Nova Ceph custom configs to properly configure ceph vms pool for libvirt. Combine nova-ceph related configurations and nova FFU related ones into a single nova-compute-extraconfig service (by design, having two dataplane services for Nova in the same node set is not supported). Note about available choises for libvirt storage backends for Nova Add nova_libvirt_backend to control either to deploy with the local or ceph storage EDPM backends Signed-off-by: Bohdan Dobrelia <[email protected]>
openstack-k8s-operators · Apr 17, 2024 · 9837f39 · 9837f39
1 parent 6209f6b
commit 9837f39
Show file tree

Hide file tree

Showing 11 changed files with 261 additions and 33 deletions.
diff --git a/docs_dev/assemblies/development_environment.adoc b/docs_dev/assemblies/development_environment.adoc
@@ -108,13 +108,16 @@ export EDPM_COMPUTE_ADDITIONAL_NETWORKS=$(jq -c . /tmp/addtional_nets.json)
 export STANDALONE_COMPUTE_DRIVER=ironic
 export NTP_SERVER=pool.ntp.org  # Only necessary if not on the RedHat network ...
 export EDPM_COMPUTE_CEPH_ENABLED=false  # Optional
+export EDPM_COMPUTE_CEPH_NOVA=false # Optional
 export EDPM_COMPUTE_SRIOV_ENABLED=false # Without this the standalone deploy fails when compute driver is ironic.
 ----
 
 [Note]
 ===
 If `EDPM_COMPUTE_CEPH_ENABLED=false` is set, TripleO configures `Glance` with
 `Swift` as a backend.
+If `EDPM_COMPUTE_CEPH_NOVA=false` is set, TripleO configures `Nova/Libvirt` with
+a local storage backend.
 ===
 '''
 
@@ -293,19 +296,20 @@ ping -c 4 $(openstack server show baremetal-test -f json -c addresses | jq -r .a
 
 ==== Virtual Machine Steps
 
+Create a test VM instance with a test volume attachement:
+
 [,bash]
 ----
 cd ~/data-plane-adoption
 bash tests/roles/development_environment/files/pre_launch.bash
 ----
 
+This also creates a test Cinder volume, a backup from it, and a snapshot of it.
+
 '''
 
 ==== Ceph Storage Steps
 
-Make sure a cinder-volume backend is properly configured, or skip below steps
-to create a test workload without volume attachments.
-
 Confirm the image UUID can be seen in Ceph's images pool.
 
 [,bash]
@@ -318,22 +322,6 @@ Create a Barbican secret
 openstack secret store --name testSecret --payload 'TestPayload'
 ```
 
-Create a Cinder volume, a backup from it, and snapshot it.
-
-[,bash]
-----
-openstack volume create --image cirros --bootable --size 1 disk
-openstack volume backup create --name backup disk
-openstack volume snapshot create --volume disk snapshot
-----
-
-Add volume to the test VM
-
-[,bash]
-----
-openstack server add volume test disk
-----
-
 == Performing the Data Plane Adoption
 
 The development environment is now set up, you can go to the https://openstack-k8s-operators.github.io/data-plane-adoption/[Adoption

diff --git a/docs_user/modules/openstack-dataplane_adoption.adoc b/docs_user/modules/openstack-dataplane_adoption.adoc
@@ -10,6 +10,7 @@
 
 * Previous Adoption steps completed.
 * Remaining source cloud xref:stopping-infrastructure-management-and-compute-services_{context}[Stopping infrastructure management and Compute services] on Compute hosts.
+* Ceph backend for Nova/Libvirt is configured xref:configuring-a-ceph-backend_{context}[Configuring a Ceph backend].
 
 ____
 *WARNING* This step is a "point of no return" in the data plane adoption
@@ -27,6 +28,7 @@ The values are just illustrative, use values that are correct for your environme
 
 ----
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //'
 
 alias openstack="oc exec -t openstackclient -- openstack"
 FIP=192.168.122.20
@@ -178,7 +180,7 @@ rm -f id*
 cd -
 ----
 
-* Create a Nova Compute Extra Config service
+* Create a Nova Compute Extra Config service (no ceph backend in use)
 +
 [source,yaml]
 ----
@@ -213,6 +215,61 @@ The secret `nova-cell<X>-compute-config` is auto-generated for each
 `cell<X>`. That secret, alongside `nova-migration-ssh-key`, should
 always be specified for each custom `OpenStackDataPlaneService` related to Nova.
 
+That service removes pre-FFU workarounds and configures Nova compute
+services for local storage backend.
+
+* Create a Nova Compute Extra Config service (with ceph backend)
++
+[source,yaml]
+----
+oc apply -f - <<EOF
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: nova-compute-extraconfig
+  namespace: openstack
+data:
+  19-nova-compute-cell1-workarounds.conf: |
+    [workarounds]
+    disable_compute_service_check_for_ffu=true
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ceph-nova
+  namespace: openstack
+data:
+  03-ceph-nova.conf: |
+    [libvirt]
+    images_type=rbd
+    images_rbd_pool=vms
+    images_rbd_ceph_conf=/etc/ceph/ceph.conf
+    images_rbd_glance_store_name=default_backend
+    images_rbd_glance_copy_poll_interval=15
+    images_rbd_glance_copy_timeout=600
+    rbd_user=openstack
+    rbd_secret_uuid=$CEPH_FSID
+---
+apiVersion: dataplane.openstack.org/v1beta1
+kind: OpenStackDataPlaneService
+metadata:
+  name: nova-compute-extraconfig
+  namespace: openstack
+spec:
+  label: nova.compute.extraconfig
+  configMaps:
+    - nova-compute-extraconfig
+    - ceph-nova
+  secrets:
+    - nova-cell1-compute-config
+    - nova-migration-ssh-key
+  playbook: osp.edpm.nova
+EOF
+----
++
+That service removes pre-FFU workarounds and configures Nova compute
+services for Ceph storage backend.
+
 ifeval::["{build}" == "downstream"]
 * Create subscription-manager and redhat-registry secrets
 +
@@ -239,6 +296,9 @@ EOF
 +
 endif::[]
 
+These should be cell-specific configurations.
+For multi-cell, config maps and OpenStack dataplane services should be named like `nova-custom-ceph-cellX` and `nova-compute-extraconfig-cellX`.
+
 * Deploy OpenStackDataPlaneNodeSet:
 +
 Make sure that ovn-controller settings configured in the OpenStackDataPlaneNodeSet are the same as were set in the compute nodes before adoption.
@@ -428,6 +488,44 @@ endif::[]
 EOF
 ----
 
+* Prepare adopted EDPM workloads to use Ceph backend for Cinder, if configured so
++
+[source,yaml]
+----
+oc patch osdpns/openstack --type=merge --patch "
+spec:
+  services:
+    - repo-setup
+    - download-cache
+    - bootstrap
+    - configure-network
+    - validate-network
+    - install-os
+    - configure-os
+    - run-os
+    - install-certs
+    - ceph-client
+    - libvirt
+    - nova-compute-extraconfig
+    - ovn
+    - neutron-metadata
+  nodeTemplate:
+    extraMounts:
+    - extraVolType: Ceph
+      volumes:
+      - name: ceph
+        secret:
+          secretName: ceph-conf-files
+      mounts:
+      - name: ceph
+        mountPath: "/etc/ceph"
+        readOnly: true
+"
+----
+
+Note that you should retain the original `OpenStackDataPlaneNodeSet` services
+composition, except the inserted `ceph-client` service.
+
 * Optionally enable neutron-sriov-nic-agent in the OpenStackDataPlaneNodeSet CR
 +
 [source,yaml]
@@ -682,6 +780,12 @@ oc exec -it nova-cell0-conductor-0 -- nova-manage db online_data_migrations
 oc exec -it nova-cell1-conductor-0 -- nova-manage db online_data_migrations
 ----
 
+* Ask Nova to discover all compute hosts
++
+----
+oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
+----
+
 * Verify if Nova services can stop the existing test VM instance:
 +
 ----

diff --git a/tests/playbooks/test_rollback_with_ceph.yaml b/tests/playbooks/test_rollback_with_ceph.yaml
@@ -9,6 +9,7 @@
     manila_backend: cephfs
     cinder_volume_backend: ceph
     cinder_backup_backend: ceph
+    nova_libvirt_backend: ceph
   module_defaults:
     ansible.builtin.shell:
       executable: /bin/bash

diff --git a/tests/playbooks/test_with_ceph.yaml b/tests/playbooks/test_with_ceph.yaml
@@ -9,6 +9,7 @@
     manila_backend: cephfs
     cinder_volume_backend: ceph
     cinder_backup_backend: ceph
+    nova_libvirt_backend: ceph
   module_defaults:
     ansible.builtin.shell:
       executable: /bin/bash

diff --git a/tests/playbooks/test_with_ironic.yaml b/tests/playbooks/test_with_ironic.yaml
@@ -25,6 +25,7 @@
       executable: /bin/bash
   vars:
     ironic_adoption: true
+    nova_libvirt_backend: local
   roles:
     - role: development_environment
     - role: backend_services

diff --git a/tests/roles/dataplane_adoption/defaults/main.yaml b/tests/roles/dataplane_adoption/defaults/main.yaml
@@ -123,3 +123,6 @@ neutron_public_interface_name: "{{ dataplane_public_iface | default('eth0') }}"
 edpm_sshd_allowed_ranges: "{{ ['192.168.122.0/24'] if dataplane_os_net_config_set_route|default(true)|bool else ['0.0.0.0/0'] }}"
 edpm_neutron_sriov_agent_enabled: true
 edpm_neutron_dhcp_agent_enabled: true
+
+nova_libvirt_backend: local
+
diff --git a/tests/roles/dataplane_adoption/tasks/main.yaml b/tests/roles/dataplane_adoption/tasks/main.yaml
@@ -1,3 +1,11 @@
+- name: set shell vars for edpm adoption
+  no_log: "{{ use_no_log }}"
+  when:
+    - ('ceph' in [nova_libvirt_backend])
+  ansible.builtin.set_fact:
+    ceph_backend_configuration_fsid_shell_vars: |
+      CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //')
+
 - name: Save ansibleee-operator pod name to be able to wait for the rollout of the new pod
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |
@@ -125,11 +133,47 @@
     rm -f id*
     cd -
 
-- name: create a Nova Compute Extra Config service
+- name: create a Nova Compute Extra Config service (no ceph backend in use)
+  when:
+    - ('ceph' not in [nova_libvirt_backend])
+  no_log: "{{ use_no_log }}"
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    oc apply -f - <<EOF
+    apiVersion: v1
+    kind: ConfigMap
+    metadata:
+      name: nova-compute-extraconfig
+      namespace: openstack
+    data:
+      19-nova-compute-cell1-workarounds.conf: |
+        [workarounds]
+        disable_compute_service_check_for_ffu=true
+    ---
+    apiVersion: dataplane.openstack.org/v1beta1
+    kind: OpenStackDataPlaneService
+    metadata:
+      name: nova-compute-extraconfig
+      namespace: openstack
+    spec:
+      label: nova.compute.extraconfig
+      configMaps:
+        - nova-compute-extraconfig
+      secrets:
+        - nova-cell1-compute-config
+        - nova-migration-ssh-key
+      playbook: osp.edpm.nova
+    EOF
+
+- name: create a Nova Compute Extra Config service (with ceph backend)
+  when:
+    - ('ceph' in [nova_libvirt_backend])
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |
     {{ shell_header }}
     {{ oc_header }}
+    {{ ceph_backend_configuration_fsid_shell_vars }}
     oc apply -f - <<EOF
     apiVersion: v1
     kind: ConfigMap
@@ -141,6 +185,23 @@
         [workarounds]
         disable_compute_service_check_for_ffu=true
     ---
+    apiVersion: v1
+    kind: ConfigMap
+    metadata:
+      name: ceph-nova
+      namespace: openstack
+    data:
+      03-ceph-nova.conf: |
+        [libvirt]
+        images_type=rbd
+        images_rbd_pool=vms
+        images_rbd_ceph_conf=/etc/ceph/ceph.conf
+        images_rbd_glance_store_name=default_backend
+        images_rbd_glance_copy_poll_interval=15
+        images_rbd_glance_copy_timeout=600
+        rbd_user=openstack
+        rbd_secret_uuid=$CEPH_FSID
+    ---
     apiVersion: dataplane.openstack.org/v1beta1
     kind: OpenStackDataPlaneService
     metadata:
@@ -150,6 +211,7 @@
       label: nova.compute.extraconfig
       configMaps:
         - nova-compute-extraconfig
+        - ceph-nova
       secrets:
         - nova-cell1-compute-config
         - nova-migration-ssh-key
@@ -269,6 +331,42 @@
             edpm_ovn_ofctrl_wait_before_clear: 8000
     EOF
 
+# TODO: Apply the ceph backend config for Cinder in the original openstack CR, via kustomize
+- name: prepare adopted EDPM workloads to use Ceph backend for Cinder, if configured so
+  no_log: "{{ use_no_log }}"
+  when: cinder_volume_backend == "ceph" or cinder_backup_backend == "ceph"
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    oc patch osdpns/openstack --type=merge --patch "
+    spec:
+      services:
+        - bootstrap
+        - download-cache
+        - configure-network
+        - validate-network
+        - install-os
+        - configure-os
+        - run-os
+        - install-certs
+        - ceph-client
+        - libvirt
+        - nova-compute-extraconfig
+        - ovn
+        - neutron-metadata
+      nodeTemplate:
+        extraMounts:
+        - extraVolType: Ceph
+          volumes:
+          - name: ceph
+            secret:
+              secretName: ceph-conf-files
+          mounts:
+          - name: ceph
+            mountPath: "/etc/ceph"
+            readOnly: true
+    "
+
 - name: set neutron-sriov-nic-agent configuration in the OpenStackDataPlaneNodeSet CR
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |