Ceph: workloads adoption with cinder volume

Enable back the cinder volume commands on the source cloud, and resume testing of the ceph-backed volume attached to the test VM. Extend volume/backup/snapshot/attachment commands to wait for the previous step results. Follow the EDPM Post Ceph steps of HCI VA to prepare adopted workloads for using Ceph backend on EDPM. Add Nova discover host command (step 5 of the HCI VA). Add Nova Ceph custom EDPM service for properly configure ceph vms pool for libvirt. Signed-off-by: Bohdan Dobrelia <[email protected]>
openstack-k8s-operators · Mar 5, 2024 · bd5eac1 · bd5eac1
1 parent 9c8f450
commit bd5eac1
Show file tree

Hide file tree

Showing 6 changed files with 218 additions and 31 deletions.
diff --git a/docs_dev/assemblies/development_environment.adoc b/docs_dev/assemblies/development_environment.adoc
@@ -282,19 +282,20 @@ ping -c 4 $(openstack server show baremetal-test -f json -c addresses | jq -r .a
 
 ==== Virtual Machine Steps
 
+Create a test VM instance with a test volume attachement:
+
 [,bash]
 ----
 cd ~/data-plane-adoption
 bash tests/roles/development_environment/files/pre_launch.bash
 ----
 
+This also creates a test Cinder volume, a backup from it, and a snapshot of it.
+
 '''
 
 ==== Ceph Storage Steps
 
-Make sure a cinder-volume backend is properly configured, or skip below steps
-to create a test workload without volume attachments.
-
 Confirm the image UUID can be seen in Ceph's images pool.
 
 [,bash]
@@ -307,22 +308,6 @@ Create a Barbican secret
 openstack secret store --name testSecret --payload 'TestPayload'
 ```
 
-Create a Cinder volume, a backup from it, and snapshot it.
-
-[,bash]
-----
-openstack volume create --image cirros --bootable --size 1 disk
-openstack volume backup create --name backup disk
-openstack volume snapshot create --volume disk snapshot
-----
-
-Add volume to the test VM
-
-[,bash]
-----
-openstack server add volume test disk
-----
-
 == Performing the Data Plane Adoption
 
 The development environment is now set up, you can go to the https://openstack-k8s-operators.github.io/data-plane-adoption/[Adoption

diff --git a/docs_user/modules/openstack-edpm_adoption.adoc b/docs_user/modules/openstack-edpm_adoption.adoc
@@ -10,6 +10,7 @@
 
 * Previous Adoption steps completed.
 * Remaining source cloud xref:stopping-infrastructure-management-and-compute-services_{context}[Stopping infrastructure management and Compute services] on Compute hosts.
+* Ceph backend for Nova/Libvirt is configured xref:configuring-a-ceph-backend_{context}[Configuring a Ceph backend].
 
 ____
 *WARNING* This step is a "point of no return" in the EDPM adoption
@@ -27,6 +28,7 @@ The values are just illustrative, use values that are correct for your environme
 
 ----
 PODIFIED_DB_ROOT_PASSWORD=$(oc get -o json secret/osp-secret | jq -r .data.DbRootPassword | base64 -d)
+CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //'
 
 alias openstack="oc exec -t openstackclient -- openstack"
 FIP=192.168.122.20
@@ -195,6 +197,48 @@ The secret `nova-cell<X>-compute-config` is auto-generated for each
 `cell<X>`. That secret, alongside `nova-migration-ssh-key`, should
 always be specified for each custom `OpenStackDataPlaneService` related to Nova.
 
+* Create a Nova service for libvirt Ceph backend
++
+[source,yaml]
+----
+oc apply -f - <<EOF
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: ceph-nova
+  namespace: openstack
+data:
+  03-ceph-nova.conf: |
+    [libvirt]
+    images_type=rbd
+    images_rbd_pool=vms
+    images_rbd_ceph_conf=/etc/ceph/ceph.conf
+    images_rbd_glance_store_name=default_backend
+    images_rbd_glance_copy_poll_interval=15
+    images_rbd_glance_copy_timeout=600
+    rbd_user=openstack
+    rbd_secret_uuid=$CEPH_FSID
+---
+apiVersion: dataplane.openstack.org/v1beta1
+kind: OpenStackDataPlaneService
+metadata:
+  name: nova-custom-ceph
+  namespace: openstack
+spec:
+  label: nova.custom.ceph
+  configMaps:
+    - ceph-nova
+  secrets:
+    - nova-cell1-compute-config
+    - nova-migration-ssh-key
+  playbook: osp.edpm.nova
+EOF
+----
++
+This should be a cell-specific configuration.
+For multi-cell, the config map should be named like `nova-custom-ceph-cellX`.
+For a single cell scenario, we name it simply `nova-custom-ceph`.
+
 * Deploy OpenStackDataPlaneNodeSet:
 +
 Make sure that ovn-controller settings configured in the OpenStackDataPlaneNodeSet are the same as were set in the compute nodes before adoption.
@@ -231,6 +275,7 @@ spec:
     - install-certs
     - libvirt
     - nova-compute-extraconfig
+    - nova-custom-ceph
     - ovn
     - neutron-metadata
   env:
@@ -392,6 +437,45 @@ endif::[]
 EOF
 ----
 
+* Prepare adopted EDPM workloads to use Ceph backend for Cinder, if configured so
++
+[source,yaml]
+----
+oc patch osdpns/openstack --type=merge --patch "
+spec:
+  services:
+    - repo-setup
+    - download-cache
+    - bootstrap
+    - configure-network
+    - validate-network
+    - install-os
+    - configure-os
+    - run-os
+    - reboot-os
+    - install-certs
+    - ceph-client
+    - libvirt
+    - nova-compute-extraconfig
+    - ovn
+    - neutron-metadata
+  nodeTemplate:
+    extraMounts:
+    - extraVolType: Ceph
+      volumes:
+      - name: ceph
+        secret:
+          secretName: ceph-conf-files
+      mounts:
+      - name: ceph
+        mountPath: "/etc/ceph"
+        readOnly: true
+"
+----
+
+Note that you should retain the original `OpenStackDataPlaneNodeSet` services
+composition, except the inserted `ceph-client` service.
+
 * Deploy OpenStackDataPlaneDeployment:
 +
 [source,yaml]
@@ -571,6 +655,12 @@ The above query should return an empty result as a completion criterion.
   oc exec -it nova-cell1-conductor-0 -- nova-manage db online_data_migrations
 ----
 
+* Ask Nova to discover all compute hosts
++
+----
+oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
+----
+
 * Verify if Nova services can stop the existing test VM instance:
 +
 ----

diff --git a/tests/roles/dataplane_adoption/tasks/main.yaml b/tests/roles/dataplane_adoption/tasks/main.yaml
@@ -1,3 +1,9 @@
+- name: set shell vars for edpm adoption
+  no_log: "{{ use_no_log }}"
+  ansible.builtin.set_fact:
+    ceph_backend_configuration_fsid_shell_vars: |
+      CEPH_FSID=$(oc get secret ceph-conf-files -o json | jq -r '.data."ceph.conf"' | base64 -d | grep fsid | sed -e 's/fsid = //'
+
 - name: use ansible-runner image built from source or latest if none is defined
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |
@@ -139,6 +145,45 @@
       playbook: osp.edpm.nova
     EOF
 
+- name: create a Nova service for libvirt Ceph backend
+  no_log: "{{ use_no_log }}"
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    {{ ceph_backend_configuration_fsid_shell_vars}}
+    oc apply -f - <<EOF
+    apiVersion: v1
+    kind: ConfigMap
+    metadata:
+      name: ceph-nova
+      namespace: openstack
+    data:
+      03-ceph-nova.conf: |
+        [libvirt]
+        images_type=rbd
+        images_rbd_pool=vms
+        images_rbd_ceph_conf=/etc/ceph/ceph.conf
+        images_rbd_glance_store_name=default_backend
+        images_rbd_glance_copy_poll_interval=15
+        images_rbd_glance_copy_timeout=600
+        rbd_user=openstack
+        rbd_secret_uuid=$CEPH_FSID
+    ---
+    apiVersion: dataplane.openstack.org/v1beta1
+    kind: OpenStackDataPlaneService
+    metadata:
+      name: nova-custom-ceph
+      namespace: openstack
+    spec:
+      label: nova.custom.ceph
+      configMaps:
+        - ceph-nova
+      secrets:
+        - nova-cell1-compute-config
+        - nova-migration-ssh-key
+      playbook: osp.edpm.nova
+    EOF
+
 - name: deploy dataplane
   ansible.builtin.shell: |
     {{ shell_header }}
@@ -164,6 +209,7 @@
         - install-certs
         - libvirt
         - nova-compute-extraconfig
+        - nova-custom-ceph
         - ovn
         - neutron-metadata
       env:
@@ -256,6 +302,42 @@
             edpm_ovn_ofctrl_wait_before_clear: 8000
     EOF
 
+- name: prepare adopted EDPM workloads to use Ceph backend for Cinder, if configured so
+  no_log: "{{ use_no_log }}"
+  when: cinder_volume_backend == "ceph" or cinder_backup_backend == "ceph"
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    oc patch osdpns/openstack --type=merge --patch "
+    spec:
+      services:
+        - bootstrap
+        - download-cache
+        - configure-network
+        - validate-network
+        - install-os
+        - configure-os
+        - run-os
+        - reboot-os
+        - install-certs
+        - ceph-client
+        - libvirt
+        - nova-compute-extraconfig
+        - ovn
+        - neutron-metadata
+      nodeTemplate:
+        extraMounts:
+        - extraVolType: Ceph
+          volumes:
+          - name: ceph
+            secret:
+              secretName: ceph-conf-files
+          mounts:
+          - name: ceph
+            mountPath: "/etc/ceph"
+            readOnly: true
+    "
+
 - name: deploy the dataplane deployment
   no_log: "{{ use_no_log }}"
   ansible.builtin.shell: |

diff --git a/tests/roles/dataplane_adoption/tasks/nova_ffu.yaml b/tests/roles/dataplane_adoption/tasks/nova_ffu.yaml
@@ -108,7 +108,7 @@
   retries: 10
   delay: 6
 
-- name: run Nova DB migrations to complete Wallaby->antelope FFU
+- name: run Nova DB online migrations to complete FFU
   ansible.builtin.shell: |
     {{ shell_header }}
     {{ oc_header }}
@@ -118,3 +118,9 @@
   until: nova_exec_result is success
   retries: 10
   delay: 6
+
+- name: ask Nova to discover all compute hosts
+  ansible.builtin.shell: |
+    {{ shell_header }}
+    {{ oc_header }}
+    oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose
diff --git a/tests/roles/development_environment/files/pre_launch.bash b/tests/roles/development_environment/files/pre_launch.bash
@@ -1,5 +1,20 @@
 set -e
 
+function wait_for_status() {
+    local time=0
+    local msg="Waiting for $2"
+    local status="${3:-available}"
+    local result
+    while [ $time -le 30 ] ; do
+        result=$(${BASH_ALIASES[openstack]} $1 -f json)
+        echo $result | jq -r ".status" | grep -q $status && break
+        echo "result=$result"
+        echo "$msg"
+        time=$(( time + 5 ))
+        sleep 5
+    done
+}
+
 # Create Image
 IMG=cirros-0.5.2-x86_64-disk.img
 URL=http://download.cirros-cloud.net/0.5.2/$IMG
@@ -50,15 +65,23 @@ export FIP=192.168.122.20
 # check connectivity via FIP
 ping -c4 ${FIP}
 
-# FIXME: Invalid volume: Volume xxx status must be available, but current status is: backing-up
-# Create a Cinder volume, a backup from it, and snapshot it.
-#${BASH_ALIASES[openstack]} volume show disk || \
-#    ${BASH_ALIASES[openstack]} volume create --image cirros --bootable --size 1 disk
-#${BASH_ALIASES[openstack]} volume backup show backup || \
-#    ${BASH_ALIASES[openstack]} volume backup create --name backup disk
-#${BASH_ALIASES[openstack]} volume snapshot show snapshot || \
-#    ${BASH_ALIASES[openstack]} volume snapshot create --volume disk snapshot
+if ! ${BASH_ALIASES[openstack]} volume show disk ; then
+    ${BASH_ALIASES[openstack]} volume create --image cirros --bootable --size 1 disk
+    wait_for_status "volume show disk" "test volume 'disk' creation"
+fi
+
+if ! ${BASH_ALIASES[openstack]} volume backup show backup; then
+    ${BASH_ALIASES[openstack]} volume backup create --name backup disk
+    wait_for_status "volume backup show backup" "test volume 'disk' backup completion"
+fi
+
+if ! ${BASH_ALIASES[openstack]} volume snapshot show snapshot ; then
+    ${BASH_ALIASES[openstack]} volume snapshot create --volume disk snapshot
+    wait_for_status "volume snapshot show snapshot" "test volume 'disk' backup snapshot availability"
+fi
+
+# Add volume to the test VM
+if ${BASH_ALIASES[openstack]} volume show disk -f json | jq -r '.status' | grep -q available ; then
+    ${BASH_ALIASES[openstack]} server add volume test disk
+fi
 
-# TODO: Add volume to the test VM, after tripleo wallaby (osp 17) isolnet network adoption implemented for storage networks
-#${BASH_ALIASES[openstack]} volume show disk -f json | jq -r '.status' | grep -q available && \
-#    ${BASH_ALIASES[openstack]} server add volume test disk
diff --git a/tests/roles/development_environment/tasks/main.yaml b/tests/roles/development_environment/tasks/main.yaml
@@ -6,5 +6,6 @@
     EDPM_PRIVATEKEY_PATH: "{{ edpm_privatekey_path }}"
   ansible.builtin.shell:
     cmd: |
+      {{ shell_header }}
       alias openstack="ssh -i {{ edpm_privatekey_path }} root@{{ edpm_node_ip }} OS_CLOUD=standalone openstack"
       {{ lookup('ansible.builtin.file', prelaunch_test_instance_script) }}