Ceph: workloads adoption with cinder volume

[bogdando]

Enable back the cinder volume commands on the source cloud, and
resume testing of the ceph-backed volume attached to the test VM.

Extend volume/backup/snapshot/attachment commands to wait for
the previous step results.

Follow the EDPM Post Ceph steps of HCI VA to prepare adopted
workloads for using Ceph backend on EDPM.

Add Nova discover host command (step 5 of the HCI VA).

Add Nova Ceph custom configs to properly configure ceph
vms pool for libvirt.

Combine nova-ceph related configurations and nova FFU related
ones into a single nova-compute-extraconfig service (by design,
having two dataplane services for Nova in the same node set is
not supported).

Note about available choises for libvirt storage backends for Nova

Add nova_libvirt_backend to control either to deploy with the local
or ceph storage EDPM backends

Depends-On: https://review.rdoproject.org/r/c/rdo-jobs/+/52932

Jira OSPRH-4217

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/bde0113029fd45b895b39e9a08d83512

❌ data-plane-adoption-osp-17-to-extracted-crc FAILURE in 1h 22m 59s
✔️ adoption-docs-preview SUCCESS in 1m 49s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/68bba9a4da9f466c8ce96f1810882314

❌ data-plane-adoption-osp-17-to-extracted-crc FAILURE in 1h 18m 40s
✔️ adoption-docs-preview SUCCESS in 1m 46s

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/70b75dc56cac404e8f5ad9c258763e11

❌ data-plane-adoption-osp-17-to-extracted-crc FAILURE in 2h 24m 52s
✔️ adoption-docs-preview SUCCESS in 2m 03s

[fultonj]

@fultonj @fmount could you please inspect this for missing bits? I believe I followed the documented steps and didn't miss anything? On my local setup, I see ceph-client EDPM service has produced these files:

[root@standalone ~]# cat /etc/ceph/ceph.client.openstack.keyring
[client.openstack]
   key = "xxx snip xxx=="
   caps mgr = allow *
   caps mon = profile rbd
   caps osd = profile rbd pool=volumes, profile rbd pool=images, profile rbd pool=backups
[root@standalone ~]# cat /etc/ceph/ceph.conf 
# minimal ceph.conf for 00d3cde9-0501-5b30-a533-e69040d6dcde
[global]
        fsid = 00d3cde9-0501-5b30-a533-e69040d6dcde
        mon_host = [v2:172.18.0.100:3300/0,v1:172.18.0.100:6789/0]

CI failure in logs :

ult default] Could not find nvme_core/parameters/multipath: FileNotFoundError: [Errno 2] No such file or directory: '/sys/module/nvme_core/parameters/multipath'
2024-02-29 17:43:46.523 209 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): findmnt / -n -o SOURCE execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:384
2024-02-29 17:43:46.536 209 DEBUG oslo_concurrency.processutils [-] CMD "findmnt / -n -o SOURCE" returned: 0 in 0.012s execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:422
2024-02-29 17:43:46.536 209 DEBUG oslo.privsep.daemon [-] privsep: reply[139919505952080]: (4, ('overlay
', '')) _call_back /usr/lib/python3.9/site-packages/oslo_privsep/daemon.py:512
2024-02-29 17:43:46.537 209 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): blkid overlay -s UUID -o value execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:384
2024-02-29 17:43:46.543 209 DEBUG oslo_concurrency.processutils [-] CMD "blkid overlay -s UUID -o value" returned: 2 in 0.006s execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:422
2024-02-29 17:43:46.543 209 DEBUG oslo_concurrency.processutils [-] 'blkid overlay -s UUID -o value' failed. Not Retrying. execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:473
2024-02-29 17:43:46.543 209 DEBUG oslo.privsep.daemon [-] privsep: Exception during request[139919505952080]: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2
Stdout: ''
Stderr: '' _process_cmd /usr/lib/python3.9/site-packages/oslo_privsep/daemon.py:490
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 487, in _process_cmd
    ret = func(*f_args, **f_kwargs)
  File "/usr/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 255, in _wrap
    return func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/os_brick/privileged/rootwrap.py", line 197, in execute_root
    return custom_execute(*cmd, shell=False, run_as_root=False, **kwargs)
  File "/usr/lib/python3.9/site-packages/os_brick/privileged/rootwrap.py", line 145, in custom_execute
    return putils.execute(on_execute=on_execute,
  File "/usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py", line 438, in execute
    raise ProcessExecutionError(exit_code=_returncode,
oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Command: blkid overlay -s UUID -o value
Exit code: 2

I don't see how this os_brick error is connected to Ceph. Let's work our way down from the top to make sure our assumptions are correct.

I understand that /etc/ceph/ceph.client.openstack.keyring exists on the container host. Do we know if the nova containers have it (along with the ceph.conf)? Was a nova secret.xml created? For exampe:

bash-5.1$ cat /etc/nova/secret.xml
<secret ephemeral='no' private='no'>
  <usage type='ceph'>
    <name>client.openstack secret</name>
  </usage>
  <uuid>604c9994-1d82-11ed-8ae5-5254003d6107</uuid>
</secret>
bash-5.1$

Can we use NovaEnableRbdBackend: true?

[fultonj]

update2: I may be missing this https://github.com/openstack-k8s-operators/ci-framework/blob/34de0c6392c1c713af46a8b405e7123eacc89950/ci_framework/roles/hci_prepare/templates/configmap-ceph-nova.yml.j2#L8

Yes, if you're missing the override to nova.conf to tell [libvirt] to use rbd that will explain the problem. Let's see how it works with those parameters.

[softwarefactory-project-zuul]

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/data-plane-adoption for 280,4a5aa2c50d7c4997ff10cd58985811ccdb367e69

[fultonj]

The ceph related changes look good.

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/2c4fedb2fb344292bfdd2c23101962c1

❌ data-plane-adoption-osp-17-to-extracted-crc FAILURE in 2h 07m 24s
✔️ adoption-docs-preview SUCCESS in 1m 44s

[bogdando]

@GIBI on my dev setup I get this:

Mar 06 16:04:26 standalone.localdomain nova_compute[509696]: ERROR:__main__:Failed to change ownership of /var/lib/nova/compute_id to 42436:42436
Mar 06 16:04:26 standalone.localdomain nova_compute[509696]: Traceback (most recent call last):
Mar 06 16:04:26 standalone.localdomain nova_compute[509696]:   File "/usr/local/bin/kolla_set_configs", line 343, in set_perms
Mar 06 16:04:26 standalone.localdomain nova_compute[509696]:     os.chown(path, uid, gid)
Mar 06 16:04:26 standalone.localdomain nova_compute[509696]: PermissionError: [Errno 1] Operation not permitted: '/var/lib/nova/compute_id'

the similar failure is when I consequently re-deploy the same EDPM node, w/o full clean up:

ssh -i /home/cloud-user/install_yamls/out/edpm/ansibleee-ssh-key-id_rsa [email protected] 'echo 940ea0e1-c87f-4731-a9ee-c96d9d1a712a | sudo tee /var/lib/nova/compute_id && sudo chown 42436:42436 /var/lib/nova/compute_id && sudo chcon -t container_file_t /var/lib/nova/compute_id'
tee: /var/lib/nova/compute_id: Operation not permitted
940ea0e1-c87f-4731-a9ee-c96d9d1a712a

this idempotency issue is related to using chattr command to lock that file. For the latter, fix is #333. Not sure about the kolla part

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/9b3b601c9793423da7853533cd268c11

❌ data-plane-adoption-osp-17-to-extracted-crc FAILURE in 2h 26m 20s
✔️ adoption-docs-preview SUCCESS in 2m 08s

[bogdando]

recheck new dep

[softwarefactory-project-zuul]

This change depends on a change that failed to merge.

Changes openstack-k8s-operators/install_yamls#757, https://review.rdoproject.org/r/c/rdo-jobs/+/52932 are needed.

[softwarefactory-project-zuul]

Merge Failed.

This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.
Warning:
Error merging github.com/openstack-k8s-operators/data-plane-adoption for 280,fc9046ebc775e25f7e226f7eda2832fbf321693f

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/aa388c68941842b1a90b5f82a262894a

✔️ data-plane-adoption-osp-17-to-extracted-crc SUCCESS in 2h 27m 26s
✔️ data-plane-adoption-osp-17-to-extracted-crc-minimal-no-ceph SUCCESS in 2h 13m 48s
❌ adoption-docs-preview RETRY_LIMIT in 1m 35s

[bogdando]

/test adoption-docs-preview

[bogdando]

recheck adoption-docs-preview

[bogdando]

good to go

[jistr]

The dependency didn't land yet? Please let's only remove the dependency label when there is no depends-on or when all of it has landed. The purpose of it isn't to block, it is to alert the reviewer about to push the merge button to check dependency order. I created it for myself after merging a few patches in wrong order because i didn't notice the dependency :).

[jistr]

The dependency is merged, merging this one too.