Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[nova]Set up ssh keys auth for the nova user #446

Merged

Conversation

gibizer
Copy link
Contributor

@gibizer gibizer commented Oct 12, 2023

The nova user needs to be able to copy files between the compute nodes.
The edpm_nova role now expects that a Secret is mounted to the AnsibleEE
CR containing an ssh keypair (ssh-privatekey and ssh-publickey). This
keypair is used to set up the auth for the nova user between the
compute nodes. On the EDPM host the nova user will have the public key
added as authorized_keys. While the nova user in the nova-compute
container will have the private key added as identity in
/var/lib/nova/.ssh. Also an ssh config file is added in the container to
ensure that this identity is used for all the outgoing ssh connection.

Depends-On: openstack-k8s-operators/dataplane-operator#471

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 12, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@gibizer gibizer changed the title Copy nova migration keys [nova]Set up ssh keys auth for the nova user Oct 12, 2023
@softwarefactory-project-zuul
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/771d8fdcd545417ea7d0d1cd596c2701

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 07m 56s
podified-multinode-edpm-deployment-crc FAILURE in 1h 52m 29s
cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 4m 04s
✔️ edpm-ansible-molecule-edpm_podman SUCCESS in 6m 12s
✔️ edpm-ansible-molecule-edpm_module_load SUCCESS in 5m 29s
✔️ edpm-ansible-molecule-edpm_kernel SUCCESS in 11m 13s
✔️ edpm-ansible-molecule-edpm_libvirt SUCCESS in 6m 03s
edpm-ansible-molecule-edpm_nova FAILURE in 5m 51s
✔️ edpm-ansible-molecule-edpm_frr SUCCESS in 6m 58s

@lewisdenny
Copy link
Contributor

recheck
Due to a CI Blocker

@softwarefactory-project-zuul
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/653e24168f0c4889a7321b9d4cbaa756

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 10m 00s
podified-multinode-edpm-deployment-crc FAILURE in 1h 55m 26s
cifmw-crc-podified-edpm-baremetal FAILURE in 1h 52m 19s
✔️ edpm-ansible-molecule-edpm_podman SUCCESS in 6m 17s
✔️ edpm-ansible-molecule-edpm_module_load SUCCESS in 5m 39s
✔️ edpm-ansible-molecule-edpm_kernel SUCCESS in 11m 31s
✔️ edpm-ansible-molecule-edpm_libvirt SUCCESS in 6m 58s
edpm-ansible-molecule-edpm_nova FAILURE in 6m 28s
✔️ edpm-ansible-molecule-edpm_frr SUCCESS in 7m 21s

@gibizer
Copy link
Contributor Author

gibizer commented Oct 13, 2023

The CI failure is relevant. I'm working on a fix to this PR

2023-10-12T15:37:21.586220014+00:00 stdout F �[0;31mfatal: [edpm-compute-0]: FAILED! => {"changed": false, "msg": "There was an issue creating /home/nova/.ssh as requested: [Errno 13] Permission denied: b'/home/nova/.ssh'", "path": "/home/nova/.ssh"}�[0m

@gibizer gibizer force-pushed the copy-nova-migration-keys branch from f15b3ba to bc01d7f Compare October 13, 2023 09:15
@softwarefactory-project-zuul
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/1018a42e375842b59e0fd361b1fa3f04

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 05m 13s
podified-multinode-edpm-deployment-crc FAILURE in 1h 51m 54s
cifmw-crc-podified-edpm-baremetal FAILURE in 1h 43m 39s
✔️ edpm-ansible-molecule-edpm_podman SUCCESS in 5m 57s
✔️ edpm-ansible-molecule-edpm_module_load SUCCESS in 5m 10s
✔️ edpm-ansible-molecule-edpm_kernel SUCCESS in 11m 01s
✔️ edpm-ansible-molecule-edpm_libvirt SUCCESS in 6m 08s
edpm-ansible-molecule-edpm_nova FAILURE in 6m 04s
✔️ edpm-ansible-molecule-edpm_frr SUCCESS in 6m 50s

@gibizer
Copy link
Contributor Author

gibizer commented Oct 13, 2023

CI failure is relevant. The nova role now expects an ssh key passed in so I first need to modify the OpenStackDataPlaneService/nova sample and/or the install_yamls / ci_framework scripting to provide an extra secret to service

TASK [osp.edpm.edpm_nova : Copy the migration ssh public key as authorized_keys to the nova user] ***
Friday 13 October 2023  11:01:34 +0000 (0:00:00.585)       0:00:16.505 ******** 
�[0;31mfatal: [edpm-compute-0]: FAILED! => {"changed": false, "msg": "Source /var/lib/openstack/config/nova/ssh-publickey not found"}�[0m

@softwarefactory-project-zuul
Copy link

This change depends on a change that failed to merge.

Change #445 is needed.

@gibizer gibizer force-pushed the copy-nova-migration-keys branch from ca21079 to c5793bc Compare October 13, 2023 15:16
@softwarefactory-project-zuul
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/2721daa9b75d4a11b786012f4f8a481c

openstack-k8s-operators-content-provider FAILURE in 10m 53s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
edpm-ansible-molecule-edpm_podman FAILURE in 13m 03s
✔️ edpm-ansible-molecule-edpm_module_load SUCCESS in 5m 43s
✔️ edpm-ansible-molecule-edpm_kernel SUCCESS in 11m 17s
edpm-ansible-molecule-edpm_libvirt FAILURE in 13m 10s
edpm-ansible-molecule-edpm_nova FAILURE in 12m 59s
edpm-ansible-molecule-edpm_frr FAILURE in 8m 13s

gibizer added a commit to gibizer/dataplane-operator that referenced this pull request Oct 16, 2023
The nova role in edpm-ansible requires[1] an ssh key-pair that is then used
as to authenticate the nova user when it copies data between compute
nodes during migration.

A placeholder Secret name for this key-pair is now added to the nova
DataplaneService definition. The deployer (human, ci_framework, or
install_yamls) needs to make sure that a secret is created with an SSH
key-pair in it.

[1] openstack-k8s-operators/edpm-ansible#446
gibizer added a commit to gibizer/dataplane-operator that referenced this pull request Oct 16, 2023
The nova role in edpm-ansible requires[1] an ssh key-pair that is then used
as to authenticate the nova user when it copies data between compute
nodes during migration.

A placeholder Secret name for this key-pair is now added to the nova
DataplaneService definition. The deployer (human, ci_framework, or
install_yamls) needs to make sure that a secret is created with an SSH
key-pair in it.

[1] openstack-k8s-operators/edpm-ansible#446
gibizer added a commit to gibizer/dataplane-operator that referenced this pull request Oct 16, 2023
The nova role in edpm-ansible requires[1] an ssh key-pair that is then used
as to authenticate the nova user when it copies data between compute
nodes during migration.

A placeholder Secret name for this key-pair is now added to the nova
DataplaneService definition. The deployer (human, ci_framework, or
install_yamls) needs to make sure that a secret is created with an SSH
key-pair in it.

[1] openstack-k8s-operators/edpm-ansible#446
@gibizer gibizer force-pushed the copy-nova-migration-keys branch from c5793bc to dec6f38 Compare October 16, 2023 13:46
@gibizer gibizer marked this pull request as ready for review October 16, 2023 13:46
@openshift-ci openshift-ci bot requested review from abays and frenzyfriday October 16, 2023 13:46
@gibizer
Copy link
Contributor Author

gibizer commented Oct 16, 2023

/hold need to land the dataplane-operator change first

gibizer added a commit to gibizer/dataplane-operator that referenced this pull request Oct 16, 2023
The nova role in edpm-ansible requires[1] an ssh key-pair that is then used
as to authenticate the nova user when it copies data between compute
nodes during migration.

A placeholder Secret name for this key-pair is now added to the nova
DataplaneService definition. The deployer (human, ci_framework, or
install_yamls) needs to make sure that a secret is created with an SSH
key-pair in it.

[1] openstack-k8s-operators/edpm-ansible#446
@softwarefactory-project-zuul
Copy link

This change depends on a change that failed to merge.

Change openstack-k8s-operators/dataplane-operator#471 is needed.

Copy link
Contributor

@rebtoor rebtoor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just few minor changes, when fixed this patch is LGTM for me.

The nova user needs to be able to copy files between the compute nodes.
The edpm_nova role now expects that a Secret is mounted to the AnsibleEE
CR containing an ssh keypair (ssh-privatekey and ssh-publickey). This
keypair is used to set up the auth for the nova user between the
compute nodes. On the EDPM host the nova user will have the public key
added as authorized_keys. While the nova user in the nova-compute
container will have the private key added as identity in
/var/lib/nova/.ssh. Also an ssh config file is added in the container to
ensure that this identity is used for all the outgoing ssh connection.
@gibizer gibizer force-pushed the copy-nova-migration-keys branch from b9ca9a5 to 2f45fb5 Compare October 19, 2023 12:22
@gibizer
Copy link
Contributor Author

gibizer commented Oct 19, 2023

Just few minor changes, when fixed this patch is LGTM for me.

Thanks for the review. I resolved your comments.

gibizer added a commit to gibizer/nova-operator that referenced this pull request Oct 19, 2023
Now that the deployment is configured to support cold migration we can
enable the relevant tempest tests as well.

Depends-On: openstack-k8s-operators/edpm-ansible#446
@openshift-ci openshift-ci bot added the lgtm label Oct 19, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 19, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gibizer, rebtoor

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gibizer
Copy link
Contributor Author

gibizer commented Oct 19, 2023

/unhold
dataplane dependency landed

@softwarefactory-project-zuul
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/2610e2b604214950a04780089bc29012

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 58m 33s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 38m 55s
cifmw-crc-podified-edpm-baremetal FAILURE in 50m 37s
✔️ edpm-ansible-molecule-edpm_podman SUCCESS in 6m 57s
✔️ edpm-ansible-molecule-edpm_module_load SUCCESS in 6m 14s
✔️ edpm-ansible-molecule-edpm_kernel SUCCESS in 12m 26s
✔️ edpm-ansible-molecule-edpm_libvirt SUCCESS in 7m 18s
✔️ edpm-ansible-molecule-edpm_nova SUCCESS in 6m 29s
✔️ edpm-ansible-molecule-edpm_frr SUCCESS in 7m 08s
✔️ edpm-ansible-molecule-edpm_iscsid SUCCESS in 5m 12s
✔️ edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 7m 45s

@gibizer
Copy link
Contributor Author

gibizer commented Oct 19, 2023

recheck
just the baremetal job failed with unrelated reasons

Warning: resource namespaces/openstack-operators is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by oc apply. oc apply should only be used on resources created declaratively by either oc create --save-config or oc apply. The missing annotation will be patched automatically.
error: unable to default to a user name: the server is currently unable to handle the request (get users.user.openshift.io ~)
gmake: *** [Makefile:450: operator_namespace] Error 1

@softwarefactory-project-zuul
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/ccf6af9709f7447a86eb5e2747d37be0

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 15m 40s
podified-multinode-edpm-deployment-crc FAILURE in 1h 55m 07s
cifmw-crc-podified-edpm-baremetal FAILURE in 1h 42m 51s
✔️ edpm-ansible-molecule-edpm_podman SUCCESS in 6m 06s
✔️ edpm-ansible-molecule-edpm_module_load SUCCESS in 5m 18s
✔️ edpm-ansible-molecule-edpm_kernel SUCCESS in 10m 46s
✔️ edpm-ansible-molecule-edpm_libvirt SUCCESS in 7m 48s
✔️ edpm-ansible-molecule-edpm_nova SUCCESS in 6m 14s
✔️ edpm-ansible-molecule-edpm_frr SUCCESS in 7m 00s
✔️ edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 53s
✔️ edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 7m 41s

@gibizer
Copy link
Contributor Author

gibizer commented Oct 19, 2023

As this was green before I think the depends-on is not working correctly. I guess when the needed dataplane-operator PR was still open the job picket it up, but now it is merged so the job thinks that it is available automatically, however it is not built into the openstack-operator yet so it is not available in the deployment.

TASK [osp.edpm.edpm_nova : Copy the migration ssh public key as authorized_keys to the nova user] ***
Thursday 19 October 2023  16:48:26 +0000 (0:00:00.707)       0:00:20.519 ****** 
�[0;31mfatal: [edpm-compute-0]: FAILED! => {"changed": false, "msg": "Source /var/lib/openstack/config/nova/ssh-publickey not found"}�[0m

@SeanMooney
Copy link
Contributor

recheck dep is now merged

@SeanMooney
Copy link
Contributor

the content provider job is proably not setup to support depends on currently.
or it is but podified-multinode-edpm-deployment-crc and cifmw-crc-podified-edpm-baremetal
are not using it properly i missed your comment about it not being built into the operator yet hopefully that is now resolved

@softwarefactory-project-zuul
Copy link

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://review.rdoproject.org/zuul/buildset/42d17eda73034e429b86a2fa415cdc45

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 45s
podified-multinode-edpm-deployment-crc FAILURE in 1h 53m 54s
cifmw-crc-podified-edpm-baremetal FAILURE in 1h 47m 29s
✔️ edpm-ansible-molecule-edpm_podman SUCCESS in 8m 20s
✔️ edpm-ansible-molecule-edpm_module_load SUCCESS in 5m 52s
✔️ edpm-ansible-molecule-edpm_kernel SUCCESS in 10m 54s
✔️ edpm-ansible-molecule-edpm_libvirt SUCCESS in 8m 40s
✔️ edpm-ansible-molecule-edpm_nova SUCCESS in 7m 22s
✔️ edpm-ansible-molecule-edpm_frr SUCCESS in 6m 55s
✔️ edpm-ansible-molecule-edpm_iscsid SUCCESS in 6m 36s
✔️ edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 8m 17s

@gibizer
Copy link
Contributor Author

gibizer commented Oct 20, 2023

recheck
openstack-operator bump merged that updated to the latest dataplane-operator. I hope this is enough

@openshift-ci openshift-ci bot merged commit 41da5bf into openstack-k8s-operators:main Oct 20, 2023
gibizer added a commit to gibizer/nova-operator that referenced this pull request Oct 24, 2023
Now that the deployment is configured to support cold migration we can
enable the relevant tempest tests as well.

Depends-On: openstack-k8s-operators/edpm-ansible#446
gibizer added a commit to gibizer/nova-operator that referenced this pull request Nov 12, 2023
Now that the deployment is configured to support cold migration we can
enable the relevant tempest tests as well.

Depends-On: openstack-k8s-operators/edpm-ansible#446
gibizer added a commit to gibizer/nova-operator that referenced this pull request Nov 12, 2023
Now that the deployment is configured to support cold migration we can
enable the relevant tempest tests as well.

Depends-On: openstack-k8s-operators/edpm-ansible#446
openshift-merge-bot bot pushed a commit to openstack-k8s-operators/nova-operator that referenced this pull request Nov 13, 2023
Now that the deployment is configured to support cold migration we can
enable the relevant tempest tests as well.

Depends-On: openstack-k8s-operators/edpm-ansible#446
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants