-
Notifications
You must be signed in to change notification settings - Fork 27
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[spec] Propose EDPM OpenStack service architecture
- Loading branch information
Showing
1 changed file
with
323 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,323 @@ | ||
| # OpenStack service deployment architecture on EDPM Node | ||
|
|
||
| ## Problem description | ||
|
|
||
| Various OpenStack componentes have agents that need to run on the External | ||
| DataPlane Node. For example OpenStack Nova's nova-api, nova-conductor | ||
| and nova-scheduler services can run in kubernetes pods independently from the | ||
| hypervisors they manage. In the other hand the nova-compute service is designed | ||
| to be collocated with the hyperviors it manages. In the current design the EDPM | ||
| Node serves as a hypervisor for Nova, therefore the nova-compute service needs | ||
| to be deployed on EDPM nodes. | ||
|
|
||
| Similar examples from other OpenStack compontents are ovn-metadata-agent, neutron-nicswitch agent, and in the HCI case the cinder-volume service. | ||
|
|
||
| ### The deployment dependency | ||
|
|
||
| The deployment of an EDPM node needs to consider multiple phases | ||
|
|
||
| * **phase -1: Bare metal provisioning**. This step transforms an empty bare | ||
| metal machine to a runnign server with a host OS. | ||
| * **phase 0: Infrastructure deployment and configuration** of storage, | ||
| networking, and computing resources. This step transforms a generic running | ||
| server to a more specific storage node, networking node, computing | ||
| hypervisor, or a mix of these. | ||
| * **phase 1: OpenStack service deployment**. This transforms the server to act | ||
| as an OpenStack storage node, networking node, or compute node (or a mix of | ||
| them) by deploying and configuring the specific OpenStack services that | ||
| connects the server to the OpenStack control plane. | ||
|
|
||
| Both phase -1 and 0 can be performed independently from the rest of the | ||
| OpenStack control plane deployment. However phase 1 needs to be coordinated | ||
| with the podified infrastructure, and OpenStack service deployment. For example | ||
| to generate the configuration of the nova-compute service on an EDPM node the | ||
| deployment engine needs to know how to connect the nova-compute service to the | ||
| podified infrasture, e.g. to which nova cell and therefore via which message | ||
| bus. Handling this dependency is one of the scope of this specification. | ||
|
|
||
| ### The OpenStack service configuration interface | ||
|
|
||
| Independently from the fact that an OpenStack service is running in a | ||
| kubernetes pod or in a container on the EDPM Node the service has a well | ||
| defined and widely known configuration interface implemented by oslo.config | ||
| configuration options in one or more *.ini like* files. | ||
|
|
||
| The configuration options can be divided to two major parts: | ||
| * configuration that is generated by the deployment engine. E.g. database | ||
| hostnames and message bus transport URLs | ||
| * configuration that is provided by the human deployer. E.g. CPU allocation | ||
| ratio, PCI device spec and alias | ||
|
|
||
| The existing podified control plane API, the OpenStackControlPlane CRD and the | ||
| service sub CRDs, provides a simple way for the human deployer to pass | ||
| configuration parameters to the OpenStack services. Each CRD that represents | ||
| an OpenStack service type provides a `CustomServiceConfig` field that is a | ||
| plain text field to carrying oslo.config formatted configuration. Then the | ||
| deployment engine makes sure that this configuration snipet is provided | ||
| to the deployed OpenStack service binary in a way that the config options | ||
| defined in it has higher priority than the automatically generated, default | ||
| service configuration. | ||
|
|
||
| On the podified control plane side kubernetes APIs are used to generate, store, | ||
| and transfer the OpenStack service configuration, and to deploy, configure, and | ||
| start the OpenStack service binary. While parts of this is visible to the human | ||
| deployer. E.g. kubernetes Pods, Deployments, ConfigMaps, VolumeMounts. The | ||
| human deployer only needs to understand the `CustomServiceConfig` text fields | ||
| to be able to pass through the desired configuration to an OpenStack service. | ||
|
|
||
| As OpenStack services are running also on the EDPM Nodes there is also a need | ||
| to pass through human defined configuration to these services as well. | ||
|
|
||
| On the EDPM side Ansible roles are used to provision and configure services. | ||
| Ansible has its own configuration passing mechanism, Ansible variables. It is | ||
| technically possible to pass through oslo.config options as Ansible variables | ||
| to configure the OpenStack services. | ||
|
|
||
| However it means that the same oslo.config configuration interface of the | ||
| OpenStack service are represented in two different formats. On the podified | ||
| control plane side it is an oslo.config snipet in the `CustomServiceConfig` | ||
| field. While on the EDPM side Ansible variables. The former does not involve | ||
| any translation. The naming and structure of the config options are exactly the | ||
| same in the `CustomServiceConfig` field as in the actual oslo.config file of | ||
| the OpenStack binary. In the other hand the latter applies a name mangling to | ||
| avoid name clashes between different oslo.config sections and between different | ||
| services using the same Ansible inventory to pass through variales. Moreover | ||
| the latter requires explicit work in the Ansible roles every time a new config | ||
| variable is added. | ||
|
|
||
| For example, enabling debug logging for the OVN metadata agent on the EDPM node | ||
| requires the following oslo.config file to be provided to the service: | ||
| ```ini | ||
| [DEFAULT] | ||
| debug = True | ||
| ``` | ||
|
|
||
| The `CustomServiceConfig` the following needs to be added to the CR | ||
| representing the given OVN metadata agent: | ||
| ```yaml | ||
| customServiceConfig: | | ||
| [DEFAULT] | ||
| debug = True | ||
| ``` | ||
| With ansible variables the following needs to be added to the CR that triggers | ||
| deployment of the given OVN metadata agent: | ||
| ```yaml | ||
| ansibleVars: | | ||
| edpm_ovn_metadata_agent_metadata_agent_DEFAULT_debug: true | ||
| ``` | ||
| ## Proposed change | ||
| Keep the existing DataPlane, DataPlaneRole, and DataPlaneNode CRDs to represent | ||
| and coordinate phase -1 and phase 0 of the EDPM deployment. But decouple | ||
| phase 1 from the existing DataPlane model. This means that the DataPlane CRDs | ||
| will not represent and therefore won't directly deploy any OpenStack services | ||
| on the EDPM node. These CRDs still expose the status of the phase -1 and | ||
| phase 0 deployment, so other operators can decide when to start deploying | ||
| OpenStack service to the EDPM node. | ||
| Define CRDs representing each OpenStack service type that needs to be run on | ||
| the EDPM node (e.g. NovaExternalCompute, OVNMetadataAgent, etc). These CRDs | ||
| are defined in and therefore reconciled by the respective service operator | ||
| (e.g. nova-operator, ovn-operator). These OpenStack service CRDs will be | ||
| created automatically based on the existence of DataPlaneRole and Node CR and | ||
| the user specified annotations on these CRs. The service operator | ||
| will ensure that both the podified control plane dependencies | ||
| (e.g. message bus) are deployed and the EDPM node phase 0 is finished before | ||
| generates the service configuration and creates AnsibleEE CRs to deploy | ||
| OpenStack service on the EDPM node. | ||
| The EDPM related OpenStack service CRDs will have `CustomServiceConfig` fields | ||
| that will have the same syntax and semantic as the same fields in the existing | ||
| podified Openstack service CRDs, while the implementation of such configuration | ||
| will use Ansible to transfer the generated configuration to the service running | ||
| on the EDPM node. | ||
|
|
||
| **Benefits**: | ||
|
|
||
| * The DataPlane CRDs and therefore the dataplane-operator implementation kept | ||
| OpenStack service agnostic. Adding support for deploying a new OpenStack | ||
| or partner service on the EDPM node can be done without changing the | ||
| DataPlane CRDs or the dataplane-operator implementation. | ||
|
|
||
| * The service operators encapsulates all the service specific logic. Every | ||
| deployment, configuration or upgrade concerns for an OpenStack component can | ||
| be handled within a single, cohesive operator implementation. | ||
|
|
||
| * The `CustomServiceConfig` field in the service CRDs simplifies and unifies | ||
| the OpenStack service configuration for the human deployer regardless where | ||
| the service is being depoyed (i.e. podified or EDPM). | ||
|
|
||
| **Drawbacks**: | ||
|
|
||
| * The mechanism to watch DataPlane CRs from service operators and auto create | ||
| service CRs adds impementation complexity and introduces possible performance | ||
| impacts that could be avoided if dataplane-operator creates the service CRs. | ||
|
|
||
| * The Ready condition of the DataPlaneRole and the DataPlaneNode can only | ||
| represent the phase -1 and phase 0 readiness of the EDPM node. The phase 1 | ||
| readiness can only be collected by checking the Ready condition of every | ||
| service CRs related to this EDPM node. | ||
|
|
||
|
|
||
| ### Example | ||
|
|
||
| NovaCell/nova-cell1 | ||
| ```yaml | ||
| apiVersion: nova.openstack.org/v1beta1 | ||
| kind: NovaCell | ||
| metadata: | ||
| name: nova-cell1 | ||
| ... | ||
| ``` | ||
| No changes in NovaCell. | ||
|
|
||
| OpenstackDataPlaneRole/edpm-compute | ||
| ```yaml | ||
| apiVersion: dataplane.openstack.org/v1beta1 | ||
| kind: OpenStackDataPlaneRole | ||
| metadata: | ||
| name: edpm-compute | ||
| annotations: | ||
| nova.openstack.org/cell: nova-cell1 | ||
| spec: | ||
| dataPlane: openstack-edpm | ||
| nodeTemplate: | ||
| ansiblePort: 22 | ||
| ... | ||
| ``` | ||
| The `novaTemplate` is removed from the `nodeTemplate` and a new user specified | ||
| annotation `nova.openstack.org/cell` is added so the human deployer can | ||
| define that this Role describe a set of EDPM nodes that are compute nodes | ||
| connected to cell1. | ||
|
|
||
| OpenstackDataPlaneNode/edpm-compute-0 | ||
| ```yaml | ||
| apiVersion: dataplane.openstack.org/v1beta1 | ||
| kind: OpenStackDataPlaneNode | ||
| metadata: | ||
| name: edpm-compute-0 | ||
| spec: | ||
| ansibleHost: 192.168.122.100 | ||
| hostName: edpm-compute-0 | ||
| node: | ||
| ansibleSSHPrivateKeySecret: dataplane-ansible-ssh-private-key-secret | ||
| role: edpm-compute | ||
| ... | ||
| ``` | ||
| The `novaTemplate` is removed from the `node`. As this DataPlaneNode referst | ||
| to a DataPlaneRole that has a `nova.openstack.org/cell` this DataPlaneNode is | ||
| a compute node connected to cell1. | ||
|
|
||
| OpenstackDataPlaneNode/edpm-compute-1 | ||
| ```yaml | ||
| apiVersion: dataplane.openstack.org/v1beta1 | ||
| kind: OpenStackDataPlaneNode | ||
| metadata: | ||
| name: edpm-compute-1 | ||
| annotations: | ||
| nova.openstack.org/cell: nova-cell2 | ||
| spec: | ||
| ansibleHost: 192.168.122.101 | ||
| hostName: edpm-compute-1 | ||
| node: | ||
| ansibleSSHPrivateKeySecret: dataplane-ansible-ssh-private-key-secret | ||
| role: edpm-compute | ||
| ... | ||
| ``` | ||
| This DataPlaneNode has an `nova.openstack.org/cell` annotation that overrides | ||
| the cell annotation of its role. So this EDPM node is a compute node connected | ||
| to cell2. | ||
|
|
||
| NovaExternalCompute=edpm-compute-0 | ||
| ```yaml | ||
| apiVersion: nova.openstack.org/v1beta1 | ||
| kind: NovaExternalCompute | ||
| metadata: | ||
| name: edpm-compute-0 | ||
| spec: | ||
| dataplaneNodeName: edpm-compute-0 | ||
| cellName: nova-cell1 | ||
| customServiceConfig: | | ||
| # human deployer can uset this to pass through service specific config | ||
| sshKeySecretName: dataplane-ansible-ssh-private-key-secret | ||
| ... | ||
| ``` | ||
| This CR is auto created by the nova-operator based on the corresponding | ||
| DataPlaneNode CR and the cell annotation set on it. The spec fields, except | ||
| `CustomServiceConfig` is populated by the nova-operator. The | ||
| `CustomServiceConfig` field is never set by the nova-operator it can be used | ||
| by the human deployer. | ||
|
|
||
| The following fields will be copied over from the DataplaneNode (or Role due to | ||
| inheritance) and changes there will be propagated to the NovaExternalCompute CR | ||
| * InventoryConfigMapName | ||
| * NetworkAttachments | ||
| * SshKeySecretName | ||
|
|
||
| The AnsibleEEContainerImage, NovaComputeContainerImage, and | ||
| NovaLibvirtContainerImage will be set based on the respective ENV value via the | ||
| defaulting webhook in nova-operator during NovaExternalCompute auto creation | ||
| but these fields can be changed later independently from their defaults. | ||
|
|
||
| The NovaInstance field is removed and the CellName field is redefined to hold | ||
| the name of the NovaCell CR. The name of the NovaCell CR is a combination of | ||
| the name of the cell from the NovaCellTemplate in the Nova CR and the name of | ||
| the Nova CR. So it uniquely identifies a cell even if there are multiple Nova | ||
| CRs exist in the namespace. | ||
|
|
||
| The upstream nova project does not support moving computes between cells. | ||
| Therefore the NovaCell field can only be set at creation and cannot be updated | ||
| later. The dataplane-operator needs to implement a validation hook to prevent | ||
| the related annotation to change after being set. | ||
|
|
||
|
|
||
| ## Alternatives | ||
|
|
||
| * Continue the pattern what we have today. The DataPlane CRDs would embed | ||
| OpenStack speific service templates and the dataplane-operator would create | ||
| service specific CRDs based on that to trigger OpenStack service deployment | ||
| on the EDPM node after phase 0 is succeded. This alternative can support | ||
| `CustomServiceConfig` based OpenStack service configuration. | ||
|
|
||
| It is not the preferred alternative as it would cause that the | ||
| dataplane-operator gather OpenStack service specific implementations. And | ||
| therefore OpenStack component specific knowledge would be split across | ||
| multiple operators. | ||
|
|
||
| * Do not represent OpenStack service types as CRDs. Keep DataPlane CRDs as | ||
| generic Ansible interfaces to trigger service deployment. | ||
|
|
||
| It is not the preferred alternative as it would duplicate the service | ||
| configuration syntax and semantic for the human deployer. | ||
| Also it would move OpenStack service specific logic to the | ||
| dataplane-operator, and therefore OpenStack component specific knowledge | ||
| would be split across multiple operators. | ||
|
|
||
| * Decouple the service CRDs from the DataPlane CRDs as in the main proposal | ||
| but instead of watching the DataPlaneNode CRs to trigger the auto creation | ||
| of the relevant service CR, the DataPlaneNode - Service CR connection is | ||
| defined by DataPlaneNode CR names part of the OpenStackControlPlane CR. | ||
| E.g. the `OpenStackControlePlane.Spec.Nova.Template.CellTemplate` can contain | ||
| a list of DataPlaneNode names to define that the given DataPlaneNode are | ||
| compute nodes and therefore NovaExternalCompute CR creation is needed. | ||
|
|
||
| This alternative would remove the need for the watch in the service operator | ||
| and therefore remove some of the complexity and performance overhead of of | ||
| the main proposal. However it would mean that the OpenstackControlPlane CR | ||
| contains data plane related information. | ||
|
|
||
| ## Implementation considerations | ||
|
|
||
| The nova-operator will have a new controller that will watch DataPlaneNode CRs, | ||
| therefore the controller will Reconcile when a DataPlaneNode CR is created or | ||
| updated. The controller will look up the CR and checks for the | ||
| `nova.openstack.org/cell` annotation. If it is not present then the reconcile | ||
| ends. If the annotation is present then the controller will CreateOrPatch a | ||
| NovaExternalCompute CR based on the fields in the DataPlaneNode CR and the | ||
| fields of the NovaCell CR pointed by the `nova.openstack.org/cell`. | ||
|
|
||
| The existing controller for NovaExternalCompute will reconcile the CR as today | ||
| but it will wait for the ReadyCondition of the DataPlaneNode CR before creating | ||
| the AnsibleEE CRs to deploy compute services on the EDPM node. |