Table of Contents generated with DocToc
- Get started with external openstack-cloud-controller-manager in Kubernetes
External cloud providers were introduced as an Alpha feature in Kubernetes release 1.6. openstack-cloud-controller-manager is the implementation of external cloud provider for OpenStack clusters. An external cloud provider is a kubernetes controller that runs cloud provider-specific loops required for the functioning of kubernetes. These loops were originally a part of the kube-controller-manager
, but they were tightly coupling the kube-controller-manager
to cloud-provider specific code. In order to free the kubernetes project of this dependency, the cloud-controller-manager
was introduced.
cloud-controller-manager
allows cloud vendors and kubernetes core to evolve independent of each other. In prior releases, the core Kubernetes code was dependent upon cloud provider-specific code for functionality. In future releases, code specific to cloud vendors should be maintained by the cloud vendor themselves, and linked to cloud-controller-manager
while running Kubernetes.
For more information about cloud-controller-manager, please see:
- https://github.com/kubernetes/enhancements/tree/master/keps/sig-cloud-provider/2392-cloud-controller-manager
- https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/#running-cloud-controller-manager
- https://kubernetes.io/docs/tasks/administer-cluster/developing-cloud-controller-manager/
NOTE: Now, the openstack-cloud-controller-manager implementation is based on OpenStack Octavia, Neutron-LBaaS has been deprecated in OpenStack since Queens release and no longer maintained in openstack-cloud-controller-manager. So make sure to use Octavia if upgrade to the latest openstack-cloud-controller-manager docker image.
The following guide has been tested to install Kubernetes v1.17 on Ubuntu 18.04.
- docker, kubeadm, kubelet and kubectl has been installed.
-
Create the kubeadm config file according to
manifests/controller-manager/kubeadm.conf
-
Bootstrap the cluster, make sure to install the CNI network plugin as well.
kubeadm init --config kubeadm.conf
-
Bootstrap worker nodes. You need to set
--cloud-provider=external
for kubelet service before runningkubeadm join
. -
Create a secret containing the cloud configuration. You can find an example config file in
manifests/controller-manager/cloud-config
. If you have certs you need put the cert file into folder/etc/ssl/certs/
and updateca-file
in the configuration file, refer toca-file
option here for further information. After that, Save the configuration to a file named cloud.conf, then:kubectl create secret -n kube-system generic cloud-config --from-file=cloud.conf
-
Create RBAC resources and openstack-cloud-controller-manager deamonset.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-roles.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
-
Waiting for all the pods in kube-system namespace up and running.
If you are already running a Kubernetes cluster (installed by kubeadm) but using in-tree openstack cloud provider, switching to openstack-cloud-controller-manager is easy by following the steps in the demo below.
Also, checkout the guide on Migrate to CCM
Implementation of openstack-cloud-controller-manager relies on several OpenStack services.
Service | API Version(s) | Deprecated | Required |
---|---|---|---|
Identity (Keystone) | v3 | No | Yes |
Compute (Nova) | v2 | No | Yes |
Load Balancing (Neutron-LBaaS) | v1, v2 | Yes | No |
Load Balancing (Octavia) | v2 | No | Yes |
Key Manager (Barbican) | v1 | No | No |
NOTE:
- Block Storage is not needed for openstack-cloud-controller-manager in favor of cinder-csi-plugin.
- Barbican is required to support creating Service of LoadBalancer type with TLS termination.
The options in Global
section are used for openstack-cloud-controller-manager authentication with OpenStack Keystone, they are similar to the global options when using openstack
CLI, see more information in openstack man page.
-
auth-url
Required. Keystone service URL, e.g. http://128.110.154.166/identity -
os-endpoint-type
Optional. Specify which type of endpoint to use from the service catalog. If not set, public endpoints are used. -
ca-file
Optional. CA certificate bundle file for communication with Keystone service, this is required when using the https protocol in the Keystone service URL. -
cert-file
Optional. Client certificate path used for the client TLS authentication. -
key-file
Optional. Client private key path used for the client TLS authentication. -
username
Keystone user name. If you are using Keystone application credential, this option is not required. -
password
Keystone user password. If you are using Keystone application credential, this option is not required. -
region
Required. Keystone region name. -
domain-id
Keystone user domain ID. If you are using Keystone application credential, this option is not required. -
domain-name
Keystone user domain name, not required ifdomain-id
is set. -
tenant-id
Keystone project ID. When using Keystone V3 - which changed the identifiertenant
toproject
- thetenant-id
value is automatically mapped to the project construct in the API.tenant-id
is not needed when usingtrust-id
or Keystone application credential -
tenant-name
Keystone project name, not required iftenant-id
is set. -
tenant-domain-id
Keystone project domain ID. -
tenant-domain-name
Keystone project domain name. -
user-domain-id
Keystone user domain ID. -
user-domain-name
Keystone user domain name. -
trust-id
Keystone trust ID. A trust represents a user's (the trustor) authorization to delegate roles to another user (the trustee), and optionally allow the trustee to impersonate the trustor. Available trusts are found under the/v3/OS-TRUST/trusts
endpoint of the Keystone API. -
trustee-id
Keystone trustee user ID. -
trustee-password
Keystone trustee user password. -
use-clouds
Set this option totrue
to get authorization credentials from a clouds.yaml file. Options explicitly set in this section are prioritized over values read from clouds.yaml, the file path can be set inclouds-file
option. Otherwise, the following order is applied:- A file path stored in the environment variable
OS_CLIENT_CONFIG_FILE
- The directory
pkg/openstack
- The directory
~/.config/openstack
- The directory
/etc/openstack
- A file path stored in the environment variable
-
clouds-file
File path of a clouds.yaml file, used together withuse-clouds=true
. -
cloud
Used to specify which named cloud in the clouds.yaml file that you want to use, used together withuse-clouds=true
. -
application-credential-id
The ID of an application credential to authenticate with. Anapplication-credential-secret
has to be set along with this parameter. -
application-credential-name
The name of an application credential to authenticate with. Ifapplication-credential-id
is not set, the user name and domain need to be set. -
application-credential-secret
The secret of an application credential to authenticate with. -
tls-insecure
If set totrue
, then the server’s certificate will not be verified. Default isfalse
.
ipv6-support-disabled
Indicates whether or not IPv6 is supported. Default: falsepublic-network-name
The name of Neutron external network. openstack-cloud-controller-manager uses this option when getting the external IP of the Kubernetes node. Can be specified multiple times. Specified network names will be ORed. Default: ""internal-network-name
The name of Neutron internal network. openstack-cloud-controller-manager uses this option when getting the internal IP of the Kubernetes node, this is useful if the node has multiple interfaces. Can be specified multiple times. Specified network names will be ORed. Default: ""
Although the openstack-cloud-controller-manager was initially implemented with Neutron-LBaaS support, Octavia is recommended now because Neutron-LBaaS has been deprecated since Queens OpenStack release cycle and no longer accepted new feature enhancements. As a result, lots of advanced features in openstack-cloud-controller-manager rely on Octavia, even the CI is running based on Octavia enabled OpenStack environment. Functionalities are not guaranteed if using Neutron-LBaaS.
-
enabled
Whether or not to enable the LoadBalancer type of Services integration at all. Default: true -
use-octavia
Whether or not to use Octavia for LoadBalancer type of Service implementation instead of using Neutron-LBaaS. Default: true -
floating-network-id
Optional. The external network used to create floating IP for the load balancer VIP. If there are multiple external networks in the cloud, either this option must be set or user must specifyloadbalancer.openstack.org/floating-network-id
in the Service annotation. -
floating-subnet-id
Optional. The external network subnet used to create floating IP for the load balancer VIP. Can be overridden by the Service annotationloadbalancer.openstack.org/floating-subnet-id
. -
floating-subnet
Optional. A name pattern (glob or regexp if starting with~
) for the external network subnet used to create floating IP for the load balancer VIP. Can be overridden by the Service annotationloadbalancer.openstack.org/floating-subnet
. If multiple subnets match the first one with still available IPs is used. -
floating-subnet-tags
Optional. Tags for the external network subnet used to create floating IP for the load balancer VIP. Can be overridden by the Service annotationloadbalancer.openstack.org/floating-subnet-tags
. If multiple subnets match the first one with still available IPs is used. -
lb-method
The load balancing algorithm used to create the load balancer pool. The value can beROUND_ROBIN
,LEAST_CONNECTIONS
, orSOURCE_IP
. Default:ROUND_ROBIN
-
lb-provider
Optional. Used to specify the provider of the load balancer, e.g. "amphora" or "octavia". Only "amphora" or "octavia" provider are officially tested, other provider will cause a warning log. -
lb-version
Optional. If specified, only "v2" is supported. -
subnet-id
ID of the Neutron subnet on which to create load balancer VIP, this ID is also used to create pool members. -
network-id
ID of the Neutron network on which to create load balancer VIP, not needed ifsubnet-id
is set. -
manage-security-groups
If the Neutron security groups should be managed separately. Default: falseThis option is not supported for Octavia. The worker nodes and the Octavia amphorae are usually in the same subnet, so it's sufficient to config the port security group rules manually for worker nodes, to allow the traffic coming from the the subnet IP range to the node port range(i.e. 30000-32767).
-
create-monitor
Indicates whether or not to create a health monitor for the service load balancer. A health monitor required for services that declareexternalTrafficPolicy: Local
. Default: false -
monitor-delay
The time, in seconds, between sending probes to members of the load balancer. Default: 5 -
monitor-max-retries
The number of successful checks before changing the operating status of the load balancer member to ONLINE. A valid value is from 1 to 10. Default: 1 -
monitor-timeout
The maximum time, in seconds, that a monitor waits to connect backend before it times out. Default: 3 -
internal-lb
Determines whether or not to create an internal load balancer (no floating IP) by default. Default: false. -
cascade-delete
Determines whether or not to perform cascade deletion of load balancers. Default: true. -
flavor-id
The id of the loadbalancer flavor to use. Uses octavia default if not set. -
availability-zone
The name of the loadbalancer availability zone to use. It is applicable if use-octavia is set to True and requires Octavia API version 2.14 or later (Ussuri release). The Octavia availability zone capabilities will not be used if it is not set. The parameter will be ignored if the Octavia version doesn't support availability zones yet. -
LoadBalancerClass "ClassName"
This is a config section including a set of config options. User can choose theClassName
by specifying the Service annotationloadbalancer.openstack.org/class
. The following options are supported:- floating-network-id. The same with
floating-network-id
option above. - floating-subnet-id. The same with
floating-subnet-id
option above. - floating-subnet. The same with
floating-subnet
option above. - floating-subnet-tags. The same with
floating-subnet-tags
option above. - network-id. The same with
network-id
option above. - subnet-id. The same with
subnet-id
option above.
- floating-network-id. The same with
-
enable-ingress-hostname
Used with proxy protocol (set by annotation
loadbalancer.openstack.org/proxy-protocol: "true"
) by adding a dns suffix (nip.io) to the load balancer IP address. Default false.This option is currently a workaround for the issue kubernetes/ingress-nginx#3996, should be removed or refactored after the Kubernetes KEP-1860 is implemented.
-
ingress-hostname-suffix
The dns suffix to the load balancer IP address when using proxy protocol. Default nip.io
This option is currently a workaround for the issue kubernetes/ingress-nginx#3996, should be removed or refactored after the Kubernetes KEP-1860 is implemented.
-
default-tls-container-ref
Reference to a tls container. This option works with Octavia, when this option is set then the cloud provider will create an Octavia Listener of type TERMINATED_HTTPS for a TLS Terminated loadbalancer.Format for tls container ref:
https://{keymanager_host}/v1/containers/{uuid}
-
max-shared-lb
The maximum number of Services that share a load balancer. Default: 2
NOTE:
- When using
ovn
provider service has limited scope -create_monitor
is not supported and only supportedlb-method
isSOURCE_IP
.
-
search-order
This configuration key influences the way that the provider retrieves metadata relating to the instance(s) in which it runs. The default value ofconfigDrive,metadataService
results in the provider retrieving metadata relating to the instance from the config drive first if available and then the metadata service. Alternative values are:configDrive
- Only retrieve instance metadata from the configuration drive.metadataService
- Only retrieve instance metadata from the metadata service.metadataService,configDrive
- Retrieve instance metadata from the metadata service first if available, then the configuration drive.
Not all OpenStack clouds provide both configuration drive and metadata service though and only one or the other may be available which is why the default is to check both. Especially, the metadata on the config drive may grow stale over time, whereas the metadata service always provides the most up to date data.
Refer to Exposing applications using services of LoadBalancer type
Refer to Metrics for openstack-cloud-controller-manager
topology.kubernetes.io/zone
is used to label node and its value comes from availability zone of the node, according to label spec it does not support blank (' ') but OpenStack availability zone supports blank. So your OpenStack availability zone must not contain blank otherwise it will lead to node that belongs to this availability zone register failure, see #1379 for further information.