CAPA- ensure api and kubelet has all flags we use in our GS product #487

alex-dabija · 2021-10-07T07:36:11Z

CAPA- ensure api and kubelet has all flags we use in our GS product

AverageMarcus · 2021-10-08T08:26:37Z

Current flags

Kubelet args

Worker:

--node-ip=${DEFAULT_IPV4} \
--config=/etc/kubernetes/config/kubelet.yaml \
--logtostderr=true \
--cloud-provider={{.Cluster.Kubernetes.CloudProvider}} \
--pod-infra-container-image={{ .Images.Pause }} \
--image-pull-progress-deadline={{.ImagePullProgressDeadline}} \
--network-plugin=cni \
--register-node=true \
--kubeconfig=/etc/kubernetes/kubeconfig/kubelet.yaml \
--node-labels="node.kubernetes.io/worker,role=worker,ip=${DEFAULT_IPV4},{{.Cluster.Kubernetes.Kubelet.Labels}}" \
--v=2

Control Plane:

--node-ip=${DEFAULT_IPV4} \
--config=/etc/kubernetes/config/kubelet.yaml \
--logtostderr=true \
--cloud-provider={{.Cluster.Kubernetes.CloudProvider}} \
--pod-infra-container-image={{ .Images.Pause }} \
--image-pull-progress-deadline={{.ImagePullProgressDeadline}} \
--network-plugin=cni \
--register-node=true \
--register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
--kubeconfig=/etc/kubernetes/kubeconfig/kubelet.yaml \
--node-labels="node.kubernetes.io/master,role=master,ip=${DEFAULT_IPV4},{{.Cluster.Kubernetes.Kubelet.Labels}}" \
--v=2

api server

- --allow-privileged=true
- --anonymous-auth=true
- --kubelet-preferred-address-types=InternalIP
- --secure-port={{.Cluster.Kubernetes.API.SecurePort}}
- --bind-address=0.0.0.0
- --etcd-prefix={{.Cluster.Etcd.Prefix}}
- --profiling=false
- --service-account-lookup=true
- --authorization-mode=RBAC
- --feature-gates=TTLAfterFinished=true
- --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,DefaultStorageClass,PersistentVolumeClaimResize,PodSecurityPolicy,Priority,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
- --cloud-provider={{.Cluster.Kubernetes.CloudProvider}}
- --service-cluster-ip-range={{.Cluster.Kubernetes.API.ClusterIPRange}}
- --etcd-servers=https://127.0.0.1:2379
- --etcd-cafile=/etc/kubernetes/ssl/etcd/server-ca.pem
- --etcd-certfile=/etc/kubernetes/ssl/etcd/server-crt.pem
- --etcd-keyfile=/etc/kubernetes/ssl/etcd/server-key.pem
- --advertise-address=$(HOST_IP)
- --runtime-config=api/all=true,scheduling.k8s.io/v1alpha1=true
- --logtostderr=true
- --tls-cert-file=/etc/kubernetes/ssl/apiserver-crt.pem
- --tls-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256
- --client-ca-file=/etc/kubernetes/ssl/apiserver-ca.pem
- --audit-log-path=/var/log/apiserver/audit.log
- --audit-log-maxage=30
- --audit-log-maxbackup=30
- --audit-log-maxsize=100
- --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
- --encryption-provider-config=/etc/kubernetes/encryption/k8s-encryption-config.yaml
- --request-timeout=1m
- --requestheader-client-ca-file=/etc/kubernetes/ssl/apiserver-ca.pem
- --requestheader-allowed-names=aggregator,{{.Cluster.Kubernetes.API.Domain}},{{.Cluster.Kubernetes.Kubelet.Domain}}
- --requestheader-extra-headers-prefix=X-Remote-Extra-
- --requestheader-group-headers=X-Remote-Group
- --requestheader-username-headers=X-Remote-User
- --service-account-issuer=https://{{.Cluster.Kubernetes.API.Domain}}
- --service-account-jwks-uri=https://{{.Cluster.Kubernetes.API.Domain}}/openid/v1/jwks
- --service-account-key-file=/etc/kubernetes/ssl/service-account-key.pem
- --service-account-signing-key-file=/etc/kubernetes/ssl/service-account-key.pem
- --proxy-client-cert-file=/etc/kubernetes/ssl/apiserver-crt.pem
- --proxy-client-key-file=/etc/kubernetes/ssl/apiserver-key.pem

If HA etcd:

--apiserver-count=3

Cluster API

kubelet flags

Workers:

--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf 
--kubeconfig=/etc/kubernetes/kubelet.conf 
--config=/var/lib/kubelet/config.yaml 
--cloud-provider=aws 
--container-runtime=remote 
--container-runtime-endpoint=/run/containerd/containerd.sock 
--healthz-bind-address=0.0.0.0 
--hostname-override=ip-10-0-45-142.eu-west-1.compute.internal 
--node-labels=role=worker,giantswarm.io/machine-pool=a4u7f 
--pod-infra-container-image=k8s.gcr.io/pause:3.2

Control plane

--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf 
--kubeconfig=/etc/kubernetes/kubelet.conf 
--config=/var/lib/kubelet/config.yaml 
--cloud-provider=aws 
--container-runtime=remote 
--container-runtime-endpoint=/run/containerd/containerd.sock 
--hostname-override=ip-10-0-69-224.eu-west-1.compute.internal 
--pod-infra-container-image=k8s.gcr.io/pause:3.2

(/var/lib/kubelet/config.yaml)

authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

api server

--advertise-address=10.0.69.224 
--allow-privileged=true 
--authorization-mode=Node,RBAC 
--client-ca-file=/etc/kubernetes/pki/ca.crt 
--cloud-provider=aws 
--enable-admission-plugins=NodeRestriction 
--enable-bootstrap-token-auth=true 
--etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt 
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt 
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key 
--etcd-servers=https://127.0.0.1:2379 
--insecure-port=0 
--kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt 
--kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key 
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 
--proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt 
--proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key 
--requestheader-allowed-names=front-proxy-client 
--requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt 
--requestheader-extra-headers-prefix=X-Remote-Extra- 
--requestheader-group-headers=X-Remote-Group 
--requestheader-username-headers=X-Remote-User 
--secure-port=6443 
--service-account-key-file=/etc/kubernetes/pki/sa.pub 
--service-cluster-ip-range=10.96.0.0/12 
--tls-cert-file=/etc/kubernetes/pki/apiserver.crt 
--tls-private-key-file=/etc/kubernetes/pki/apiserver.key

Missing flags

kubelet - control plane

--node-ip=${DEFAULT_IPV4} - https://github.com/giantswarm/giantswarm/issues/19133
--image-pull-progress-deadline={{.ImagePullProgressDeadline}}
--register-with-taints=node-role.kubernetes.io/master=:NoSchedule
--node-labels="node.kubernetes.io/master,role=master,ip=${DEFAULT_IPV4},{{.Cluster.Kubernetes.Kubelet.Labels}}" - CAPI worker node role labeling #374
--v=2
--network-plugin=cni [Kubelet DEPRECATED]
--register-node=true [Kubelet DEFAULT]
--logtostderr=true [Kubelet DEFAULT]

kubelet - worker

--node-ip=${DEFAULT_IPV4} - https://github.com/giantswarm/giantswarm/issues/19133
--image-pull-progress-deadline={{.ImagePullProgressDeadline}}
--v=2
--logtostderr=true [Kubelet DEFAULT]
--network-plugin=cni [Kubelet DEPRECATED]
--register-node=true [Kubelet DEFAULT]

api server

Different values

api server

--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,DefaultStorageClass,PersistentVolumeClaimResize,PodSecurityPolicy,Priority,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
--kubelet-preferred-address-types=InternalIP
--requestheader-allowed-names=aggregator,{{.Cluster.Kubernetes.API.Domain}},{{.Cluster.Kubernetes.Kubelet.Domain}}

AverageMarcus · 2021-10-08T08:35:29Z

The CAPA values were taken from doing a kubectl gs template ... with default values provided.

Do we want to update the flags generated by kubectl-gs template or do we want to set some of the values with Kyverno? (Or maybe a mix of both?)

alex-dabija · 2021-10-11T07:39:06Z

They should be set in Kyverno because not all clusters will be created with kubectl-gs. Customers will be able to submit their own CRs directly. In this situation we should mutate or validate what changes are accepted by our system depending on the situation.

AverageMarcus · 2021-10-13T07:27:46Z

Released as https://github.com/giantswarm/kyverno-policies/releases/tag/v0.8.0

Until we're able to upgrade the version of Kyverno we're using (currently unable to due to a regression bug) the current policies can cause duplicate values if the flags are already present on the cluster resources. Once Kyverno is upgraded we can switch to better handling of duplicates.

AverageMarcus · 2021-10-20T13:26:50Z

Reopening after the discussion on giantswarm/kyverno-policies#123 (comment)

Need to understand if the encryption-provider-config is working on AWS and if so how the file is getting onto the machine image (and how can we for Azure).

AverageMarcus · 2021-11-15T10:27:17Z

Closing in favour of specific issues:

AverageMarcus assigned AverageMarcus and unassigned AverageMarcus Oct 8, 2021

alex-dabija added provider/cluster-api-aws Cluster API based running on AWS team/firecracker labels Oct 11, 2021

AverageMarcus self-assigned this Oct 11, 2021

AverageMarcus mentioned this issue Oct 11, 2021

Added kubelet and api server args giantswarm/kyverno-policies#123

Merged

1 task

alex-dabija added the topic/capi label Oct 18, 2021

AverageMarcus closed this as completed Oct 19, 2021

AverageMarcus reopened this Oct 20, 2021

AverageMarcus closed this as completed Nov 15, 2021

AverageMarcus mentioned this issue Nov 15, 2021

CAPI - enable audit-policy-file api-server flag #563

Closed

cornelius-keller mentioned this issue Jan 10, 2022

Configure kubelet and API server flags for CAPO clusters #687

Closed

alex-dabija mentioned this issue Mar 29, 2022

[EPIC] Cluster API for AWS (CAPA) #951

Open

31 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CAPA- ensure api and kubelet has all flags we use in our GS product #487

CAPA- ensure api and kubelet has all flags we use in our GS product #487

alex-dabija commented Oct 7, 2021

AverageMarcus commented Oct 8, 2021 •

edited

Loading

AverageMarcus commented Oct 8, 2021

alex-dabija commented Oct 11, 2021

AverageMarcus commented Oct 13, 2021

AverageMarcus commented Oct 20, 2021

AverageMarcus commented Nov 15, 2021 •

edited

Loading

CAPA- ensure api and kubelet has all flags we use in our GS product #487

CAPA- ensure api and kubelet has all flags we use in our GS product #487

Comments

alex-dabija commented Oct 7, 2021

AverageMarcus commented Oct 8, 2021 • edited Loading

Current flags

Kubelet args

api server

Cluster API

kubelet flags

api server

Missing flags

kubelet - control plane

kubelet - worker

api server

Different values

api server

AverageMarcus commented Oct 8, 2021

alex-dabija commented Oct 11, 2021

AverageMarcus commented Oct 13, 2021

AverageMarcus commented Oct 20, 2021

AverageMarcus commented Nov 15, 2021 • edited Loading

AverageMarcus commented Oct 8, 2021 •

edited

Loading

AverageMarcus commented Nov 15, 2021 •

edited

Loading