Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

leases: add more details on API Server Identity #38196

Merged
merged 1 commit into from
Dec 5, 2022
Merged

leases: add more details on API Server Identity #38196

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 41 additions & 1 deletion content/en/docs/concepts/architecture/leases.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ This is used by control plane components like `kube-controller-manager` and `kub
HA configurations, where only one instance of the component should be actively running while the other
instances are on stand-by.

## kube-apiserver identity
## API Server Identity
sftim marked this conversation as resolved.
Show resolved Hide resolved

{{< feature-state for_k8s_version="v1.26" state="beta" >}}

Expand All @@ -38,3 +38,43 @@ rest of the system. While not particularly useful on its own, this provides a me
discover how many instances of `kube-apiserver` are operating the Kubernetes control plane.
Existence of kube-apiserver leases enables future capabilities that may require coordination between
each kube-apiserver.
Comment on lines 38 to 40
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
discover how many instances of `kube-apiserver` are operating the Kubernetes control plane.
Existence of kube-apiserver leases enables future capabilities that may require coordination between
each kube-apiserver.
The API server creates these leases in the `kube-system` namespace.
The existence of kube-apiserver Leases also enables adding future capabilities to Kubernetes,
that may require coordination between each kube-apiserver (for example, during cluster
upgrades).


You can inspect Leases owned by each kube-apiserver by checking for lease objects in the `kube-system` namespace
Copy link
Contributor

@sftim sftim Dec 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit)

Suggested change
You can inspect Leases owned by each kube-apiserver by checking for lease objects in the `kube-system` namespace
You can inspect Leases owned by each kube-apiserver by checking for Leases in the `kube-system` namespace

with the name `kube-apiserver-<sha256-hash>`. Alternatively you can use the label selector `k8s.io/component=kube-apiserver`:

```shell
$ kubectl -n kube-system get lease -l k8s.io/component=kube-apiserver
NAME HOLDER AGE
Comment on lines +46 to +47
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please split this - see don't include the command prompt and separate commands from output.

kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a_9cbf54e5-1136-44bd-8f9a-1dcd15c346b4 5m33s
sftim marked this conversation as resolved.
Show resolved Hide resolved
kube-apiserver-dz2dqprdpsgnm756t5rnov7yka kube-apiserver-dz2dqprdpsgnm756t5rnov7yka_84f2a85d-37c1-4b14-b6b9-603e62e4896f 4m23s
kube-apiserver-fyloo45sdenffw2ugwaz3likua kube-apiserver-fyloo45sdenffw2ugwaz3likua_c5ffa286-8a9a-45d4-91e7-61118ed58d2e 4m43s
```

The SHA256 hash used in the lease name is based on the OS hostname as seen by kube-apiserver. Each kube-apiserver should be
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which is true:

Suggested change
The SHA256 hash used in the lease name is based on the OS hostname as seen by kube-apiserver. Each kube-apiserver should be
The SHA256 hash used in the Lease name is based on the OS hostname as seen by kube-apiserver.
The API server coerces its hostname string to lowercase before hashing it. Each kube-apiserver
should be

or

Suggested change
The SHA256 hash used in the lease name is based on the OS hostname as seen by kube-apiserver. Each kube-apiserver should be
The SHA256 hash used in the Lease name is based on the OS hostname as seen by kube-apiserver.
Each kube-apiserver should be

?

configured to use a hostname that is unique within the cluster. New instances of kube-apiserver that use the same hostname
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we document the use of the kubernetes.io/hostname label as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #38196 (comment)

will take over existing Leases using a new holder identity, as opposed to instantiating new lease objects. You can check the
Copy link
Contributor

@sftim sftim Dec 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit)

Suggested change
will take over existing Leases using a new holder identity, as opposed to instantiating new lease objects. You can check the
will take over existing Leases using a new holder identity, as opposed to instantiating new Leases.
You can check the

hostname used by kube-apisever by checking the value of the `kubernetes.io/hostname` label:

```shell
$ kubectl -n kube-system get lease kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a -o yaml
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit)

Suggested change
$ kubectl -n kube-system get lease kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a -o yaml
kubectl -n kube-system get lease kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a -o yaml

Maybe use a JSON path to get just the one label?

```

```yaml
apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
creationTimestamp: "2022-11-30T15:37:15Z"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) Ideally, pick a date that is close to the v1.26 release.

labels:
k8s.io/component: kube-apiserver
kubernetes.io/hostname: kind-control-plane
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://kubernetes.io/docs/reference/labels-annotations-taints/ lists that this label is used on Nodes (only)

Either:

name: kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a
namespace: kube-system
resourceVersion: "18171"
uid: d6c68901-4ec5-4385-b1ef-2d783738da6c
spec:
holderIdentity: kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a_9cbf54e5-1136-44bd-8f9a-1dcd15c346b4
leaseDurationSeconds: 3600
renewTime: "2022-11-30T18:04:27.912073Z"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) Ideally, pick a date that is close to the v1.26 release.

```

Expired leases from kube-apiservers that no longer exist are garbage collected by new kube-apiservers after 1 hour.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this right:

Suggested change
Expired leases from kube-apiservers that no longer exist are garbage collected by new kube-apiservers after 1 hour.
### API server Lease garbage collection
Expired Leases from kube-apiservers that no longer exist are garbage collected by live kube-apiservers.
The control plane leaves the expired Leases for at least one hour before removing them.

?

We should update https://kubernetes.io/docs/concepts/architecture/garbage-collection/ to link to that heading, which is why I added it.