Configure kubelet Shutdown Grace Period

[mnitchev]

What this PR does / why we need it

This PR configures both the shutdownGracePeriod and shutdownGracePeriodCriticalPods. The grace period is set to 5 minutes while the critical pods grace period is set to 1 minute. This means that kubelet will start terminating critical pods (pods with Priority class set to system-cluster-critical or system-node-critical) in the last 1 minute of the shutdownGracePeriod. This is important since our CNI (cilium) is a critical pod that needs to terminated last.

This feature is implemented using systemd Inhibitor Locs. The maximum inhibit delay is controlled by the InhibitDelayMaxSec in logind. On our AWS nodes the default is set to 30 seconds, so we need to override it. This is why the /lib/systemd/logind.conf.d/zzz-kubelet-graceful-shutdown.conf is mounted in the nodes (the zzz prefix is so it always lands as the last evaluated file).

Checklist

• Update changelog in CHANGELOG.md.

References

1. Graceful Node Shutdown docs
2. Graceful Node Shutdown Proposal docs (there's some more info on how it works in here)
3. logind.conf docs
4. systemd inhibit docs

Trigger e2e tests

/test create
/test upgrade
/run cluster-test-suites

[tityosbot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[mnitchev]

/test all

[tityosbot]

@mnitchev: No jobs can be run with /test all.
The following commands are available to trigger jobs:

• /test create
• /test upgrade

In response to this:

/test all

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mnitchev]

/test create
/test upgrade

[AndiDog]

LGTM apart from comments

[mnitchev]

/test create
/test upgrade