Pull sandbox image periodically

[cartermckinnon]

Issue #, if available:

Helps workaround #1597, proper fix will be an updated containerd from Amazon Linux.

Description of changes:

As a hotfix for accidental garbage collections of the sandbox container image, we'll check every minute and re-pull it if necessary.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Testing Done

Tested live on a node, the timer has the desired effect:

[ec2-user@ip-172-31-37-28 ~]$ sudo journalctl -u sandbox-image -f
...
Jan 30 20:22:37 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Starting pull sandbox image defined in containerd config.toml...
Jan 30 20:22:37 ip-172-31-37-28.us-west-2.compute.internal sudo[12946]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/ctr#040--namespace#040k8s.io#040image#040ls
Jan 30 20:22:37 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Started pull sandbox image defined in containerd config.toml.
Jan 30 20:24:37 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Starting pull sandbox image defined in containerd config.toml...
Jan 30 20:24:37 ip-172-31-37-28.us-west-2.compute.internal sudo[12961]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/ctr#040--namespace#040k8s.io#040image#040ls
Jan 30 20:24:37 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Started pull sandbox image defined in containerd config.toml.
Jan 30 20:26:17 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Starting pull sandbox image defined in containerd config.toml...
Jan 30 20:26:17 ip-172-31-37-28.us-west-2.compute.internal sudo[13007]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/ctr#040--namespace#040k8s.io#040image#040ls
Jan 30 20:26:17 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Started pull sandbox image defined in containerd config.toml.
Jan 30 20:27:37 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Starting pull sandbox image defined in containerd config.toml...
Jan 30 20:27:37 ip-172-31-37-28.us-west-2.compute.internal sudo[13034]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/ctr#040--namespace#040k8s.io#040image#040ls
Jan 30 20:27:37 ip-172-31-37-28.us-west-2.compute.internal systemd[1]: Started pull sandbox image defined in containerd config.toml.

[cartermckinnon]

/ci

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[cartermckinnon]

The CI job for 1.29 will fail, but will be fixed by #1602. I'll re-run in a bit

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

Kubernetes version	Build	Launch	Test
1.23	success ✅	success ✅	success ✅
1.24	success ✅	success ✅	success ✅
1.25	success ✅	success ✅	success ✅
1.26	success ✅	success ✅	success ✅
1.27	success ✅	success ✅	success ✅
1.28	success ✅	success ✅	success ✅
1.29	failure ❌	skipped ⏭️	skipped ⏭️

[cartermckinnon]

/ci

giving 1.29 another go after the CI fix...

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

Kubernetes version	Build	Launch	Test
1.23	success ✅	success ✅	success ✅
1.24	success ✅	success ✅	success ✅
1.25	success ✅	success ✅	success ✅
1.26	success ✅	success ✅	success ✅
1.27	success ✅	failure ❌	skipped ⏭️
1.28	success ✅	success ✅	success ✅
1.29	success ✅	success ✅	success ✅

[cartermckinnon]

1.27 hit a resource limit in the CI account, disregard

[Idan-Lazar]

Could someone please merge it?

[avisaradir]

Is there an expected date and time for the distribution of this solution to #1597 ?

[spatelwearpact]

Please merge this in, we're dealing with a production outage right now because of this issue! None of our pods and jobs are able to deploy into the cluster!

[dims]

Please merge this in, we're dealing with a production outage right now because of this issue! None of our pods and jobs are able to deploy into the cluster!

@spatelwearpact Samir, this is NOT the right forum for a production outage discussion, please escalate through your AWS support folks.

[mmerkes]

LGTM

[spatelwearpact]

@dims your comment is not helpful, the issue we're having is in production and is directly related to this PR. So how about you guys get cracking on this and get it merged instead of telling me to report things to my TAM!

[dims]

@spatelwearpact thanks for the tip!

[dims]

@cartermckinnon fyi kubernetes/kubernetes#118544

[bryantbiggs]

FYI - the Kubernetes version skew policy does allow you to run 1.28 nodes with a 1.29 control plane if that helps folks get around any immediate issues until a fix is pushed out

[dims]

cc @henry118

[RobCannon]

About how long does it take for the new AMI to appear in the AWS EKS console now that this fix is merged?

[dims]

@RobCannon please see #1597 (comment)