Automate updates for k8s.io redirector service

[ixdy]

Currently, changes to the k8s.io redirector service (i.e any changes to the configs under the k8s.io subdirectory) require @ixdy or @thockin to manually update the cluster, basically following something like the following process:

cd k8s.io/
kubectl -n k8s-io-canary apply -f configmap-nginx.yaml
# pick up new configs by forcing nginx to restart
kubectl -n k8s-io-canary scale deployment k8s-io --replicas=0
kubectl -n k8s-io-canary scale deployment k8s-io --replicas=1
TARGET_IP=[canary namespace service IP] make test

# if tests pass, deploy to production
kubectl -n k8s-io-prod apply -f configmap-nginx.yaml
# pick up new configs by forcing nginx to restart
kubectl -n k8s-io-prod scale deployment k8s-io --replicas=0
# note we scale back to 2, not 1
kubectl -n k8s-io-prod scale deployment k8s-io --replicas=2
# verify everything on prod
make test

There are lots of steps to automate here:

• restarting nginx manually
• testing on the canary namespace before updating the prod namespace
  • ideally we test before merge even, which is currently a manual process
• requiring a human to deploy changes (rather than automatically updating on merge)

[bartsmykla]

Hi @ixdy what do you think about me helping with that. Can you give me some hints or directions about people who could give me more informations? :-)

[ixdy]

Take one of the steps I listed above and figure out some solution? e.g. figure out how to automatically update nginx when pushing a new configmap. (There are several approaches with different tradeoffs, and nobody's taken the time to figure out what makes the most sense here.)

[bartsmykla]

In one of my past projects we were using the approach of creating a sidecar container which was watching changes inside specified paths, and when they appeared it was reading configmap, parsing it and sending to specified endpoint., but here it would be different.

One of the approaches would be to use feature, which is currently in beta -> Share Process Namespace between Containers in a Pod. We would create a sidecar component which will be watching configmap changes and if they will appear we could send a HUP signal to the nginx.

It's good because we are separating logic related to watching changes and sending appropriate signals to separate place, and not touching basic nginx image. Problem can appear if we don't want to use beta feature, or if our kubernetes cluster's version will be lower than 1.13.

Another approach will be related to running two binaries/scripts inside a container. One for nginx and second for our watcher which will send HUP signal to nginx process when changes appear.
Cons:

• We'll face a problem of multiple entrypoints, which can be solved by creating some wrapping bash scripts, or using something like supervisord
  • the first approach won't give us a lot of control when something won't go smoothly but it's easy and lightweight solution;
  • the second approach expect from us to install environment for supervisord (python, pip) or using it's go's equivalent (ie. https://github.com/ochinchina/supervisord)
  • We will also have to write script to watch changes in configmap files (ie. using inotify-tools) or some solution in go
• We'll have to create and maintain our own image with all that changes

Actually I succeeded to create a solution using https://github.com/ochinchina/supervisord and https://github.com/fsnotify/fsnotify so I can clean it and push somewhere to test.

If it would be possible I think I would choose the first option, but I don't have experience with that feature so probably I couldn't see some downsides.

[bartsmykla]

@ixdy I have created simple Go app using: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/ which can be added as a sidecar container. I have to figure out how to test it yet, but for tests we can play with it: https://github.com/bartsmykla/nginx-reloader

[bartsmykla]

Hi @ixdy did you have some time to look at my suggestions maybe?

[dims]

@ixdy WDYT? ^

[ixdy]

@bartsmykla I also think I prefer the sidecar option, since that seems like a cleaner, more generalizable pattern, though I don't think GKE supports kubernetes 1.13 yet.

Do you have access to a kubernetes cluster for testing? You could try updating the manifests in the k8s.io directory of this repo to try out your approach.

[ixdy]

though, a counterpoint: it'd be nice to take advantage of the fact that we're using a Deployment right now, and so we should perform a rolling update of the config (in case it causes nginx to crash).

I've seen patterns where the ConfigMap containing the nginx config is somehow munged to work with a Deployment rolling update - maybe something like this?

(I think I've seen similar but different patterns elsewhere, but I'm not immediately finding them now.)

[bartsmykla]

Let me dig into it a little bit more

[dims]

pending getting a cluster up

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stp-ip]

/remove-lifecycle stale

[stp-ip]

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stp-ip]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stp-ip]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[stp-ip]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/remove-priority backlog
/priority important-longterm
/milestone v1.21

Since we're going to want to make changes to dl.k8s.io as part of #1569, now might be the right time to re-examine how this is deployed/managed

[spiffxp]

/remove-lifecycle stale

[spiffxp]

FYI @ameukam @nikhita

[spiffxp]

/milestone v1.22

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
Still relevant. We are close with the deploy.sh script but it's not as simple as run once parameterless. Not that far off from something an appropriately privileged prowjob could run though

[spiffxp]

/close
This was accomplished by kubernetes/test-infra#22970 as part of #2151

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
This was accomplished by kubernetes/test-infra#22970 as part of #2151

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.