Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move host-network services off of port 8080 #9355

Merged
merged 2 commits into from
Jun 16, 2020
Merged

Move host-network services off of port 8080 #9355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4bf8302
Move kube-apiserver-healthcheck to port 3990
johngmyers Jun 6, 2020
5babf00
Move node-local-dns healthcheck to port 3989
johngmyers Jun 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion cmd/kube-apiserver-healthcheck/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ go_library(
srcs = ["main.go"],
importpath = "k8s.io/kops/cmd/kube-apiserver-healthcheck",
visibility = ["//visibility:private"],
deps = ["//vendor/k8s.io/klog:go_default_library"],
deps = [
"//pkg/wellknownports:go_default_library",
"//vendor/k8s.io/klog:go_default_library",
],
)

go_binary(
Expand Down
4 changes: 2 additions & 2 deletions cmd/kube-apiserver-healthcheck/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This is a small sidecar container that allows for health-checking the
kube-apiserver without enabling anonymous authentication and without
enabling the unauthenticated port.

It listens on port 8080 (http), and proxies a few known-safe requests
It listens on port 3990 (http), and proxies a few known-safe requests
to the real apiserver listening on 443. It uses a client certificate
to authenticate itself to the apiserver.

Expand All @@ -14,5 +14,5 @@ it also lets us have better load-balancer health-checks.
Because it runs as a sidecar next to kube-apiserver, it is in the same
network namespace, and thus it can reach apiserver on
https://127.0.0.1 . The kube-apiserver-healthcheck process listens on
8080, but the health checks for the apiserver container are configured
3990, but the health checks for the apiserver container are configured
for :8080 and actually go via the sidecar.
3 changes: 2 additions & 1 deletion cmd/kube-apiserver-healthcheck/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import (
"os"

"k8s.io/klog"
"k8s.io/kops/pkg/wellknownports"
)

// healthCheckServer is the http server
Expand Down Expand Up @@ -129,7 +130,7 @@ func (s *healthCheckServer) proxyRequest(w http.ResponseWriter, forwardRequest *
}

func run() error {
listen := ":8080"
listen := fmt.Sprintf(":%d", wellknownports.KubeAPIServerHealthCheck)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to make sure, does the health check have to listen to all ips? I did a quick check to see where reqs were coming from and seemed to be all localhost. It was a very quick test though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is not changing that aspect.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough


clientCert := ""
clientKey := ""
Expand Down
26 changes: 13 additions & 13 deletions cmd/kube-apiserver-healthcheck/proxy_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,19 +27,19 @@ func TestBuildProxyRequest(t *testing.T) {
In string
Out string
}{
{In: "http://127.0.0.1:8080/readyz", Out: "https://127.0.0.1/readyz"},
{In: "http://127.0.0.1:8080/livez", Out: "https://127.0.0.1/livez"},
{In: "http://127.0.0.1:8080/healthz", Out: "https://127.0.0.1/healthz"},
{In: "http://127.0.0.1:8080/ready", Out: ""},
{In: "http://127.0.0.1:8080/", Out: ""},
{In: "http://127.0.0.1:8080/readyz/foo", Out: ""},
{In: "http://127.0.0.1:8080/readyzfoo", Out: ""},
{In: "http://127.0.0.1:8080/readyz?", Out: "https://127.0.0.1/readyz"},
{In: "http://127.0.0.1:8080/readyz?foo=bar", Out: "https://127.0.0.1/readyz"},
{In: "http://127.0.0.1:8080/readyz?exclude=1", Out: "https://127.0.0.1/readyz?exclude=1"},
{In: "http://127.0.0.1:8080/readyz?exclude=1&exclude=2", Out: "https://127.0.0.1/readyz?exclude=1&exclude=2"},
{In: "http://127.0.0.1:8080/readyz?exclude=1&verbose", Out: "https://127.0.0.1/readyz?exclude=1"},
{In: "http://127.0.0.1:8080/readyz?exclude", Out: "https://127.0.0.1/readyz?exclude="},
{In: "http://127.0.0.1:3990/readyz", Out: "https://127.0.0.1/readyz"},
{In: "http://127.0.0.1:3990/livez", Out: "https://127.0.0.1/livez"},
{In: "http://127.0.0.1:3990/healthz", Out: "https://127.0.0.1/healthz"},
{In: "http://127.0.0.1:3990/ready", Out: ""},
{In: "http://127.0.0.1:3990/", Out: ""},
{In: "http://127.0.0.1:3990/readyz/foo", Out: ""},
{In: "http://127.0.0.1:3990/readyzfoo", Out: ""},
{In: "http://127.0.0.1:3990/readyz?", Out: "https://127.0.0.1/readyz"},
{In: "http://127.0.0.1:3990/readyz?foo=bar", Out: "https://127.0.0.1/readyz"},
{In: "http://127.0.0.1:3990/readyz?exclude=1", Out: "https://127.0.0.1/readyz?exclude=1"},
{In: "http://127.0.0.1:3990/readyz?exclude=1&exclude=2", Out: "https://127.0.0.1/readyz?exclude=1&exclude=2"},
{In: "http://127.0.0.1:3990/readyz?exclude=1&verbose", Out: "https://127.0.0.1/readyz?exclude=1"},
{In: "http://127.0.0.1:3990/readyz?exclude", Out: "https://127.0.0.1/readyz?exclude="},
}

for _, g := range grid {
Expand Down
5 changes: 0 additions & 5 deletions docs/boot-sequence.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,6 @@ that have hostNetwork will work - so all the "core" containers run with hostNetw

## api-server bringup

The api-server will listen on localhost:8080 on the master. This is an unsecured endpoint,
but is only reachable from the master, and only for pods running with hostNetwork=true. This
is how components like kube-scheduler and kube-controller-manager can reach the API without
requiring a token.

APIServer also listens on the HTTPS port (443) on all interfaces. This is a secured endpoint,
and requires valid authentication/authorization to use it. This is the endpoint that node kubelets
will reach, and also that end-users will reach.
Expand Down
1 change: 1 addition & 0 deletions nodeup/pkg/model/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ go_library(
"//pkg/rbac:go_default_library",
"//pkg/systemd:go_default_library",
"//pkg/tokens:go_default_library",
"//pkg/wellknownports:go_default_library",
"//pkg/wellknownusers:go_default_library",
"//upup/pkg/fi:go_default_library",
"//upup/pkg/fi/cloudup/awsup:go_default_library",
Expand Down
5 changes: 3 additions & 2 deletions nodeup/pkg/model/kube_apiserver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import (
"k8s.io/kops/pkg/k8scodecs"
"k8s.io/kops/pkg/kubeconfig"
"k8s.io/kops/pkg/kubemanifest"
"k8s.io/kops/pkg/wellknownports"
"k8s.io/kops/pkg/wellknownusers"
"k8s.io/kops/upup/pkg/fi"
"k8s.io/kops/upup/pkg/fi/nodeup/nodetasks"
Expand Down Expand Up @@ -409,10 +410,10 @@ func (b *KubeAPIServerBuilder) buildPod() (*v1.Pod, error) {
probeAction := &v1.HTTPGetAction{
Host: "127.0.0.1",
Path: "/healthz",
Port: intstr.FromInt(8080),
Port: intstr.FromInt(wellknownports.KubeAPIServerHealthCheck),
}
if useHealthcheckProxy {
// kube-apiserver-healthcheck sidecar container runs on port 8080
// kube-apiserver-healthcheck sidecar container runs on port 3990
} else if kubeAPIServer.InsecurePort != 0 {
probeAction.Port = intstr.FromInt(int(kubeAPIServer.InsecurePort))
} else if kubeAPIServer.SecurePort != 0 {
Expand Down
2 changes: 1 addition & 1 deletion pkg/model/components/apiserver.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,7 @@ func (b *KubeAPIServerOptionsBuilder) BuildOptions(o interface{}) error {
c.AnonymousAuth = fi.Bool(false)

if b.IsKubernetesGTE("1.17") {
// We query via the kube-apiserver-healthcheck proxy, which listens on port 8080
// We query via the kube-apiserver-healthcheck proxy, which listens on port 3990
c.InsecurePort = 0
} else {
// Older versions of kubernetes continue to rely on the insecure port: kubernetes issue #43784
Expand Down
1 change: 1 addition & 0 deletions pkg/model/components/kubeapiserver/BUILD.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ go_library(
"//pkg/assets:go_default_library",
"//pkg/k8scodecs:go_default_library",
"//pkg/model:go_default_library",
"//pkg/wellknownports:go_default_library",
"//upup/pkg/fi:go_default_library",
"//upup/pkg/fi/fitasks:go_default_library",
"//vendor/k8s.io/api/core/v1:go_default_library",
Expand Down
5 changes: 3 additions & 2 deletions pkg/model/components/kubeapiserver/model.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"k8s.io/kops/pkg/assets"
"k8s.io/kops/pkg/k8scodecs"
"k8s.io/kops/pkg/model"
"k8s.io/kops/pkg/wellknownports"
"k8s.io/kops/upup/pkg/fi"
"k8s.io/kops/upup/pkg/fi/fitasks"
)
Expand Down Expand Up @@ -100,7 +101,7 @@ spec:
# The sidecar serves a healthcheck on the same port,
# but with a .kube-apiserver-healthcheck prefix
path: /.kube-apiserver-healthcheck/healthz
port: 8080
port: %d
host: 127.0.0.1
initialDelaySeconds: 5
timeoutSeconds: 5
Expand All @@ -126,7 +127,7 @@ spec:
func (b *KubeApiserverBuilder) buildHealthcheckSidecar() (*corev1.Pod, error) {
// TODO: pull from bundle
bundle := "(embedded kube-apiserver-healthcheck manifest)"
manifest := []byte(defaultManifest)
manifest := []byte(fmt.Sprintf(defaultManifest, wellknownports.KubeAPIServerHealthCheck))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I normally try to avoid using Sprintf with our templates, favoring {{ HealthCheckPort }} and strings.ReplaceAll, because the %d is harder to read, and has more gotchas (repeated placeholders don't work, it's vaguely possible that a % placeholder could appear 'accidentally')


var pod *corev1.Pod
var container *corev1.Container
Expand Down
2 changes: 1 addition & 1 deletion pkg/model/components/kubeapiserver/tests/minimal/tasks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Contents:
httpGet:
host: 127.0.0.1
path: /.kube-apiserver-healthcheck/healthz
port: 8080
port: 3990
initialDelaySeconds: 5
timeoutSeconds: 5
name: healthcheck
Expand Down
8 changes: 8 additions & 0 deletions pkg/wellknownports/wellknownports.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,14 @@ const (

// DNSControllerGossipMemberlist is the port where dns-controller listens for the memberlist-backed gossip
DNSControllerGossipMemberlist = 3993

// 4001 is etcd main, 4002 is etcd events, 4003 is etcd cilium

// KubeAPIServerHealthCheck is the port where kube-apiserver-healthcheck listens.
KubeAPIServerHealthCheck = 3990

// NodeLocalDNSHealthCheck is the port where the node-local-dns health check listens.
NodeLocalDNSHealthCheck = 3989
)

type PortRange struct {
Expand Down
4 changes: 2 additions & 2 deletions upup/models/bindata.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions upup/models/cloudup/resources/addons/nodelocaldns.addons.k8s.io/k8s-1.12.yaml.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ data:
force_tcp
}
prometheus :9253
health {{ KubeDNS.NodeLocalDNS.LocalIP }}:8080
health {{ KubeDNS.NodeLocalDNS.LocalIP }}:{{ NodeLocalDNSHealthCheck }}
}
in-addr.arpa:53 {
errors
Expand Down Expand Up @@ -154,7 +154,7 @@ spec:
httpGet:
host: {{ .KubeDNS.NodeLocalDNS.LocalIP }}
path: /health
port: 8080
port: {{ NodeLocalDNSHealthCheck }}
initialDelaySeconds: 60
timeoutSeconds: 5
volumeMounts:
Expand Down
3 changes: 3 additions & 0 deletions upup/pkg/fi/cloudup/template_functions.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,9 @@ func (tf *TemplateFunctions) AddTo(dest template.FuncMap, secretStore fi.SecretS
}
return tf.cluster.Spec.KubeDNS.ServerIP
}
dest["NodeLocalDNSHealthCheck"] = func() string {
return fmt.Sprintf("%d", wellknownports.NodeLocalDNSHealthCheck)
}

dest["KopsControllerArgv"] = tf.KopsControllerArgv
dest["KopsControllerConfig"] = tf.KopsControllerConfig
Expand Down