Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

scheduler: enable secure port and authn/z #69663

Merged
merged 5 commits into from
Nov 9, 2018
Merged

scheduler: enable secure port and authn/z #69663

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
cb95eda
kube-scheduler: enable secure ports 10259
Sep 5, 2018
d91feb6
kube-scheduler: move stopCh creation out of scheduler factory code
sttts Oct 11, 2018
98d8e87
integration: rename test/integration/{controllermanager -> serving}
sttts Oct 11, 2018
0ba6613
kube-scheduler: split out cmd Run func
sttts Nov 8, 2018
1588af4
kube-scheduler: add secure serving and authn/z integration tests
sttts Nov 8, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cmd/kube-scheduler/app/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ filegroup(
":package-srcs",
"//cmd/kube-scheduler/app/config:all-srcs",
"//cmd/kube-scheduler/app/options:all-srcs",
"//cmd/kube-scheduler/app/testing:all-srcs",
],
tags = ["automanaged"],
)
1 change: 1 addition & 0 deletions cmd/kube-scheduler/app/config/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ go_library(
"//staging/src/k8s.io/client-go/informers/core/v1:go_default_library",
"//staging/src/k8s.io/client-go/kubernetes:go_default_library",
"//staging/src/k8s.io/client-go/kubernetes/typed/core/v1:go_default_library",
"//staging/src/k8s.io/client-go/rest:go_default_library",
"//staging/src/k8s.io/client-go/tools/leaderelection:go_default_library",
"//staging/src/k8s.io/client-go/tools/record:go_default_library",
],
Expand Down
6 changes: 6 additions & 0 deletions cmd/kube-scheduler/app/config/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ import (
coreinformers "k8s.io/client-go/informers/core/v1"
clientset "k8s.io/client-go/kubernetes"
v1core "k8s.io/client-go/kubernetes/typed/core/v1"
restclient "k8s.io/client-go/rest"
"k8s.io/client-go/tools/leaderelection"
"k8s.io/client-go/tools/record"
kubeschedulerconfig "k8s.io/kubernetes/pkg/scheduler/apis/config"
Expand All @@ -32,6 +33,9 @@ type Config struct {
// config is the scheduler server's configuration object.
ComponentConfig kubeschedulerconfig.KubeSchedulerConfiguration

// LoopbackClientConfig is a config for a privileged loopback connection
LoopbackClientConfig *restclient.Config

InsecureServing *apiserver.DeprecatedInsecureServingInfo // nil will disable serving on an insecure port
InsecureMetricsServing *apiserver.DeprecatedInsecureServingInfo // non-nil if metrics should be served independently
Authentication apiserver.AuthenticationInfo
Expand Down Expand Up @@ -70,5 +74,7 @@ func (c *Config) Complete() CompletedConfig {
c.InsecureMetricsServing.Name = "metrics"
}

apiserver.AuthorizeClientBearerToken(c.LoopbackClientConfig, &c.Authentication, &c.Authorization)

return CompletedConfig{&cc}
}
1 change: 1 addition & 0 deletions cmd/kube-scheduler/app/options/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ go_library(
"//cmd/kube-scheduler/app/config:go_default_library",
"//pkg/api/legacyscheme:go_default_library",
"//pkg/client/leaderelectionconfig:go_default_library",
"//pkg/master/ports:go_default_library",
"//pkg/scheduler/apis/config:go_default_library",
"//pkg/scheduler/apis/config/scheme:go_default_library",
"//pkg/scheduler/apis/config/v1alpha1:go_default_library",
Expand Down
12 changes: 6 additions & 6 deletions cmd/kube-scheduler/app/options/insecure_serving.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ import (
// CombinedInsecureServingOptions sets up to two insecure listeners for healthz and metrics. The flags
// override the ComponentConfig and DeprecatedInsecureServingOptions values for both.
type CombinedInsecureServingOptions struct {
Healthz *apiserveroptions.DeprecatedInsecureServingOptions
Metrics *apiserveroptions.DeprecatedInsecureServingOptions
Healthz *apiserveroptions.DeprecatedInsecureServingOptionsWithLoopback
Metrics *apiserveroptions.DeprecatedInsecureServingOptionsWithLoopback
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to understand why is it replacing one deprecated option with another deprecated one?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it seems that is it extending it to loopback? though still not sure why to extend a deprecated option?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not extending it in the code sense. We have support for insecure ports using these two option structs (not new for scheduler). The whole insecure port code is deprecated, to be removed when all components switched over to secure ports. The difference of DeprecatedInsecureServingOptionsWithLoopback is just that it add the loopback authenticator which helps testing. Otherwise it is the same.


BindPort int // overrides the structs above on ApplyTo, ignored on ApplyToFromLoadedConfig
BindAddress string // overrides the structs above on ApplyTo, ignored on ApplyToFromLoadedConfig
Expand Down Expand Up @@ -60,11 +60,11 @@ func (o *CombinedInsecureServingOptions) applyTo(c *schedulerappconfig.Config, c
return err
}

if err := o.Healthz.ApplyTo(&c.InsecureServing); err != nil {
if err := o.Healthz.ApplyTo(&c.InsecureServing, &c.LoopbackClientConfig); err != nil {
return err
}
if o.Metrics != nil && (c.ComponentConfig.MetricsBindAddress != c.ComponentConfig.HealthzBindAddress || o.Healthz == nil) {
if err := o.Metrics.ApplyTo(&c.InsecureMetricsServing); err != nil {
if err := o.Metrics.ApplyTo(&c.InsecureMetricsServing, &c.LoopbackClientConfig); err != nil {
return err
}
}
Expand Down Expand Up @@ -108,7 +108,7 @@ func (o *CombinedInsecureServingOptions) ApplyToFromLoadedConfig(c *schedulerapp
return o.applyTo(c, componentConfig)
}

func updateAddressFromDeprecatedInsecureServingOptions(addr *string, is *apiserveroptions.DeprecatedInsecureServingOptions) error {
func updateAddressFromDeprecatedInsecureServingOptions(addr *string, is *apiserveroptions.DeprecatedInsecureServingOptionsWithLoopback) error {
if is == nil {
*addr = ""
} else {
Expand All @@ -124,7 +124,7 @@ func updateAddressFromDeprecatedInsecureServingOptions(addr *string, is *apiserv
return nil
}

func updateDeprecatedInsecureServingOptionsFromAddress(is *apiserveroptions.DeprecatedInsecureServingOptions, addr string) error {
func updateDeprecatedInsecureServingOptionsFromAddress(is *apiserveroptions.DeprecatedInsecureServingOptionsWithLoopback, addr string) error {
if is == nil {
return nil
}
Expand Down
28 changes: 14 additions & 14 deletions cmd/kube-scheduler/app/options/insecure_serving_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1234",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{},
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{},
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindPort: 0,
},
},
Expand All @@ -61,7 +61,7 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1234",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{},
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindPort: 0,
},
},
Expand All @@ -79,7 +79,7 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1234",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{},
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindPort: 0,
},
},
Expand All @@ -97,8 +97,8 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1234",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{},
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{},
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindPort: 0,
},
},
Expand All @@ -118,8 +118,8 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1235",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{},
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{},
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindPort: 0,
},
},
Expand All @@ -141,8 +141,8 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1234",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{},
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{},
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindPort: 1236,
BindAddress: "1.2.3.4",
},
Expand All @@ -163,8 +163,8 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1234",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{},
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{},
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindAddress: "2.3.4.5",
BindPort: 1234,
},
Expand All @@ -185,8 +185,8 @@ func TestOptions_ApplyTo(t *testing.T) {
MetricsBindAddress: "1.2.3.4:1234",
},
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{},
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{},
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{}).WithLoopback(),
BindAddress: "2.3.4.5",
BindPort: 0,
},
Expand Down
46 changes: 34 additions & 12 deletions cmd/kube-scheduler/app/options/options.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ import (
schedulerappconfig "k8s.io/kubernetes/cmd/kube-scheduler/app/config"
"k8s.io/kubernetes/pkg/api/legacyscheme"
"k8s.io/kubernetes/pkg/client/leaderelectionconfig"
"k8s.io/kubernetes/pkg/master/ports"
kubeschedulerconfig "k8s.io/kubernetes/pkg/scheduler/apis/config"
kubeschedulerscheme "k8s.io/kubernetes/pkg/scheduler/apis/config/scheme"
"k8s.io/kubernetes/pkg/scheduler/apis/config/validation"
Expand All @@ -56,7 +57,7 @@ type Options struct {
// The default values. These are overridden if ConfigFile is set or by values in InsecureServing.
ComponentConfig kubeschedulerconfig.KubeSchedulerConfiguration

SecureServing *apiserveroptions.SecureServingOptions
SecureServing *apiserveroptions.SecureServingOptionsWithLoopback
CombinedInsecureServing *CombinedInsecureServingOptions
Authentication *apiserveroptions.DelegatingAuthenticationOptions
Authorization *apiserveroptions.DelegatingAuthorizationOptions
Expand Down Expand Up @@ -85,25 +86,34 @@ func NewOptions() (*Options, error) {

o := &Options{
ComponentConfig: *cfg,
SecureServing: nil, // TODO: enable with apiserveroptions.NewSecureServingOptions()
SecureServing: apiserveroptions.NewSecureServingOptions().WithLoopback(),
CombinedInsecureServing: &CombinedInsecureServingOptions{
Healthz: &apiserveroptions.DeprecatedInsecureServingOptions{
Healthz: (&apiserveroptions.DeprecatedInsecureServingOptions{
BindNetwork: "tcp",
},
Metrics: &apiserveroptions.DeprecatedInsecureServingOptions{
}).WithLoopback(),
Metrics: (&apiserveroptions.DeprecatedInsecureServingOptions{
BindNetwork: "tcp",
},
}).WithLoopback(),
BindPort: hport,
BindAddress: hhost,
},
Authentication: nil, // TODO: enable with apiserveroptions.NewDelegatingAuthenticationOptions()
Authorization: nil, // TODO: enable with apiserveroptions.NewDelegatingAuthorizationOptions()
Authentication: apiserveroptions.NewDelegatingAuthenticationOptions(),
Authorization: apiserveroptions.NewDelegatingAuthorizationOptions(),
Deprecated: &DeprecatedOptions{
UseLegacyPolicyConfig: false,
PolicyConfigMapNamespace: metav1.NamespaceSystem,
},
}

o.Authentication.RemoteKubeConfigFileOptional = true
o.Authorization.RemoteKubeConfigFileOptional = true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in addition to tolerating missing kubeconfig, this needed to tolerate forbidden failures in lookupMissingConfigInCluster... nothing granted the scheduler delegated authentication permissions by default

xref #71743

o.Authorization.AlwaysAllowPaths = []string{"/healthz"}

// Set the PairName but leave certificate directory blank to generate in-memory by default
o.SecureServing.ServerCert.CertDirectory = ""
o.SecureServing.ServerCert.PairName = "kube-scheduler"
o.SecureServing.BindPort = ports.KubeSchedulerPort

return o, nil
}

Expand Down Expand Up @@ -173,13 +183,19 @@ func (o *Options) ApplyTo(c *schedulerappconfig.Config) error {
}
}

if err := o.SecureServing.ApplyTo(&c.SecureServing); err != nil {
if err := o.SecureServing.ApplyTo(&c.SecureServing, &c.LoopbackClientConfig); err != nil {
return err
}
if err := o.Authentication.ApplyTo(&c.Authentication, c.SecureServing, nil); err != nil {
return err
if o.SecureServing != nil && (o.SecureServing.BindPort != 0 || o.SecureServing.Listener != nil) {
if err := o.Authentication.ApplyTo(&c.Authentication, c.SecureServing, nil); err != nil {
return err
}
if err := o.Authorization.ApplyTo(&c.Authorization); err != nil {
return err
}
}
return o.Authorization.ApplyTo(&c.Authorization)

return nil
}

// Validate validates all the required options.
Expand All @@ -200,6 +216,12 @@ func (o *Options) Validate() []error {

// Config return a scheduler config object
func (o *Options) Config() (*schedulerappconfig.Config, error) {
if o.SecureServing != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no nil check needed here, MaybeDefaultWithSelfSignedCerts no-ops if SecureServing is nil

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I appear to have led you astray... sorry:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1b607db]

goroutine 86 [running]:
testing.tRunner.func1(0xc00030b200)
	GOROOT/src/testing/testing.go:792 +0x6a7
panic(0x1d2c320, 0x2fd61e0)
	GOROOT/src/runtime/panic.go:513 +0x1b9
k8s.io/kubernetes/cmd/kube-scheduler/app/options.(*Options).Config(0xc0001b5c80, 0xc000073e01, 0x41d0f8, 0x568f5b)
	cmd/kube-scheduler/app/options/options.go:219 +0xcb

if err := o.SecureServing.MaybeDefaultWithSelfSignedCerts("localhost", nil, []net.IP{net.ParseIP("127.0.0.1")}); err != nil {
return nil, fmt.Errorf("error creating self-signed certificates: %v", err)
}
}

c := &schedulerappconfig.Config{}
if err := o.ApplyTo(c); err != nil {
return nil, err
Expand Down
75 changes: 73 additions & 2 deletions cmd/kube-scheduler/app/options/options_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/diff"
apiserverconfig "k8s.io/apiserver/pkg/apis/config"
apiserveroptions "k8s.io/apiserver/pkg/server/options"
kubeschedulerconfig "k8s.io/kubernetes/pkg/scheduler/apis/config"
)

Expand Down Expand Up @@ -175,6 +176,29 @@ users:
}
return *cfg
}(),
SecureServing: (&apiserveroptions.SecureServingOptions{
ServerCert: apiserveroptions.GeneratableKeyCert{
CertDirectory: "/a/b/c",
PairName: "kube-scheduler",
},
HTTP2MaxStreamsPerConnection: 47,
}).WithLoopback(),
Authentication: &apiserveroptions.DelegatingAuthenticationOptions{
CacheTTL: 10 * time.Second,
ClientCert: apiserveroptions.ClientCertAuthenticationOptions{},
RequestHeader: apiserveroptions.RequestHeaderAuthenticationOptions{
UsernameHeaders: []string{"x-remote-user"},
GroupHeaders: []string{"x-remote-group"},
ExtraHeaderPrefixes: []string{"x-remote-extra-"},
},
RemoteKubeConfigFileOptional: true,
},
Authorization: &apiserveroptions.DelegatingAuthorizationOptions{
AllowCacheTTL: 10 * time.Second,
DenyCacheTTL: 10 * time.Second,
RemoteKubeConfigFileOptional: true,
AlwaysAllowPaths: []string{"/healthz"}, // note: this does not match /healthz/ or /healthz/*
},
},
expectedUsername: "config",
expectedConfig: kubeschedulerconfig.KubeSchedulerConfiguration{
Expand Down Expand Up @@ -233,6 +257,29 @@ users:
cfg.ClientConnection.Kubeconfig = flagKubeconfig
return *cfg
}(),
SecureServing: (&apiserveroptions.SecureServingOptions{
ServerCert: apiserveroptions.GeneratableKeyCert{
CertDirectory: "/a/b/c",
PairName: "kube-scheduler",
},
HTTP2MaxStreamsPerConnection: 47,
}).WithLoopback(),
Authentication: &apiserveroptions.DelegatingAuthenticationOptions{
CacheTTL: 10 * time.Second,
ClientCert: apiserveroptions.ClientCertAuthenticationOptions{},
RequestHeader: apiserveroptions.RequestHeaderAuthenticationOptions{
UsernameHeaders: []string{"x-remote-user"},
GroupHeaders: []string{"x-remote-group"},
ExtraHeaderPrefixes: []string{"x-remote-extra-"},
},
RemoteKubeConfigFileOptional: true,
},
Authorization: &apiserveroptions.DelegatingAuthorizationOptions{
AllowCacheTTL: 10 * time.Second,
DenyCacheTTL: 10 * time.Second,
RemoteKubeConfigFileOptional: true,
AlwaysAllowPaths: []string{"/healthz"}, // note: this does not match /healthz/ or /healthz/*
},
},
expectedUsername: "flag",
expectedConfig: kubeschedulerconfig.KubeSchedulerConfiguration{
Expand Down Expand Up @@ -264,8 +311,32 @@ users:
},
},
{
name: "overridden master",
options: &Options{Master: insecureserver.URL},
name: "overridden master",
options: &Options{
Master: insecureserver.URL,
SecureServing: (&apiserveroptions.SecureServingOptions{
ServerCert: apiserveroptions.GeneratableKeyCert{
CertDirectory: "/a/b/c",
PairName: "kube-scheduler",
},
HTTP2MaxStreamsPerConnection: 47,
}).WithLoopback(),
Authentication: &apiserveroptions.DelegatingAuthenticationOptions{
CacheTTL: 10 * time.Second,
RequestHeader: apiserveroptions.RequestHeaderAuthenticationOptions{
UsernameHeaders: []string{"x-remote-user"},
GroupHeaders: []string{"x-remote-group"},
ExtraHeaderPrefixes: []string{"x-remote-extra-"},
},
RemoteKubeConfigFileOptional: true,
},
Authorization: &apiserveroptions.DelegatingAuthorizationOptions{
AllowCacheTTL: 10 * time.Second,
DenyCacheTTL: 10 * time.Second,
RemoteKubeConfigFileOptional: true,
AlwaysAllowPaths: []string{"/healthz"}, // note: this does not match /healthz/ or /healthz/*
},
},
expectedUsername: "none, http",
},
{
Expand Down
Loading