From 32f24bc4113d38036a8b2b7e14bf3dd28ccad077 Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Thu, 26 Oct 2017 11:19:43 -0700 Subject: [PATCH 1/8] Add proposal for configurable pod resolv.conf --- .../network/pod-resolv-conf.md | 167 ++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100644 contributors/design-proposals/network/pod-resolv-conf.md diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md new file mode 100644 index 00000000000..c04384c57de --- /dev/null +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -0,0 +1,167 @@ +# Custom /etc/resolv.conf +* Status: pending +* Version: alpha +* Implementation owner: Bowei Du <[bowei@google.com](mailto:bowei@google.com)>, Zihong Zheng <[zihongz@google.com](mailto:zihongz@google.com)> + +# Overview +The `/etc/resolv.conf` in a pod is managed by Kubelet and its contents are generated based on `pod.dnsPolicy`. For `dnsPolicy: Default`, `resolv.conf` is copied from the node where the pod is running. If the `dnsPolicy` is `ClusterFirst`, the contents of the resolv.conf is the hosts `resolv.conf` augmented with the following options: + +* Search paths to add aliases for domain names in the same namespace and cluster suffix. +* `options ndots` to 5 to ensure the search paths are searched for all potential matches. + +The configuration of both search paths and `ndots` results in query amplification of five to ten times for non-cluster internal names. This is due to the fact that each of the search path expansions must be tried before the actual result is found. This order of magnitude increase of query rate imposes a large load on the kube-dns service. At the same time, there are user applications do not need the convenience of the name aliases and do not wish to pay this performance cost. + + +## Existing workarounds + +The current work around for this problem is to specify an FQDN for name resolution. Any domain name that ends with a period (e.g. `foo.bar.com.`) will not be search path expanded. However, use of FQDNs is not well-known practice and imposes application-level changes. Cluster operators may not have the luxury of enforcing such a change to applications that run on their infrastructure. + +It is also possible for the user to insert a short shell script snippet that rewrites `resolv.conf` on container start-up. This has the same problems as the previous approach and is also awkward for the user. This also forces the container to have additional executable code such as a shell or scripting engine which increases the applications security surface area. + + +# Proposal sketch + +Add a new `pod.dnsPolicy: Custom` that allows for user customization of `resolv.conf`. + + +## Pod API example + +In the example below, the user wishes to add the pod namespace and a custom expansion to the search path, as they do not use the other name aliases: + +```yaml +# Pod spec +apiVersion: v1 +kind: Pod +metadata: + namespace: ns1 + name: example +spec: + containers: + - name: example + image: example + dnsPolicy: Custom + dnsParams: + searchPaths: + - $NAMESPACE.svc.$CLUSTER + - my.dns.search.suffix + - $HOST + options: + - ndots: 2 +``` + +Given the following host `/etc/resolv.conf`: + +```bash +nameserver 1.2.3.4 +search foo.com +options: ndots: 1 +``` + +The pod will get the following `/etc/resolv.conf`: + +```bash +nameserver 10.240.0.10 +# Populated from searchPaths +search ns1.svc.cluster.local my.dns.search.suffix foo.com +# Populated from options +options ndots: 2 +``` + +## More examples + +The following is a Pod dnsParams that only contains the host search paths: + +```yaml +dnsParams: + searchPaths: + - $HOST +``` + +Override `ndots` and add custom search path. Note that overriding the ndot may break the functionality of some of the search paths the + +```yaml +dnsParams: + searchPaths: + - my.custom.suffix + - $HOST + options: + - ndots: 3 +``` + +# API changes + +```go +type PodSpec struct { + ... + DNSPolicy string + DNSParams *PodDNSParams +} + +type PodDNSParams struct { + SearchPaths: []string + Options: []string +} + +// This will not appear in types.go but is here for explication purposes. +type DNSParamsSubstitution string +const ( + DNSParamsSearchPathNamespace = "$NAMESPACE" + DNSParamsSearchPathClusterDomain = "$CLUSTER" + DNSParamsSearchPathHostPaths = "$HOST" +) +``` + +## Semantics +### searchPath +If `dnsPolicy: Custom` is used, then the `search` line will be constructed from the entries listed in `dnsParams.searchPath`: +```go +// Note: pseudocode does not include input validation. +func SearchPath(params *PodDNSParams) []string { + var searchPaths []string + for _, entry := range params.SearchPaths { + if entry == "$HOST" { + searchPaths = append(searchPaths, HostSearchPaths...) + break + } + searchPaths = append(searchPaths, expand(entry)) + } + return searchPaths +} + +func expand(s string) string { + labels := strings.Split(s, ".") + for i := range labels { + if labels[0] == "$" { + labels[i] = Substitute(labels[i]) + } + } + return strings.Join(labels, ".") +} +``` + +#### Substitutions +`Substitute` will replace labels that begin with `$` with values for the given Pod: + +| Substitution | Description | +| ---- | ---- | +| `$NAMESPACE` | Namespace of the Pod | +| `$CLUSTER` | Kubernetes cluster domain (e.g. `cluster.local`) | +| `$HOST` | Host search paths. This is a special substitution that must appear at the end of the `searchPaths` list | + +### options +Each element of options will be copied unmodified as an options line. + +### Invalid configurations + +The follow configurations will result in an invalid Pod spec: + +* An invalid domain name/substitution appears in `searchPaths`. +* `dnsParams` is MUST be empty unless `dnsPolicy: Custom` is used. +* Number of final search paths exceeds 5 (glibc limit). +* `$HOST` is not the last item in `searchPaths`. +* `ndots` is greater than 15 (glibc limit). + +# References + +* [Kubernetes DNS name specification](https://github.com/kubernetes/dns/blob/master/docs/specification.md) +* [`/etc/resolv.conf manpage`](http://manpages.ubuntu.com/manpages/zesty/man5/resolv.conf.5.html) From 1171859a0ed97142badb98d330b639060ce8f308 Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Mon, 30 Oct 2017 16:29:17 -0700 Subject: [PATCH 2/8] More feedback --- .../network/pod-resolv-conf.md | 135 +++++++++++++----- 1 file changed, 96 insertions(+), 39 deletions(-) diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md index c04384c57de..13094389a4a 100644 --- a/contributors/design-proposals/network/pod-resolv-conf.md +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -1,32 +1,57 @@ # Custom /etc/resolv.conf + * Status: pending * Version: alpha -* Implementation owner: Bowei Du <[bowei@google.com](mailto:bowei@google.com)>, Zihong Zheng <[zihongz@google.com](mailto:zihongz@google.com)> +* Implementation owner: Bowei Du <[bowei@google.com](mailto:bowei@google.com)>, + Zihong Zheng <[zihongz@google.com](mailto:zihongz@google.com)> # Overview -The `/etc/resolv.conf` in a pod is managed by Kubelet and its contents are generated based on `pod.dnsPolicy`. For `dnsPolicy: Default`, `resolv.conf` is copied from the node where the pod is running. If the `dnsPolicy` is `ClusterFirst`, the contents of the resolv.conf is the hosts `resolv.conf` augmented with the following options: -* Search paths to add aliases for domain names in the same namespace and cluster suffix. -* `options ndots` to 5 to ensure the search paths are searched for all potential matches. +The `/etc/resolv.conf` in a pod is managed by Kubelet and its contents are +generated based on `pod.dnsPolicy`. For `dnsPolicy: Default`, the `search` and +`nameserver` fields are taken from the `resolve.conf` on the node where the pod +is running. If the `dnsPolicy` is `ClusterFirst`, the search contents of the +resolv.conf is the hosts `resolv.conf` augmented with the following options: + +* Search paths to add aliases for domain names in the same namespace and + cluster suffix. +* `options ndots` to 5 to ensure the search paths are searched for all + potential matches. -The configuration of both search paths and `ndots` results in query amplification of five to ten times for non-cluster internal names. This is due to the fact that each of the search path expansions must be tried before the actual result is found. This order of magnitude increase of query rate imposes a large load on the kube-dns service. At the same time, there are user applications do not need the convenience of the name aliases and do not wish to pay this performance cost. +The configuration of both search paths and `ndots` results in query +amplification of five to ten times for non-cluster internal names. This is due +to the fact that each of the search path expansions must be tried before the +actual result is found. This order of magnitude increase of query rate imposes a +large load on the kube-dns service. At the same time, there are user +applications do not need the convenience of the name aliases and do not wish to +pay this performance cost. ## Existing workarounds -The current work around for this problem is to specify an FQDN for name resolution. Any domain name that ends with a period (e.g. `foo.bar.com.`) will not be search path expanded. However, use of FQDNs is not well-known practice and imposes application-level changes. Cluster operators may not have the luxury of enforcing such a change to applications that run on their infrastructure. +The current work around for this problem is to specify an FQDN for name +resolution. Any domain name that ends with a period (e.g. `foo.bar.com.`) will +not be search path expanded. However, use of FQDNs is not well-known practice +and imposes application-level changes. Cluster operators may not have the luxury +of enforcing such a change to applications that run on their infrastructure. -It is also possible for the user to insert a short shell script snippet that rewrites `resolv.conf` on container start-up. This has the same problems as the previous approach and is also awkward for the user. This also forces the container to have additional executable code such as a shell or scripting engine which increases the applications security surface area. +It is also possible for the user to insert a short shell script snippet that +rewrites `resolv.conf` on container start-up. This has the same problems as the +previous approach and is also awkward for the user. This also forces the +container to have additional executable code such as a shell or scripting engine +which increases the applications security surface area. # Proposal sketch -Add a new `pod.dnsPolicy: Custom` that allows for user customization of `resolv.conf`. +Add a new `pod.dnsPolicy: Custom` that allows for user customization of +`resolv.conf`. ## Pod API example -In the example below, the user wishes to add the pod namespace and a custom expansion to the search path, as they do not use the other name aliases: +In the example below, the user wishes to add the pod namespace and a custom +expansion to the search path, as they do not use the other name aliases: ```yaml # Pod spec @@ -41,30 +66,37 @@ spec: image: example dnsPolicy: Custom dnsParams: - searchPaths: - - $NAMESPACE.svc.$CLUSTER - - my.dns.search.suffix - - $HOST - options: - - ndots: 2 + custom: + search: + - $(NAMESPACE).svc.$(CLUSTER) + - my.dns.search.suffix + - $(HOST) + options: + - "ndots:2" ``` Given the following host `/etc/resolv.conf`: ```bash nameserver 1.2.3.4 -search foo.com -options: ndots: 1 +search foo.com bar.com +options ndots:1 ``` The pod will get the following `/etc/resolv.conf`: ```bash nameserver 10.240.0.10 -# Populated from searchPaths -search ns1.svc.cluster.local my.dns.search.suffix foo.com -# Populated from options -options ndots: 2 +# Comments only for explication purposes. +# +# $(NAMESPACE).svc.$CLUSTER +# | my.dns.search.suffix +# | | $(HOST) +# | | | +# V V V +search ns1.svc.cluster.local my.dns.search.suffix foo.com bar.com +# Populated from custom.options +options ndots:2 ``` ## More examples @@ -73,19 +105,22 @@ The following is a Pod dnsParams that only contains the host search paths: ```yaml dnsParams: - searchPaths: - - $HOST + custom: + searchPaths: + - $HOST ``` -Override `ndots` and add custom search path. Note that overriding the ndot may break the functionality of some of the search paths the +Override `ndots` and add custom search path. Note that overriding the ndot may +break the functionality of some of the search paths the ```yaml dnsParams: - searchPaths: - - my.custom.suffix - - $HOST - options: - - ndots: 3 + custom: + searchPaths: + - my.custom.suffix + - $HOST + options: + - "ndots:3" ``` # API changes @@ -95,25 +130,33 @@ type PodSpec struct { ... DNSPolicy string DNSParams *PodDNSParams + ... } type PodDNSParams struct { - SearchPaths: []string - Options: []string + Custom PodDNSParamsCustom +} + +type PodDNSParamsCustom struct { + Search []string + Options []string } // This will not appear in types.go but is here for explication purposes. type DNSParamsSubstitution string const ( - DNSParamsSearchPathNamespace = "$NAMESPACE" - DNSParamsSearchPathClusterDomain = "$CLUSTER" - DNSParamsSearchPathHostPaths = "$HOST" + DNSParamsSearchPathNamespace = "$(NAMESPACE)" + DNSParamsSearchPathClusterDomain = "$(CLUSTER)" + DNSParamsSearchPathHostPaths = "$(HOST)" ) ``` ## Semantics ### searchPath -If `dnsPolicy: Custom` is used, then the `search` line will be constructed from the entries listed in `dnsParams.searchPath`: + +If `dnsPolicy: Custom` is used, then the `search` line will be constructed from +the entries listed in `dnsParams.custom.search`: + ```go // Note: pseudocode does not include input validation. func SearchPath(params *PodDNSParams) []string { @@ -137,18 +180,32 @@ func expand(s string) string { } return strings.Join(labels, ".") } + +var LabelSubstitutions map[string]string + +func Substitute(l label) string { + if s, ok := LabelSubstitutions[l]; ok { + return s + } + return l +} ``` #### Substitutions -`Substitute` will replace labels that begin with `$` with values for the given Pod: + +`Substitute` will replace DNS labels that begin with `$` with values for the +given Pod. Note: the DNS label MUST be an EXACT string match. E.g. if the pod +namespace is `my-ns`, `abc$(NAMESPACE)1234` will NOT be expanded to +`abcmy-ns1234`, but will be an invalid configuration. | Substitution | Description | | ---- | ---- | -| `$NAMESPACE` | Namespace of the Pod | -| `$CLUSTER` | Kubernetes cluster domain (e.g. `cluster.local`) | -| `$HOST` | Host search paths. This is a special substitution that must appear at the end of the `searchPaths` list | +| `$(NAMESPACE)` | Namespace of the Pod | +| `$(CLUSTER)` | Kubernetes cluster domain (e.g. `cluster.local`) | +| `$(HOST)` | Host search paths. This is a special substitution that MUST ONLY appear at the end of the `searchPaths` list | ### options + Each element of options will be copied unmodified as an options line. ### Invalid configurations From fbe7024507a5011a9b483a3c00e060328874a217 Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Tue, 7 Nov 2017 11:11:05 -0800 Subject: [PATCH 3/8] More updates --- .../network/pod-resolv-conf.md | 44 ++++++++++--------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md index 13094389a4a..58e09c3236d 100644 --- a/contributors/design-proposals/network/pod-resolv-conf.md +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -50,8 +50,9 @@ Add a new `pod.dnsPolicy: Custom` that allows for user customization of ## Pod API example -In the example below, the user wishes to add the pod namespace and a custom -expansion to the search path, as they do not use the other name aliases: +In the example below, the user wishes to use their own DNS resolver and add the +pod namespace and a custom expansion to the search path, as they do not use the +other name aliases: ```yaml # Pod spec @@ -67,12 +68,14 @@ spec: dnsPolicy: Custom dnsParams: custom: + nameservers: + - 1.2.3.4 search: - $(NAMESPACE).svc.$(CLUSTER) - my.dns.search.suffix - - $(HOST) options: - "ndots:2" + ``` Given the following host `/etc/resolv.conf`: @@ -91,7 +94,7 @@ nameserver 10.240.0.10 # # $(NAMESPACE).svc.$CLUSTER # | my.dns.search.suffix -# | | $(HOST) +# | | [from host] # | | | # V V V search ns1.svc.cluster.local my.dns.search.suffix foo.com bar.com @@ -106,48 +109,49 @@ The following is a Pod dnsParams that only contains the host search paths: ```yaml dnsParams: custom: - searchPaths: - - $HOST ``` -Override `ndots` and add custom search path. Note that overriding the ndot may -break the functionality of some of the search paths the +Override `ndots` and add custom search path and do not include host search +paths. Note that overriding the ndot may break the functionality of some of the +search paths. ```yaml dnsParams: custom: searchPaths: - my.custom.suffix - - $HOST options: - "ndots:3" + excludeHostSearchPaths: true ``` + + # API changes ```go type PodSpec struct { - ... - DNSPolicy string - DNSParams *PodDNSParams + ... + DNSPolicy string + DNSParams *PodDNSParams ... } type PodDNSParams struct { - Custom PodDNSParamsCustom + Custom PodDNSParamsCustom } type PodDNSParamsCustom struct { - Search []string - Options []string + Search []string + Options []string + ExcludeHostSearchPaths bool } // This will not appear in types.go but is here for explication purposes. type DNSParamsSubstitution string const ( - DNSParamsSearchPathNamespace = "$(NAMESPACE)" - DNSParamsSearchPathClusterDomain = "$(CLUSTER)" - DNSParamsSearchPathHostPaths = "$(HOST)" + DNSParamsSearchPathNamespace DNSParamsSubstitution = "$(NAMESPACE)" + DNSParamsSearchPathClusterDomain DNSParamsSubstitution = "$(CLUSTER)" ) ``` @@ -162,7 +166,7 @@ the entries listed in `dnsParams.custom.search`: func SearchPath(params *PodDNSParams) []string { var searchPaths []string for _, entry := range params.SearchPaths { - if entry == "$HOST" { + if !params.ExcludeHostSearchPaths { searchPaths = append(searchPaths, HostSearchPaths...) break } @@ -202,7 +206,6 @@ namespace is `my-ns`, `abc$(NAMESPACE)1234` will NOT be expanded to | ---- | ---- | | `$(NAMESPACE)` | Namespace of the Pod | | `$(CLUSTER)` | Kubernetes cluster domain (e.g. `cluster.local`) | -| `$(HOST)` | Host search paths. This is a special substitution that MUST ONLY appear at the end of the `searchPaths` list | ### options @@ -215,7 +218,6 @@ The follow configurations will result in an invalid Pod spec: * An invalid domain name/substitution appears in `searchPaths`. * `dnsParams` is MUST be empty unless `dnsPolicy: Custom` is used. * Number of final search paths exceeds 5 (glibc limit). -* `$HOST` is not the last item in `searchPaths`. * `ndots` is greater than 15 (glibc limit). # References From 9433baeec6003f98857ac2c9f0a2a1c7088d72de Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Tue, 7 Nov 2017 11:29:15 -0800 Subject: [PATCH 4/8] Feedback --- .../network/pod-resolv-conf.md | 33 ++++++++++++------- 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md index 58e09c3236d..002324850ad 100644 --- a/contributors/design-proposals/network/pod-resolv-conf.md +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -71,7 +71,7 @@ spec: nameservers: - 1.2.3.4 search: - - $(NAMESPACE).svc.$(CLUSTER) + - $(NAMESPACE).svc.$(CLUSTER_SUBDOMAIN) - my.dns.search.suffix options: - "ndots:2" @@ -92,7 +92,7 @@ The pod will get the following `/etc/resolv.conf`: nameserver 10.240.0.10 # Comments only for explication purposes. # -# $(NAMESPACE).svc.$CLUSTER +# $(NAMESPACE).svc.$(CLUSTER_SUBDOMAIN) # | my.dns.search.suffix # | | [from host] # | | | @@ -111,9 +111,20 @@ dnsParams: custom: ``` -Override `ndots` and add custom search path and do not include host search -paths. Note that overriding the ndot may break the functionality of some of the -search paths. +Don't include host search paths, override the nameservers. Note: this will for +all intents and purposes disable Kubernetes DNS. + +```yaml +dnsParams: + custom: + nameservers: + - 1.2.3.4 + - 1.2.3.5 + excludeHostSearchPaths: true +``` + +Override `ndots` and add custom search path. Note that overriding the ndot may +break the functionality of some of the search paths. ```yaml dnsParams: @@ -122,11 +133,8 @@ dnsParams: - my.custom.suffix options: - "ndots:3" - excludeHostSearchPaths: true ``` - - # API changes ```go @@ -142,8 +150,9 @@ type PodDNSParams struct { } type PodDNSParamsCustom struct { - Search []string - Options []string + Namerservers []string + Search []string + Options []string ExcludeHostSearchPaths bool } @@ -151,7 +160,7 @@ type PodDNSParamsCustom struct { type DNSParamsSubstitution string const ( DNSParamsSearchPathNamespace DNSParamsSubstitution = "$(NAMESPACE)" - DNSParamsSearchPathClusterDomain DNSParamsSubstitution = "$(CLUSTER)" + DNSParamsSearchPathClusterDomain DNSParamsSubstitution = "$(CLUSTER_SUBDOMAIN)" ) ``` @@ -205,7 +214,7 @@ namespace is `my-ns`, `abc$(NAMESPACE)1234` will NOT be expanded to | Substitution | Description | | ---- | ---- | | `$(NAMESPACE)` | Namespace of the Pod | -| `$(CLUSTER)` | Kubernetes cluster domain (e.g. `cluster.local`) | +| `$(CLUSTER_SUBDOMAIN)` | Kubernetes cluster domain (e.g. `cluster.local`) | ### options From 4a60bf307e393dc36ee7a441b132401eb27f1365 Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Tue, 7 Nov 2017 11:33:45 -0800 Subject: [PATCH 5/8] Nameservers --- contributors/design-proposals/network/pod-resolv-conf.md | 1 + 1 file changed, 1 insertion(+) diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md index 002324850ad..3b537c8ccf9 100644 --- a/contributors/design-proposals/network/pod-resolv-conf.md +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -224,6 +224,7 @@ Each element of options will be copied unmodified as an options line. The follow configurations will result in an invalid Pod spec: +* Invalid number of nameservers (more than three). * An invalid domain name/substitution appears in `searchPaths`. * `dnsParams` is MUST be empty unless `dnsPolicy: Custom` is used. * Number of final search paths exceeds 5 (glibc limit). From e6df2a7933094ffa913b46fef077863fb869309d Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Mon, 13 Nov 2017 16:34:24 -0800 Subject: [PATCH 6/8] Remove template, make fields append/replace instead of just replace --- .../network/pod-resolv-conf.md | 213 +++++++----------- 1 file changed, 87 insertions(+), 126 deletions(-) diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md index 3b537c8ccf9..5a66d5282d3 100644 --- a/contributors/design-proposals/network/pod-resolv-conf.md +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -44,11 +44,40 @@ which increases the applications security surface area. # Proposal sketch -Add a new `pod.dnsPolicy: Custom` that allows for user customization of -`resolv.conf`. +This proposal gives users a way to overlay tweaks into the existing +`DnsPolicy`. A new PodSpec field `dnsParams` will contains fields that are +merged with the settings currently selected with `DnsPolicy`. +The fields of `DnsParams` are: -## Pod API example +* `nameservers` is a list of additional nameservers to use for resolution. On + `resolv.conf` platforms, these are entries to `nameserver`. +* `search` is a list of additional search path subdomains. On `resolv.conf` + platforms, these are entries to the `search` setting. These domains will be + appended to the existing search path. +* `options` that are an OS-dependent list of options. For containers that use + `/etc/resolv.conf` style configuration, these correspond to the parameters + passed to the `option` lines. Options will override if their names coincide, + i.e, if the `DnsPolicy` sets `ndots:5` and `ndots:1` appears in the `Spec`, + then the final value will be `ndots:1`. + +For users that want to completely customize their resolution configuration, we +add a new `DnsPolicy: Custom` that does not define any settings. This is +essentially an empty `resolv.conf` with no fields defined. + +## Pod API examples + +### Host `/etc/resolv.conf` + +Assume in the examples below that the host has the following `/etc/resolv.conf`: + +```bash +nameserver 10.1.1.10 +search foo.com +options ndots:1 +``` + +### Override DNS server and search paths In the example below, the user wishes to use their own DNS resolver and add the pod namespace and a custom expansion to the search path, as they do not use the @@ -58,81 +87,42 @@ other name aliases: # Pod spec apiVersion: v1 kind: Pod -metadata: - namespace: ns1 - name: example +metadata: {"namespace": "ns1", "name": "example"} spec: - containers: - - name: example - image: example + ... dnsPolicy: Custom dnsParams: - custom: - nameservers: - - 1.2.3.4 - search: - - $(NAMESPACE).svc.$(CLUSTER_SUBDOMAIN) - - my.dns.search.suffix - options: - - "ndots:2" - -``` - -Given the following host `/etc/resolv.conf`: - -```bash -nameserver 1.2.3.4 -search foo.com bar.com -options ndots:1 + nameservers: ["1.2.3.4"] + search: + - ns1.svc.cluster.local + - my.dns.search.suffix + options: ["ndots:2"] ``` The pod will get the following `/etc/resolv.conf`: ```bash -nameserver 10.240.0.10 -# Comments only for explication purposes. -# -# $(NAMESPACE).svc.$(CLUSTER_SUBDOMAIN) -# | my.dns.search.suffix -# | | [from host] -# | | | -# V V V -search ns1.svc.cluster.local my.dns.search.suffix foo.com bar.com -# Populated from custom.options +nameserver 1.2.3.4 +search ns1.svc.cluster.local my.dns.search.suffix options ndots:2 ``` -## More examples - -The following is a Pod dnsParams that only contains the host search paths: +## Overriding `ndots` -```yaml -dnsParams: - custom: -``` - -Don't include host search paths, override the nameservers. Note: this will for -all intents and purposes disable Kubernetes DNS. +Override `ndots:5` in `ClusterFirst` with `ndots:1`. This keeps all of the +settings intact: ```yaml -dnsParams: - custom: - nameservers: - - 1.2.3.4 - - 1.2.3.5 - excludeHostSearchPaths: true +dnsPolicy: ClusterFirst +dnsParams: {"options": "ndots:1"} ``` -Override `ndots` and add custom search path. Note that overriding the ndot may -break the functionality of some of the search paths. +Resulting `resolv.conf`: -```yaml -dnsParams: - custom: - searchPaths: - - my.custom.suffix - options: - - "ndots:3" +```bash +nameserver 10.0.0.10 +search default.svc.cluster.local svc.cluster.local cluster.local foo.com +options ndots:1 ``` # API changes @@ -146,89 +136,60 @@ type PodSpec struct { } type PodDNSParams struct { - Custom PodDNSParamsCustom -} - -type PodDNSParamsCustom struct { - Namerservers []string - Search []string - Options []string - ExcludeHostSearchPaths bool + Nameservers []string + Search []string + Options []string } - -// This will not appear in types.go but is here for explication purposes. -type DNSParamsSubstitution string -const ( - DNSParamsSearchPathNamespace DNSParamsSubstitution = "$(NAMESPACE)" - DNSParamsSearchPathClusterDomain DNSParamsSubstitution = "$(CLUSTER_SUBDOMAIN)" -) ``` ## Semantics -### searchPath -If `dnsPolicy: Custom` is used, then the `search` line will be constructed from -the entries listed in `dnsParams.custom.search`: +Let the following be the Go representation of the `resolv.conf`: ```go -// Note: pseudocode does not include input validation. -func SearchPath(params *PodDNSParams) []string { - var searchPaths []string - for _, entry := range params.SearchPaths { - if !params.ExcludeHostSearchPaths { - searchPaths = append(searchPaths, HostSearchPaths...) - break - } - searchPaths = append(searchPaths, expand(entry)) - } - return searchPaths +type ResolvConf struct { + Nameserver []string // "nameserver" entries + Search []string // "search" entries + Options []string // "options" entries } +``` -func expand(s string) string { - labels := strings.Split(s, ".") - for i := range labels { - if labels[0] == "$" { - labels[i] = Substitute(labels[i]) - } - } - return strings.Join(labels, ".") -} +Let `var HostResolvConf ResolvConf` be the host `resolv.conf`. -var LabelSubstitutions map[string]string +Then the final Pod `resolv.conf` will be: -func Substitute(l label) string { - if s, ok := LabelSubstitutions[l]; ok { - return s - } - return l +```go +func podResolvConf() ResolvConf { + var podResolv ResolvConf + + switch (pod.DNSPolicy) { + case "Default": + podResolv = HostResolvConf + case "ClusterFirst: + podResolv.Nameservers = []string{ KubeDNSClusterIP } + podResolv.Search = ... // populate with ns.svc.suffix, svc.suffix, suffix, host entries... + podResolv.Options = []string{ "ndots:5" } + case "Custom": // start with empty `resolv.conf` + break + } + + // Append the additional nameservers. + podResolv.Nameservers = append(Nameservers, pod.DNSParams.Nameservers...) + // Append the additional search paths. + podResolv.Search = append(Search, pod.DNSParams.Search...) + // Overlay the DnsParams.Options with the default options. + podResolv.Options = mergeOptions(pod.Options, pod.DNSParams.Options) + + return podResolv } ``` -#### Substitutions - -`Substitute` will replace DNS labels that begin with `$` with values for the -given Pod. Note: the DNS label MUST be an EXACT string match. E.g. if the pod -namespace is `my-ns`, `abc$(NAMESPACE)1234` will NOT be expanded to -`abcmy-ns1234`, but will be an invalid configuration. - -| Substitution | Description | -| ---- | ---- | -| `$(NAMESPACE)` | Namespace of the Pod | -| `$(CLUSTER_SUBDOMAIN)` | Kubernetes cluster domain (e.g. `cluster.local`) | - -### options - -Each element of options will be copied unmodified as an options line. - ### Invalid configurations The follow configurations will result in an invalid Pod spec: -* Invalid number of nameservers (more than three). -* An invalid domain name/substitution appears in `searchPaths`. -* `dnsParams` is MUST be empty unless `dnsPolicy: Custom` is used. -* Number of final search paths exceeds 5 (glibc limit). -* `ndots` is greater than 15 (glibc limit). +* nameservers or search paths exceed system limits. (Three nameservers, six + search paths, 256 characters for `glibc`). # References From 77610bba016ea41afaedc8ccbee5ce9f7fa3b759 Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Mon, 13 Nov 2017 17:21:44 -0800 Subject: [PATCH 7/8] Make options mergeable --- .../network/pod-resolv-conf.md | 41 +++++++++++++------ 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md index 5a66d5282d3..f927c23e108 100644 --- a/contributors/design-proposals/network/pod-resolv-conf.md +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -55,8 +55,10 @@ The fields of `DnsParams` are: * `search` is a list of additional search path subdomains. On `resolv.conf` platforms, these are entries to the `search` setting. These domains will be appended to the existing search path. -* `options` that are an OS-dependent list of options. For containers that use - `/etc/resolv.conf` style configuration, these correspond to the parameters + +* `options` that are an OS-dependent list of (name, value) options. These values + are NOT expected to be generally portable across platforms. For containers that + use `/etc/resolv.conf` style configuration, these correspond to the parameters passed to the `option` lines. Options will override if their names coincide, i.e, if the `DnsPolicy` sets `ndots:5` and `ndots:1` appears in the `Spec`, then the final value will be `ndots:1`. @@ -96,7 +98,10 @@ spec: search: - ns1.svc.cluster.local - my.dns.search.suffix - options: ["ndots:2"] + options: + - name: ndots + value: 2 + - name: edns0 ``` The pod will get the following `/etc/resolv.conf`: @@ -105,6 +110,7 @@ The pod will get the following `/etc/resolv.conf`: nameserver 1.2.3.4 search ns1.svc.cluster.local my.dns.search.suffix options ndots:2 +options edns0 ``` ## Overriding `ndots` @@ -114,7 +120,10 @@ settings intact: ```yaml dnsPolicy: ClusterFirst -dnsParams: {"options": "ndots:1"} +dnsParams: +- options: + - name: ndots + - value: 1 ``` Resulting `resolv.conf`: @@ -130,15 +139,20 @@ options ndots:1 ```go type PodSpec struct { ... - DNSPolicy string - DNSParams *PodDNSParams + DNSPolicy string `json:"dnsPolicy,omitempty"` + DNSParams *PodDNSParams `json:"dnsParams,omitempty"` ... } type PodDNSParams struct { - Nameservers []string - Search []string - Options []string + Nameservers []string `json:"nameservers,omitempty"` + Search []string `json:"search,omitempty"` + Options []PodDNSParamsOption `json:"options,omitempty" patchStrategy:"merge" patchMergeKey:"name"` +} + +type PodDNSParamsOption struct { + Name string `json:"name"` + Value *string `json:"value,omitempty"` } ``` @@ -150,7 +164,7 @@ Let the following be the Go representation of the `resolv.conf`: type ResolvConf struct { Nameserver []string // "nameserver" entries Search []string // "search" entries - Options []string // "options" entries + Options []PodDNSParamsOption // "options" entries } ``` @@ -168,7 +182,7 @@ func podResolvConf() ResolvConf { case "ClusterFirst: podResolv.Nameservers = []string{ KubeDNSClusterIP } podResolv.Search = ... // populate with ns.svc.suffix, svc.suffix, suffix, host entries... - podResolv.Options = []string{ "ndots:5" } + podResolv.Options = []PodDNSParamsOption{{"ndots","5" }} case "Custom": // start with empty `resolv.conf` break } @@ -177,7 +191,7 @@ func podResolvConf() ResolvConf { podResolv.Nameservers = append(Nameservers, pod.DNSParams.Nameservers...) // Append the additional search paths. podResolv.Search = append(Search, pod.DNSParams.Search...) - // Overlay the DnsParams.Options with the default options. + // Merge the DnsParams.Options with the options derived from the given DNSPolicy. podResolv.Options = mergeOptions(pod.Options, pod.DNSParams.Options) return podResolv @@ -188,8 +202,9 @@ func podResolvConf() ResolvConf { The follow configurations will result in an invalid Pod spec: -* nameservers or search paths exceed system limits. (Three nameservers, six +* Nameservers or search paths exceed system limits. (Three nameservers, six search paths, 256 characters for `glibc`). +* Invalid option appears for the given platform. # References From eb79043a77170cc4e340d47f2835de5fb75d0d31 Mon Sep 17 00:00:00 2001 From: Bowei Du Date: Thu, 16 Nov 2017 16:44:02 -0800 Subject: [PATCH 8/8] fix options --- contributors/design-proposals/network/pod-resolv-conf.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/contributors/design-proposals/network/pod-resolv-conf.md b/contributors/design-proposals/network/pod-resolv-conf.md index f927c23e108..04a97df48e7 100644 --- a/contributors/design-proposals/network/pod-resolv-conf.md +++ b/contributors/design-proposals/network/pod-resolv-conf.md @@ -55,7 +55,6 @@ The fields of `DnsParams` are: * `search` is a list of additional search path subdomains. On `resolv.conf` platforms, these are entries to the `search` setting. These domains will be appended to the existing search path. - * `options` that are an OS-dependent list of (name, value) options. These values are NOT expected to be generally portable across platforms. For containers that use `/etc/resolv.conf` style configuration, these correspond to the parameters @@ -109,8 +108,7 @@ The pod will get the following `/etc/resolv.conf`: ```bash nameserver 1.2.3.4 search ns1.svc.cluster.local my.dns.search.suffix -options ndots:2 -options edns0 +options ndots:2 edns0 ``` ## Overriding `ndots`