Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CoreDNS support for ExternalName services is broken in EKS 1.11 #129

Closed
bhang opened this issue Jan 24, 2019 · 17 comments
Closed

CoreDNS support for ExternalName services is broken in EKS 1.11 #129

bhang opened this issue Jan 24, 2019 · 17 comments
Labels
EKS Amazon Elastic Kubernetes Service

Comments

@bhang
Copy link

bhang commented Jan 24, 2019 •
edited
Loading

Environment

Kubernetes version: 1.11
Platform version: eks.1
Cluster DNS provider: CoreDNS
CoreDNS image: 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns:v1.1.3

Problem

The version of coreDNS being deployed to 1.11 EKS clusters suffers from this issue - coredns/coredns#2038 (repro steps available here)

This issue has been fixed in coredns 1.2.1 but EKS is still running the older version (i.e. 1.1.3) and so ExternalName services that serve as an alias for in-cluster kubernetes services in another namespace cannot currently be resolved.

This problem does not exist in 1.10 i.e. w/ kubedns
@abby-fuller abby-fuller added the EKS Amazon Elastic Kubernetes Service label Jan 26, 2019
@eswarbala
Copy link

Thanks for reporting this @bhang. We will get this fixed.

@mogaal
Copy link

mogaal commented Feb 14, 2019

I am facing the same problem, any workaround? Maybe remove CoreDNS and come back to kube-dns again?

@ghost
Copy link

ghost commented Feb 14, 2019

@mogaal we just run newer version of coredns. Version 1.3.1.

@0verc1ocker 0verc1ocker mentioned this issue Feb 25, 2019
Closed
@stafot
Copy link

stafot commented Feb 26, 2019

@spronin-aurea so is it fine to use upstream releases there aren't any "compatibility" problems with eks?
/cc @eswarbala

@shyamjvs
Copy link

shyamjvs commented Mar 7, 2019

@bhang You should now be able to edit your coredns deployment to new image 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/coredns:v1.2.2. Though note that upstream tested k8s 1.11 with 1.1.3 coredns and is the recommended one for 1.11. So can you try and let us know if new one works?

@ghost
Copy link

ghost commented Mar 7, 2019

@spronin-aurea so is it fine to use upstream releases there aren't any "compatibility" problems with eks?
/cc @eswarbala

Yes, it is fine. We run it in heavy prod for some time now.

@shyamjvs
Copy link

shyamjvs commented Mar 7, 2019

Ok, I believe we can mark this one fixed then. Re-open as apt.

@stafot
Copy link

stafot commented Mar 7, 2019

Yes I used also latest. Seems fine.

@bhang
Copy link
Author

bhang commented Mar 29, 2019

@shyamjvs Apologies for the delay. Can confirm that switching to the version you indicated solves the problem

@whereisaaron
Copy link

eksctl 0.1.26 has a eksctl utils update-coredns command now to make updating existing clusters easy.
eksctl-io/eksctl#678
https://github.com/weaveworks/eksctl/releases/tag/0.1.26

@StevenACoffman
Copy link

StevenACoffman commented Apr 23, 2019 •
edited
Loading

The 1.1.3 manifest lives at https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2019-02-11/dns.yaml

Which we updated to use 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/coredns:v1.2.2

Do we have a repository or tracking issue where we can find the approved manifest as it is updated?

@StevenACoffman
Copy link

@whereisaaron ref #266

@richiesgr richiesgr mentioned this issue May 12, 2019
Closed
@AndiDog
Copy link

AndiDog commented Aug 5, 2019

I tried to patch my EKS 1.13 coredns deployment as follows:

kubectl patch -n kube-system deployment/coredns --type json --patch '[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"coredns/coredns:1.3.1"}]'

But none of the versions 1.3.1 and 1.4.0 worked (1.6.1 fails to start). I cannot resolve service names of type ExternalName.

The above comments read like upstream releases should work. Which steps did you take to fix this?

@PFacheris
Copy link

@AndiDog, I had a similar issue but doing this resolved it for me: coredns/coredns#2038 (comment)

@AndiDog
Copy link

AndiDog commented Sep 19, 2019

I'd rather not use workarounds to fix stuff in production, especially on strongly audited systems. Would be very surprising if an application then doesn't work anymore after a cluster version upgrade.

Today, I have upgraded my EKS cluster to 1.14 including setting all the recommended image versions (core-dns etc.), but it still fails to resolve ExternalName services.

apiVersion: v1
kind: Service
metadata:
  name: exampledotcom
spec:
  type: ExternalName
  externalName: 127.0.0.1

Attempt to access from some pod fails:

$ curl exampledotcom.default.svc
curl: (6) Could not resolve host: exampledotcom.default.svc

$ curl -k https://my-non-external-svc.my-namespace.svc:443
<!doctype html>...

@ctran
Copy link

ctran commented Sep 22, 2019 •
edited
Loading

Per https://kubernetes.io/docs/concepts/services-networking/service/#externalname, it has to be a canonical DNS name:

Note: ExternalName accepts an IPv4 address string, but as a DNS names comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name. To hardcode an IP address, consider using headless Services.

@AndiDog
Copy link

AndiDog commented Sep 24, 2019

Indeed, using an FQDN instead of an IP works! On EKS with Kubernetes 1.14 (platform version eks.1) at least, cannot test other versions right now. Maybe other people can verify if this works for them.

apiVersion: v1
kind: Service
metadata:
  name: exampledotcom
spec:
  type: ExternalName
  externalName: example.com

$ curl -sS -H 'Host: example.com' exampledotcom.default | tail

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is established to be used for illustrative examples in documents. You may use this
    domain in examples without prior coordination or asking for permission.</p>
    <p><a href="http://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
EKS Amazon Elastic Kubernetes Service
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests