Setting CoreDNS version to 1.8.3 did not update its RBAC

[xrl]

1. What kops version are you running? The command kops version, will display
this information.

kops 1.19.2 built from the release-1.19 branch

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

kubernetes 1.19.10

3. What cloud provider are you using?

aws

4. What commands did you run? What is the simplest way to reproduce this issue?

kops replace $my_cluster_config.yaml
kops update
kops update --yes

the config part that I think matters:

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2019-03-26T00:00:00Z
  name: elided.io
spec:
  docker:
    skipInstall: true
  kubeDNS:
    provider: CoreDNS
    coreDNSImage: k8s.gcr.io/coredns/coredns:v1.8.3

but the problem is that the coredns ClusterRole was not updated. I looked at the upstream config and it was different from what was in my kube cluster:

  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch

but in my config map it was still on the 1.6.3 style I had before:

- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get

How is the coredns release managed? Could this ClusterRole be handled by kops?

5. What happened after the commands executed?

Kubernetes 1.19 was rolled out and most of the pods worked but eventually coredns became inoperable and I lost DNS.

6. What did you expect to happen?

Setting the coredns version should have synced all the parameters.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2019-03-26T00:00:00Z
  name: cluster.io
spec:
  docker:
    skipInstall: true
  kubeDNS:
    provider: CoreDNS
    coreDNSImage: k8s.gcr.io/coredns/coredns:v1.8.3
  kubelet:
    featureGates:
      ExpandInUsePersistentVolumes: "true"
  kubeControllerManager:
    clusterCIDR: 10.43.202.0/23
    allocateNodeCIDRs: false
  kubeProxy:
    clusterCIDR: 10.43.202.0/23
  api:
    loadBalancer:
      type: Internal
      idleTimeoutSeconds: 1800 # 30 minutes
  authorization:
    rbac: {}
  authentication:
    aws: {}
  additionalPolicies:
    master: |
      [
        {
          "Effect": "Allow",
          "Action": ["s3:ListObjects"],
          "Resource": ["arn:aws:s3:::storage/cluster.io/*"]
        },
        {
          "Effect": "Allow",
          "Action": ["elasticloadbalancing:DeregisterTargets"],
          "Resource": ["arn:aws:elasticloadbalancing:us-west-2:123:targetgroup/*"]
        }
      ]
    node: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "ec2:CreateNetworkInterface",
            "ec2:AttachNetworkInterface",
            "ec2:DeleteNetworkInterface",
            "ec2:DetachNetworkInterface",
            "ec2:DescribeNetworkInterfaces",
            "ec2:DescribeInstances",
            "ec2:ModifyNetworkInterfaceAttribute",
            "ec2:AssignPrivateIpAddresses",
            "ec2:UnassignPrivateIpAddresses"
          ],
          "Resource": ["*"]
        },
        {
          "Effect": "Allow",
          "Action": [
            "ec2:CreateTags"
          ],
          "Resource": [
            "arn:aws:ec2:*:*:network-interface/*"
          ]
        }
      ]
  channel: stable
  cloudProvider: aws
  configBase: s3://config/cluster.io
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-west-2c
      name: c
    name: main
    enableEtcdTLS: true
    version: 3.2.24
  - etcdMembers:
    - instanceGroup: master-us-west-2c
      name: c
    name: events
    enableEtcdTLS: true
    version: 3.2.24
  iam:
    allowContainerRegistry: true
    legacy: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.19.10
  masterPublicname: api.cluster.io
  networkCIDR: 10.43.202.0/23
  networkID: vpc-0b94766c8a8c871a4
  networking:
    amazonvpc:
      imageName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.7.10
      env:
      - name: WARM_IP_TARGET
        value: "1"
      - name: WARM_ENI_TARGET
        value: "0"
  nonMasqueradeCIDR: 10.43.202.0/23
  # We're borrowing a subnet from prod. They should never meet, but just in case.
  serviceClusterIPRange: 10.43.175.0/24
  sshAccess:
  - "10.0.0.0/8"
  - "172.16.0.0/12"
  - "192.168.0.0/16"
  subnets:
  - cidr: 10.43.202.0/23
    id: subnet-03fa682131e608935
    name: us-west-2c
    type: Private
    zone: us-west-2c
    egress: External
  topology:
    dns:
      type: Private
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

[dntosas]

fixed with #11459

[estahn]

@dntosas Is there a workaround? We see coredns not coming up when upgrading from 1.18 to 1.19.

[dntosas]

@estahn you need Kops version >=1.21.0 to get this patch ^^

as you can see from here this is meant for 1.22 release but also backported on the 1.21 one

[estahn]

@dntosas We used the latest version available via brew:

kops version
Version 1.21.0

I'll try to get the error message.

[estahn]

We had a custom coredns config that didn't include ready. This was the source of our issue.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[olemarkus]

/remove-lifecycle rotten
/close

[k8s-ci-robot]

@olemarkus: Closing this issue.

In response to this:

/remove-lifecycle rotten
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.