Restrict IAM Roles permissions

[ffjia]

According to this document, the permissions look like too open. Could we just grant the minimum privileges (and list all the necessary permissions)?

For Master, change the ec2, route53, and elasticloadbalancing to:

ec2:AllocateAddress
ec2:AssociateAddress
ec2:AssociateRouteTable
ec2:AttachInternetGateway
ec2:AuthorizeSecurityGroupEgress
ec2:AuthorizeSecurityGroupIngress
ec2:CreateInternetGateway
ec2:CreateNatGateway
ec2:CreateRoute
ec2:CreateRouteTable
ec2:CreateSecurityGroup
ec2:CreateSubnet
ec2:CreateTags
ec2:CreateVpc
ec2:CreateVolume
ec2:DescribeAddresses
ec2:DescribeAvailabilityZones
ec2:DescribeInstances
ec2:DescribeInternetGateways
ec2:DescribeKeyPairs
ec2:DescribeNatGateways
ec2:DescribeRegions
ec2:DescribeRouteTables
ec2:DescribeSecurityGroups
ec2:DescribeSubnets
ec2:DescribeVpcs
ec2:ModifySubnetAttribute
ec2:ModifyVpcAttribute
ec2:RevokeSecurityGroupEgress
ec2:RunInstances

route53:ChangeResourceRecordSets
route53:GetChange
route53:GetHostedZone
route53:ListHostedZones
route53:ListResourceRecordSets

elasticloadbalancing:CreateLoadBalancer
elasticloadbalancing:ConfigureHealthCheck
elasticloadbalancing:ModifyLoadBalancerAttributes
elasticloadbalancing:DescribeLoadBalancers
elasticloadbalancing:RegisterInstancesWithLoadBalancer
elasticloadbalancing:SetLoadBalancerPoliciesOfListener

For Node, update route53 to:

route53:ChangeResourceRecordSets
route53:GetChange
route53:GetHostedZone
route53:ListHostedZones
route53:ListResourceRecordSets

Those are simply borrowed from AWS policy settings for Tectonic, should be sufficient?

[zytek]

Related to #1577 #1871 #1870

BTW I believe master also need ability to remove ELBs although that probably should be scoped to the ones with KubernetesCluster tags.

[ffjia]

Thanks for the information. Glad that you have done some awesome work to push this :)

We could add this to Master:

elasticloadbalancing:DeleteLoadBalancer

[chrislovecnm]

@ffjia have you tested this?

[ffjia]

@chrislovecnm not yet, can kops support provision cluster with a existing IAM role?

[yissachar]

@ffjia You can't yet pass in a role, but you can add policies. I know it's awkward for this use case, but it could probably still be used for testing.

[ffjia]

@yissachar In that case, I need to add a bunch of actions of Effect: Deny? It's better if we could allow some permissions explicitly 😄

[yissachar]

Yes, that's why I said it was awkward.

It isn't really intended for overriding the base permissions, more for adding extra permissions that aren't included by default.

[chrislovecnm]

Proposal

BYO IAM roles. Need to scope if this would work, but it would allow a transition to allowing users to create there own IAM roles, and not have kops create roles. Keep the same functionality that we have now and add the advanced user option into the API / YAML.

Build json examples for each role, and document those examples.

Improvements

Once the use of your own roles is tested we can determine if we want to tighten down the permissions. I would recommend that we look at VPC level isolation for the nodes. Currently, if I understand the default IAM permissions, we are not reducing the master and node perms to just the VPC to which the cluster is deployed into.

Leg Work

So here is some leg work that I have completed. NOT tested.

Kops user / CLI permissions
Kops k8s master permissions
Kops k8s node permissions

[chrislovecnm]

Here is another issue with good information: #1577

[hubt]

It'd be great if kops updated the permissions based on some configuration information. Primarily I'm thinking that if someone selects a CNI, they probably don't need/want VPC modification policies. This would make kops better for organizations that have tight VPC control policies. (This of course would make it hard to switch between CNI and VPC routing, but there could be a flag there too).

There are probably other examples of things like this as well, but VPC creation and route53 modifications are things that frequently are in separate areas of an organization.

[chrislovecnm]

I going to keep this issue going, but I am going to start breaking it into short issues, which we can PR on.

Code / Documentation good ...

[chrislovecnm]

@fridiculous just worked through the s3 perms:

    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::kops.dev.example.com",
                "arn:aws:s3:::kops.dev.example.com/*"
            ],
            "Condition": {
                "StringNotLike": {
                    "aws:userId": [
                        "AIDAUSER1",   --user1
                        "123456789",            --root
                        "AROAROLE1:*", --masters.dev.example.com 
                        "AROAROLE2:*", --nodes.dev.example.com
                    ]
                }
            }
        }
    ]
}```

[chrislovecnm]

http://docs.aws.amazon.com/IAM/latest/APIReference/API_SimulateCustomPolicy.html was mentioned :)

[chrislovecnm]

Some feedback on a CF tmplate that is working:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Custom masters Role",
  "Parameters": {
  },
  "Mappings": {
  },
  "Resources": {
    "MastersRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": "cmasters.kops.lab",
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "ec2.amazonaws.com"
                ]
              },
              "Action": [
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path": "/"
      }
    },
    "MastersPolicy": {
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "cmasters.kops.lab",
        "Roles": [ { "Ref": "MastersRole" } ],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "ec2:Describe*",
                "ec2:Attach*",
                "ec2:Detach*",
                "ec2:ModifyInstanceAttribute",
                "ec2:CreateRoute",
                "ec2:DeleteRoute",
                "ec2:*SecurityGroup*",
                "ec2:CreateTags"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "elasticloadbalancing:*"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets",
                "route53:GetHostedZone"
              ],
              "Resource": [
                "arn:aws:route53:::hostedzone/XXXXXXXX"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "route53:GetChange"
              ],
              "Resource": [
                "arn:aws:route53:::change/*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "route53:ListHostedZones"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "s3:*"
              ],
              "Resource": [
                "arn:aws:s3:::kops.lab/*kops.lab",
                "arn:aws:s3:::kops.lab/*kops.lab/*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
              ],
              "Resource": [
                "arn:aws:s3:::kops.lab"
              ]
            }
          ]
        }
      }
    }
  },
  "Outputs": {
  }
}
The nodes one:

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "Custom nodes Role",
  "Parameters": {
  },
  "Mappings": {
  },
  "Resources": {
    "NodesRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": "cnodes.kops.lab",
        "AssumeRolePolicyDocument": {
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "ec2.amazonaws.com"
                ]
              },
              "Action": [
                "sts:AssumeRole"
              ]
            }
          ]
        },
        "Path": "/"
      }
    },
    "NodesPolicy": {
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": "cnodes.kops.lab",
        "Roles": [ { "Ref": "NodesRole" } ],
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "ec2:Describe*"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "route53:ChangeResourceRecordSets",
                "route53:ListResourceRecordSets",
                "route53:GetHostedZone"
              ],
              "Resource": [
                "arn:aws:route53:::hostedzone/XXXXXXXXXX"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "route53:GetChange"
              ],
              "Resource": [
                "arn:aws:route53:::change/*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "route53:ListHostedZones"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "s3:*"
              ],
              "Resource": [
                "arn:aws:s3:::kops.lab/*kops.lab",
                "arn:aws:s3:::kops.lab/*kops.lab/*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "s3:GetBucketLocation",
                "s3:ListBucket"
              ],
              "Resource": [
                "arn:aws:s3:::kops.lab"
              ]
            }
          ]
        }
      }
    }
  },
  "Outputs": {
  }
}

[chrislovecnm]

Hey IAM gurus. Please take a peek and comment on https://github.com/kubernetes/kops/pull/2497/files#diff-55ece441df560384483b9d78ed8785fdR265

I am working on a PR to create IAM policies for

• admin, someone that uses kops
• master
• node

With tightened permissions.

[chrislovecnm]

Here is where I am at. A kubernetes e2e test using these policies is passing. I have not tested ecr or autoscaling.

Master Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AttachVolume",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:CreateRoute",
        "ec2:CreateSecurityGroup",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteRoute",
        "ec2:DeleteVolume",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSubnets",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeVolumes",
        "ec2:DetachVolume",
        "ec2:ModifyInstanceAttribute",
        "ec2:RevokeSecurityGroupIngress"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:AttachLoadBalancerToSubnets",
        "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:CreateLoadBalancerPolicy",
        "elasticloadbalancing:CreateLoadBalancerListeners",
        "elasticloadbalancing:ConfigureHealthCheck",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DeleteLoadBalancerListeners",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:DetachLoadBalancerFromSubnets",
        "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
        "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:GetAsgForInstance",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "autoscaling:UpdateAutoScalingGroup"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:ListServerCertificates",
        "iam:GetServerCertificate"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::aws.k8spro.com",
        "arn:aws:s3:::aws.k8spro.com/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::aws.k8spro.com"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey",
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:BatchGetImage"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets",
        "route53:GetHostedZone"
      ],
      "Resource": [
        "arn:aws:route53:::hostedzone/Z151KI3YMRFBLY"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetChange"
      ],
      "Resource": [
        "arn:aws:route53:::change/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ListHostedZones"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Node Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets",
        "route53:GetHostedZone"
      ],
      "Resource": [
        "arn:aws:route53:::hostedzone/Z151KI3YMRFBLY"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetChange"
      ],
      "Resource": [
        "arn:aws:route53:::change/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ListHostedZones"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:BatchGetImage"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::aws.k8spro.com",
        "arn:aws:s3:::aws.k8spro.com/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::aws.k8spro.com"
      ]
    }
  ]
}

Admin installer policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AllocateAddress",
        "ec2:AssociateAddress",
        "ec2:AssociateDhcpOptions",
        "ec2:AssociateRouteTable",
        "ec2:AttachInternetGateway",
        "ec2:AttachVolume",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateDhcpOptions",
        "ec2:CreateInternetGateway",
        "ec2:CreateNatGateway",
        "ec2:CreateKeyPair",
        "ec2:CreateRoute",
        "ec2:CreateRouteTable",
        "ec2:CreateSecurityGroup",
        "ec2:CreateSubnet",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:CreateVpc",
        "ec2:DeleteDhcpOptions",
        "ec2:DeleteInternetGateway",
        "ec2:DeleteKeyPair",
        "ec2:DeleteNatGateway",
        "ec2:DeleteRoute",
        "ec2:DeleteRouteTable",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSubnet",
        "ec2:DeleteVolume",
        "ec2:DeleteVpc",
        "ec2:DeleteTags",
        "ec2:DescribeAddresses",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeDhcpOptions",
        "ec2:DescribeHosts",
        "ec2:*DescribeImage*",
        "ec2:*DescribeInstances",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeNatGateways",
        "ec2:DescribeRegions",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ec2:*DescribeVolume*",
        "ec2:DescribeVpcAttribute",
        "ec2:DescribeVpc",
        "ec2:DescribeVpcs",
        "ec2:DetachInternetGateway",
        "ec2:DetachVolume",
        "ec2:DisassociateRouteTable",
        "ec2:ImportKeyPair",
        "ec2:ModifyVpcAttribute",
        "ec2:ReleaseAddress",
        "ec2:ReplaceRoute",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:RunInstances",
        "ec2:StopInstances",
        "ec2:TerminateInstances"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:AddTags",
        "elasticloadbalancing:ApplySec*",
        "elasticloadbalancing:ConfigureHealthCheck",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:CreateLoadBalancerListeners",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
        "elasticloadbalancing:RemoveTags",
        "elasticloadbalancing:SetSecurityGroups"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:AttachInstances",
        "autoscaling:AttachLoadBalancers",
        "autoscaling:CreateAutoScalingGroup",
        "autoscaling:CreateLaunchConfiguration",
        "autoscaling:CreateOrUpdateTags",
        "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:DeleteLaunchConfiguration",
        "autoscaling:DeleteTags",
        "autoscaling:Describe*",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "autoscaling:UpdateAutoScalingGroup"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:AddRoleToInstanceProfile",
        "iam:CreateInstanceProfile",
        "iam:CreateRole",
        "iam:DeleteInstanceProfile",
        "iam:DeleteRole",
        "iam:DeleteRolePolicy",
        "iam:GetRole",
        "iam:GetRolePolicy",
        "iam:GetInstanceProfile",
        "iam:ListInstanceProfiles",
        "iam:ListRolePolicies",
        "iam:ListRoles",
        "iam:ListInstanceProfiles",
        "iam:PassRole",
        "iam:PutRolePolicy",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:UpdateAssumeRolePolicy"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey",
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ChangeResourceRecordSets",
        "route53:ListResourceRecordSets",
        "route53:GetHostedZone"
      ],
      "Resource": [
        "arn:aws:route53:::hostedzone/Z151KI3YMRFBLY"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetChange"
      ],
      "Resource": [
        "arn:aws:route53:::change/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "route53:ListHostedZones"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:CreateBucket",
        "s3:DeleteBucket",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:GetBucketLocation",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::aws.k8spro.com",
        "arn:aws:s3:::aws.k8spro.com/*"
      ]
    }
  ]
}

@faraazkhan and others ... comments?

[faraazkhan]

@chrislovecnm Understood, we have not tested with ECR/Autoscaling yet, but I'd imagine the master node will also require autoscaling:DescribeLaunchConfigurations why not allow autoscaling:Describe* to the master nodes as well. Same thing with ECR a typical read only policy for ECR includes the ecr:DescribeImages action. See http://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr_managed_policies.html

[chrislovecnm]

I am not seeing autoscaling:DescribeLaunchConfigurations in the autoscaling code, and I have not changed the ecr policies from what we have now. Let me double check, as we may be missing something in our current codebase.

[ajohnstone]

As discussed @chrislovecnm

1. kms should be locked down to the relevant resource (ideally use the cluster names resource).
2. route53 changesets not required for nodes only masters.
3. s3:* should be avoided and have explicit rules.
4. use sids so we can reference sections and indicate the statements use.
5. master missing elasticloadbalancing:AddTags + elasticloadbalancing:DescribeTags, elasticloadbalancing:RemoveTags

[chrislovecnm]

@ajohnstone awesome feedback. In regards to kms can you be more specific about the arn to use are a resource?

[jonhosmer]

Adding the additional policies worked like a charm, thank you! One thing to note though:

Upgrading an existing cluster will use the legacy IAM privileges to reduce risk of potential regression.

This does not appear to be the case. My cluster spec has legacy: true but is still updated with the new stricter IAM policies. Want me to make a separate issue for this?

[KashifSaadat]

A kops update .. will still show updates to the IAM roles, which may just be slight reordering and Statement IDs being added in. The permissions themselves should be mostly the same / identical.

If some permissions are being incorrectly dropped and are causing problems then yes please raise a new issue and feel free to assign me.

[chrislovecnm]

I am going to close this very long long long issue. We are code complete, and please file issues on any other IAM improvements/problems.

[michaelajr]

Hi. I moved to the new stricter IAM roles and had volume issues on the nodes (deployments that had to create volumes or needed to claim existing volumes did not work). Not sure if this was an oversight. Masters of course had everything they needed. For now will add the volume privileges to the additional policies in the cluster yaml. Maybe the ability to create and claim volumes should be considered as part of the "standard" cluster deployment?

[michaelajr]

Interesting. This seems to be a permission issue with the masters because of this condition on the AttachVolume action: ec2:ResourceTag/KubernetesCluster = MY_CLUSTER_NAME. I know that these strict policies are for cluster creation, but again, maybe the basic policies should allow volume attachment as it is a standard thing for K8s to do. Just a thought.

[chrislovecnm]

@michaelajr if I understand you correctly you are having pods on nodes attach volumes? First, you can accomplish this by adding an additional role to the nodes see https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md

I get what you mean, but aren’t volume attach and detach usually handled by the masters? Seems like this is a perm needed by a custom application and not k8s. Please let me know if I am incorrect.

[michaelajr]

@chrislovecnm Hey. So I initially thought the nodes needed privileges - that was incorrect. As you stated the masters attach volumes. The issue is probably not a big deal. But here it is. The PersistentVolume object has an option to specify a pre-existing EBS volume:

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: MyVol
spec:
  capacity:
    storage: 32Gi
  accessModes:
  - ReadWriteMany
  awsElasticBlockStore:
    volumeID: vol-xxxxxxxx
  storageClassName: my-volume

But because the AttachVolume IAM action that kops creates for the masters has a condition on the volume being tagged with KubernetesCluster: CLUSTER_NAME, it will not attach without the tag. It took me a while to realize the issue was because of the action's condition. So folks either have to tag their pre-existihng volumes with the cluster they intend to use it with, or override all the volume IAM actions in the additionalPolicies field in the cluster YAML to not have any conditions. Hope that makes sense.

[chrislovecnm]

But because the AttachVolume IAM action that kops creates for the masters has a condition on the volume being tagged with KubernetesCluster: CLUSTER_NAME

Oh, good catch! Hrmmm wondering if we should remove the tag restriction.

@KashifSaadat thoughts about loosening the policy so that EBS volumes do not have to have the cluster tag? @michaelajr has a valid use case where existing volumes may not have those tags.

[zytek]

Although this is a valid use case I'd opt for secure by default approach. Good documentations and error logs should help debug use cases such as @michaelajr - which is valid, but so is deploying apps that need tons of other IAM perms, for which we have kube2iam.

[michaelajr]

On one hand, I think this is just part of what the masters should do out of the box. Just like being able to enter Route53 records or launching instances. But on the other... I also like that it is locked down to JUST the volumes created by the cluster. I.e., if you do not BYOV, then the PersistentVolume is created with the tag and claimed just fine. So maybe the 80/20 rule is fine here. As long as it is well documented that you'll need to loosen the restrictions when bringing your own. I would NOT tell folks to tag their volumes as Kops might delete it when the cluster is deleted.

[KashifSaadat]

Both good points. I'm not keen on opening AttachVolume for any volume, as the cluster could be in a shared AWS Account with other resources and so there's risk that you interfere with volumes used for other deployments.

The "secure by default" approach should be sufficient and then users have at least the following options to support this extra use-case (which we should ensure is appropriately documented):

• Add an additional IAM policy within the Kops ClusterSpec to grant open access to AttachVolume call
• Tag their volume with KubernetesCluster so it is associated with the Cluster (should test whether this means the volume is deleted as part of kops delete cluster.. as you've mentioned, and include in docs as a warning)

[chrislovecnm]

Tag their volume with KubernetesCluster so it is associated with the Cluster (should test whether this means the volume is deleted as part of kops delete cluster.. as you've mentioned, and include in docs as a warning)

Yah don't use that tag.

Wondering if we should add another tag name that we can use to enable this? We can document the tag name, os that a user can utilize that tag, and not run into deletion issues.

[jordanjennings]

@chrislovecnm Yes I like that idea, roughly similar to the shared subnets tag approach (although kops sets the shared subnet tag itself, whereas here it wouldn't set a tag). Then kops can set up the permissions out of the box and users can add tags to opt-in for specific volumes to be accessible.

[chrislovecnm]

@jordanjennings actually maybe we can re-use the shared tags ...

[d-mcd]

K8s noob here;
Just ran into this issue today, during some testing of Docker Swarm to K8s migration. Found it was the Master IAM role being used, by decoding the authorization message, I saw via kubectl describe pod. Dug into the Master IAM role policy and saw the tag requirement. Would be nice to have the alternate tag option, so volumes can be attached by Master IAM role but are not deleted on cluster delete. Thanks everyone for the hard work you put in on this stuff! Kops v1.8.1 K8s v1.8.8

[bitfusion-brian]

@chrislovecnm This is way old now, but is there a way to specify your own IAM role that kops will use cluster wide?

Related to this comment about BYO IAM #1873 (comment)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[berniechiu]

Well, here's my limited create/destroy policies if anyone is concerned about the permission issue, it works fine in my cases since I also encountered the issue with security teams not being able to land me that much of the permissions to create the clusters on production AWS.

// Creation

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetLifecycleConfiguration",
                "s3:GetBucketTagging",
                "s3:GetBucketWebsite",
                "s3:GetBucketLogging",
                "s3:ListBucket",
                "s3:GetAccelerateConfiguration",
                "s3:GetBucketVersioning",
                "s3:GetReplicationConfiguration",
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:GetEncryptionConfiguration",
                "s3:PutBucketTagging",
                "s3:GetBucketRequestPayment",
                "s3:GetBucketCORS",
                "s3:GetObjectTagging",
                "s3:PutBucketAcl",
                "s3:PutObjectTagging",
                "s3:GetBucketLocation",
                "s3:PutObjectAcl",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::your_bucket/*",
                "arn:aws:s3:::your_bucket"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateDhcpOptions",
                "ec2:AuthorizeSecurityGroupIngress",
                "ec2:DescribeInstances",
                "ec2:CreateKeyPair",
                "route53:GetHostedZone",
                "iam:CreateRole",
                "ec2:AttachInternetGateway",
                "iam:PutRolePolicy",
                "iam:AddRoleToInstanceProfile",
                "ec2:AssociateRouteTable",
                "ec2:DescribeInternetGateways",
                "elasticloadbalancing:DescribeLoadBalancers",
                "ec2:CreateRoute",
                "ec2:CreateInternetGateway",
                "autoscaling:DescribeAutoScalingGroups",
                "ec2:DescribeVolumes",
                "route53:ListResourceRecordSets",
                "ec2:DescribeAccountAttributes",
                "autoscaling:UpdateAutoScalingGroup",
                "route53:UpdateHostedZoneComment",
                "ec2:DescribeKeyPairs",
                "elasticloadbalancing:DescribeInstanceHealth",
                "iam:ListRolePolicies",
                "ec2:DescribeNetworkAcls",
                "ec2:DescribeRouteTables",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                "iam:GetRole",
                "route53:CreateHostedZone",
                "ec2:ImportKeyPair",
                "ec2:DescribeVpcClassicLinkDnsSupport",
                "ec2:CreateTags",
                "ec2:DescribeReservedInstancesOfferings",
                "ec2:ModifyNetworkInterfaceAttribute",
                "autoscaling:DescribeTags",
                "ec2:CreateRouteTable",
                "route53:ChangeResourceRecordSets",
                "ec2:RunInstances",
                "ec2:DescribeVpcClassicLink",
                "ec2:CreateVolume",
                "elasticloadbalancing:DescribeLoadBalancerAttributes",
                "elasticloadbalancing:AddTags",
                "route53:ChangeTagsForResource",
                "ec2:CreateSubnet",
                "ec2:AssociateAddress",
                "ec2:DescribeSubnets",
                "elasticloadbalancing:ModifyLoadBalancerAttributes",
                "iam:GetRolePolicy",
                "autoscaling:CreateAutoScalingGroup",
                "iam:CreateInstanceProfile",
                "ec2:DescribeAddresses",
                "route53:GetChange",
                "ec2:CreateNatGateway",
                "ec2:DescribeInstanceAttribute",
                "elasticloadbalancing:ConfigureHealthCheck",
                "ec2:DescribeRegions",
                "autoscaling:DescribeLaunchConfigurations",
                "ec2:CreateVpc",
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeVpcAttribute",
                "ec2:ModifySubnetAttribute",
                "iam:ListInstanceProfilesForRole",
                "iam:PassRole",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeAvailabilityZones",
                "autoscaling:DescribeScalingActivities",
                "ec2:CreateSecurityGroup",
                "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
                "ec2:ModifyVpcAttribute",
                "ec2:ModifyInstanceAttribute",
                "autoscaling:AttachLoadBalancers",
                "ec2:AuthorizeSecurityGroupEgress",
                "ec2:AssociateDhcpOptions",
                "elasticloadbalancing:CreateLoadBalancer",
                "elasticloadbalancing:AttachLoadBalancerToSubnets",
                "autoscaling:EnableMetricsCollection",
                "iam:GetInstanceProfile",
                "ec2:DescribeTags",
                "elasticloadbalancing:DescribeTags",
                "route53:ListHostedZones",
                "iam:ListRoles",
                "ec2:DescribeNatGateways",
                "iam:ListInstanceProfiles",
                "route53:ListTagsForResource",
                "ec2:AllocateAddress",
                "ec2:DescribeSecurityGroups",
                "elasticloadbalancing:CreateLoadBalancerListeners",
                "ec2:DescribeImages",
                "autoscaling:CreateLaunchConfiguration",
                "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
                "ec2:DescribeVpcs",
                "iam:GetUser"
            ],
            "Resource": "*"
        }
    ]
}

// Destroy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:DeleteObjectTagging",
                "s3:DeleteObjectVersion",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::your_bucket/*",
                "arn:aws:s3:::your_bucket"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:DeleteSubnet",
                "ec2:ReplaceRouteTableAssociation",
                "iam:RemoveRoleFromInstanceProfile",
                "ec2:DeleteRouteTable",
                "elasticloadbalancing:DeleteLoadBalancer",
                "ec2:DeleteVolume",
                "ec2:RevokeSecurityGroupEgress",
                "iam:DeleteRolePolicy",
                "ec2:DeleteInternetGateway",
                "route53:DeleteHostedZone",
                "ec2:ReleaseAddress",
                "iam:DeleteInstanceProfile",
                "ec2:TerminateInstances",
                "ec2:DeleteRoute",
                "iam:DeleteRole",
                "ec2:DetachInternetGateway",
                "ec2:DisassociateRouteTable",
                "ec2:RevokeSecurityGroupIngress",
                "autoscaling:DeleteLaunchConfiguration",
                "ec2:DeleteSecurityGroup",
                "ec2:DeleteDhcpOptions",
                "ec2:DeleteNatGateway",
                "autoscaling:DeleteAutoScalingGroup",
                "ec2:DeleteVpc",
                "ec2:DeleteKeyPair"
            ],
            "Resource": "*" // Should be your cluster resources
        }
    ]
}

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[steven-friedman]

As I mentioned, my list was developed by verifying that I had no errors. My initial problems were with a fully open policy. I did this lock-down when I was making no progress.