Feature request: Option to use existing IAM Role in AWS

[egeland]

Use case

Possibly relating to kube2iam ( #921 ) and other AWS IAM Role related tickets (like #379 , #223 ), it may be a useful thing to be able to have a pre-configured IAM Role (with the trust relationships required by kube2iam already defined) and give this to kops to use, rather than kops generating its own, and then needing to add to this and set up trust relationships with potentially hundreds of kube2iam "instance roles".

This way, one could spin up new clusters with kops, and have kube2iam work immediately.

It would probably require that the pre-configured Role either be manually set up with the access kops needs, or that kops edits it as needed.

Something like --worker-iam-role=$ROLENAMEHERE and --master-iam-role=$OTHERROLEHERE ?

[chrislovecnm]

@kris-nova I think this is going to hit out radar pretty quick.

[chrislovecnm]

@egeland do you have it documented the requirement for master and nodes in IAM? This would be a huge help.

Also which role are we using to run kops? We have an initiative to move kops functionality into a http API server, and I am wondering if masters should have the same functionality that kops requires.

[chrislovecnm]

Oh and btw thanks, this is a big one for security!!

[egeland]

Caveat: I've only tried kube2iam in a cluster I spun up for testing it, so my experience is limited - it does seem to solve the IAM role problem very well, though.

The reason I created this feature request, is that in a large environment, you might have to set up dozens of roles, for dozens of app deployments. If you wanted to spin down the kops-created cluster and spin up a new one, you'd need to go through these roles and add the Trust relationship to each (see kube2iam documentation), which would be tedious to say the least.
If one could set up the Role first, and have any and all kops-created clusters use it, that would save a lot of effort.

I hope that makes some sort of sense?

[tiadobatima]

@sp-borja-juncosa submitted #2139 addressing this. Would love to see this merged.

[chrislovecnm]

We are waiting on a couple of changes and we will merge. We really need tested IAM roles ;)

[chrislovecnm]

We are waiting on a couple of changes and we will merge. We really need tested IAM roles ;)

Started some work on this:

Installer notes
https://gist.github.com/chrislovecnm/c4adac1316bd0a3b9bba4e5d5c79e856
Master notes
https://gist.github.com/chrislovecnm/4f5de81adcc0e37d290198190e981ed4
Node notes
https://gist.github.com/chrislovecnm/c6cff95c0d13d14d7f9c3cb362bc01e1

I am thinking we add a kops feature that would allow an admin to create the proper roles. Thoughts?

[Miyurz]

This makes sense to me. Currently, there is no way in kops to use an existing IAM role for master and nodes. Since kube2IAM came too late in our project, we are now struggling to find a way to assign required IAM roles to the pods. I think kops could provide a way to consume exiting role like kube2IAM or create it on the fly(may be) as we spin the master and nodes?

[chrislovecnm]

Copied from a PR @justinsb

Sorry for the delay on this one. I've been thinking about it, and I think the issue is that we need to make sure that we don't create an unmaintainable product. At the same time, we want to let users customize everything.

I think the way we keep this maintainable is that we have to solve the problem of "how do I use a custom IAM role". So this is half of it, in that we can use a custom IAM role, but we should document and probably create a command for how that role should be created. We previously had kops update cluster --models which let you specify that only certain subsets of the configuration should run; I think we want something like that (although maybe we call it phases or stages).

This has a few advantages:

we can validate that the configuration is correct with dryrun
we can output to terraform, cloudformation, or any other systems we support in future
docs will likely fall out of date. If need be we can create a --target=human to produce docs or something like that :-)
Again I apologize for the delay - I've been trying to formulate this, and it's taken a while and some discussions for it to crystallize... I do think this is a great feature, we just have to make sure that we don't add a feature and then all progress stops because it is too hard to use.

For the record, I think we probably need to do the same thing for networking, because that also causes a lot of support time, because it is hard :-)

[vendrov]

I would like to suggest solution. let's make the IAM role customizable in the IG level instead of the cluster level (like it's now), and in addition we will automatically attach the "kops basic policy" to all the roles, since kops uses only "allow" permissions I see no reason why we can't attach them to the custom role. thus we can assure that all the IAM roles will work properly with kops and in the same time IG-nodes that needs the extra permission will have them

[chrislovecnm]

My challenge is that kops pemissions are too open. Does your solution help at all with this?

[egeland]

Looks like #2440 will be merged fairly soon... 👍

[chrislovecnm]

We are working on getting this in ;)

/assign

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[kenden]

PR #4171 is still being worked on
/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[egeland]

/reopen

[egeland]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[egeland]

/remove-lifecycle stale

[andrelohmann]

Hi, I need to use kops within an enterprise environment (what a surprise) and handing over the permission (especially full permissions) to create and manage IAM Roles is not an option (IAM roles are explicitly only managed by an operations team, handing over the AWS OUs to the specific DevOps Teams). So what I need is to create the IAM Permissions/Roles upfront and hand over the explicit ARNs to the kops script. Since I'm a beginner with kops I haven't found any solution to my problem so far, that why I want to ask the simple question here, if this kind of functionality is allready implemented?

[rifelpet]

Hi @andrelohmann, it is indeed possible to have IAM roles be created externally and provide their ARNs to kops. The documentation walks you through that process.

Closing because this has been implemented
/close

[k8s-ci-robot]

@rifelpet: Closing this issue.

In response to this:

Hi @andrelohmann, it is indeed possible to have IAM roles be created externally and provide their ARNs to kops. The documentation walks you through that process.

Closing because this has been implemented
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andrelohmann]

Hi there,

I was also trying to use kops without the necessity to use IAMFullAccess and just link to previously (by an administrator) created roles.

Sadly the Documentation https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md#use-existing-aws-instance-profiles is unclear about how to setup a greenfield cluster with kops, without IAMFullAccess for the kops user. It just showcases how to change the IAMRoles later on, when using kops update. The suggested parameters for lifecycle-overrides to not work on kops create. Instead it is failing on

W0114 10:57:37.317828 23282 executor.go:130] error running task "IAMRole/masters....

Any Suggestions or Help? Ist it might be an alternative to create a terraform target and then "manually" remove the aws_iam_role resources and just link them to the previously created resources?

kind regards

[andrelohmann]

@rifelpet , as you suggested the documented way, may I ask you, if I do anything wrong, or if I have an error in my thinking?

[ludwigprager]

@andrelhmann The suggested way doesn't work. But I found a workaround. Basically: edit the cluster config in the bucket and two more steps. It took me quite a while to figure that out.
I finally got a working cluster with a user that doesn't need IamFullAccess but only a few minor IAM permissions.
Shall I document that in more detail?

[kb18951452]

@ludwigprager Yes, please document that. Thank you very much.

[anoopwebs]

We still don't have an option to specify instanceProfile while creating the cluster?
I got the following error when I add existing profile arn in node.spec.iam.profile
error creating IAMInstanceProfileRole: LimitExceeded: Cannot exceed quota for InstanceSessionsPerInstanceProfile: 1