Skip to content

Commit

Permalink
Update multitenancy proposal
Browse files Browse the repository at this point in the history
  • Loading branch information
Naadir Jeewa committed Apr 15, 2021
1 parent 69cf50c commit 5746097
Showing 1 changed file with 56 additions and 56 deletions.
112 changes: 56 additions & 56 deletions docs/proposal/20200506-single-controller-multitenancy.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ superseded-by: []
- [Controller Changes](#controller-changes)
- [Clusterctl changes](#clusterctl-changes)
- [Validating webhook changes](#validating-webhook-changes)
- [Principal Type Credential Provider Behaviour](#principal-type-credential-provider-behaviour)
- [Identity Type Credential Provider Behaviour](#identity-type-credential-provider-behaviour)
- [Security Model](#security-model)
- [Roles](#roles)
- [RBAC](#rbac)
Expand All @@ -57,7 +57,7 @@ superseded-by: []
- [CAPA Controller Requirements](#capa-controller-requirements)
- [Alternative Approaches Considered](#alternative-approaches-considered)
- [Using only secrets or RoleARN field on AWSCluster](#using-only-secrets-or-rolearn-field-on-awscluster)
- [1:1 mapping one namespace to one AWSPrincipal](#11-mapping-one-namespace-to-one-awsprincipal)
- [1:1 mapping one namespace to one AWSIdentity](#11-mapping-one-namespace-to-one-awsidentity)
- [Risks and Mitigations](#risks-and-mitigations)
- [Network assumptions are made explicit](#network-assumptions-are-made-explicit)
- [Caching and handling refresh of credentials](#caching-and-handling-refresh-of-credentials)
Expand All @@ -74,7 +74,7 @@ superseded-by: []

## Glossary

* Principal Type - One of several ways to provide a form of identity that is ultimately resolved to an AWS access key ID,
* Identity Type - One of several ways to provide a form of identity that is ultimately resolved to an AWS access key ID,
secret access key and optional session token tuple.
* Credential Provider - An implementation of the interface specified in the [AWS SDK for
Go][aws-sdk-go-credential-provider].
Expand Down Expand Up @@ -108,8 +108,8 @@ the existing behavior with no changes to user configuration required.
For large organizations, especially highly-regulated organizations, there is a need to be able to perform separate
duties at various levels of infrastructure - permissions, networks and accounts. VPC sharing is a model which provides
separation at the AWS account level. Within this model it is appropriate for tooling running within the 'management' account
to manage infrastructure within the 'workload' accounts, which requires a principal in the management account which can
assume a principal within a workload account. For CAPA to be most useful within these organizations it will need to
to manage infrastructure within the 'workload' accounts, which requires a identity in the management account which can
assume a identity within a workload account. For CAPA to be most useful within these organizations it will need to
support multi-account models.

Some organizations may also delegate the management of clusters to another third-party. In that case, the boundary
Expand Down Expand Up @@ -155,7 +155,7 @@ clusters in dedicated AWS accounts.
The current configuration exists: AWS Account 'management':
* Vpc, subnets shared with 'workload' AWS Account
* EC2 instance profile linked to the IAM role 'ClusterAPI-Mgmt'
* A management Kubernetes cluster running Cluster API Provider AWS controllers using the 'ClusterAPI-Owner' IAM principal.
* A management Kubernetes cluster running Cluster API Provider AWS controllers using the 'ClusterAPI-Owner' IAM identity.

AWS Account 'workload':
* Vpc and subnets provided by 'Owner' AWS Account
Expand Down Expand Up @@ -197,9 +197,9 @@ a single instance of CAPA to cover multiple organisations.
<a name="FR4">FR4.</a> CAPA MUST prevent privilege escalation allowing users to create clusters in AWS accounts they should
not be able to.

<a name="FR5">FR5.</a> CAPA SHOULD support credential refreshing when principal data is modified.
<a name="FR5">FR5.</a> CAPA SHOULD support credential refreshing when identity data is modified.

<a name="FR6">FR6.</a> CAPA SHOULD provide validation for principal data submitted by users.
<a name="FR6">FR6.</a> CAPA SHOULD provide validation for identity data submitted by users.

<a name="FR7">FR7.</a> CAPA COULD support role assumption using OIDC projected volume service account tokens.

Expand Down Expand Up @@ -279,21 +279,21 @@ AWS accounts whilst preventing privilege escalation as per [FR4](#FR4). Reasons

<em>Namespace scoped resources</em>

* `AWSServiceAccountPrincipal` represents the use of a Kubernetes service account for access
* `AWSServiceAccountIdentity` represents the use of a Kubernetes service account for access
to AWS using federated identity.

<strong><em>Changes to AWSCluster</em></strong>

A new field is added to the `AWSClusterSpec` to reference a principal. We intend to use `corev1.LocalObjectReference` in
A new field is added to the `AWSClusterSpec` to reference a identity. We intend to use `corev1.LocalObjectReference` in
order to ensure that the only objects that can be references are either in the same namespace or are scoped to the
entire cluster.

```go
// AWSPrincipalKind defines allowed AWS principal types
type AWSPrincipalKind string
// AWSIdentityKind defines allowed AWS identity types
type AWSIdentityKind string

type AWSIdentityRef struct {
Kind AWSPrincipalKind `json:"kind"`
Kind AWSIdentityKind `json:"kind"`
Name string `json:"name"`
}

Expand Down Expand Up @@ -333,35 +333,35 @@ The `IdentityRef` field will be mutable in order to support `clusterctl move`
scenarios where a user instantiates a cluster on their laptop and then makes
the cluster self-managed.
<strong><em>Principal CRDs</em></strong>
<strong><em>Identity CRDs</em></strong>
<em>Common elements</em>
All AWSCluster*Principal types will have Spec structs with an `AllowedNamespaces`
All AWSCluster*Identity types will have Spec structs with an `AllowedNamespaces`
field as follows:
```go

type AWSClusterIdentitySpec struct {
// AllowedNamespaces is used to identify which namespaces are allowed to use the principal from.
// AllowedNamespaces is used to identify which namespaces are allowed to use the identity from.
// Namespaces can be selected either using an array of namespaces or with label selector.
// An empty allowedNamespaces object indicates that AWSClusters can use this principal from any namespace.
// An empty allowedNamespaces object indicates that AWSClusters can use this identity from any namespace.
// If this object is nil, no namespaces will be allowed (default behaviour, if this field is not provided)
// A namespace should be either in the NamespaceList or match with Selector to use the principal.
// A namespace should be either in the NamespaceList or match with Selector to use the identity.
//
// +optional
AllowedNamespaces *AllowedNamespaces `json:"allowedNamespaces"`
}

type AllowedNamespaces struct {
// An nil or empty list indicates that AWSClusters cannot use the principal from any namespace.
// An nil or empty list indicates that AWSClusters cannot use the identity from any namespace.
//
// +optional
// +nullable
NamespaceList []string `json:"list"`

// AllowedNamespaces is a selector of namespaces that AWSClusters can
// use this ClusterPrincipal from. This is a standard Kubernetes LabelSelector,
// use this ClusterIdentity from. This is a standard Kubernetes LabelSelector,
// a label query over a set of resources. The result of matchLabels and
// matchExpressions are ANDed.
//
Expand Down Expand Up @@ -542,8 +542,8 @@ type AWSClusterRoleIdentity struct {
Spec AWSClusterRoleIdentitySpec `json:"spec,omitempty""`
}

// AWSClusterIdentityKind defines allowed cluster-scoped AWS principal types
type AWSClusterIdentityKind AWSPrincipalKind
// AWSClusterIdentityKind defines allowed cluster-scoped AWS identity types
type AWSClusterIdentityKind AWSIdentityKind

type AWSClusterIdentityReference struct {
Kind AWSClusterIdentityKind `json:"kind"`
Expand All @@ -566,7 +566,7 @@ type AWSClusterRoleIdentitySpec struct {
// +kubebuilder:validation:Pattern:=[\w+=,.@:\/-]*
ExternalID *string `json:"externalID,omitempty"`

// SourceIdentityRef is a reference to another principal which will be chained to do
// SourceIdentityRef is a reference to another identity which will be chained to do
// role assumption.
SourceIdentityRef AWSClusterIdentityReference `json:"sourceIdentityRef,omitempty"`
}
Expand Down Expand Up @@ -596,7 +596,7 @@ metadata:
name: "test-account-role"
spec:
allowedNamespaces:
list: # allows only "test" namespace to use this principal
list: # allows only "test" namespace to use this identity
"test"
roleARN: "arn:aws:iam::123456789:role/CAPARole"
# Optional settings
Expand All @@ -611,23 +611,23 @@ spec:
name: test-account-creds
```
<em>Future implementation: AWSServiceAccountPrincipal</em>
<em>Future implementation: AWSServiceAccountIdentity</em>
This would not be implemented in the first instance, but opens the possibility to use Kubernetes service accounts
together with `STS::AssumeRoleWithWebIdentity`, supporting [FR7](#FR7).
Definition:
```go
type AWSServiceAccountPrincipal struct {
type AWSServiceAccountIdentity struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`

// Spec for this AWSServiceAccountPrincipal.
Spec AWSServiceAccountPrincipalSpec `json:"spec,omitempty""`
// Spec for this AWSServiceAccountIdentity.
Spec AWSServiceAccountIdentitySpec `json:"spec,omitempty""`
}

type AWSServiceAccountPrincipalSpec struct {
type AWSServiceAccountIdentitySpec struct {
AWSRoleSpec

// Audience is the intended audience of the token. A recipient of a token
Expand Down Expand Up @@ -659,7 +659,7 @@ The account owner would be expected to set up an appropriate IAM role with the f
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Identity": {
"Federated": "<Provider ARN for the management cluster OIDC configuration>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
Expand All @@ -686,11 +686,11 @@ metadata:
spec:
region: "eu-west-1"
identityRef:
kind: AWSServiceAccountPrincipal
kind: AWSServiceAccountIdentity
name: test-service-account
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSServiceAccountPrincipal
kind: AWSServiceAccountIdentity
metadata:
name: "test-service-account"
namespace: "test"
Expand All @@ -706,10 +706,10 @@ with the requisite parameters and receive a new JWT token to use with `STS::Assu
### Controller Changes
* If identityRef is specified, the CRD is fetched and unmarshalled into a AWS SDK credential.Provider for the principal type.
* If identityRef is specified, the CRD is fetched and unmarshalled into a AWS SDK credential.Provider for the identity type.
* The controller will compare the hash of the credential provider against the same secret’s provider in a cache ([NFR 8](#NFR8)).
* The controller will take the newer of the two and instantiate AWSClients with the selected credential provider.
* The controller will set an the principal resource as one of the OwnerReferences of the AWSCluster.
* The controller will set an the identity resource as one of the OwnerReferences of the AWSCluster.
* The controller and defaulting webhook will default `nil` `identityRef` field in AWSClusters to `AWSClusterControllerIdentity`.
This flow is shown below:
Expand All @@ -721,20 +721,20 @@ This flow is shown below:
Today, `clusterctl move` operates by tracking objectreferences within the same namespace, since we are now proposing to
use cluster-scoped resources, we will need to add requisite support to clusterctl's object graph to track ownerReferences
pointing at cluster-scoped resources, and ensure they are moved. We will naively not delete cluster-scoped resources
during a move, as they maybe referenced across namespaces. Because we will be tracking the AWS Account ID for a given principal, it is expected, that for this proposal, this provides sufficient protection against the possibility of a cluster-scoped
principal after one move, and being copied again.
during a move, as they maybe referenced across namespaces. Because we will be tracking the AWS Account ID for a given identity, it is expected, that for this proposal, this provides sufficient protection against the possibility of a cluster-scoped
identity after one move, and being copied again.
#### Validating webhook changes
A validating webhook could potentially handle some of the cross-resource validation necessary for the [security
model](#security-model) and provide more immediate feedback to end users. However, it would be imperfect. For example, a
change to a `AWSCluster*Principal` could affect the validity of corresponding AWSCluster.
change to a `AWSCluster*Identity` could affect the validity of corresponding AWSCluster.
The singleton `AWSClusterControllerIdentity` resource will be immutable to avoid any unwanted overrides to the allowed namespaces, especially during upgrading clusters.
#### Principal Type Credential Provider Behaviour
#### Identity Type Credential Provider Behaviour
Implementations for all principal types will implement the `credentials.Provider` interface in the AWS SDK to support [FR5](#FR5) as well as an
Implementations for all identity types will implement the `credentials.Provider` interface in the AWS SDK to support [FR5](#FR5) as well as an
additional function signature to support caching:
```go
Expand All @@ -748,10 +748,10 @@ type Provider interface {
IsExpired() bool
}

type AWSPrincipalTypeProvider interface {
type AWSIdentityTypeProvider interface {
credentials.Provider
// Hash returns a unique hash of the data forming the credentials
// for this principal
// for this identity
Hash() (string, error)
}
```
Expand All @@ -771,7 +771,7 @@ forcing a refresh.
The authors have implemented a similar mechanism [based on the Ruby AWS SDK][aws-assume-role].
This could be further optimised by having the controller maintain a watch on all Secrets matching the principal types.
This could be further optimised by having the controller maintain a watch on all Secrets matching the identity types.
Upon receiving an update event, the controller will update lookup the key in the cache and update the relevant provider.
This may be implemented as its own interface. Mutexes will ensure in-flight updates are completed prior to SDK calls are
made. This would require changes to RBAC, and maintaining a watch on secrets of a specific type will require further
Expand Down Expand Up @@ -805,11 +805,11 @@ defined above. In most cases, it will be desirable to have all resources be
readable by most roles, so instead we'll focus on write access for this model.
##### Write Permissions
| | AWSCluster*Principal | AWSServiceAccountPrincipal | AWS IAM API | Cluster |
| ---------------------------- | -------------------- | -------------------------- | ----------- | ------- |
| Infrastructure Provider | Yes | Yes | Yes | Yes |
| Management Cluster Operators | Yes | Yes | Yes | Yes |
| Workload Cluster Operator | No | Yes | No | Yes |
| | AWSCluster*Identity | AWSServiceAccountIdentity | AWS IAM API | Cluster |
| ---------------------------- | ------------------- | ------------------------- | ----------- | ------- |
| Infrastructure Provider | Yes | Yes | Yes | Yes |
| Management Cluster Operators | Yes | Yes | Yes | Yes |
| Workload Cluster Operator | No | Yes | No | Yes |
Because Kubernetes service accounts necessarily encode the namespace into the JWT subject, we can allow workload cluster
operators to create their own `AWSServiceAccountIdentities`. Whether they have actual permissions on AWS IAM to set up
Expand All @@ -821,11 +821,11 @@ The extra configuration options are not possible to control with RBAC. Instead,
they will be controlled with configuration fields on GatewayClasses:
* **allowedNamespaces**: This field is a selector of namespaces that
Gateways can use this `AWSCluster*Principal` from. This is a standard Kubernetes
Gateways can use this `AWSCluster*Identity` from. This is a standard Kubernetes
LabelSelector, a label query over a set of resources. The result of
matchLabels and matchExpressions are ANDed. CAPA will not support
AWSClusters in namespaces outside this selector. An empty selector (default)
indicates that AWSCluster can use this `AWSCluster*Principal` from any namespace. This
indicates that AWSCluster can use this `AWSCluster*Identity` from any namespace. This
field is intentionally not a pointer because the nil behavior (no namespaces)
is undesirable here.
Expand All @@ -834,10 +834,10 @@ they will be controlled with configuration fields on GatewayClasses:
The CAPA controller will need to:
* Populate condition fields on AWSClusters and indicate if it is
compatible with `AWS*Principal`.
* Not implement invalid configuration. Fore example, if a `AWSCluster*Principal` is referenced in
compatible with `AWS*Identity`.
* Not implement invalid configuration. Fore example, if a `AWSCluster*Identity` is referenced in
an invalid namespace for it, it should be ignored.
* Respond to changes in `AWS*Principal` configuration that may change.
* Respond to changes in `AWS*Identity` configuration that may change.
### Alternative Approaches Considered
Expand All @@ -862,13 +862,13 @@ API server KMS.
**Downsides**
By allowing workload cluster operators to create the various principals, or referencing them directly in the
By allowing workload cluster operators to create the various identitys, or referencing them directly in the
AWSCluster as a field they could potentially escalate privilege when assuming role across AWS accounts as the CAPA
controller itself may be running with privileged trust.
#### 1:1 mapping one namespace to one AWSPrincipal
#### 1:1 mapping one namespace to one AWSIdentity
The mapping of a singular AWSPrincipal to a single namespace such that there is a 1:1 mapping
The mapping of a singular AWSIdentity to a single namespace such that there is a 1:1 mapping
was considered, via either some implicitly named secret or other metadata on the namespace.
**Benefits**
Expand Down Expand Up @@ -923,7 +923,7 @@ The data changes are additive and optional, except `AWSClusterControllerIdentity
`AWSClusterControllerIdentity` singleton instance restricts the usage of controller credentials only from `allowedNamespaces`.
AWSClusters that do not have an assigned `IdentityRef` is defaulted to use `AWSClusterControllerIdentity`, hence existing clusters needs to have
`AWSClusterControllerIdentity` instance. In order to make existing AWSClusters to continue to reconcile as before, a new controller is added as experimental feature
and gated with **Feature gate:** AutoControllerPrincipalCreator=true. By default, this feature is enabled. This controller creates `AWSClusterControllerIdentity` singleton instance (if missing) that allows all namespaces to use the principal.
and gated with **Feature gate:** AutoControllerIdentityCreator=true. By default, this feature is enabled. This controller creates `AWSClusterControllerIdentity` singleton instance (if missing) that allows all namespaces to use the identity.
During v1alpha4 releases, since breaking changes will be allowed, this feature will become obsolete.
## Additional Details
Expand Down

0 comments on commit 5746097

Please sign in to comment.