Skip to content

Commit

Permalink
Merge pull request #877 from stealthybox/integrations-registry-creden…
Browse files Browse the repository at this point in the history
…tials-sync

Add ACR auth to Image Updates examples
  • Loading branch information
stefanprodan authored Feb 8, 2021
2 parents afffdfb + e99b1c3 commit ec2a834
Show file tree
Hide file tree
Showing 33 changed files with 1,034 additions and 2 deletions.
87 changes: 85 additions & 2 deletions docs/guides/image-update.md
Original file line number Diff line number Diff line change
Expand Up @@ -657,7 +657,7 @@ spec:

#### Using access token [short-lived]

!!!note "Workload Identity"
!!! note "Workload Identity"
Please ensure that you enable workload identity for your cluster, create a GCP service account that has
access to the container registry and create an IAM policy binding between the GCP service account and
the Kubernetes service account so that the pods created by the cronjob can access GCP APIs and get the token.
Expand Down Expand Up @@ -792,4 +792,87 @@ or [Sealed Secrets](sealed-secrets.md) , commit and push the encypted file to gi
### Azure Container Registry
TODO
AKS clusters are not able to pull and run images from ACR by default.
Read [Integrating AKS /w ACR](https://docs.microsoft.com/en-us/azure/aks/cluster-container-registry-integration) as a potential pre-requisite
before integrating Flux `ImageRepositories` with ACR.
Note that the resulting ImagePullSecret for Flux could also be specified by Pods within the same Namespace to pull and run ACR images as well.
#### Generating Tokens for Managed Identities [short-lived]
With [AAD Pod-Identity](https://azure.github.io/aad-pod-identity/docs/), we can create Pods that have their own
cloud credentials for accessing Azure services like ACR.
Your cluster should have `--enable-managed-identity` configured.
This software can be [installed via Helm](https://azure.github.io/aad-pod-identity/docs/getting-started/installation/) not managed by Azure.
Use Flux's `HelmRepository` and `HelmRelease` object to manage the aad-pod-identity installation from a bootstrap repository.
!!! As an alternative to Helm, the `--enable-aad-pod-identity` flag for the `az aks create` is currently in Preview.
Follow the Azure guide for [Creating an AKS cluster with AAD Pod Identity](https://docs.microsoft.com/en-us/azure/aks/use-azure-ad-pod-identity) if you would like to enable this feature with the Azure CLI.
Once we have AAD Pod Identity installed, we can create a Deployment that frequently refreshes an image pull secret into
our desired Namespace.
Create a directory in your control repository and save this `kustomization.yaml`:
```yaml
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/fluxcd/flux2/archive/main.zip//manifests/integrations/registry-credentials-sync/azure
patchesStrategicMerge:
- config-patches.yaml
```
Save and configure the following patch -- note the instructional comments for configuring matching Azure resources:
```yaml
# config-patches.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: credentials-sync
data:
ACR_NAME: my-registry
KUBE_SECRET: my-registry # does not yet exist -- will be created in the same Namespace
SYNC_PERIOD: "3600" # ACR tokens expire every 3 hours; refresh faster than that

# Create an identity in Azure and assign it a role to pull from ACR (note: the identity's resourceGroup should match the desired ACR):
# az identity create -n acr-sync
# az role assignment create --role AcrPull --assignee-object-id "$(az identity show -n acr-sync -o tsv --query principalId)"
# Fetch the clientID and resourceID to configure the AzureIdentity spec below:
# az identity show -n acr-sync -otsv --query clientId
# az identity show -n acr-sync -otsv --query resourceId
---
apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentity
metadata:
name: credentials-sync # name must match the stub-resource in az-identity.yaml
namespace: flux-system
spec:
clientID: 4ceaa448-d7b9-4a80-8f32-497eaf3d3287
resourceID: /subscriptions/8c69185e-55f9-4d00-8e71-a1b1bb1386a1/resourcegroups/stealthybox/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-sync
type: 0 # user-managed identity
```
Verify that `kustomize build .` works, then commit the directory to you control repo.
Flux will apply the Deployment and it will use the AAD managed identity for that Pod to regularly fetch ACR tokens into your configured `KUBE_SECRET` name.
Reference the `KUBE_SECRET` value from any `ImageRepository` objects for that ACR registry.

This example uses the `fluxcd/flux2` github archive as a remote base, but you may copy the [./manifests/integrations/registry-credentials-sync/azure](github.com/fluxcd/flux2/tree/main/manifests/integrations/registry-credentials-sync/azure)
folder into your own repository or use a git submodule to vendor it if preferred.

#### Using Static Credentials [long-lived]

!!! Using a static credential requires a Secrets management solution compatible with your GitOps workflow.

Follow the official Azure documentation for [Creating an Image Pull Secret for ACR](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-kubernetes).

Instead of creating the Secret directly into your Kubernetes cluster, encrypt it using [Mozilla SOPS](mozilla-sops.md)
or [Sealed Secrets](sealed-secrets.md), then commit and push the encypted file to git.

This Secret should be in the same Namespace as your flux `ImageRepository` object.
Update the `ImageRepository.spec.secretRef` to point to it.

It is also possible to create [Repository Scoped Tokens](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-repository-scoped-permissions).

!!! Note that this feature is in preview and does have limitations.
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
app: credentials-sync

resources:
- sync.yaml

vars:
- name: KUBE_SECRET
objref:
kind: ConfigMap
name: credentials-sync
apiVersion: v1
fieldref:
fieldpath: data.KUBE_SECRET

configurations:
- kustomizeconfig.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
varReference:
- path: rules/resourceNames
kind: Role
125 changes: 125 additions & 0 deletions manifests/integrations/registry-credentials-sync/_base/sync.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: credentials-sync
data:
# Patch this ConfigMap with additional values needed for your cloud
KUBE_SECRET: my-registry-token # does not yet exist -- will be created in the same Namespace
SYNC_PERIOD: "3600" # tokens expire; refresh faster than that

---
# This Deployment frequently fetches registry tokens and applies them as an imagePullSecret.
# It's done as a 1-replica Deployment rather than a CronJob, because CronJob scheduling can
# block cluster bootstraps and cold-reboots from obtaining registry tokens for a considerable time.
# This deployment will immediately fetch a token, which reduces latency for working image updates.
apiVersion: apps/v1
kind: Deployment
metadata:
name: credentials-sync
namespace: flux-system
spec:
replicas: 1
strategy:
type: Recreate
template:
spec:
serviceAccount: credentials-sync
containers:
- image: busybox # override this with a cloud-specific image
name: sync
envFrom:
- configMapRef:
name: credentials-sync
env:
- name: RECONCILE_SH # override this env var with a shell function in a kustomize patch
value: |-
reconcile() {
echo reconciling...
}
command:
- bash
- -ceu
- |-
# template reconcile() into the script
# env var is expanded by k8s before the pod starts
$(RECONCILE_SH)
apply-secret() {
/kbin/kubectl create secret docker-registry "${1}" \
--docker-server="${2}" \
--docker-username="${3}" \
--docker-password="${4}" \
--dry-run=client -o=yaml \
| grep -v "creationTimestamp:" \
| /kbin/kubectl apply -f -
}
pause_loop() {
sleep "${SYNC_PERIOD:-3600}" || true
}
graceful_exit() {
echo "Trapped signal -- $(date)"
job_ids="$(
jobs \
| grep "pause_loop" \
| cut -d] -f1 \
| tr [ %
)"
# shellcheck disable=SC2086
if [ "${job_ids}" ]; then
kill ${job_ids}
fi
wait
echo "Graceful exit -- $(date)"
}
trap graceful_exit INT TERM
echo "Loop started (period: ${SYNC_PERIOD} s) -- $(date)"
while true; do
reconcile & wait $!
pause_loop & wait $!
done
resources: {}


# RBAC necessary for our Deployment to apply our imagePullSecret
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: credentials-sync
namespace: flux-system
rules:
- apiGroups: [""]
resources:
- secrets
verbs:
- get
- create
- update
- patch
# # Lock this down to the specific Secret name (Optional)
resourceNames:
- $(KUBE_SECRET) # templated from kustomize vars referencing ConfigMap, also see kustomizeconfig.yaml
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: credentials-sync
namespace: flux-system
subjects:
- kind: ServiceAccount
name: credentials-sync
roleRef:
kind: Role
name: credentials-sync
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: credentials-sync
namespace: flux-system
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

commonLabels:
app: credentials-sync

resources:
- sync.yaml

vars:
- name: KUBE_SECRET
objref:
kind: ConfigMap
name: credentials-sync
apiVersion: v1
fieldref:
fieldpath: data.KUBE_SECRET

configurations:
- kustomizeconfig.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
varReference:
- path: rules/resourceNames
kind: Role
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
---
apiVersion: v1
kind: ConfigMap
metadata:
name: credentials-sync
data:
# Patch this ConfigMap with additional values needed for your cloud
KUBE_SECRET: my-registry-token # does not yet exist -- will be created in the same Namespace

---
# This CronJob frequently fetches registry tokens and applies them as an imagePullSecret.
# note: CronJob scheduling can block cluster bootstraps and cold-reboots from obtaining registry tokens for a considerable time.
# To run the job immediately, do `kubectl create job --from=cronjob/credentials-sync -n flux-system credentials-sync-init`
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: credentials-sync
namespace: flux-system
spec:
suspend: false
schedule: 0 */6 * * *
failedJobsHistoryLimit: 1
successfulJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
serviceAccountName: credentials-sync
restartPolicy: Never
containers:
- image: busybox # override this with a cloud-specific image
name: sync
envFrom:
- configMapRef:
name: credentials-sync
env:
- name: RECONCILE_SH # override this env var with a shell function in a kustomize patch
value: |-
reconcile() {
echo reconciling...
}
command:
- bash
- -ceu
- |-
# template reconcile() into the script
# env var is expanded by k8s before the pod starts
$(RECONCILE_SH)
apply-secret() {
/kbin/kubectl create secret docker-registry "${1}" \
--docker-server="${2}" \
--docker-username="${3}" \
--docker-password="${4}" \
--dry-run=client -o=yaml \
| grep -v "creationTimestamp:" \
| /kbin/kubectl apply -f -
}
reconcile
resources: {}


# RBAC necessary for our Deployment to apply our imagePullSecret
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: credentials-sync
namespace: flux-system
rules:
- apiGroups: [""]
resources:
- secrets
verbs:
- get
- create
- update
- patch
# # Lock this down to the specific Secret name (Optional)
resourceNames:
- $(KUBE_SECRET) # templated from kustomize vars referencing ConfigMap, also see kustomizeconfig.yaml
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: credentials-sync
namespace: flux-system
subjects:
- kind: ServiceAccount
name: credentials-sync
roleRef:
kind: Role
name: credentials-sync
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: credentials-sync
namespace: flux-system
Loading

0 comments on commit ec2a834

Please sign in to comment.