[EPIC]: Blob-CSI implementation & Boathouse replacement

[blairdrummond]

Based on PoC work here

https://github.com/blairdrummond/kind-blob-csi-mock-aaw

Architecture

The architecture has a nice split:

• Create or Find Storage Containers, Create PVs (readwrite or readonly), and create PVCs in the user namespace

• Automatically discover PVCs and bind them to Notebooks, Workflows, or S3Proxy pods. (Goofys Webhook)

Refreshes/Updates

• Users will need to restart their notebooks to trigger refresh of mounts (in the situation where an FDI bucket gets added, for example).

• Recommend that S3Proxy comes with a CronJob that rolls out a restart on an interval. (Similar to the AAD Pod refresher.)

TODO: What happens if the storage account credentials get rotated?

BlobCSI controller ✔️

The BlobCSI Profile Controller loops through profiles and AAW Storage accounts:

• Creates a container (bucket) in each storage account if it doesn't exist.
• Creates a PersistentVolume binding to it using the Azure Blob CSI Driver. The driver authenticates using a secret within the azure-blob-csi-system
• Creates a PersistentVolumeClaim binding to the PersistentVolume in the profile's namespace.[^A Gatekeeper policy is in place to ensure that no other PVCs are allowed to bind to this PV]

FDI Submodule ❌

Use OPA sdk to check if a user has access to a given bucket, and determine what permissions they have. (We can only implement RW/RO)

Gatekeeper Policies ✔️

The PersistentVolumes are created with a profile label, matching the users profile. A Gatekeeper policy ensure that pvc.metadata.namespace == pv.metadata.labels.profile. So that users cannot bind other users volumes. This is resolved using a claimRef on the PV.

Also, the classification of the PV and PVCs must match. (Still need to check this.)
Alternatively, prevent users from creating these PVCs themselves.

Goofys Blob CSI Injector ✔️

The Goofys injector is repurposed:

• Instead of using a fixed list of mounts, it uses the blob.aaw.statcan.gc.ca/automount label to select volumes to mount.
• It further differentiates between unclassified and protected-b PVCs, only mounting to the correct pods.

Tasks

PoC

• [Epic Blob-CSI] Test out blob-csi implementation viability #1014

Setup

• [Epic Blob-CSI] Install the Blob-CSI-Driver helm chart #1002
• [Epic Blob-CSI] Fix ~~Blob-CSI access via Azure RBAC~~ Azure Subnet Issue (Need to add AKS node pool subnet to allow list for storage accounts) #1033
• [Epic Blob-CSI] Provision storage account secrets in dedicated namespace (azure-blob-csi-system) #1003
• [Epic Blob-CSI] Configure namespace network policies #1004
• [Epic Blob-CSI] Remove or greatly increase the storage cap on namespaces #1005
• [Epic Blob-CSI] Add s3proxy & nginx to contianer-allowed-images #1020
• [Epic Blob-CSI] Implement the gatekeeper policies for PVs v.s. PVCs #1006
• [Epic Blob-CSI] Create Gatekeeper Constraint for PV/PVC classification #1016
• [Epic Blob-CSI] Create a new profile controller (Blob,PV,PVC) using code written in the PoC #1007
• [Epic Blob-CSI] Add new blob-csi component to profile-controller helm chart #1022
• [Epic Blob-CSI] Deploy the new mutating webhook #1008
• [Epic Blob-CSI] Add all updates to aaw-argocd-manifests #1024
• [Blob csi] Fix mount options #1067

Testing

• Blob-CSI: Deploy example notebook that utilizes the mounts #1009
• Blob-CSI: Change container configuration in profiles controller to be ADLS for AAW containers #1275
• [Epic Blob-CSI] Load test and speed test the tool #1010

Switch-over/Teardown

• Blob-CSI: RMI - Fill out SOS Document #1315
• Blob-CSI: RMI - Fill out ConOps #1316
• Blob-CSI: RMI - Create ticket in NETA Heat System #1317
• Remove MinIO Credential Injector #1064
• Remove profile-configurator role from Vault, and all MinIO mounts/roles #1063
• [Epic Blob-CSI] UX fix: remove blob PVCs from the notebook spawner UI #1088

S3Proxy

• [Epic Blob-CSI] Create non-root gaul/s3proxy image (or see if current can work) #1021
• Blob-CSI: Investigate local S3proxy deployment and verify it works #1244
• Blob-CSI: Add deployment to aaw-argocd-manifests repository #1267
• Blob-CSI: Write an S3Proxy controller (with automated restarts) #1037
• Blob-CSI: Implement UI replacement #1023
• Blob-CSI: Add i18n #1062
• Epic Blob-CSI: Integrate into central dashboard #1268

FDI integration

• [Epic Blob-CSI] Configure PVs with FDI credentials #1019
• [Epic Blob-CSI] Mock the FDI OPA setup #1083
• [Epic Blob-CSI] Query the OPA instance via controller to get buckets. Add this to controller. #1084
• Blob-CSI: Deploy FDI Integration to Dev #1125
• [Epic Blob-CSI] Minio in Dev not working with FDI #1130
• Blob-CSI: Setup firewall for protected-b in dev #1139
• Blob-CSI: Setup Infra in Terraform  #1186
• Blob-CSI: Update helm chart in charts repo #1187
• Blob-CSI: Update profiles controller in aaw-argocd-manifests repo #1188
• Blob-CSI: Manually test FDI firewall #1252

Production Deployment

• Blob-CSI: Verify that storage quotas is per storage class and not global
• Blob-CSI: Investigate container configuration in profiles controller for AAW containers #1273
• [Epic Blob-CSI] Finalize FDI firewall rules #1201
• Blob-CSI: Update kubecost to more accurately aggregate storage cost #1269
• Prod: Configure Prot-b Firewall Rules for FDI Integration #1218
• Blob-CSI: Update argocd-manifests deployment for profiles-controller in production #1219
• Prod: Configure terraform with secrets for the profiles-controller #1220
• Prod: Update and confirm that the AAD refresher is working with the FDI prod deployment #1221
• Prod: Setup netpols for connection between profiles-controller and opa-gateways #1222
• Prod: Deploy blob-csi-driver #1233
• Prod: Deploy blob-csi-injector #1234
• Blob-CSI: PV size should be 10 TB when deploying to prod #1260
• Replace standard and premium with unclassified volume and remove standard storage account #1309

RMI

• Blob-CSI: RMI - Fill out SOS Document #1315
• Blob-CSI: RMI - Fill out ConOps #1316
• Blob-CSI: RMI - Create ticket in NETA Heat System #1317

Docs

• Profiles Controller: documentation for debugging the controller #1118
• Blob-CSI: Architecture diagram for blob-csi indicating which components are involved with this feature #1113
• Add information pertinent to blob-csi feature aaw-security-proposal#10
• Update the docs under /en, /fr, and /docs in here to include:
  • Blob-CSI: Update S3Proxy information in daaas docs #1194
  • Blob-CSI: Update Blob-CSI information in daaas docs #1195

UX

• https://github.com/StatCan/jupyter-apis Remove Kubeflow UI menu options that allow users to mount PVCs to their notebook (this happens transparently now with the new Blob CSI implementation) e.g. https://github.com/StatCan/jupyter-apis/blob/main/persistentvolumeclaims.go

[Collinbrown95]

IMPORTANT

• Investigate methods for backing up or enabling 'soft deleting' prior to doing anything
• DO NOT DELETE THESE https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L5-25
• DO NOT DELETE THIS https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L56-72
• Make Azure object replication policy doing a back-up of the entire prod / dev storage accounts while we do this migration - then delete once migration is successfully complete.

Deprecating old profiles controller

• when we're deprecating MinIO, we should also do the migration of the old profiles controller - there will only be a few components left, so we can use this opportunity to migrate the various functionality over to aaw-kubeflow-profiles-controller.

Roadmap to Deprecation

• Archive minio-credential-injector repo

• Remove minio-credential-injector argo-cd-manifest

• Remove the boathouse MinIO ingresses

• Archive goofys injector

• Delete deployment for goofys injector in argocd manfiests

• Archive boathouse repo

• Delete the entire terraform config file for boathouse (dev and prod)

• Delete goofys-injector terraform https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/goofys_injector.tf

• Delete everything in this range https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/istio_clusterrbacconfig.tf#L28-36 - do not delete fdi-gateway exclusion yet b/c OPA gateway is in this namespace and needs to communicate with other namespaces.

• Delete the read_instances and oidc_instance (https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L26-54)

• Delete these https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/minio_gateway.tf#L74-200

• Delete helm chart for boathouse https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/tree/main/charts/boathouse

• Add rbac components in kubeflow-controller to the rbac.go file in the kubeflow-profiles-controller repo.

• Archive kubeflow controller repo https://github.com/StatCan/aaw-kubeflow-controller

• Remove the kubeflow controller deployment from argocd manifests https://github.com/StatCan/aaw-argocd-manifests/tree/aaw-dev-cc-00/daaas-system/profile-controllers/kubeflow-controller

• Flatten directory structure of profiles controller here https://github.com/StatCan/aaw-argocd-manifests/tree/aaw-dev-cc-00/daaas-system/profile-controllers (no need for two deployments anymore)

• Delete minio.go from https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/minio.go

• Delete all minio-related network policies https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/network.go

• Delete deployment-buckets https://github.com/StatCan/charts/blob/master/stable/profiles-controller/templates/deployment-buckets.yaml from charts repo

• Remove any network.go references to vault in aaw-kubeflow-profiles-controller

• Upgrade chart https://github.com/StatCan/charts/blob/master/stable/profiles-controller/Chart.yaml (uptick chart version)

• Update values.yaml file (delete any references to buckets or vault agent)

• Delete vault minio config map https://github.com/StatCan/charts/blob/master/stable/profiles-controller/templates/configmap-vaultagent.yaml

• https://github.com/StatCan/aaw-kubeflow-profiles-controller/blob/main/cmd/notebook.go#L109-L131 delete mounting minio to fs

• https://github.com/StatCan/aaw-argocd-manifests/blob/aaw-dev-cc-00/daaas-system/profile-controllers/profiles-controller/application.jsonnet This no longer needs to be parameterized as jsonnet (no need for vault/minio) - can just make this a plain manifest.yaml file. Delete anything unused and update the helm chart

• Delete entire folder storage-system https://github.com/StatCan/aaw-argocd-manifests/tree/aaw-dev-cc-00/storage-system -

• Delete goofys injector network policies https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/goofys-injector-system.yaml (note: none of the network policies are deployed by themselves; need to edit the kustomize file in https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/environments/aaw-dev-cc-00/kustomization.yaml for dev and prod)

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/boathouse-system.yaml

• Delete network policies for minio-credential-injector and goofys-injector https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/daaas-system.yaml#L139-L169

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/minio-legacy-system.yaml

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/minio-premium-legacy-system.yaml

• Delete https://github.com/StatCan/aaw-network-policies/blob/aaw-dev-cc-00/vault-system.yaml#L18-L54

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/secret_minio_standard_tenant_1.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/secret_minio_premium_tenant_1.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_profile_configurator.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_minio.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_goofys_injector.tf

• Delete https://github.com/StatCan/terraform-aaw-vault/blob/master/policy_boathouse.tf

• https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/terraform-aaw-dev-cc-00-vault Delete policy_boathouse.tf and policy_profile_configurator.tf, do additional pass through and delete any old references

• TODO: https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/terraform-aaw-prod-cc-00-vault get vault token so that we can swtich entirely to the gitlab version of this repo and archive the prod github version

• Delete https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/tree/main/modules/minio-gateway minio gateway terraform module

• Delete minio_gateway unclassified and protected-b in here https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/fdi_gateway.tf

• Delete https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/goofys_injector.tf

• Any variables https://gitlab.k8s.cloud.statcan.ca/cloudnative/aaw/daaas-infrastructure/aaw-dev-cc-00/-/blob/main/variables.tf that relate to minio

• Unregister MinIO from OIDC app regsitry

• Update DAaaS docs end user documentation for blob csi

  • Remove old references to MinIO that are no longer valid
  • Add user-facing explanation of new Blob-CSI driver + S3Proxy approach
  • Add Blob CSI documentation to the daaas/docs/dev folder