Add Default Backingstore for AWS STS with CCO

[shirady]

Explain the changes

1. Edit the volumeMount and volume to the operator deployment, endpoint deployment, and statefulset noobaa-core so they can assume the role with web identity.
2. Add annotation to the CSV to claim support for STS
3. Get role ARN and web identity token path to the credentials request for the CCO (cloud credential operator).
   • Get the role ARN from the environment variable set on the pod by the subscription config and set the webIdentityTokenPath as const.
   • Add the role ARN and web identity token path to the credentials request and apply it during operator initialization.
4. Add the option aws-sts-arn to pass the role ARN (for testing).
5. In file operator yaml we had the filed volumeMounts twice, so I removed the second time it appeared and moved the key-values that we had there using the convention name and then mountPath.

Issues: Fixed #xxx / Gap #xxx

1. none

Testing Instructions:

1. Deploy noobaa on AWS STS cluster and make sure that the backinstore type is aws-s3 with awsSTSRoleARN value in Phase Ready (more details for developers in the comment below).

• Doc added/updated
• Tests added

[shirady]

Detailed Testing Instructions:

Using AWS cluster with cluster-bot

Those are informal testing instructions until we have an AWS STS cluster.

1. In cluster-bot launch 4.14 aws.
2. Code changes in the operator repository:

• In the file pkg/system/phase2_creating.go: Add this line: r.IsAWSSTSCluster = true as the first line in ReconcileAWSCredentials.
  Since we use an AWS cluster (and not AWS STS cluster) we add this condition to fail the creation of the default backingstore on the AWS cluster.
• In the file pkg/system/phase4_configuring.go: And change minutesToWaitForDefaultBSCreation from 10 to 180.
  Since we have a fallback to pv-pool, we wish to avoid it (and the system will stuck in phase 'configuring' on purpose).

3. Code changes in the core repository:

• In the file: src/util/cloud_utils.js Edit the path of projectedServiceAccountToken to /var/run/secrets/openshift/serviceaccount/token
  We changed it according to internal instructions in the field path under serviceAccountToken from oidc-token to token.

4. Install noobaa with the images with the changes above using: noobaa install --dev --noobaa-image <core-image-you-build> --operator-image <operator-image-you-build> -n <your-namespace> --aws-sts-arn <role-ARN>.

• Role ARN structure: arn:aws:iam::<account-id>:role/<role-name>. If you haven't create the role yet, still provide the details: can be found using aws sts get-caller-identity --query "Account" --output text, will be the name of the role for example <your-name>-<current-month>-<current-day>.
• The images should be for linux/amd64 platform.
  It would stuck in phase configuring and the default backingstore would not be created.

5. Follow the instructions steps-to-in-place-migrate-an-openshift-cluster-to-sts

• At step 7 restart all pods - before running it make sure you already have the noobaa pods (operator, db, endpoint).
• In the end of the instructions there is a Cleanup AWS resources after uninstalling the cluster, please use it when finishing working with the cluster (notice that the role was not created by the ccoctl if you wish to remove it as well).

8. Remove the credential request and secret that were created for AWS platform:
   kubectl delete credentialsrequest noobaa-aws-cloud-creds -n <your-namespace>
kubectl delete secret noobaa-aws-cloud-creds-secret -n <your-namespace>
   They were created for the AWS cluster, in AWS STS case we use different arguments.
9. Default backingstore type should be aws-s3 with Role ARN in spec:
   kubectl get backingstore noobaa-default-backing-store -n test1 -o json | grep "awsSTSRoleARN"

Additional instructions for MacOS users:

In the instruction there is a use of a tool ccoctl, currently it only have a Linux binary.
The workaround I used is:

1. Clone the CCO project locally: git clone https://github.com/openshift/cloud-credential-operator.git.
2. Define the env: ENV GO_PACKAGE http://github.com/openshift/cloud-credential-operator.
3. In the cloud-credential-operator dir run:
   go build -ldflags "-X $GO_PACKAGE/pkg/version.versionFromGit=0.0.1" ./cmd/cloud-credential-operator
go build -ldflags "-X $GO_PACKAGE/pkg/version.versionFromGit=0.0.1" ./cmd/ccoctl
4. Then you can see the binary ccoctl.
5. Change the permissions of the binary: chmod 775 ./ccoctl.
6. To make sure you can use it run: ./ccoctl aws --help.

Images for cluster-bot:

I have MacOS, for using the cluster-bot I need the operator image for linux/amd64.

Noobaa Operator repository:

1. Build the image: DOCKER_DEFAULT_PLATFORM=linux/amd64 GOARCH=amd64 make all.
2. Tag it to your Quay account and push the image:
   docker tag noobaa/noobaa-operator:5.15.0 quay.io/<my-user>/noobaa-operator:<my-tag>
docker push quay.io/<my-user>/noobaa-operator:<my-tag>

Noobaa Core repository:

1. Build the image: CONTAINER_PLATFORM=linux/amd64 make noobaa NOOBAA_TAG=noobaa-core:<your-tag>
2. Tag it to your Quay account and push the image:
   docker tag noobaa/noobaa-core:<your-tag>.0 quay.io<my-user>/noobaa-core:<my-tag>
docker push quay.io<my-user>/noobaa-core:<my-tag>