CI/CD - Error when running 01-resman GitHub workflow (tf-config, PERMISSION_DENIED)

[mgfeller]

When running the GitHub workflow for 01-resman (fast-resman), the step "Copy Terraform output files" fails with

Run gcloud alpha storage cp -r \
ERROR: (gcloud.alpha.storage.cp) There was a problem refreshing your current auth tokens: ('Unable to acquire impersonated credentials: No access token or invalid expiration in response.', '{\n  "error": {\n    "code": 403,\n    "message": "The caller does not have permission",\n    "status": "PERMISSION_DENIED"\n  }\n}\n')
Please run:
  $ gcloud auth login
to obtain new credentials.
If you have already logged in with a different account:
    $ gcloud config set account ACCOUNT
to select an already authenticated account to use.
Error: Process completed with exit code 1.

The code is based on v16.0.0.

The 00-bootstrap workflow works fine.

Unfortunately, I cannot figure out what is wrong.

Is it just my setup that might be causing this?

[ludoo]

Hey Michael, there were some bugs in the CI/CD implementation for stage 1, we sent a PR to fix them last Friday. Can you try patching with that version?

#682

[mgfeller]

Thanks for the fast feedback.I'll give it a try as soon as possible.

[mgfeller]

I've updated my fast-modules, fast-bootstrap, and fast-resman to daily-2022.06.17 (hopefully correctly).
Running the fast-bootstrap action on github still works, but for fast-resman it still fails with the same error.

I was trying to compare the configuration of the service accounts prefix-prod-bootstrap-1 and prefix-prod-resman-1, but couldn't spot differences in the places I looked. Any hints?

[ludoo]

did you also update the workflow? one the errors we had was using the wrong SA in stage1+ CI/CD

[mgfeller]

Hmm, I cannot see any changes in the workflow. That would be resman-workflow.yaml generated in 00-bootstrap, right?

Shouldn't prefix-prod-resman-1@prefix-prod-iac-core-0.iam.gserviceaccount.com be the SA?

name: "FAST resman stage"

on:
  pull_request:
    branches:
      - main
    types:
      - closed
      - opened
      - synchronize

env:
  FAST_OUTPUTS_BUCKET: prefix-prod-iac-core-outputs-0
  FAST_SERVICE_ACCOUNT: prefix-prod-resman-1@prefix-prod-iac-core-0.iam.gserviceaccount.com
  FAST_WIF_PROVIDER: projects/254628875009/locations/global/workloadIdentityPools/prefix-bootstrap/providers/prefix-bootstrap-provider
  SSH_AUTH_SOCK: /tmp/ssh_agent.sock
  TF_PROVIDERS_FILE: 01-resman-providers.tf
  TF_VAR_FILES: 00-bootstrap.auto.tfvars.json
    globals.auto.tfvars.json
  TF_VERSION: 1.1.7

[ludoo]

the SA is the right one prefix-prod-resman-1, I can try and reproduce on my side when I have 30m later today

[mgfeller]

Thanks!

[ludoo]

I just tested it and it's working fine for me:

I had to do a small patch to add an extra attribute to the token mapping to support my condition, I will push it in a few minutes but I don't think it matters for you.

Can you paste here the tfvars for

• federated_identity_providers
• cicd_repositories

My supsicionis your workload identity federation provider is not working for both repos.

[ludoo]

Actually wait, I was still using an older version, going to try again and report back.

[ludoo]

I can confirm it works.

[mgfeller]

Thanks for your feedback and for testing this!
I changed the branch name for resman from main to null, but it still didn't work.
CI/CD for bootstrap works.
I'll try to check that once more once I'm back at the office with a proper PC and bandwidth, in a couple of days.

federated_identity_providers = {
 github-mgfdev = {
  attribute_condition = "attribute.repository_owner==\"mgfeller-dev\""
  issuer              = "github"
 }
}

cicd_repositories = {
 bootstrap = {
  branch            = null
  identity_provider = "github-mgfdev"
  name              = "mgfeller-dev/fast-bootstrap"
  type              = "github"
 }
 resman = {
  branch            = null
  identity_provider = "github-mgfdev"
  name              = "mgfeller-dev/fast-resman"
  type              = "github"
 }
}

[ludoo]

Yours looks ok, mine is here for reference (with the extra repo and attributes we recently added):

cicd_repositories = {
  bootstrap = {
    branch            = "main"
    identity_provider = "gitlab-ludomagno"
    name              = "ludomagno/fast-test"
    type              = "gitlab"
  }
  cicd = null
  resman = {
    branch            = null
    identity_provider = "github-ludomagno"
    name              = "ludomagno/fast-0-01-resman"
    type              = "github"
  }
}
federated_identity_providers = {
  github-ludomagno = {
    attribute_condition = "attribute.repository_owner==\"ludomagno\""
    issuer              = "github"
    custom_settings     = null
  }
  gitlab-ludomagno = {
    attribute_condition = "attribute.namespace_path==\"ludomagno\""
    issuer              = "gitlab"
    custom_settings     = null
  }
}

I would try aligning everything to our main branch, then applying bootstrap, and fetching the generated workflow from scratch.

[mgfeller]

Thanks, I'll do that, it might take a couple of days though.

It is quite a lot of work to update the repositories. It could be useful to have a section with recommendations about that as well as the versioning approach used in the docs - I might have overlooked it though. I'll create an issue for that.

Do you consider the daily releases production ready?

[ludoo]

It is quite a lot of work to update the repositories. It could be useful to have a section with recommendations about that as well as the versioning approach used in the docs - I might have overlooked it though. I'll create an issue for that.

Yes, we don't have a good solution yet for updating the repos. If you have ideas, let's discuss them. And +1 on improving our documentation.

Do you consider the daily releases production ready?

I would, but I also would expect the code to be maintained and evolved independently once cloned off master. This is how most of us work with our customers and how this entire repo is designed: we clone master and start integrating local customizations whenever we start a new project.

Then sometimes new features we create to meet specific requirements are generalized and backported here, when it makes sense to have them in FAST or modules. The opposite (integrating new FAST features in existing customer code) is rarer and requires more work, and it typically happens at the module and not stage level.

[mgfeller]

I have updated to daily-2022.06.24 and now it works 👍 What exactly did the trick I didn't have time to investigate. The workflow was the same.

Thanks for your help, @ludoo!