Represent all GH org secrets in pulumi

[soapy1]

This PR represents all the conda-forge github org secrets in pulumi. Now, to update any github secret the process is:

• find the secret in the pulumi 1password vault
• update it to the desired value
• run the push-secrets workflow

[beckermr]

Before I double check each entry in detail, is there a way with pulumi to only run commands against one of the entries in the file, or do we have to do the whole thing?

[soapy1]

It looks like you can use the --target flag to run against a particular resource.

If it's helpful, there are also all ci these runs that apply only 1-3 resources at a time (I batched changes so that I could verify things were not breaking as secrets got swapped out).

[beckermr]

Awesome. A question for the group, would all of this be clearer if we used separate files per secret?

cc @jaimergp

[soapy1]

would all of this be clearer if we used separate files per secret?

Ya, I think that's could be nice. Pulumi.yaml can have all the repos and providers defined, and then each secret can have a file like

# azure-token.yaml
variables:
  azure-token:
    fn::invoke:
      function: onepassword:getItem
      arguments:
        title: azure-token
        vault: pulumi
resources:
  gh-org-secret-azure-token:
    type: github:ActionsOrganizationSecret
    options:
      protect: false
      retainOnDelete: true
      deleteBeforeReplace: false
    properties:
      secretName: AZURE_TOKEN
      plaintextValue: ${azure-token.credential}
      visibility: selected
      selectedRepositoryIds:
        - ${repo-admin-requests.repoId}
        - ${repo-admin-migrations.repoId}

[beckermr]

That looks good to me. Let's go for it and make sure to put instructions for the updated scheme in the readme.

[soapy1]

Ok so, I was briefly looking into splitting out the pulumi config into different files. But it looks like there is no good, built-in way to do this. Other users have used templating/yaml merging tools to achieve this end.

I think that's ok, but maybe a little clunky. Before spending the time to get that working, re-gauging interest in separating out secrets out into their own files.

ref:
https://archive.pulumi.com/t/13396200/is-there-anyway-to-split-pulumi-lt-stack-gt-yaml-to-multiple

[beckermr]

That seems clunky. Is there a reason we need one single stack as opposed to separate stacks?

[beckermr]

Yeah I was thinking about this and let's just use one file for now. We can figure out if something better can be done later.

[soapy1]

Is there a reason we need one single stack as opposed to separate stacks?

Could do, though I would lean towards not adopting a stack per secret. I think we get a lot of operational overhead as we adopt more stacks. Mostly around ensuring stack configuration stays consistent, and ensuring users access. And, adding secrets becomes more of a process.

Each stack has its own separate configuration and secrets, role-based access controls (RBAC) and policies, and concurrent deployments.
ref: https://www.pulumi.com/docs/iac/packages-and-automation/organizing-projects-stacks/

If we adopt a stack per secret, I think the workflow for adding a new secret would be something like:

1. Create a new stack in pulumi pulumi stack init conda-forge/sync-secrets/my-new-secret (user must have access to the pulumi org)
2. Add a new directory for the secret eg sync-secrets/my-new-secret
3. Add the secret to sync-secrets/my-new-secret/Pulumi.yaml (would be similar to as outlined above)
4. Add a step to the github workflow push-1password-secrets-to-gha.yaml to run the new new automation against the new stack

Using something like the automation api could automate most of these steps away. But, yaml is not one of it's supported languages.