-
Notifications
You must be signed in to change notification settings - Fork 104
Keypair Rotation
This document describes what secret credentials we maintain and how to rotate them.
We currently have two Ansible Vaults: production and sandbox, each with their own vault password.
You'll need both the existing key and the new key. Adjust the parameters for the new secret and run ansible rekey
in https://github.com/GSA/data.gov. Don't forget to add the new key to Google Drive.
Vault re-keys should be done as a hotfix to minimize the chance of merge conflicts within vaults and so all branches (master
and develop
) aren't using multiple key versions.
For production, you'll want to rekey all the vault files for production, staging, and mgmt inventories (i.e. filter out the sandbox vault files).
$ vault_id=production
$ vault_password_file=.secrets/production-v$(date +%Y%m%d).txt
$ pwgen -s 64 1 > ${vault_password_file} # Generate a random 64-character string and put it in the key file.
$ find . -name vault.yml | grep -v 'sandbox\|local' | xargs pipenv run ansible-vault rekey --new-vault-id ${vault_id}@${vault_password_file}
$ pipenv run ansible-vault rekey --new-vault-id ${vault_id}@${vault_password_file} ansible/files/mgmt/saml-key.jks
To rekey the sandbox vault:
$ vault_id=sandbox
$ vault_password_file=.secrets/sandbox-v$(date +%Y%m%d).txt
$ pwgen -s 64 1 > ${vault_password_file} # Generate a random 64-character string and put it in the key file.
$ find . -name vault.yml | grep 'sandbox\|local' | xargs pipenv run ansible-vault rekey --new-vault-id ${vault_id}@${vault_password_file}
$ pipenv run ansible-vault rekey --new-vault-id ${vault_id}@${vault_password_file} ansible/files/sandbox/saml-key.jks
If you are using the ansible-vault textconv to see plaintext diffs, you should see an empty diff when you run git show --stat
. If you run git show --stat --no-textconv
, you should see the ciphertext changed.
Before merging, give a heads up to other developers. Any in-progress vault changes will need to be rebased in order to pick up the new key by the author.
Immediately after merging, all environments must be updated with the new secret.
For each environment, update the jumpbox with the new key. Add a new file for the key (e.g. /etc/datagov/sandbox-v20200709.txt
) with permissions root:operators 0640
. Update the symlink to point to the new key file.
$ ln -snf sandbox-v20200709.txt /etc/datagov/ansible-secret.txt
Validate the change with ansible and remove all old keys.
$ pipenv run ansible -m ping all
Finally, update the ansible-vault credential in Jenkins.
The root ssh keys allow access to the ubuntu
user. These keys are only used for the initial provision of the jumpbox. After that, operators use their personal keys to access all hosts.
-
Generate a new SSH key. When prompted, set a 16+ character password. For the comment, use the key name (datagov-environment-vX).
$ ssh-keygen -f ~/.ssh/datagov-environment-vX
-
Upload the private and public keys to the shared Drive. Share the password with teammates over a secure channel.
-
For BSP environments (production and staging), create a Service Now ticket to update the SSH key in the CloudFormation templates. Attach the public key to the ticket. BSP does not need the private key.
-
For AWS Sandbox environments, import the key in the AWS Console. Update the keypair name in datagov-infrastructure-live.