Welcome to the GSSAR Product! đź‘‹
- Overview
- How this works
- Design
- Pre-Reqs
- Initial Installation
- Configuring Remediators
- Issues and Feedback
- Contributing
- FAQs
GSSAR is an open-source initiative helping teams automatically revoke secrets discovered by GitHub's Secret Scanning tool.
Right now, whenever a new secret is discovered, you are notified straight away within the Secret Scanning tab. You can then make a conscious decision over whether to revoke the secret or mark it as a false positive, etc.
With some secrets, it's okay not to take immediate action and only revoke after the right people have reviewed. However, for some other types of secrets (such as AWS Access Keys), more of an immediate (and automated) approach may be required in revoking these secrets. GSSAR is an initiative to solve that use case. 🚨
GitHub Secret Scanner Auto Remediator (GSSAR) is an initiative that revokes certain secret types automatically. GSSAR takes an agnostic approach to the kinds of secrets and leaves it up to each GitHub organization to decide what secret types should and should not be automatically revoked.
Whenever a secret is discovered within a repository, a webhook is triggered, and a process kicks off. That process starts by collecting information about that secret. It then looks within the webhook to see what type of secret it has found (e.g. AWS, Azure, GCP, Dropbox, etc.). Then, if the secret type is one you have a remediator for, the secret will automatically be revoked. You will then be notified via a GitHub Issue on the repository where the secret has been found. If the secret type is one you do not have a remediator for, the process will finish.
Whenever a new secret alert is opened
, resolved
or reopened
, a webhook will fire from a GitHub App and be sent to an API Gateway within AWS. The API's first step is passing the context of the payload to a Lambda Authorizer, ensuring the Webhook has come from GitHub. If valid, the API will pass the event payload (from the webhook) to a Lambda to confirm the secret within the webhook is valid and expected. After both authentication methods pass, an AWS State Machine is kicked off to revoke the secret defined within the alert.
The state machine firstly goes and filters out the resolved
and reopened
events. The state machine will automatically finish for any secret alert action that != created
. If the action type does === created
, it kicks off the first lambda, which fetches the value of the secret found. The secret itself doesn't come as part of the webhook payload, so we have to make an API call to get the value.
Once the value has been collected, we then kick off a Choice. This choice will look at the alert.secret_type
and, based on the value, will decide to pass it onto a remediator or exit the state machine if the secret type isn't defined within the state machine.
If the secret within the alert.secret_type
matches a secret type within the state machine, it will send the payload to a remediator. A remediator is a function that revokes a secret, custom to a type of secret. Once revoked, it will send the result back to the state machine, then mark that secret as closed within GitHub. Finally, once all steps are complete, it will run a final step which notifies people the secret has been remediated. It informs by opening an issue on the repository where the secret was leaked.
The design has been implemented in a way to be very plug and play. Where you, as an organization, only have to focus on building remediators. The architecture can be found below:
This solution is designed in a way where the remediators are the only part you need to add/remove.
What are remediators? As mentioned above, remediators are functions that revoke specific secret types. There should be a 1:1 mapping between secret type and remediator.
For example, if a company wanted to remediate the following secret types automatically:
- Dropbox Access Token
- Amazon AWS Access Key ID
- Google Cloud Private Key ID
There would need to be three remediators. One for Dropbox Access Tokens, one for Amazon AWS Access Keys and one for Google Cloud Private Keys.
The rest of the state machine (outside of remediators) is secret type agnostic. Meaning it will work for any current and future secret types. It is worth understanding how the rest of the design works, but you do not directly need to edit ot change anything outside of the remediators.
The following technologies are used throughout this solution:
- AWS
- Lambda is used for compute power.
- Cloud Formation is used as our IaC (Infrastructure as Code).
- HTTP API Gateway is used for ingress into AWS.
- Cloud Watch is used for logging and monitoring.
- IAM is used to connect resources and allow deployments into AWS from GitHub Actions
- S3 is used by AWS SAM to deploy the stack, and therefore deploy it into the AWS ecosystem using Cloud Formation.
- AWS Systems Manager Parameter Store is used to store parameters.
- AWS Step Functions is used to co-ordinate the end-to-end process
- GitHub
- GitHub App is used as our egress method out of GitHub.
- GitHub Actions is used to deploy the solution into AWS.
AWS SAM is used for the lambda & HTTP API Gateway resources.
Note: Even though this solution is deployed to AWS, the code can be changed to work with the likes of Azure and GCP (Azure Function, Google Functions, etc.).
- Access to a cloud environment (AWS would be the quickest to get started)
- Access to a GitHub environment.
- A repository where the code for this solution is going to live.
The below steps show the path of least resistance way of deploying this solution into AWS. There are many different ways to deploy this. Every organization likely has different processes (especially with deploying into AWS), meaning you may have to pivot during these steps to accommodate organization-specific processes. This is okay. Please treat these instructions as an example and reference; if they work end-to-end, great; if not, please adjust to your company policies (and if needed, contribute back!).
If you get an error you cannot get around, please log an issue on this repository.
Create an IAM User. The IAM User will need to have the capability to do the following:
- CRUD access over S3 Resources.
- CRUD access over IAM Resources.
- CRUD access over API Gateway Resources.
- CRUD acess over Lambda Resources.
- CRUD access over CloudWatch Resources.
- CRUD access over Step Functions
From that user, create an AWS Access key and secret. Once you have both, create a GitHub Enviroment called main and within that environment create two secrets AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
with the relevant information from AWS in. Set the environment to only deploy from the main
branch. (This can be changed later at any time).
NOTE: If your organization doesn't allow the use of IAM Users, this isn't a problem. We use the official configure-aws-credentials GitHub action. Meaning you can head to the .github/workflows/deploy.yaml
file and swap out the AWS User method to assuming an AWS Role. Or, if you have a custom GitHub Action which authenticates into AWS, remove the configure-AWS-credentials
action and swap it out for your custom one.
Create a GitHub Application. You will need to be an administrator of your GitHub organization to do this. During the creation of the application, you only need to enter:
- GitHub App Name: GSSAR
- Homepage URL: https://donotknowthisurlyet.com
- Webhook URL: https://donotknowthisurlyet.com
- Webhook Secret: enter secret of your choice - keep this value secret but note it down for later
- Permissions:
- Secret scanning alerts
- Issues
- Subscribe to events:
- Secret scanning alert
- Where can this integration be installed: Only on this account
The rest of the fields you do not need to enter. Right now, you don't know what the URL's are going to be, so put any value in there.
Once the application is created, you need to install the GitHub App on your organization and then add the repositories you would like secrets to be auto-remediated. Follow the instructions here: Installing your private GitHub App on your repository.
NOTE: When you install the GitHub App on your GitHub Organisation, I would advise you do not have it connected to every repository to start with. To get familiar with the process, only install on a few repositories and once comfortable, you can install across the organization if you like.
Once it's installed, we need to collect some information:
- GitHub App Private Key. Follow the instructions here: Generating a private key to do that.
- Client Secret: Just above where you generated the private key, there will be an option for you to generate a client secret. Click the Generate a new Client Secret button and note down the secret.
- Client ID: Just above where you generated the client secret, you will see the Client ID; take a note of the id.
- App ID: Just above where you generated the client secret, you will see the App ID; take a note of the id.
- Installation ID: The Installation ID is in a different location; head to your Organizations GitHub App's page (https://github.com/organizations/${orgName}/settings/installations). Click Configure next to the GitHub App you created. If you look at the URL, at the end of the URL, you will see a number. It should be after the
installations/
part of the URL. Copy down that number.
Log into AWS, head to AWS Systems Manager, then AWS Parameter Store. In total, you will need to create seven parameters.
/GSSAR/APP_CLIENT_ID
: The GitHub App Client ID you got from Step Three./GSSAR/APP_CLIENT_SECRET
: The GitHub App Client Secret you got from Step Three./GSSAR/APP_ID
: The GitHub App ID you got from Step Three./GSSAR/APP_INSTALLATION_ID
: The GitHub App Installation ID you got from Step Three./GSSAR/APP_PRIVATE_KEY
: The GitHub App Private Key you got from Step Three. (The first part when you created the GitHub App)/GSSAR/GITHUB_WEBHOOKS_SECRET
: The secret you assigned to the webhook.
NOTE: It is recommended you make the: /GSSAR/APP_CLIENT_SECRET
, /GSSAR/APP_PRIVATE_KEY
, /GSSAR/GITHUB_WEBHOOKS_SECRET
values SecureString
within Parameter Store. The rest can be String
types.
Second to last step! Before we do this, let's check a few things:
- An environment is created with two GitHub Secrets in which can deploy the solution to AWS.
- A GitHub app is created, connected to the repositories where you would like to auto remediate secrets of a certain type
- AWS Parameters have been created.
If the above is complete, pull the contents of this codebase and push it into the repository where you configured the GitHub Environment and Secrets. Make sure you push to the main branch (or the branch you configured in the environment to deploy from).
GitHub Actions should now trigger! You can watch the workflow within the Actions tab of your repository, but what it is doing is:
- Linting
- Building (Typescript -> Javascript)
- Building (SAM)
- Deploying (SAM)
The first time you deploy, it should take about 5-6 minutes. As long as the role you created in Step One has the correct permissions mentioned above, your deployment should succeed. Log into AWS, head to Cloud Formation, look for the GSSAR
stack, head to outputs, and you should see an output called: HttpApiUrl
. Note down this URL.
Head back to the GitHub App you created in Step Four. Head down to the Webhook URL, enter the URL from Step Five and add /gssar
onto the end of the URI. The URL you got from the output is the domain, but not the full URI where webhooks should be sent. So make sure to put the /gssar
endpoint onto that URL.
Click Save
Done! From now on, whenever a Secret Scanning Alert gets: created
, fixed
and closed_by_user
, an event will be fired to be processed.
Once you have installed GSSAR, you now have to configure your required remediators. There is an example remediator found within the functions/remediators
directory. This remediator (as it is named) revokes AWS Access Keys. You are welcome to leave that remediator in if you would like to revoke AWS Access Keys in your environment, but you can also remove it.
The following steps need to be made when creating a new remediator:
- Create a new directory within the
functions/remediators
directory (name it something logical). - Within the folder you created, but the logic of the remediator. You can write it in whatever language your cloud provider allows. For AWS Lambda, you can build your remediator in any of these languages: Runtimes.
- Update the
.github/workflow/deploy.yaml
to include your remediator directory. (You can copy and paste a previous example and change the DIR name). - Update the
template.yml
. You will need to:- Create a new
AWS::Serverless::Function
. You are welcome to copy and paste a previous example (e.g.CloseSecret
). Update theAWS::Serverless::StateMachine
. Within thePolicies
, add your function to a newLambdaInvokePolicy
and in theDefinitionSubstitutions
create a new substitution for your lambda.
- Create a new
- Update the
stepfunctions/secretscanner.yml
. You will need to:- Create a new Task. To do this, copy and paste the
Remediate AWS Access Key ID
task. Name the task something suitable (e.g. Remediate XYZ). Update theResource
field within the task with the name you assigned in thetemplate.yml
(e.g.AWS::Serverless::StateMachine
->DefinitionSubstitutions
section). - Add or update a new choice within the
Find Secret Type To Remediate
section. You can copy and paste the"aws_access_key_id"
example. In the new choice, leave theVariable
the same. Update theStringEquals
field to the type of secret, and theNext
field the name of the task you created in the previous step.
- Create a new Task. To do this, copy and paste the
You can look at the code and copy and paste from previous examples.
Important: Within your new remediator, the event
object received by your lambda handler will be structured as the following:
{
"number": "number", // The Secret Scanner Alert Number
"name": "string", // The name of the GitHub Repository where the secret scanner alert was found.
"login": "string", // The name of the GitHub Organisation where the secret scanner alert was found.
"secret": "string", // The value of the Secret found.
"secret_type": "string", // The type of Secret found.
"html_url": "string", // The URL to the specific alert.
"created_at": "string" // When the GitHub Secret was discovered.
}
Within your remediator, use the provided values to revoke the secret.
On the success of remediation, return the same payload to the state machine that you received. This structure is required downstream within the state machine.
throw an Error and the state machine will fail accordingly.
If you find any issues, or have any feedback regarding this initative, please open an issue on this repository.
Please open an issue and let's discuss your thoughts on what can be made better. Then feel free to raise a pull request on this repository. This initative welcomes feedback from the community!
Not a problem. The reason why AWS was chosen is due to the market popularity. However, we understand that not every company has AWS. The codebase will require some reconfiguration to meet whatever requirements your cloud/hosting provider has. The codebase structure can stay the same; you will likely have to change the template.yml
, for example. However, a large part of the code within the functions/*
directory can stay the same.
I would advise if you don't use AWS. Use this codebase as a reference. It is a great template to copy and paste snippets from and put into your solution.
Again, not a problem. Take a look at the .github/workflow/deploy.yaml
and translate that to whatever CI engine you are using. You shouldn't need to make any changes to the actual codebase, just the workflow file.