Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ci): add trufflehog secrets detection #31344

Merged
merged 4 commits into from
Jun 12, 2024
Merged

Conversation

McPatate
Copy link
Member

@McPatate McPatate commented Jun 10, 2024

What does this PR do?

Adding a GH action to scan for leaked secrets on each commit.

Context

trufflehog will scan the commit that triggered the CI for any token leak. trufflehog works with a large number of what they call "detectors", each of which will read the text from the commit to see if there is match for a token. For example, the hugging face detector will check for hf tokens and then query our /api/whoami{-v2} endpoint to check if the token is valid. If it detects a valid token, the CI will fail, informing you that you need to rotate the token given it leaked.

What to do if the CI fails

If the CI errors, you should check the logs to see which token was leaked. After that, depending on the origin of the token, you should go to the token issuer and delete it, recreate it or rotate it (depending on the available features).
Once this is done, remove the token from the code and use an environment variable to load the secret or any other external configuration management system.

References

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding!

  • Would be good to have some guidelines added to let users know what to do if this pipeline fails e.g. in the contributing guidelines
  • What's the chance of having false positives on these runs?

@McPatate
Copy link
Member Author

Would be good to have some guidelines added to let users know what to do if this pipeline fails e.g. in the contributing guidelines

Would you like me to address this in this PR?

What's the chance of having false positives on these runs?

From my experience I would say pretty low, on some occasions a sha could be misinterpreted as a secret. But it will not be considered "verified", so in most cases safe to ignore.

@amyeroberts
Copy link
Collaborator

Would you like me to address this in this PR?

Ideally yes. At least, a comment within this PR which we can link on issues. As, tbh, I'm not 100% sure what I should tell people to do if this fails 😅

From my experience I would say pretty low, on some occasions a sha could be misinterpreted as a secret. But it will not be considered "verified", so in most cases safe to ignore.

OK!

@McPatate
Copy link
Member Author

At least, a comment within this PR which we can link on issues.

Updated the PR description, lmk what you think.

Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spice it up 😄

on:
push:

name: Secret Leaks
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: Secret Leaks
name: Are you leaking you github token

😈

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't do a suggestion on your suggestion 😄

Suggested change
name: Secret Leaks
name: Are you leaking a token?

Copy link
Collaborator

@amyeroberts amyeroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding! 🤐

@McPatate McPatate merged commit 348e229 into main Jun 12, 2024
22 checks passed
@McPatate McPatate deleted the feat/add_trufflehog_ci branch June 12, 2024 16:00
zucchini-nlp pushed a commit to zucchini-nlp/transformers that referenced this pull request Jun 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants