Enhancement request: OAuth with Github to scan repos for Access Keys

[mikeatlas]

This is even something I'd consider building myself / adding to the project. Even secure keys on AWS can be insecure and exposed on a public github repo (even in deleted commit histories). Any interest?

[mikeatlas]

There's also the potential to actively scan https://api.github.com/events while running (or as part of a premium service) for leaked keys.

[matthewdfuller]

This is a really awesome idea. I'm trying to think through some logistics, but I imagine by using OAuth, we could just scan the user's public repos (or even private for a less severe warning) for known keys from their accounts. Hitting /events may be a bit resource intensive (although I have seen scripts out there that do something similar).

[mikeatlas]

It's actually real-world, not pointing fingers but I've indirectly been bit by this problem twice now (users pushing their .bashrc or .zshrc profiles and config settings to their public repos).

[mikeatlas]

Well, even hitting private repos (organizations can have 100s of repos, 1000s of files, 1000s of commits, plus 100s of users with public non-org repos multiplied by N commits...) with all git history commits to scan.... it's definitely intensive even at a small scale.

[mikeatlas]

Quick search by hand even yielded a key:
https://github.com/yukiyasunasu/pinroom/blob/25316c25e041661533734ef4b3432e5841b831e5/.env

[matthewdfuller]

Agreed, it's well beyond the scale of the existing scans, so I'm going to have to think on this one in order to fit it into the setup of the existing plugins. Perhaps GitHub offers a search API which could be hit directly on behalf of a user and include history, deleted files, etc. There was a hacker news thread once on GitHub searches for locating all sorts of things (SSH keys, AWS access keys, etc.). Will try to find it.

[mikeatlas]

I recall that HN posting as well but can't find that repo quickly either. https://github.com/michenriksen/gitrob maybe?

[mikeatlas]

yeah I think this is it it... gluing these two projects together would be magical - the output of potential sensitive keys from cloudsploit-scan is compared with output of gitrob scan.

[mikeatlas]

Played around with gitrob internally. It's a nice tool but the author has not accepted PRs or bugfixes in nearly 7-8 months. Consider abandonware, I'd recommend you fork and salvage useful bits for iterating Github repos and source code and ditch the rest rather than try to integrate. It comes with a large singleton (for the most part) class that could be picked apart without a lot of struggles.

Gitrob also flags a lot of odd false-positives, combining this code with CloudSploit might really want to focus on using the iteration + oauth features from gitrob and narrow them down to look for AWS-specific matches.

[mikeatlas]

I also wrapped it in a Dockerized container if you want to try it out with potentially less setup:

https://github.com/weftio/weft-gitrob

[matthewdfuller]

Thanks for locating these projects, they'll be useful in doing something similar here. I took a look at we could do a combination of Oauth and GitHub search: https://developer.github.com/v3/search/

The only issue is, it's rate limited to 30 searches per minute, so accounts with large numbers of keys may take awhile to complete.

Also, AWS does not allow you to retrieve the secrets, so we may wind up with a lot of false positives (user hard-coded the access key, but not the secret).

[mikeatlas]

Well, general good practice policiies are that all keys (including access keys) are considered secrets and belong in the .env file that that file is in .gitignore always, and they're kept out of source code.

I wonder... with the IAM SDK, could you just use these two:

http://docs.aws.amazon.com/sdkforruby/api/Aws/IAM/Types/ListUsersRequest.html combined with
http://docs.aws.amazon.com/sdkforruby/api/Aws/IAM/Client.html#list_access_keys-instance_method

Wouldn't matter if the secret keys were in github or not, but usually they're either both there, or neither of them are there.

[tleyden]

I've been working on a project called keynuker that is really similar to what @mikeatlas described. It uses the github events API and pretty much the exact AWS SDK calls described.

[mikeatlas]

Revivalist @tleyden takes a post from 2yr ago and turns it into a project!