Massive refactoring

[firefart]

Changes

• massive refactoring. libgobuster can now be used as a library from other programs too (but it's still not perfect because it still contains some command line option logic inside the library which should be extracted)
• status on wordlist progress is now printed
• a context is now passed to the main function to handle cancelation (CTRL+C). Also all HTTP requests now receive the context so they will be canceled immediately.
• connections are now actually reused. This speeds up gobuster by factor 3 (at least on HTTPS connections)
• new -to parameter to specify request timeout (default 10s)
• status codes are now sorted on output
• a default user agent of gobuster version is set on every request (old: Go-http-client/1.1, changeable with command line option)
• added status 403 to the default status list
• new STDIN handling, you now have to pass a - to use STDIN
• extracted both gobuster modes (dir and dns) as plugins implementing an interface. This way we can simply add more plugins in the future
• lower memory usage
• only exported the types that are necessary to export and documented the rest (run go doc to check)
• extension list can now also contains dots and spaces
• replaced uuid package (see Replace package satori/go.uuid #87)

Memory usage comparison

current master

HTOP

ps

root      2942 50.4  0.1 1127080 15416 pts/0   Sl+  11:19   0:07  |   |   \_ /root/go/bin/gobuster -u https://firefart.at/ -w /usr/share/wordlists/dirb/common.txt

this pr

HTOP

ps

root      2970 47.3  0.1  14784  9860 pts/0    Sl+  11:20   0:04  |   |   \_ ./gobuster -u https://firefart.at/ -w /usr/share/wordlists/dirb/common.txt

Example runs

during execution

finished

Test instructions

• go get -u github.com/OJ/gobuster
• cd $GOPATH/src/github.com/OJ/gobuster
• git remote add firefart https://github.com/FireFart/gobuster.git
• git fetch firefart
• git checkout -b status firefart/status
• go build

[eur0pa]

This PR fails to compile atm:

# _/Users/europa/pentest/gobuster-timeout
./main.go:42:19: s.URL undefined (type libgobuster.State has no field or method URL, but does have Url)
./main.go:49:21: s.Timeout undefined (type libgobuster.State has no field or method Timeout)

[firefart]

@eur0pa because you need to change "github.com/OJ/gobuster/libgobuster" in main.go to "github.com/FireFart/gobuster/libgobuster" if you are using my repo (or any other path you are using)

[eur0pa]

Tested this PR for a few days; it still hangs from time to time but, worst, it will still sometime quit inexplicably after a few hundred requests

[firefart]

I'm currently working on a big refactor so I hope this error is also fixed (stumbled across it also multiple times)

[firefart]

@eur0pa I pushed a new commit and did a lot of refactoring. Can you please try the new version and see if the unexpected quit errors are now fixed?

[eur0pa]

Seems to be working well for now, except when using extensions: the status progress won't take them into account, as it only counts the number of lines in the wordlist and not the amount of requests-to-be

[firefart]

thx fixed it to now be a request counter instead of a line counter

[anshumanbh]

@firefart I believe this breaks because both gobusterdir.go and gobusterdns.go use github.com/OJ/gobuster/libgobuster as one of the imports and I believe that still uses the old uuid package because I see this while building the Docker image:

vendor/github.com/OJ/gobuster/libgobuster/dir.go:116:19: not enough arguments in call to uuid.Must
	have (uuid.UUID)
	want (uuid.UUID, error)
vendor/github.com/OJ/gobuster/libgobuster/dns.go:13:19: not enough arguments in call to uuid.Must
	have (uuid.UUID)
	want (uuid.UUID, error)

[firefart]

@anshumanbh it looks like your code uses the "real" libgobuster from current master (that's not going to work). I added some test instructions at the end of the PR description to get it up and running. Which Docker image are you trying to build?

[anshumanbh]

@firefart Doesn't the status branch of your forked repo also use the real libgobuster. https://github.com/FireFart/gobuster/blob/status/gobusterdns/gobusterdns.go#L10. The import statement here github.com/OJ/gobuster/libgobuster uses the master branch of gobuster, corret? Am I missing something?

I have a Dockerfile that looks like this:

# Build Container
FROM golang:1.9.4-alpine3.7 AS build-env
RUN apk add --no-cache --upgrade git openssh-client ca-certificates
RUN go get -u github.com/golang/dep/cmd/dep

WORKDIR /go/src/github.com/anshumanbh/gobuster

# Cache the dependencies early
COPY Gopkg.toml Gopkg.lock ./
RUN dep ensure -vendor-only -v

COPY gobusterdir/ gobusterdir/
COPY gobusterdns/ gobusterdns/
COPY libgobuster/ libgobuster/
COPY main.go ./

# Install the binary
RUN go install

# Final Container
FROM alpine:3.7
COPY --from=build-env /go/bin/gobuster /usr/bin/gobuster

# Copy wordlists
COPY wordlists wordlists

ENTRYPOINT ["/usr/bin/gobuster"]

And, I am using dep to manage the dependencies.

[firefart]

no the import statement only says to use libgobuster (in this case the local one). If it's not found locally it will use the one from github (which is bad). Just add the buildsteps from above to your Dockerfile and it should build correctly

[eur0pa]

There seems to be a problem with the http library, no matter how you shuffle this code, it won't reuse connections:

~ netstat -antup|grep -c TIME_WAIT
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
402

This was with 50 threads. I tried implementing a simpler bruteforcer in go and still go the same behavior, no matter how many tricks I try:

	tr = &http.Transport{
		MaxIdleConnsPerHost: *threads,
		MaxIdleConns:        100,
		IdleConnTimeout: time.Duration(*timeout) * time.Second,
		TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
	}

Also reading / discarding body doesn't change this behavior, neither in my tool nor gobuster. Any ideas?

edit: "fixed" it with

net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

edit 2: apparently, I didn't fix anything. It's a Go long-standing bug: not only all connections will linger in TIME_WAIT, but once the host runs out of filedescriptors, the Go resolver will also fail silently and the requests dropped. That's why dirsearch and gobuster always yield different results when used on a long list of hosts: golang/go#18588
I have found no way around this but to set request.Close = true which slows down the enumeration considerably but will prevent the resolver bug from triggering. @firefart, any idea?

[firefart]

@eur0pa I think TIME_WAIT is not directly related to connection reuse, it's a socket thing. Try watching the requests in Wireshark and have a look at the source ports from your machine. If they stay the same, connections are reused. That's how I checked it after implementing it.

[eur0pa]

@firefart yeah I went the sysctl way to solve TIME_WAIT both for gobuster and other tools. What really boggles my mind though, is how bad the Go resolver really is. Even in gobuster, any request will also open a socket to to a random resolver picked from /etc/resolv.conf. After a certain amount of time, even with all sockets closing down properly, tools will fail with lookup [DOMAIN] on [DNS]:53: no such host: this will cause requests to fail and files to be missed, a bunch at a time.
In order to reproduce this quickly, serve a large list of hosts to gobuster via parallel and see what happens, ie:

[ assuming hosts.txt has, say, 200 entries ]
~ cat hosts.txt | parallel --bar -j 20 "gobuster -t 10 -w wordlist.txt -u {}"

You'll see that everything will start to fail pretty soon. Even by lowering the jobs amount down to 1 or 2 will eventually see it fail. This doesn't happen in Python, and seems to be a long-standing issue with Go (golang/go#18588). Really no idea how to solve this, even in gobuster.

[joohoi]

What's the current state of this PR in general? I'm thinking of starting to write a new PR, but unsure if I should base it on the refactored code in this branch, or the current master. Sorry for the static!

[OJ]

The status is that this PR is a monster that includes a lot more than a code refactor. It's taking a lot of time for me to go through it. Time is something in particularly short on at the moment. I would hold off submitting anything until this is out of the way. Sorry mate.

…

On Sun., 12 Aug. 2018, 06:40 Joona Hoikkala, ***@***.***> wrote: What's the current state of this PR in general? I'm thinking of starting to write a new PR, but unsure if I should base it on the refactored code in this branch, or the current master. Sorry for the static! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#79 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABw4K0nGz2Fl1WNjGscXW2bWzu3apW3ks5uP0EvgaJpZM4T9JYk> .