--discover-backup not working as advertised, looks for files even if they weren't found

[heinosasshallik]

I noticed that without the --discover-backup flag, I had 800k requests lined up in the progress bar, but with the --discover-backup option, I had ~6 million. Reading the help text, it says that backup files are only enumerated if a file is found. Given that, wouldn't it make sense to display only the 800k requests in the progress bar, and update the amount of requests when files get found?

I'm using Kali's version of gobuster.

└─$ gobuster version               
3.1.0

I initially thought the progress bar would maybe jump to 100% when all 800k requests have been completed without a single found file.

However, after playing around with it, it seems the help text is wrong. --discover-backup will search for files even if they haven't been found.

Proof: (look at the time stamps and how long it took to run the scans - 3 seconds vs 20 seconds)

┌──(x90slide㉿kali)-[~/…/jeeves/testing/autorecon/scans]
└─$ gobuster dir -u http://jeeves.htb:80/ -w /tmp/small.txt --discover-backup --no-tls-validation --status-codes "200,204,301,302,307,403,500" | tee "/tmp/tcp_80_http_gobuster_words_combined.txt"        
===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     http://jeeves.htb:80/
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                /tmp/small.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.1.0
[+] Timeout:                 10s
===============================================================
2021/06/17 18:25:38 Starting gobuster in directory enumeration mode
===============================================================
                              
===============================================================
2021/06/17 18:25:58 Finished
===============================================================
┌──(x90slide㉿kali)-[~/…/jeeves/testing/autorecon/scans]
└─$ gobuster dir -u http://jeeves.htb:80/ -w /tmp/small.txt  --no-tls-validation --status-codes "200,204,301,302,307,403,500" | tee "/tmp/tcp_80_http_gobuster_words_combined.txt"     
===============================================================
Gobuster v3.1.0
by OJ Reeves (@TheColonial) & Christian Mehlmauer (@firefart)
===============================================================
[+] Url:                     http://jeeves.htb:80/
[+] Method:                  GET
[+] Threads:                 10
[+] Wordlist:                /tmp/small.txt
[+] Negative Status codes:   404
[+] User Agent:              gobuster/3.1.0
[+] Timeout:                 10s
===============================================================
2021/06/17 18:26:12 Starting gobuster in directory enumeration mode
===============================================================
                            
===============================================================
2021/06/17 18:26:15 Finished
===============================================================

The help entry says:

 -d, --discover-backup                 Upon finding a file search for backup files

Given that not a single file was found, I would have expected the directory enumeration to take the same amount of time on both occasions.

[firefart]

the total requests are calculated before gobuster is run, so this shows the maximum requests that would be issued. That's a hard problem because updating the total requests during the run also comes with problems as you can't estimate anything because the number is not fixed and subject to change anytime. This problem is somehow solved with the next version as the disover-backup option checks for those files everytime and not only on a found entry

[heinosasshallik]

I'm not sure I follow. Just to confirm:

1. The total requests are calculated before gobuster is run, so the requests are calculated pessimistically. That's understandable.
2. The fact that --discover-backup took 7 times longer than not using that flag is indeed a bug. For some reason it checks the files every time and not only on a found entry
3. That bug will be somehow fixed (as in, you're not sure why it will start working?) in the next version of gobuster, version 3.2.0, which has not been released yet

Is all of this correct?

[firefart]

The current version checks for backup versions of all files, not only if the original was found. That's why the calculation is now correct because the overall requests are known before running the enumeration

[heinosasshallik]

The current version checks for backup versions of all files, not only if the original was found.

Oh. So you decided to skip the "only check if the original was found" part and always look for backups of all files?

Well that sucks. I guess I won't be using it then.

Maybe you could add a separate option to check for backup files only if the original was found? Otherwise the scans start to take too long (7x increase in scan time) and it becomes impractical.

[IppSec]

I am not exactly sure what was changed, but if you pull from the version where I introduced the feature (https://github.com/IppSec/gobuster), it behaves as you expect (only downloading backups after the original file was found).

[IppSec]

It appears it broke back in June 20, 2020. The commit which appears to changed the logic and broke how I pulled successful pages is: aa3f514

Unfortunately, out of time to look into this issue.

[firefart]

Yes we've changed the behaviour so the request calculation is correct and there are also cases where only the backup file exists without the original

[IppSec]

If you just want to discover if the backup exists without the original, can't you just use the pattern flag? The purpose of DiscoverBackup was to only trigger on successful find otherwise your wordlist gets enormous. #200

[heinosasshallik]

Yeah you've basically taken the best thing about the --discover-backups flag and ruined it. As it is, that flag is practically unusable, because a 7x increase in scan times just isn't worth it.

Please revert it so that it only checks for backup versions of files it has already found.

[firefart]

IK will have a look at this at the dev branch to find a clean way to implement this

[dombg1337]

Thanks for having a look at it, I just wanted to create the same issue. I even went back a version to be able to use the feature again. Now I'm missing --exclude-length^^ Really looking forward to seeing it working again.