Add probe-all-ips feature #427

zerodivisi0n · 2021-10-23T16:17:24Z

Added flag to support probe all ips #245.

Example output:
For default output, I just implicitly enable IP address output:

$ echo hackerone.com | ./httpx -sc -scan-all-ips
https://hackerone.com [302] [104.16.99.52]
https://hackerone.com [302] [104.16.100.52]

JSON (as described in the original issue):

{
  "timestamp": "2021-10-23T19:10:52.740432132+03:00",
  "scheme": "https",
  "port": "443",
  "path": "/",
  "body-sha256": "bf697861898d0a7fabf4886f0eb238a440f45622e062ef40ac266e5575796347",
  "header-sha256": "883abfaf1ec8e7c9270389ceb45e0c4b8a16264c933174b93b14766b6f66c85c",
  "a": [
    "104.16.100.52",
    "104.16.99.52"
  ],
  "url": "https://hackerone.com:443",
  "input": "hackerone.com",
  "location": "https://www.hackerone.com/",
  "webserver": "cloudflare",
  "content-type": "text/html",
  "method": "GET",
  "host": "104.16.100.52",
  "content-length": 92,
  "status-code": 302,
  "csp": {
    "domains": [
      "a5s.hackerone-ext-content.com",
      "b5s.hackerone-ext-content.com",
      "errors.hackerone.net",
      "profile-photos.hackerone-user-content.com",
      "www.youtube-nocookie.com",
      "www.google-analytics.com",
      "cover-photos.hackerone-user-content.com",
      "hackathon-photos.hackerone-user-content.com",
      "hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com",
      "https://errors.hackerone.net/api/30/csp-report/?sentry_key=374aea95847f4040a69f9c8d49a3a59d"
    ]
  },
  "response-time": "963.997528ms",
  "failed": false
}
{
  "timestamp": "2021-10-23T19:10:52.774901606+03:00",
  "scheme": "https",
  "port": "443",
  "path": "/",
  "body-sha256": "bf697861898d0a7fabf4886f0eb238a440f45622e062ef40ac266e5575796347",
  "header-sha256": "e129249144cc96d0bcf0273eb03e42f002169966d76179ffbd5d359264e5ea35",
  "a": [
    "104.16.100.52",
    "104.16.99.52"
  ],
  "url": "https://hackerone.com:443",
  "input": "hackerone.com",
  "location": "https://www.hackerone.com/",
  "webserver": "cloudflare",
  "content-type": "text/html",
  "method": "GET",
  "host": "104.16.99.52",
  "content-length": 92,
  "status-code": 302,
  "csp": {
    "domains": [
      "profile-photos.hackerone-user-content.com",
      "https://errors.hackerone.net/api/30/csp-report/?sentry_key=374aea95847f4040a69f9c8d49a3a59d",
      "www.youtube-nocookie.com",
      "b5s.hackerone-ext-content.com",
      "www.google-analytics.com",
      "errors.hackerone.net",
      "cover-photos.hackerone-user-content.com",
      "hackathon-photos.hackerone-user-content.com",
      "hackerone-us-west-2-production-attachments.s3.us-west-2.amazonaws.com",
      "a5s.hackerone-ext-content.com"
    ]
  },
  "response-time": "1.005377776s",
  "failed": false
}

ehsandeep

Thank you @zerodivisi0n for adding this feature, upon further testing I've noticed with the following example it's not working as expected, i.e using the same IP for all probes.

echo docs.hackerone.com | ./httpx -probe-all-ips

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.1.4-dev

		projectdiscovery.io

Use with caution. You are responsible for your actions.
Developers assume no liability and are not responsible for any misuse or damage.
http://docs.hackerone.com [185.199.109.153]
http://docs.hackerone.com [185.199.109.153]
http://docs.hackerone.com [185.199.109.153]
http://docs.hackerone.com [185.199.109.153]

zerodivisi0n · 2021-10-23T16:53:35Z

Thanks for the quick feedback @ehsandeep . I also found a problem with the following input:

$ echo https://www.hackerone.com/login | ./httpx -sc -probe-all-ips
https://www.hackerone.com/login [404] [104.16.100.52]

I'm not sure if it should be supported, probably it should.

I'll try to deal with both cases tomorrow.

zerodivisi0n · 2021-10-23T16:56:12Z

Your case gives the following output on my laptop:

$ echo docs.hackerone.com | ./httpx -probe-all-ips
https://docs.hackerone.com [185.199.108.153]
https://docs.hackerone.com [185.199.111.153]
https://docs.hackerone.com [185.199.109.153]
https://docs.hackerone.com [185.199.110.153]

I'll try to dig deeper.

ehsandeep · 2021-10-23T16:59:52Z

Thanks for the quick feedback @ehsandeep . I also found a problem with the following input:
$ echo https://www.hackerone.com/login | ./httpx -sc -probe-all-ips
https://www.hackerone.com/login [404] [104.16.100.52]
I'm not sure if it should be supported, probably it should.

I'll try to deal with both cases tomorrow.

Thank you, with this modification, the only difference is that we are probing the same request with different available IPs; the rest of the features and flags should remain functional as before.

zerodivisi0n · 2021-10-23T17:30:41Z

Fixed issue with probing IPs for complex URLs:

$ echo https://hackerone.com/users/sign_in | ./httpx -sc -probe-all-ips
https://hackerone.com/users/sign_in [200] [104.16.100.52]
https://hackerone.com/users/sign_in [200] [104.16.99.52]

Still can't reproduce the problem with the same IPs. Maybe it is some sort of DNS caching problem? Anyway, I'll try to dig deeper tomorrow.

zerodivisi0n · 2021-10-26T22:36:27Z

@ehsandeep Unfortunately, I have not been able to reproduce your issue. So I need your help in debugging.
What operating system did you test on? I have tested on Ubuntu and on Mac OS and also on AWS instance with Ubuntu. And I got the correct output everywhere:

$ echo docs.hackerone.com | ./httpx -probe-all-ips -sc      
https://docs.hackerone.com [200] [185.199.110.153]
https://docs.hackerone.com [200] [185.199.109.153]
https://docs.hackerone.com [200] [185.199.111.153]
https://docs.hackerone.com [200] [185.199.108.153]

What is the output of the command host docs.hackerone.com on your machine?
I have one like this:

$ host docs.hackerone.com
docs.hackerone.com is an alias for hacker0x01.github.io.
hacker0x01.github.io has address 185.199.110.153
hacker0x01.github.io has address 185.199.111.153
hacker0x01.github.io has address 185.199.109.153
hacker0x01.github.io has address 185.199.108.153
hacker0x01.github.io has IPv6 address 2606:50c0:8002::153
hacker0x01.github.io has IPv6 address 2606:50c0:8003::153
hacker0x01.github.io has IPv6 address 2606:50c0:8000::153
hacker0x01.github.io has IPv6 address 2606:50c0:8001::153

Are you using default fastdialer module from your repository?
Can you also provide the json output of the program? Any other information will also be useful.

ehsandeep · 2021-10-30T19:10:26Z

@zerodivisi0n I'm adding all the information, let me know any more information is required.

$host docs.hackerone.com

docs.hackerone.com is an alias for hacker0x01.github.io.
hacker0x01.github.io has address 185.199.111.153
hacker0x01.github.io has address 185.199.110.153
hacker0x01.github.io has address 185.199.108.153
hacker0x01.github.io has address 185.199.109.153
hacker0x01.github.io has IPv6 address 2606:50c0:8000::153
hacker0x01.github.io has IPv6 address 2606:50c0:8001::153
hacker0x01.github.io has IPv6 address 2606:50c0:8003::153
hacker0x01.github.io has IPv6 address 2606:50c0:8002::153

$echo docs.hackerone.com | ./httpx -probe-all-ips -silent

http://docs.hackerone.com [185.199.108.153]
http://docs.hackerone.com [185.199.108.153]
http://docs.hackerone.com [185.199.108.153]
http://docs.hackerone.com [185.199.108.153]

$echo docs.hackerone.com | ./httpx -probe-all-ips -silent -json | jq .

{
  "timestamp": "2021-10-31T00:36:15.466334+05:30",
  "scheme": "http",
  "port": "80",
  "path": "/",
  "body-sha256": "9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a",
  "header-sha256": "41c5ae0a07f5f84615b3093d40b78a94b2a74763994771fc55a817afde8f2fd4",
  "a": [
    "185.199.111.153",
    "185.199.108.153",
    "185.199.109.153",
    "185.199.110.153"
  ],
  "cnames": [
    "hacker0x01.github.io"
  ],
  "url": "http://docs.hackerone.com:80",
  "input": "docs.hackerone.com",
  "location": "https://docs.hackerone.com/",
  "title": "301 Moved Permanently",
  "webserver": "GitHub.com",
  "content-type": "text/html",
  "method": "GET",
  "host": "185.199.111.153",
  "content-length": 162,
  "status-code": 301,
  "response-time": "122.40125ms",
  "failed": false
}
{
  "timestamp": "2021-10-31T00:36:15.466339+05:30",
  "scheme": "http",
  "port": "80",
  "path": "/",
  "body-sha256": "9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a",
  "header-sha256": "0bca1be5120ce3f9e68fe1ea9fa9b59c07370d4cfba3564d0a46a477ec480cd8",
  "a": [
    "185.199.111.153",
    "185.199.108.153",
    "185.199.109.153",
    "185.199.110.153"
  ],
  "cnames": [
    "hacker0x01.github.io"
  ],
  "url": "http://docs.hackerone.com:80",
  "input": "docs.hackerone.com",
  "location": "https://docs.hackerone.com/",
  "title": "301 Moved Permanently",
  "webserver": "GitHub.com",
  "content-type": "text/html",
  "method": "GET",
  "host": "185.199.111.153",
  "content-length": 162,
  "status-code": 301,
  "response-time": "126.154625ms",
  "failed": false
}
{
  "timestamp": "2021-10-31T00:36:15.468841+05:30",
  "scheme": "http",
  "port": "80",
  "path": "/",
  "body-sha256": "9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a",
  "header-sha256": "bc7cf34f5129a2f04b19c6d95db65a26156787696545468dca7d73bead710655",
  "a": [
    "185.199.111.153",
    "185.199.108.153",
    "185.199.109.153",
    "185.199.110.153"
  ],
  "cnames": [
    "hacker0x01.github.io"
  ],
  "url": "http://docs.hackerone.com:80",
  "input": "docs.hackerone.com",
  "location": "https://docs.hackerone.com/",
  "title": "301 Moved Permanently",
  "webserver": "GitHub.com",
  "content-type": "text/html",
  "method": "GET",
  "host": "185.199.111.153",
  "content-length": 162,
  "status-code": 301,
  "response-time": "125.287167ms",
  "failed": false
}
{
  "timestamp": "2021-10-31T00:36:15.469921+05:30",
  "scheme": "http",
  "port": "80",
  "path": "/",
  "body-sha256": "9e17cb15dd75bbbd5dbb984eda674863c3b10ab72613cf8a39a00c3e11a8492a",
  "header-sha256": "b43e5812be88a0a5588f967d6c741e2a400f465bf83ef9b4788ef1df462c089b",
  "a": [
    "185.199.111.153",
    "185.199.108.153",
    "185.199.109.153",
    "185.199.110.153"
  ],
  "cnames": [
    "hacker0x01.github.io"
  ],
  "url": "http://docs.hackerone.com:80",
  "input": "docs.hackerone.com",
  "location": "https://docs.hackerone.com/",
  "title": "301 Moved Permanently",
  "webserver": "GitHub.com",
  "content-type": "text/html",
  "method": "GET",
  "host": "185.199.111.153",
  "content-length": 162,
  "status-code": 301,
  "response-time": "118.357ms",
  "failed": false
}

uname -a
Darwin Sandeeps-MacBook-Pro-3.local 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm64

zerodivisi0n · 2021-10-31T10:15:54Z

Oh, you have a new MacBook! Nice!

I continue to investigate and so far I have found one oddity - in your examples, requests are made via the http protocol, but mine via https. And I have such behavior on the dev and master branches as well:

$ echo docs.hackerone.com | ./httpx -silent -sc                                     
https://docs.hackerone.com [200]

I don't know if this is directly related to the current problem. Maybe you have some custom config (~/.config/httpx/config.yaml)?

And one more clarification - are you using a build from the current branch with the last commit (2559dce)?

Thanks in advance for your reply, I will continue to figure it out.

zerodivisi0n · 2021-11-03T14:51:33Z

I tried another approach to probe all the IPs.
Since I have not been able to reproduce the issue, can you please check this on your side?

ehsandeep · 2021-11-17T08:35:52Z

@zerodivisi0n it works as expected after the change, but noticed an unexpected wait time/delay during the scan even with a single input, will add more specific details.

@Mzack9999 FYI, this is related to #245, we settled with the output UI, and @zerodivisi0n proposed the above changes.

Mzack9999

Everything seems to work correctly, nice work!

Mzack9999 · 2021-11-17T20:09:28Z

After further analysis, some servers reject TLS requests having an SNI name equal to the IP. This is why some results were found valid only for http, for example, gslink.hackerone.com. The request should use the domain name to generate the URL, but somehow we need to find a way to tell fastdialer to use a specific IP as a destination. Prototype of working solution using https://github.com/projectdiscovery/fastdialer/tree/issue-28-sni-server-name-and-custom-ip and changing the code as follows in runner.go:

..
if scanopts.ProbeAllIPS && customIP != "" {
    customHost = URL.Host
    // URL.Host = customIP <= don't overwrite Host header, will be handled via request context
}
..
if customIP != "" {
    req = req.WithContext(context.WithValue(req.Context(), "ip", customIP))
}
resp, err := hp.Do(req, httpx.UnsafeOptions{URIPath: reqURI})

zerodivisi0n · 2021-11-18T05:44:50Z

@ehsandeep Waiting for your details.
@Mzack9999 I understand what you're talking about. Let me think about how to fix it.

ehsandeep · 2021-11-22T08:31:11Z

@zerodivisi0n it was related to the issue mentioned by mzack, looks like I got some hosts not returning any results because of the ssl issue and took some time before getting errored out.

ehsandeep · 2021-12-01T07:12:02Z

@zerodivisi0n everything is now working well, thank you for working on this.

zerodivisi0n · 2021-12-01T08:24:07Z

@ehsandeep Sorry, I've been very busy in the last few weeks, and haven't been able to get to this. It's good that it's finally done. Thank you!

zerodivisi0n added 2 commits October 23, 2021 18:59

Extract getDNSData method

8ef1c06

Add probe-all-ips flag

cf231c0

ehsandeep linked an issue Oct 23, 2021 that may be closed by this pull request

Adding support for probe-all-ips #245

Closed

ehsandeep self-assigned this Oct 23, 2021

ehsandeep reviewed Oct 23, 2021

View reviewed changes

Fix all ips probing for urls

2559dce

ehsandeep assigned zerodivisi0n and unassigned ehsandeep Oct 26, 2021

ehsandeep added the Status: In Progress This issue is being worked on, and has someone assigned. label Oct 26, 2021

Another approach to probe all ips

cc49b3a

ehsandeep added the Status: Review Needed The issue has a PR attached to it which needs to be reviewed label Nov 15, 2021

ehsandeep requested a review from Mzack9999 November 17, 2021 08:26

Mzack9999 approved these changes Nov 17, 2021

View reviewed changes

Mzack9999 self-requested a review November 18, 2021 11:31

ehsandeep mentioned this pull request Nov 19, 2021

v1.1.4 Release preparation #437

Merged

2 tasks

implementing custom dialed ip via fastdialer

b2525f7

Mzack9999 approved these changes Nov 30, 2021

View reviewed changes

Mzack9999 requested a review from ehsandeep November 30, 2021 11:15

Mzack9999 added Type: Enhancement Most issues will probably ask for additions or changes. and removed Status: In Progress This issue is being worked on, and has someone assigned. labels Nov 30, 2021

Mzack9999 and others added 2 commits November 30, 2021 20:58

adding missing output info

76ae47c

minor flag updates

256b050

ehsandeep added Status: Completed Nothing further to be done with this issue. Awaiting to be closed. and removed Status: Review Needed The issue has a PR attached to it which needs to be reviewed labels Dec 1, 2021

ehsandeep approved these changes Dec 1, 2021

View reviewed changes

ehsandeep merged commit 9481594 into projectdiscovery:dev Dec 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add probe-all-ips feature #427

Add probe-all-ips feature #427

zerodivisi0n commented Oct 23, 2021

ehsandeep left a comment

zerodivisi0n commented Oct 23, 2021

zerodivisi0n commented Oct 23, 2021

ehsandeep commented Oct 23, 2021

zerodivisi0n commented Oct 23, 2021

zerodivisi0n commented Oct 26, 2021

ehsandeep commented Oct 30, 2021

zerodivisi0n commented Oct 31, 2021

zerodivisi0n commented Nov 3, 2021

ehsandeep commented Nov 17, 2021

Mzack9999 left a comment

Mzack9999 commented Nov 17, 2021 •

edited

Loading

zerodivisi0n commented Nov 18, 2021

ehsandeep commented Nov 22, 2021

ehsandeep commented Dec 1, 2021

zerodivisi0n commented Dec 1, 2021

Add probe-all-ips feature #427

Add probe-all-ips feature #427

Conversation

zerodivisi0n commented Oct 23, 2021

ehsandeep left a comment

Choose a reason for hiding this comment

zerodivisi0n commented Oct 23, 2021

zerodivisi0n commented Oct 23, 2021

ehsandeep commented Oct 23, 2021

zerodivisi0n commented Oct 23, 2021

zerodivisi0n commented Oct 26, 2021

ehsandeep commented Oct 30, 2021

zerodivisi0n commented Oct 31, 2021

zerodivisi0n commented Nov 3, 2021

ehsandeep commented Nov 17, 2021

Mzack9999 left a comment

Choose a reason for hiding this comment

Mzack9999 commented Nov 17, 2021 • edited Loading

zerodivisi0n commented Nov 18, 2021

ehsandeep commented Nov 22, 2021

ehsandeep commented Dec 1, 2021

zerodivisi0n commented Dec 1, 2021

Mzack9999 commented Nov 17, 2021 •

edited

Loading