[Issue] httpx v1.0.6 does not process protocl://host:port value in stdin

[osamahamad]

Sometimes httpx is not only for probing hosts and checking ports, it is nice to see that some of features can be used instead of building another tools from scratch to do the same job.

For example using -match-string flag on a list of URLs to check for specific string in response:

Stdin example:

https://example.com
http://example.com
http://example.com/good/
http://example.com/goo/bad?url=<h1>asdasd</h1>
.
.

Will go through, but when stdin have something like protocol + port number , i.e: http://example:80/good/bad?url=<h1>asdasd</h1>

It seems on my side that this will not be processed and the request will not sent to http://example:80/good/bad?url=<h1>asdasd</h1>

Example:
qsreplace simply change parameter values in a list of URLs: https://github.com/tomnomnom/qsreplace

gau testphp.vulnweb.com | grep '=' | qsreplace '"><h1>asdasd</h1>' | while read host;do curl -s --path-as-is -k -L "$host" | grep -qs "<h1>asdasd</h1>" && echo "$host" ;done

Will return ( Ignoring other 5 URLs ) :

http://testphp.vulnweb.com:80/hpp/?pp=%22%3E%3Ch1%3Easdasd%3C%2Fh1%3E

As it seems it sent a request to the last URL , found a string , print it.

But in

gau testphp.vulnweb.com | grep '=' | qsreplace '"><h1>asdasd</h1>' > 1.test

cat 1.test | httpx --follow-redirects -match-string "<h1>asdasd</h1>"

Supposed to return the same result of the above curl command, It miss the mentioned URL that have protocol specified.

http://testphp.vulnweb.com:80/hpp/?pp=%22%3E%3Ch1%3Easdasd%3C%2Fh1%3E

The expected behavior is to send request to http://testphp.vulnweb.com:80/hpp/?pp=%22%3E%3Ch1%3Easdasd%3C%2Fh1%3E like the other URLs in stdin and print the URL if the specified string in response found or simply just treating http://testphp.vulnweb.com:80 same as http://testphp.vulnweb.com or testphp.vulnweb.com since last two will be processed without problems.

I did not test it on other than port 80 but if my assumption is true then it will miss the other URLs like http://example.com:8000 ..etc.

[ehsandeep]

@osamahamad looks this is a bug introduced in the last release of httpx v1.0.6 and related to #276

Working as expected with the previous version, for example, with httpx v1.0.5

echo 'http://testphp.vulnweb.com:80/hpp/?pp=%22%3E%3Ch1%3Easdasd%3C%2Fh1%3E' | httpx

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.0.5

		projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
http://testphp.vulnweb.com:80/hpp/?pp=%22%3E%3Ch1%3Easdasd%3C%2Fh1%3E

[ehsandeep]

This is now fixed in #282

[ehsandeep]

@osamahamad thanks for reporting, this is now supported in the latest release of httpx v1.0.7