Proxy daemon SSL bug

[andresriancho]

w3af's proxy daemon has SSL unittests and they pass, but I've found an issue when trying to run test_https_via_proxy in TestExtendedUrllibProxy. It seems that for some reason the extended urllib doesn't play well with the daemon proxy; BUT since the same test passes if I use tinyproxy(apt-get install tinyproxy) I'm flagging this as a proxy.py bug.

How to reproduce:

• Remove the SkipTest from test_xurllib_proxy.py
• Run the unittest

nosetests --config=nose.cfg \
core/data/url/tests/test_xurllib_proxy.py:TestExtendedUrllibProxy.test_https_via_proxy

• Expected output: PASS (in less than 1 second)
• Result: Long wait and a timeout

[andresriancho]

Related with #170

[andresriancho]

This is what you get when removing the SkipTest:

AssertionError: '<title>moth: vulnerable web application</title>' not found in
u'w3af proxy error: Invalid URL "/"<br/><br/>Traceback for this error: <br/><br/>Traceback (most recent call last):<br/>  File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/daemons/proxy.py", line 127, in do_ALL<br/>    res = self._send_to_server()<br/> 
File "/home/pablo/PycharmProjects/w3af/w3af/core/controllers/daemons/proxy.py", line 150, in _send_to_server<br/>    uri_instance = URL(path)<br/> 
File "/home/pablo/PycharmProjects/w3af/w3af/core/data/parsers/url.py", line 196, in __init__<br/>    raise ValueError, \'Invalid URL "%s"\' % (data,)<br/>ValueError: Invalid URL "/"<br/>'

We could get rid of our proxy code all together if we use libmproxy #1269

[andresriancho]

Related with #1269 (Analyze possibility to migrate to libmproxy)

[andresriancho]

This is the tinyproxy configuration:

##
## tinyproxy.conf -- tinyproxy daemon configuration file
##
## This example tinyproxy.conf file contains example settings
## with explanations in comments. For decriptions of all
## parameters, see the tinproxy.conf(5) manual page.
##

#
# User/Group: This allows you to set the user and group that will be
# used for tinyproxy after the initial binding to the port has been done
# as the root user. Either the user or group name or the UID or GID
# number may be used.
#
#User nobody
#Group nobody

#
# Port: Specify the port which tinyproxy will listen on.  Please note
# that should you choose to run on a port lower than 1024 you will need
# to start tinyproxy using root.
#
Port 8888

#
# Listen: If you have multiple interfaces this allows you to bind to
# only one. If this is commented out, tinyproxy will bind to all
# interfaces present.
#
#Listen 192.168.0.1
Listen 127.0.0.1

#
# Bind: This allows you to specify which interface will be used for
# outgoing connections.  This is useful for multi-home'd machines where
# you want all traffic to appear outgoing from one particular interface.
#
#Bind 192.168.0.1

#
# BindSame: If enabled, tinyproxy will bind the outgoing connection to the
# ip address of the incoming connection.
#
#BindSame yes

#
# Timeout: The maximum number of seconds of inactivity a connection is
# allowed to have before it is closed by tinyproxy.
#
Timeout 600

#
# ErrorFile: Defines the HTML file to send when a given HTTP error
# occurs.  You will probably need to customize the location to your
# particular install.  The usual locations to check are:
#   /usr/local/share/tinyproxy
#   /usr/share/tinyproxy
#   /etc/tinyproxy
#
#ErrorFile 404 "/usr/local/share/tinyproxy/404.html"
#ErrorFile 400 "/usr/local/share/tinyproxy/400.html"
#ErrorFile 503 "/usr/local/share/tinyproxy/503.html"
#ErrorFile 403 "/usr/local/share/tinyproxy/403.html"
#ErrorFile 408 "/usr/local/share/tinyproxy/408.html"

#
# DefaultErrorFile: The HTML file that gets sent if there is no
# HTML file defined with an ErrorFile keyword for the HTTP error
# that has occured.
#
DefaultErrorFile "/usr/local/share/tinyproxy/default.html"

#
# StatHost: This configures the host name or IP address that is treated
# as the stat host: Whenever a request for this host is received,
# Tinyproxy will return an internal statistics page instead of
# forwarding the request to that host.  The default value of StatHost is
# tinyproxy.stats.
#
#StatHost "tinyproxy.stats"
#

#
# StatFile: The HTML file that gets sent when a request is made
# for the stathost.  If this file doesn't exist a basic page is
# hardcoded in tinyproxy.
#
StatFile "/usr/local/share/tinyproxy/stats.html"

#
# LogFile: Allows you to specify the location where information should
# be logged to.  If you would prefer to log to syslog, then disable this
# and enable the Syslog directive.  These directives are mutually
# exclusive.
#
#LogFile "/var/log/tinyproxy/tinyproxy.log"

#
# Syslog: Tell tinyproxy to use syslog instead of a logfile.  This
# option must not be enabled if the Logfile directive is being used.
# These two directives are mutually exclusive.
#
#Syslog On

#
# LogLevel: 
#
# Set the logging level. Allowed settings are:
#   Critical    (least verbose)
#   Error
#   Warning
#   Notice
#   Connect     (to log connections without Info's noise)
#   Info        (most verbose)
#
# The LogLevel logs from the set level and above. For example, if the
# LogLevel was set to Warning, then all log messages from Warning to
# Critical would be output, but Notice and below would be suppressed.
#
LogLevel Info

#
# PidFile: Write the PID of the main tinyproxy thread to this file so it
# can be used for signalling purposes.
#
#PidFile "/var/run/tinyproxy/tinyproxy.pid"

#
# XTinyproxy: Tell Tinyproxy to include the X-Tinyproxy header, which
# contains the client's IP address.
#
#XTinyproxy Yes

#
# Upstream:
#
# Turns on upstream proxy support.
#
# The upstream rules allow you to selectively route upstream connections
# based on the host/domain of the site being accessed.
#
# For example:
#  # connection to test domain goes through testproxy
#  upstream testproxy:8008 ".test.domain.invalid"
#  upstream testproxy:8008 ".our_testbed.example.com"
#  upstream testproxy:8008 "192.168.128.0/255.255.254.0"
#
#  # no upstream proxy for internal websites and unqualified hosts
#  no upstream ".internal.example.com"
#  no upstream "www.example.com"
#  no upstream "10.0.0.0/8"
#  no upstream "192.168.0.0/255.255.254.0"
#  no upstream "."
#
#  # connection to these boxes go through their DMZ firewalls
#  upstream cust1_firewall:8008 "testbed_for_cust1"
#  upstream cust2_firewall:8008 "testbed_for_cust2"
#
#  # default upstream is internet firewall
#  upstream firewall.internal.example.com:80
#
# The LAST matching rule wins the route decision.  As you can see, you
# can use a host, or a domain:
#  name     matches host exactly
#  .name    matches any host in domain "name"
#  .        matches any host with no domain (in 'empty' domain)
#  IP/bits  matches network/mask
#  IP/mask  matches network/mask
#
#Upstream some.remote.proxy:port

#
# MaxClients: This is the absolute highest number of threads which will
# be created. In other words, only MaxClients number of clients can be
# connected at the same time.
#
MaxClients 100

#
# MinSpareServers/MaxSpareServers: These settings set the upper and
# lower limit for the number of spare servers which should be available.
#
# If the number of spare servers falls below MinSpareServers then new
# server processes will be spawned.  If the number of servers exceeds
# MaxSpareServers then the extras will be killed off.
#
MinSpareServers 5
MaxSpareServers 20

#
# StartServers: The number of servers to start initially.
#
StartServers 10

#
# MaxRequestsPerChild: The number of connections a thread will handle
# before it is killed. In practise this should be set to 0, which
# disables thread reaping. If you do notice problems with memory
# leakage, then set this to something like 10000.
#
MaxRequestsPerChild 0

#
# Allow: Customization of authorization controls. If there are any
# access control keywords then the default action is to DENY. Otherwise,
# the default action is ALLOW.
#
# The order of the controls are important. All incoming connections are
# tested against the controls based on order.
#
Allow 127.0.0.1

#
# AddHeader: Adds the specified headers to outgoing HTTP requests that
# Tinyproxy makes. Note that this option will not work for HTTPS
# traffic, as Tinyproxy has no control over what headers are exchanged.
#
#AddHeader "X-My-Header" "Powered by Tinyproxy"

#
# ViaProxyName: The "Via" header is required by the HTTP RFC, but using
# the real host name is a security concern.  If the following directive
# is enabled, the string supplied will be used as the host name in the
# Via header; otherwise, the server's host name will be used.
#
ViaProxyName "tinyproxy"

#
# DisableViaHeader: When this is set to yes, Tinyproxy does NOT add
# the Via header to the requests. This virtually puts Tinyproxy into
# stealth mode. Note that RFC 2616 requires proxies to set the Via
# header, so by enabling this option, you break compliance.
# Don't disable the Via header unless you know what you are doing...
#
#DisableViaHeader Yes

#
# Filter: This allows you to specify the location of the filter file.
#
#Filter "/usr/local/etc/tinyproxy/filter"

#
# FilterURLs: Filter based on URLs rather than domains.
#
#FilterURLs On

#
# FilterExtended: Use POSIX Extended regular expressions rather than
# basic.
#
#FilterExtended On

#
# FilterCaseSensitive: Use case sensitive regular expressions.
#
#FilterCaseSensitive On

#
# FilterDefaultDeny: Change the default policy of the filtering system.
# If this directive is commented out, or is set to "No" then the default
# policy is to allow everything which is not specifically denied by the
# filter file.
#
# However, by setting this directive to "Yes" the default policy becomes
# to deny everything which is _not_ specifically allowed by the filter
# file.
#
#FilterDefaultDeny Yes

#
# Anonymous: If an Anonymous keyword is present, then anonymous proxying
# is enabled.  The headers listed are allowed through, while all others
# are denied. If no Anonymous keyword is present, then all headers are
# allowed through.  You must include quotes around the headers.
#
# Most sites require cookies to be enabled for them to work correctly, so
# you will need to allow Cookies through if you access those sites.
#
#Anonymous "Host"
#Anonymous "Authorization"
#Anonymous "Cookie"

#
# Configure one or more ReversePath directives to enable reverse proxy
# support. With reverse proxying it's possible to make a number of
# sites appear as if they were part of a single site.
#
# If you uncomment the following two directives and run tinyproxy
# on your own computer at port 8888, you can access Google using
# http://localhost:8888/google/ and Wired News using
# http://localhost:8888/wired/news/. Neither will actually work
# until you uncomment ReverseMagic as they use absolute linking.
#
#ReversePath "/google/" "http://www.google.com/"
#ReversePath "/wired/"  "http://www.wired.com/"

#
# When using tinyproxy as a reverse proxy, it is STRONGLY recommended
# that the normal proxy is turned off by uncommenting the next directive.
#
#ReverseOnly Yes

#
# Use a cookie to track reverse proxy mappings. If you need to reverse
# proxy sites which have absolute links you must uncomment this.
#
#ReverseMagic Yes

#
# The URL that's used to access this reverse proxy. The URL is used to
# rewrite HTTP redirects so that they won't escape the proxy. If you
# have a chain of reverse proxies, you'll need to put the outermost
# URL here (the address which the end user types into his/her browser).
#
# If not set then no rewriting occurs.
#
#ReverseBaseURL "http://localhost:8888/"

And you start it with sudo tinyproxy -d -c /tmp/tinyproxy.conf

[andresriancho]

Confirmed that latest change only works with tinyproxy and not with w3af's internal proxy

[andresriancho]

With debugging enabled and in 4627bfc you see:

[pablo:/home/pablo/pch/w3af] [w3af] feature/poodle(+1/-7) 59s ± nosetests -s w3af/core/data/url/tests/test_xurllib_proxy.py:TestExtendedUrllibProxy.test_https_via_proxy
keepalive: added one connection,len(self._hostmap["127.0.0.1:8001"]): 1
keepalive: added one connection,len(self._hostmap["127.0.0.1:8001:8001"]): 1

Which is really ugly (note the 8001:8001!)

[andresriancho]

Yet another test that fails because of this proxy uglyness is w3af/plugins/attack/db/tests/test_sqlmap_wrapper.py:TestSQLMapWrapper.test_verify_vulnerability_ssl

[andresriancho]

Console 1:

mitmproxy -T -p 8888

Console 2:

nosetests -s w3af/plugins/attack/db/tests/test_sqlmap_wrapper.py:TestSQLMapWrapper.test_verify_vulnerability_ssl

Console 1:

Exception in thread SlaveThread (0.0.0.0:8888):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/home/pablo/pch/virtual-envs/w3af/local/lib/python2.7/site-packages/libmproxy/controller.py", line 77, in run
  File "/home/pablo/pch/virtual-envs/w3af/local/lib/python2.7/site-packages/netlib/tcp.py", line 479, in serve_forever
  File "/usr/lib/python2.7/socket.py", line 202, in accept
error: [Errno 24] Too many open files

[andresriancho]

mitmproxy/mitmproxy#383 , but works well with mitmproxy -p 8888

Something that's strange is that it seems that I'm checking SSL certs?

<mhils> ping __apr__
<__apr__> mhils, pong
<__apr__> that was quick
<__apr__> :)
<mhils> ah, excellent. :)
<__apr__> how may I help you?
* alener has quit (Quit: Leaving)
<mhils> answer my questions ;-)
<__apr__> sure
<mhils> Any specific reason why you are using mitmproxy in transparent mode?
<mhils> i.e. why not just `mitmproxy -p 8888`?
<__apr__> no special reason
<mhils> I haven't dug into the w3af codebase, but I suspect you're expecting a regular proxy and not a transparent one?
<__apr__> we're not using libmitm still
<__apr__> but we're in the process of doing so
<__apr__> https://github.com/andresriancho/w3af/issues/1269
<__apr__> https://github.com/andresriancho/w3af/pull/4165
<__apr__> and I was testing (for the first time) mitm from the console
<mhils> ok, what's the output if you just run mitmproxy without -T?
<__apr__> let me check
* mhils reads through the issues in the meantime
<__apr__> mhils, strange stuff, doesn't seem to work
<__apr__> might be a w3af thing
<__apr__> without the -T, I'm getting no error in the mitm console
<__apr__> but I don't see any traffic either
<mhils> anything in the mitmproxy event log? (press `e`)
<__apr__> w3af can't connect to the host, but wget can
<__apr__> this works
<__apr__> https_proxy=127.0.0.1:8888 wget https://127.0.0.1:8001/ --no-check-certificate
<__apr__> and I do see the traffic in the mitm console (for wget)
<mhils> there should be at least a clientconnect message in the console, otherwise we're not even seeing a TCP connection.
<__apr__> for w3af I see:
<__apr__> 127.0.0.1:54903: clientconnect
<__apr__> 127.0.0.1:54903: SSL handshake error: The client may not trust the proxy's certificate.
<__apr__> 127.0.0.1:54903: clientdisconnect
<mhils> so, I suspect you're not trusting the CA certificate :-)
<__apr__> I see that message three times, which I suspect is for the different ssl protos, let me show you
<mhils> located in ~/.mitmproxy/
<mhils> you probably want to replace the CA mitmproxy generates with the one you use for w3af
<__apr__> https://github.com/andresriancho/w3af/blob/feature/poodle/w3af/core/data/url/handlers/keepalive/connections.py#L116
<mhils> we generate a unique CA on every user's machine, w3af doesn't know that one, you're getting an SSL error because w3af rejects the proxy.
<mhils> if you're starting mitmproxy from the command line, you'll be using the cert in ~/.mitmproxy.
<__apr__> hmmm
<__apr__> I believe I'm not checking certs
<__apr__> give me a sec
<mhils> does it work if you use `mitmproxy --ignore .*` ?
<mhils> if so, you're checking certs.
<__apr__> oh, it seems to work
<__apr__> mitmproxy --ignore '.*' -p 8888 
<__apr__> that's the command you wanted me to run?
<__apr__> what's the ignore?
<mhils> http://mitmproxy.org/doc/features/passthrough.html
<mhils> basically not intercepting SSL connections and passing everything through as-is.
<mhils> in other words, if that works, your client is most likely checking cers.
<mhils> certs*
<__apr__> hmmm
<__apr__> the test server I'm connecting to through the proxy doesn't have a valid ssl cert
<__apr__> [pablo:/home/pablo/pch/w3af] [w3af] feature/poodle(+2/-2,1) 5 ± wget https://127.0.0.1:8001/
<__apr__> --2015-02-03 12:31:50--  https://127.0.0.1:8001/
<__apr__> Connecting to 127.0.0.1:8001... connected.
<__apr__> ERROR: cannot verify 127.0.0.1's certificate, issued by ‘/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd’:
<__apr__>   Self-signed certificate encountered.
<__apr__>     ERROR: certificate common name ‘’ doesn't match requested host name ‘127.0.0.1’.

https://github.com/andresriancho/w3af/blob/feature/poodle/w3af/core/data/url/openssl/ssl_wrapper.py#L112

[andresriancho]

<__apr__> ok, so...
<__apr__> the thing is that w3af is wrapping sqlmap in this test
<__apr__> and when wrapping sqlmap it runs it like this:
<__apr__> /home/pablo/pch/virtual-envs/w3af/bin/python ./w3af/plugins/attack/db/sqlmap/sqlmap.py --output-dir=/tmp --batch --disable-coloring --proxy=http://127.0.0.1:8888/ --batch --url=https://127.0.0.1:8001/audit/sql_injection/where_string_single_qs.py?uname=pablo
<__apr__> when run as expected by w3af, this will proxy all sqlmap's traffic via w3af's proxy
<__apr__> where we can add custom headers
<__apr__> cookies, etc
<__apr__> during these tests we're routing this via mitmproxy
<__apr__> and for some reason sqlmap (not w3af, that code I mentioned before doesn't have anything to do with this test)
<__apr__> doesn't like mitmproxy
<__apr__> it might be checking certs, but I doubt it
<__apr__> to sum up:
<__apr__> 1- sqlmap and mitmproxy don't seem to get along unless we --ignore
<__apr__> 2- there is still a bug in mitmproxy with -T, but I don't really needed that option
<__apr__> 3- mitmproxy support rocks :)

[andresriancho]

This bug will be fixed at Complete libmproxy implementation #10429