Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tldsextract library causing SSL verify errors in MITM environment #1

Closed
liquidsec opened this issue Apr 2, 2022 · 1 comment
Closed

tldsextract library causing SSL verify errors in MITM environment #1

liquidsec opened this issue Apr 2, 2022 · 1 comment
Assignees
@TheTechromancer
Labels
bug Something isn't working

Comments

@liquidsec
Copy link
Collaborator

liquidsec commented Apr 2, 2022 •
edited
Loading

The tldsextract library is causing SSL verify errors even with sslverify=False in config when in an ssl-proxied (mitm) environment
As shown in the stack trace when operating in an environment where all traffic is being intercepted this module makes a call via requests to https://publicsuffix.org/list/public_suffix_list.dat and https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat. These calls do not respect the sslverify=False setting which the rest of bbot respects (such as web request helper).

At the end of the stack trace, --current-config was run to show that ssl_verify is currently false.

Issue appears to be rooted in the tldsextract library.

root@c0965812342f:/opt/bbot# poetry run bbot -d -m httpx -t https://icanhazip.com 
[INFO] bbot.cli: Command: bbot -d -m httpx -t https://icanhazip.com
[DBUG] bbot.core.event: Autodetected event type "URL" based on data: "https://icanhazip.com"
[INFO] bbot.scanner: Loading 1 modules: httpx
[INFO] bbot.scanner: Loaded module "httpx"
[SUCC] bbot.scanner: Loaded 1 modules
[INFO] bbot.scanner: Starting scan asdf
[INFO] bbot.scanner: Setting up modules
[DBUG] bbot.modules.httpx: Setting up module httpx
[DBUG] bbot.modules.httpx: Finished setting up module httpx
[INFO] bbot.scanner: Finished setting up modules
[INFO] bbot.scanner: Target: Event("URL", "https://icanhazip.com")
{"type": "URL", "data": "https://icanhazip.com", "module": "module", "source": "f762108ca727f0bcd961fc7467c248887dfa70dc:TARGET", "id": "baed85c983e6564d59e4ca4cbf62df4d78d3492d:URL", "tags": ["target"]}
[INFO] bbot.scanner: Starting modules
[INFO] bbot.scanner: 1 modules started
Exception reading Public Suffix List url https://publicsuffix.org/list/public_suffix_list.dat
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: publicsuffix.org-tlds key: {'urls': ('https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: urls key: {'url': 'https://publicsuffix.org/list/public_suffix_list.dat'}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='publicsuffix.org', port=443): Max retries exceeded with url: /list/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/suffix_list.py", line 30, in find_first_response
    return cache.cached_fetch_url(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 199, in cached_fetch_url
    return self.run_and_cache(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 192, in run_and_cache
    result = func(**kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 209, in _fetch_url
    response = session.get(url, timeout=timeout)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='publicsuffix.org', port=443): Max retries exceeded with url: /list/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
Exception reading Public Suffix List url https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat
Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: publicsuffix.org-tlds key: {'urls': ('https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 190, in run_and_cache
    result = self.get(namespace=namespace, key=key_args)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 93, in get
    raise KeyError("namespace: " + namespace + " key: " + repr(key))
KeyError: "namespace: urls key: {'url': 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'}"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 414, in connect
    self.sock = ssl_wrap_socket(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/local/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /publicsuffix/list/master/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/suffix_list.py", line 30, in find_first_response
    return cache.cached_fetch_url(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 199, in cached_fetch_url
    return self.run_and_cache(
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 192, in run_and_cache
    result = func(**kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/tldextract/cache.py", line 209, in _fetch_url
    response = session.get(url, timeout=timeout)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/root/.cache/pypoetry/virtualenvs/bbot-IFSyk-JB-py3.9/lib/python3.9/site-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /publicsuffix/list/master/public_suffix_list.dat (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')))
[VERB] bbot.scanner: Events queued: 0 (None)
[VERB] bbot.scanner: Tasks queued: 0 (None)
[DBUG] bbot.modules.httpx: Handling batch of 1 events for module "httpx"
[SUCC] bbot.scanner: Scan asdf completed with status FINISHED
root@c0965812342f:/opt/bbot# poetry run bbot --current-config
modules:
  httpx:
    allow_skip_portscan: true
  nuclei: {}
  sublist3r: {}
  sslcert:
    timeout: 4.0
  dnsresolve:
    max_hosts: 65536
  naabu: {}
  aspnet_viewstate: {}
  dnsx:
    wordlist: https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/DNS/subdomains-top1million-20000.txt
  wayback: {}
  dnsdumpster: {}
max_threads: 250
http_proxy: null
http_timeout: 30
ssl_verify: false
user_agent: Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15
  (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1
dns_wildcard_tests: 5
url_extension_blacklist:
- png
- jpg
- jpeg
- gif
- svg
- css
- woff
- woff2
- ttf

root@c0965812342f:/opt/bbot#
@liquidsec liquidsec added the bug Something isn't working label Apr 2, 2022
@TheTechromancer
Copy link
Collaborator

Fixed in cff6a7b

@timlaw71 timlaw71 mentioned this issue Dec 5, 2022
Closed
TheTechromancer added a commit that referenced this issue Aug 2, 2024
@TheTechromancer
Merge pull request #1603 from blacklanternsecurity/fix-dns-parents
2e571b7 
DEF CON Update #1
@colin-stubbs colin-stubbs mentioned this issue Aug 17, 2024
Open
@liquidsec liquidsec mentioned this issue Aug 24, 2024
Closed
@mikismaos mikismaos mentioned this issue Oct 17, 2024
Closed
TheTechromancer pushed a commit that referenced this issue Nov 17, 2024
@pixeebot
Use Safe Defaults for lxml Parsers (#1)
a54199b 
Co-authored-by: pixeebot[bot] <104101892+pixeebot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TheTechromancer TheTechromancer

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@TheTechromancer @liquidsec