HELP Getting Started

[gilbert-mjp]

Hi Everyone
thanks for the great work John. this might sound silly, but we need help getting started. we've followed the steps..working on windows 10, got node, python, mitmproxy and bleak setup with a minimal config file.
however when we run bleak, all we see is a blank black screen.
in mitmproxy, we also note
1270.0.1:.....t: clientconnect
1270.0.1:.....: clientdisconnect
immediately. nothing else. Can you please help?
sorry if this seems trivial, but we really need this working.

Many thanks in advance
Gilbert

[y-jono]

Running BLeak on Windows 10 is difficult. There are multiple reasons.

1. mitmproxy supports operation on WSL (Linux emulator on Windows 10).
2. mitmproxy does not plan to support Windows native binaries.
3. It is necessary to hook up communication with mitmproxy (on WSL) by connecting BLeak node process and chrome process.

At the end of last year, I worked on these issues for several days. However, it was easier to run BLeak with Linux on VM than working on the above task, so I gave up.
If you are comfortable working with Linux or macOS, I recommend working with those operating systems.

(This sentence was created by Google translation.)

[gilbert-mjp]

Oh wow. thanks for the insights. i will give the Linux VM a try, nevertheless you shouldn't give up :-)
many windows developers out there who would likely benefit from this working.
thanks all the same, i'll get back to you with the results

[gilbert-mjp]

Hi
I tried with Ubuntu. Here is what i get when i run bleak

MITMProxy not running; starting up mitmproxy.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 574, in _build_master
ws.require(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 892, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 783, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (urwid 2.0.1 (/usr/lib/python3/dist-packages), Requirement.parse('urwid<1.4,>=1.3.1'), {'mitmproxy'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/bin/mitmdump", line 6, in
from pkg_resources import load_entry_point
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3088, in
@_call_aside
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3072, in _call_aside
f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 3101, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 576, in _build_master
return cls._build_from_requirements(requires)
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 589, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/lib/python3/dist-packages/pkg_resources/init.py", line 778, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'urwid<1.4,>=1.3.1' distribution was not found and is required by mitmproxy
(node:1694) UnhandledPromiseRejectionWarning: Error: Unable to start mitmproxy: 1
at Function. (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:336:35)
at Generator.throw ()
at rejected (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:5:65)
at
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
UnhandledPromiseRejectionWarning: Error: Unable to start mitmproxy: 1
at Function. (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:336:35)
at Generator.throw ()
at rejected (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:5:65)
at
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
at emitWarning (internal/process/promises.js:65:17)
at emitPendingUnhandledRejections (internal/process/promises.js:109:11)
at process._tickDomainCallback (internal/process/next_tick.js:229:7)
(node:1694) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
Error: Unable to start mitmproxy: 1
at Function. (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:336:35)
at Generator.throw ()
at rejected (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:5:65)
at
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
(node:1694) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
at emitWarning (internal/process/promises.js:92:15)
at emitPendingUnhandledRejections (internal/process/promises.js:109:11)
at process._tickDomainCallback (internal/process/next_tick.js:229:7)

Any guidance on this?

[jvilk]

It looks like the version of mitmproxy we depend on has a dependency that's not satisfied:

Requirement.parse('urwid<1.4,>=1.3.1'), {'mitmproxy'})

So, mitmdump isn't starting up at all. What happens if you run mitmdump --help?

[jvilk]

If mitmdump fails with an error, you might want to consider looking at installation instructions for your OS:

https://mitmproxy.org/

[jvilk]

Actually, it looks like you'll have to install mitmproxy via pip:

jvilk/mitmproxy-node#7

pip install mitmproxy (or pip3 install mitmproxy if pip is python 2)

[gilbert-mjp]

Hi John
thanks for the help. i tried running both installs for mitmproxy..but still get nowhere...
Here is the output this time WebSockets error pops up..

gibbs@MJP:~$ bleak run --config ./Documents/bleakconfig.js --out bleak-output --debug
MITMProxy not running; starting up mitmproxy.
Loading script: /usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/scripts/proxy.py
Script error: Traceback (most recent call last):
File "/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/scripts/proxy.py", line 16, in
import websockets
ModuleNotFoundError: No module named 'websockets'

mitmdump: Invalid replacement specifier: onlyInterceptTextFiles=false
(node:6291) UnhandledPromiseRejectionWarning: Error: Unable to start mitmproxy: 1
at Function. (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:336:35)
at Generator.throw ()
at rejected (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:5:65)
at
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
UnhandledPromiseRejectionWarning: Error: Unable to start mitmproxy: 1
at Function. (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:336:35)
at Generator.throw ()
at rejected (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:5:65)
at
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
at emitWarning (internal/process/promises.js:65:17)
at emitPendingUnhandledRejections (internal/process/promises.js:109:11)
at process._tickDomainCallback (internal/process/next_tick.js:229:7)
(node:6291) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
Error: Unable to start mitmproxy: 1
at Function. (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:336:35)
at Generator.throw ()
at rejected (/usr/local/lib/node_modules/bleak-detector/node_modules/mitmproxy/dist/index.js:5:65)
at
at process._tickDomainCallback (internal/process/next_tick.js:228:7)
(node:6291) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
at emitWarning (internal/process/promises.js:92:15)
at emitPendingUnhandledRejections (internal/process/promises.js:109:11)
at process._tickDomainCallback (internal/process/next_tick.js:229:7)

[gilbert-mjp]

It looks like the version of mitmproxy we depend on has a dependency that's not satisfied:

Requirement.parse('urwid<1.4,>=1.3.1'), {'mitmproxy'})

So, mitmdump isn't starting up at all. What happens if you run mitmdump --help?

mitmdump --help produces the following...

gibbs@MJP:~$ mitmdump --help
usage: mitmdump [options] [filter]

positional arguments:
filter Filter view expression, used to only show flows that
match a certain filter. See help in mitmproxy for
filter expression syntax.

optional arguments:
-h, --help show this help message and exit
--conf PATH Configuration file
--version
--shortversion show program's short version number and exit
--anticache Strip out request headers that might cause the server
to return 304-not-modified.
--cadir CADIR Location of the default mitmproxy CA files.
(~/.mitmproxy)
--host Use the Host header to construct URLs for display.
-q, --quiet Quiet.
-r RFILE, --read-flows RFILE
Read flows from file.
-s "script.py --bar", --script "script.py --bar"
Run a script. Surround with quotes to pass script
arguments. Can be passed multiple times.
-t FILTER, --stickycookie FILTER
Set sticky cookie filter. Matched against requests.
-u FILTER, --stickyauth FILTER
Set sticky auth filter. Matched against requests.
-v, --verbose Increase log verbosity.
-w STREAMFILE, --wfile STREAMFILE
Write flows to file.
-a STREAMFILE, --afile STREAMFILE
Append flows to file.
-z, --anticomp Try to convince servers to send us un-compressed data.
-Z SIZE, --body-size-limit SIZE
Byte size limit of HTTP request and response bodies.
Understands k/m/g suffixes, i.e. 3m for 3 megabytes.
--stream SIZE Stream data to the client if response body exceeds the
given threshold. If streamed, the body will not be
stored in any way. Understands k/m/g suffixes, i.e. 3m
for 3 megabytes.
--upstream-auth UPSTREAM_AUTH
Add HTTP Basic authentcation to upstream proxy and
reverse proxy requests. Format: username:password
--keepserving Continue serving after client playback or file read.
We exit by default.
-d, --detail Increase flow detail display level. Can be passed
multiple times.

Proxy Modes:
-R REVERSE_PROXY, --reverse REVERSE_PROXY
Forward all requests to upstream HTTP server:
http[s]://host[:port]. Clients can always connect both
via HTTPS and HTTP, the connection to the server is
determined by the specified scheme.
--socks Set SOCKS5 proxy mode.
-T, --transparent Set transparent proxy mode.
-U UPSTREAM_PROXY, --upstream UPSTREAM_PROXY
Forward all requests to upstream proxy server:
http://host[:port]

Proxy Options:
-b ADDR, --bind-address ADDR
Address to bind proxy to (defaults to all interfaces)
-I HOST, --ignore HOST
Ignore host and forward all traffic without processing
it. In transparent mode, it is recommended to use an
IP address (range), not the hostname. In regular mode,
only SSL traffic is ignored and the hostname should be
used. The supplied value is interpreted as a regular
expression and matched on the ip or the hostname. Can
be passed multiple times.
--tcp HOST Generic TCP SSL proxy mode for all hosts that match
the pattern. Similar to --ignore, but SSL connections
are intercepted. The communication contents are
printed to the log in verbose mode.
-n, --no-server Don't start a proxy server.
-p PORT, --port PORT Proxy service port.
--no-http2
--http2 Explicitly enable/disable HTTP/2 support. HTTP/2
support is enabled by default.
--http2-priority
--no-http2-priority Explicitly enable/disable PRIORITY forwarding for
HTTP/2 connections. PRIORITY forwarding is disabled by
default, because some webservers fail at implementing
the RFC properly.
--no-websocket
--websocket Explicitly enable/disable WebSocket support. WebSocket
support is enabled by default.
--raw-tcp
--no-raw-tcp Explicitly enable/disable experimental raw tcp
support. Disabled by default. Default value will
change in a future version.
--spoof-source-address
Use the client's IP for server-side connections.
Combine with --upstream-bind-address to spoof a fixed
source address.
--upstream-bind-address UPSTREAM_BIND_ADDRESS
Address to bind upstream requests to (defaults to
none)

SSL:
--cert SPEC Add an SSL certificate. SPEC is of the form
"[domain=]path". The domain may include a wildcard,
and is equal to "*" if not specified. The file at path
is a certificate in PEM format. If a private key is
included in the PEM, it is used, else the default key
in the conf dir is used. The PEM file should contain
the full certificate chain, with the leaf certificate
as the first entry. Can be passed multiple times.
--ciphers-client CIPHERS_CLIENT
Set supported ciphers for client connections. (OpenSSL
Syntax)
--ciphers-server CIPHERS_SERVER
Set supported ciphers for server connections. (OpenSSL
Syntax)
--client-certs CLIENTCERTS
Client certificate file or directory.
--no-upstream-cert Don't connect to upstream server to look up
certificate details.
--add-upstream-certs-to-client-chain
Add all certificates of the upstream server to the
certificate chain that will be served to the proxy
client, as extras.
--insecure Do not verify upstream server SSL/TLS certificates.
--upstream-trusted-cadir SSL_VERIFY_UPSTREAM_TRUSTED_CADIR
Path to a directory of trusted CA certificates for
upstream server verification prepared using the
c_rehash tool.
--upstream-trusted-ca SSL_VERIFY_UPSTREAM_TRUSTED_CA
Path to a PEM formatted trusted CA certificate.
--ssl-version-client {all,secure,SSLv2,SSLv3,TLSv1,TLSv1_1,TLSv1_2}
Set supported SSL/TLS versions for client connections.
SSLv2, SSLv3 and 'all' are INSECURE. Defaults to
secure, which is TLS1.0+.
--ssl-version-server {all,secure,SSLv2,SSLv3,TLSv1,TLSv1_1,TLSv1_2}
Set supported SSL/TLS versions for server connections.
SSLv2, SSLv3 and 'all' are INSECURE. Defaults to
secure, which is TLS1.0+.

Onboarding App:
--no-onboarding Disable the mitmproxy onboarding app.
--onboarding-host ONBOARDING_HOST
Domain to serve the onboarding app from. For
transparent mode, use an IP when a DNS entry for the
app domain is not present. Default: mitm.it
--onboarding-port 80 Port to serve the onboarding app from.

Client Replay:
-c PATH, --client-replay PATH
Replay client requests from a saved file.

Server Replay:
-S PATH, --server-replay PATH
Replay server responses from a saved file.
-k, --replay-kill-extra
Kill extra requests during replay.
--server-replay-use-header SERVER_REPLAY_USE_HEADERS
Request headers to be considered during replay. Can be
passed multiple times.
--norefresh Disable response refresh, which updates times in
cookies and headers for replayed responses.
--no-pop Disable response pop from response flow. This makes it
possible to replay same response multiple times.
--replay-ignore-content
Ignore request's content while searching for a saved
flow to replay
--replay-ignore-payload-param SERVER_REPLAY_IGNORE_PAYLOAD_PARAMS
Request's payload parameters (application/x-www-form-
urlencoded or multipart/form-data) to be ignored while
searching for a saved flow to replay. Can be passed
multiple times.
--replay-ignore-param SERVER_REPLAY_IGNORE_PARAMS
Request's parameters to be ignored while searching for
a saved flow to replay. Can be passed multiple times.
--replay-ignore-host Ignore request's destination host while searching for
a saved flow to replay

Replacements:
Replacements are of the form "/pattern/regex/replacement", where the
separator can be any character. Please see the documentation for more
information.

--replace PATTERN Replacement pattern.
--replace-from-file PATH
Replacement pattern, where the replacement clause is a
path to a file.

Set Headers:
Header specifications are of the form "/pattern/header/value", where the
separator can be any character. Please see the documentation for more
information.

--setheader PATTERN Header set pattern.

Proxy Authentication:
Specify which users are allowed to access the proxy and the method used
for authenticating them.

--nonanonymous Allow access to any user long as a credentials are
specified.
--singleuser USER Allows access to a a single user, specified in the
form username:password.
--htpasswd PATH Allow access to users specified in an Apache htpasswd
file.
gibbs@MJP:~$