Node.js requests don't display in Proxyman

[fishcharlie]

Proxyman version? (Ex. Proxyman 1.4.3)

Proxyman 1.4.7

macOS Version? (Ex. mac 10.14)

macOS 10.15 Beta (19A526h)

Steps to reproduce

1. Open Proxyman
2. Create request in Node.js (ex. https://github.com/axios/axios)
3. Run Node.js script
4. Observe that request doesn't display in Proxyman

Expected behavior

Request from Node.js to display in Proxyman

Screenshots (optional)

NA

[NghiaTranUIT]

Hey, thank for the report. I will check it out today 👍

[NghiaTranUIT]

Hey @fishcharlie, I've checked and the answer is that we have to explicitly point the axios to Proxyman.

const axios = require('axios');

// Proxy to Proxyman
axios.defaults.proxy = {
    host: '127.0.0.1',
    port: 9090,
}

// Make a request for a user with a given ID
axios.get('https://httpbin.org/get?data=123&value=nghia')
  .then(function (response) {
    // handle success
    console.log(response.data);
  })
  .catch(function (error) {
    // handle error
    console.log(error);
  })
  .finally(function () {
    // always executed
  });

then, the request will appear in Proxyman

It's relevant to #175 (comment), since some CLI doesn't use System Proxy until we explicitly override it.

Additionally, we have to do same for Charles Proxy too.

Hope that could help you 😄 🌮

[NghiaTranUIT]

If you don't want to explicitly point the axios to Proxyman, we can use

$ export http_proxy=http://127.0.01:9090
$ export https_proxy=http://127.0.01:9090

Then all CLIs (curl, nodejs, ...) will automatically proxy to Proxyman, but make sure we remove it when we don't use Proxyman. Or the app will be failed 😄

[shirshak55]

actually its working on mine. If we use terminal it wont work directly we need to explicitly say axios to use proxy and that is how it work in other cli apps too :) Every cli supports proxy like curl etc.. And as u have already given implicit way using environment variable its fine.

But please add it in request so other don't get same problem.

[alnorris]

I'm getting this when ever I try both the environmental vars and the axios proxy. Any ideas?

Error: socket hang up
    at connResetException (internal/errors.js:610:14)
    at Socket.socketOnEnd (_http_client.js:453:23)
    at Socket.emit (events.js:327:22)
    at Socket.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ECONNRESET',

[shirshak55]

@alnorris it can be your server.

[NghiaTranUIT]

@alnorris it might be your port is different than the listening port of Proxyman ( default is 9090)

You can go to proxyman Preference and see the port, then setting it correctly

[NghiaTranUIT]

If you don’t mind, pls do the same config with other proxy tools to see where the bug is 😄 because there are many hidden factors could affect

[dvcrn]

Is there a workaround to enable the proxy for everything? I sometimes have no influence on what the node app is doing, so having an option to transparently proxy everything would be nice.

Right now I am falling back to mitmproxy which has a workaround using pfctl and transparent mode to route traffic system-wide to the proxy. Works pretty good, but if Proxyman could do that out of the box, it would be reaaaally nice. Setup like this: https://docs.mitmproxy.org/stable/howto-transparent/#macos

[shirshak55]

@dvcrn it still wont work if they enable certs pinning (Like zoom app enable cert pinning). And transparent proxy is different thing than http/https proxy. And regarding node app you will have source code right so you can change proxy easily from source code? But yes transparent proxy can help in many situations like some app that don't obey proxy but it still isn't easy today due to certs pinning, android certs being hard to place, and we can forget ios app as most of them use certs pinning anyway. I know we can use frida but thats not easy.

[NghiaTranUIT]

I haven't tested pfctl, but wondering what happens if we use 9090 in the following script

rdr pass on en0 inet proto tcp to any port {80, 443} -> 127.0.0.1 port 9090

9090 is where Proxyman serves

[dvcrn]

I don't always have access to the source. For example I was pentesting one mac app but it just wasn't showing up in proxyman/charles and I had no idea why. It took me a while to figure out that the app was spawning a node worker in the background that does the connection and that's why nothing was showing up, which got me wondering what else is happening that I'm missing with Proxyman.

Using mitmproxy + pfctl worked fine even for SSL after trusting the certificate globally

[shirshak55]

@dvcrn did u use pfctl to forward to proxyman? BSD uses pf firewall which is a bit different than iptables unfortunately.

[NghiaTranUIT]

He forwarded it to mitmproxy @shirshak55. Btw, it's a cool idea to use pfctl to do transparent Proxy.

From what I see, if pfctl can work with Proxyman, we can fix the VPN issue 😄

[shirshak55]

@NghiaTranUIT pfctl is just a firewall :) And you should have transparent proxy support on proxyman (I think at the moment proxyman supports Http/Https proxy only). pfctl isn't any magic its like iptables for osx. Buy the way if you land transparant proxy soon let me know I can help you with docs regarding how to connect andriod with that transparent proxy.

[NghiaTranUIT]

There is a workaround by using proxychains to swamp your app under proxychain umbrella , and all traffic will appear on Proxyman. Read more at #544

It's quite difficult to set up and requires to disable SIP on your mac, so I would not recommend this approach.

[shirshak55]

@dvcrn I don't know why you are saying charles proxy doesn't work. Charles proxy has transparent proxy so it should have covered your usecase. Its hidden in ssl settings second tab. I haven't used it for year probably but charles proxy does support transparent proxy.

[Cykelero]

Seems like Node.js doesn't respect the http_proxy/https_proxy variables; instead, you need to use something like global-agent. This is what ended up working in my case.

Step by step:

1. Install global agent: npm install global-agent
2. Import it in your code: require("global-agent/bootstrap") (or see their docs for how to inject it from the CLI)
3. Set its env variable to Proxyman's server: export GLOBAL_AGENT_HTTP_PROXY=http://127.0.01:9090

This should be it!

For those interested, there are lengthy issues on the subject of adding support for the standard proxy vars to Node: nodejs/node#8381, nodejs/node#15620

[NghiaTranUIT]

Thank you so much for the contribution 🎉 I will add your comment to official Proxyman doc

[CyberMew]

Has this been fixed yet such that even javascript requests in browsers or some apps which I don't have code access to will work, and have the network requests show up?

[NghiaTranUIT]

@CyberMew it's not a bug, so I can't fix it. It's how NodeJS works.

If you could not access the source code to change the HTTP Proxy, please try to override on your terminal.

export http_proxy=127.0.0.1:9090
export HTTP_PROXY=127.0.0.1:9090
export https_proxy=127.0.0.1:9090
export HTTPS_PROXY=127.0.0.1:9090

It is not 100% work if your app doesn't respect the env http_proxy

Ref: https://www.serverlab.ca/tutorials/osx/administration-osx/configuring-a-network-proxy-for-osx/

[djpowers]

I wanted to check if people have since found a workaround for this. I had tried exporting HTTP(S) proxy variables, and using the global-agent approach, but didn't have any luck. Have people been successful with any other techniques?

Also it looks like the issues referenced above about adding proxy support to Node have been closed, so I'm wondering if people are aware of any other discussions about making that happen.

[NghiaTranUIT]

@Geczy can you share with me how your setup your Electron app 🤔

You have the source code, which allows you to start the Electron app from the Terminal, right? Or you'd like to capture traffic from other Electron app?

---

From what I know, I need to support ElectronJS. The above Beta build is only for NodeJS, Ruby, and Python.

[Geczy]

well previously in the production release, the electron app did not even display in proxyman as an "app" or any of the domains or anything. it's only captured if i use the terminal that proxyman opens to open the app.

i start this app by doing open /System/Volumes/Data/Users/matt/Applications/xyz.app in the spawned terminal

in this build, it does display. the electron app uses axios under the hood, if that matters. i don't have the source code for it.

[NghiaTranUIT]

I see, if you don't mind, can you share with me the name of your Electron app via [email protected] ? I'd like to download production build, test it and fix it 👍

[Geczy]

sure, i sent you the email. thanks for taking a look. super sexy app

[NghiaTranUIT]

New Beta build 🎉

#1611 (comment)

[NghiaTranUIT]

Just a friendly reminder that we've released the automatic / manual setup 🎉

cc @gkatsanos @sonic1981 @fishcharlie @dvcrn @shirshak55 @Cykelero

You can checkout the latest build or from this comment: #236 (comment)

[gkatsanos]

Thank you!

[NghiaTranUIT]

@Geczy I've fixed the ElectronJS issue, please get the Beta build from #1618 (comment)

[Geczy]

Can confirm it's fixed !

[ChiuMungZitAlexander]

My question is how can I disable it?

[NghiaTranUIT]

@ChiuMungZitAlexander Automatic Script doesn't start if you don't open the Terminal app. It only affects the current Terminal Session, not permanently change your OS. So, no need to disable it.

[gkatsanos]

Hey @NghiaTranUIT :

version 3.6.1
----------------------------------------------
------PROXYMAN AUTOMATIC SETUP SCRIPTS--------
----------------------------------------------
⭐️ Auto capture HTTP/HTTPS traffic from this Terminal app.
Support NodeJS: (axios, fetch, got, request, superagent)
Support Ruby: (http, net/http, net/https)
Support Python: (http, httplib, httplib2)

✅ Injected Proxyman variable environments to current shell

👉 Please start your local server or run your script from this Terminal.
set: +a: invalid variable name. See `help identifiers`

(Type 'help set' for related documentation)
 ✘  py[learning]  ~ 

I get this error when trying to open up a new 'automatic setup' terminal. I use

 fish -v
fish, version 3.6.1

[gkatsanos]

another issue I have (unrelated?) is even though I see one request, I dont see another one which happens to an external API endpoint but which passes through my VPN client.. could this be the reason?

[NghiaTranUIT]

@gkatsanos yes it can be a problem. If you're using fish, when opening the Automatic Setup Terminal -> Can you print the echo $PATH, to see if it contains the Proxyman path?

It's where Proxyman overrides the NodeJS env.

➜  ~ echo $PATH            
/Applications/Proxyman.app/Contents/Frameworks/ProxymanCore.framework/Versions/A/Resources/overrides/path

The reason is: there is a small warning from your log.

set: +a: invalid variable name. See `help identifiers`

[gkatsanos]

 ✘  py[learning]  ~  echo $PATH         
/Applications/Proxyman.app/Contents/Frameworks/ProxymanCore.framework/Versions/A/Resources/overrides/path /Users/gkatsanos/Library/pnpm /Users/gkatsanos/.pyenv/shims /opt/homebrew/Cellar/pyenv-virtualenv/1.2.1/shims /Users/gkatsanos/.pyenv/bin /Users/gkatsanos/.nvm/versions/node/v16.19.0/bin /Users/gkatsanos/.rbenv/shims /Users/gkatsanos/.pyenv/bin /opt/homebrew/bin /usr/local/bin /Users/gkatsanos/.yarn/bin /usr/bin /bin /usr/sbin /sbin
 py[learning]  ~  

[leohxj]

add export NODE_TLS_REJECT_UNAUTHORIZED=0 after manual setup.

[NghiaTranUIT]

@leohxj a Beta build for you: https://download.proxyman.io/beta/Proxyman_4.11.0_Improve_UI_v1.dmg

Changelog

• Add export NODE_TLS_REJECT_UNAUTHORIZED=0

[leohxj]

thanks @NghiaTranUIT , it works well

[shirshak55]

actually I don't recommend this. Last week , I think from Egypt government was hacked because of http. Although, you may not be the target, I don't recommend messing with security.

I think proxy man should explicitly ask if they want to reduce this security?

[NghiaTranUIT]

@shirshak55 it doesn't globally modify your entire system. It only takes effect on the current Terminal app session when you need to debug your NodeJS project.

[leohxj]

yes, only for develop and use it locally

[tonyxiao]

Does this work with the built in node.js fetch now? Or is the node-fetch 3rd party lib required to use with proxyman?

[NghiaTranUIT]

It works for both. Native fetch and node-fetch

[tonyxiao]

Which env variable does native fetch use? When I look at the node docs it seems that fetch doesn't support proxy / agent agt all.

[NghiaTranUIT]

Auto Setup from Proxyman does more things than just the env. Proxyman uses global-agent to auto override the proxy of NodeJS Fetch in runtime.

You simply:

1. Open Proxyman, make sure you setup the certificate properly in Certificate menu -> Install for Mac.
2. Setup menu -> Auto -> start the Termnial.
3. Run your NodeJS script and this terminal and it's done ✅

[tonyxiao]

Hmm but the global-agent lib doesn't actually seem to support node 18 fetch. @see gajus/global-agent#52 Does proxyman use the undici workaround too?

[NghiaTranUIT]

Hmm but the global-agent lib doesn't actually seem to support node 18 fetch. @see gajus/global-agent#52 Does proxyman use the undici workaround too?

Yes, it's.

[NghiaTranUIT]

FYI: We can capture all HTTPS Traffic from NodeJS with the Auto Setup (https://docs.proxyman.io/debug-devices/nodejs)

No need to config the Proxy or manually trust the self-signed certificate on your NodeJS code 👍