Cypress behind corporate proxy with cross-domain static content times out

[Carst-Tankink]

This is probably one of the instances of problems caused by cypress behind a corporate proxy (#1469), but I haven't seen a bug report that is similar to this, so let me file it here :-)

Current behavior:

We are running cypress in a docker container (started from jenkins) behind a corporate proxy. I am passing the proper http_proxy, https_proxy and no_proxy env variables.

As a minimal test case, we are doing:

const logo = '[data-test="main-logo"]';

describe('basic test', function () {
    it('Connect to test via direct url', function(){
        cy.visit('https://www.bol.com/nl/');
        cy.get(logo).should('be.visible');
    });
});

This fails with a timeout, but it does seem able to fetch the HTML (see screenshot)

I think the problem comes from the fact that we are serving our static content, including javascript, fonts and css, from a different superdomain (https://s.s-bol.com). In the debug logging of cypress, we see repeats of this:

07:24:55 Mon, 06 Aug 2018 05:24:55 GMT cypress:server:server Got CONNECT request from s.s-bol.com:443
07:24:55 Mon, 06 Aug 2018 05:24:55 GMT cypress:https-proxy Writing socket connection headers for URL: s.s-bol.com:443
07:24:55 Mon, 06 Aug 2018 05:24:55 GMT cypress:server:cors Parsed URL { port: '443', tld: 'com', domain: 's-bol' }
07:24:55 Mon, 06 Aug 2018 05:24:55 GMT cypress:server:server HTTPS request does not match URL: https://s.s-bol.com:443 with props: { port: '443', tld: 'com', domain: 'bol' }
07:24:55 Mon, 06 Aug 2018 05:24:55 GMT cypress:https-proxy Making direct connection to s.s-bol.com:443

With the direct connection not being passed through our proxy, and eventually timing out. This is supported by the fact that a slightly different test setup where the static content is served from the superdomain do pass. The setup of our proxy seems okay: when I do a curl for static content from the same docker container, it passes through (and times out when the proxy-settings are absent)

Desired behavior:

The basic test succeeds

Versions

Cypress: 3.0.3
OS: Linux (dockerized)
Browser: Electron

If there is anything you want me to try or add, please ask! We are absolutely loving cypress for testing standalone components in isolation and really want to roll it out to doing integration tests.

[promosjotte]

i have the same problem :(

[timpur]

Ive found that forcing this line to be true cypress/packages/server/lib/server.coffee:179 isMatching=true, means all traffic goes through the cypress proxy and through corporate proxy (if you start cypress with node proxy env's set)

So i believe this supports the above statements, that doing a direct connection means that the proxy env settings are not being respected. Doing a truely "direct direct" connection is not right, we need a "direct via corp proxy" :P

Now i have no idea how to test this in cypress as i only just picked up the code yesterday :P so looking for some help !!!, id really like to get this working asap.

If anyone can point me in the right direction, please and thanks

[jphchen]

I'm having the same issue, 504 timeout when trying to download cdn

• behind corporate firewall
• testing e2e on server not localhost
• server trying to download google api from cdn
• server getting http status code 504 for cdn downloads
• env and npm config all contains proxy https_proxy and noproxy

[timpur]

Did some more digging and i think issue is in packages/https-proxy/lib/server.coffee:72 _makeDirectConnection. think this makes a direct connection but not respecting the current env proxy settings thus doing a direct to gateway connection which wont work and it needs to do a direct to proxy connection for it to work. That said i couldn't get it to work, dont know the code well enough ....

https://github.com/cypress-io/cypress/blob/develop/packages/https-proxy/lib/server.coffee#L72

if you dont do a direct connection the lib uses https://www.npmjs.com/package/request which has env proxy stuff built in. Thus if you do this cypress/packages/server/lib/server.coffee:179 isMatching=true it uses the request lib which automatically picks up on the proxy settings from envs and does the right thing (sadly i think this is a accidental feature from cypress).

https://github.com/cypress-io/cypress/blob/develop/packages/server/lib/server.coffee#L186

My question is why not just use the request lib everywhere since it does the right thing ?

Wondering when this will be fixed? @jennifer-shehane. Is there any place i can go to get more depth details on the workings of cypress internals to hopefully help you out ?

[Gonzo17]

With the workaround @timpur described we proofed that it's not a problem of our proxy. But having this manual change in the code wouldn't allow us to update if there is a newer version of Cypress, so this is a blocker for us to use Cypress. Are there plans to fix this issue?

[meinaart]

You can fix this by using redsocks transparent proxy. But that's quite a bit of work to configure properly. But it does work.

[lflpowell]

Same issue

[lflpowell]

Just to confirm that @timpur's workaround fixes the problem for us - keen to know if this is on the teams radar as currently it's a blocker stopping us from using Cypress.

[lflpowell]

Another update - just to confirm for us this seems to be around using cdn assets with https protocol. Non-https assets are working fine.

[fadomire]

same issue here, direct connect should respect proxy settings i guess

[leepowelldev]

Any movement on this? Currently it's blocking us adopting in house for a number of projects.

[jennifer-shehane]

@Carst-Tankink I am able to run the test with the url from the screenshot https://www.test2.bol.com/nl/ in cypress open and the visit command does timeout, but I'm also unable to load this website in a normal browser. It spins forever trying to connect - so this doesn't appear to be specific to Cypress.

If you have a new example test I am supposed to be running, please comment here.

Possible related issue: #1039
Possible lead on where the problem is: #2295 (comment)

[leepowelldev]

Seems odd that the current logic is to not respect the proxy on direct connections. I would suggest this could be decided by the user with the NO_PROXY environment variable if they wanted that to happen.

[ddesna]

Having the same problem behind corporate proxy.
Resources from CDN: CSS and JS files wan't load in Chrome runned by Cypress due to ERR_TIMED_OUD.
If CDN domains is defined in cypress.json as blacklistHosts the error is 503.

In our case is blocking because our AngularJS app is using Google Maps library that must be loaded.

[Carst-Tankink]

@jennifer-shehane Could you try with the url https://www.bol.com/nl/? I've edited the original to contain that test (I accidentally included www.test2.bol.com, which only resolves through our internal dns :) )

For us, this issue isn't that pressing anymore, as we're using the solution/workaround that @meinaart mentioned.

[maapteh]

Same issue like mentioner of this bug report. We are behind proxy PAC and Cypress is not respecting the node request http(s)_proxy settings at all (see the above pointer from @timpur). It is an issue for some years now (#672) and you could have many more adopters when fixed.

[leepowelldev]

@maapteh - Sadly we've had to look elsewhere now because of the slowness to fix these proxy issues.

[ddesna]

In my case it was only Google Maps that tried to load from Google CDN.
Workaround was:

• Add CDN hosts to blacklistHosts in cypress.json
• Add new command (or inside beforeEach hook) google maps api definition, like:

cy.on("window:before:load", win => {
        win.google = {
            maps: {
                places: {
                    Autocomplete: {}
                }
            }
        }
      })

In my case it's Angular 1.x app that need to load Google Maps API on startup, otherwise it's failing.

[maapteh]

Thats a workaround but it will not work if your assets are needed for your app to bootstrap (Angular, React, Vue, Etc) :) And @leepowellcouk it seems from a quick glance PR #3531 is fixing it, but i don't understand why you would OS a JavaScript project written in Cofee script, less people will add fixes.

[cypress-bot]

The code for this is done in cypress-io/cypress#3531, but has yet to be released.
We'll update this issue and reference the changelog when it's released.

[leepowelldev]

Just tried this on the develop branch - and sadly the updates don't seem to fix this issue.

[leepowelldev]

Seems that if proxy settings are set in uppercase things don't work, so have to to use lowercase:

// Good
http_proxy= ...
https_proxy= ...

// Bad
HTTP_PROXY=...
HTTPS_PROXY=...

[flotwig]

@leepowellcouk Which part of it isn't working with uppercase variable names? It was tested with the expectation that users would use uppercase variable names, so it should work. Can you let me know what type of URL you're trying to load (HTTP/HTTPS) and your operating system?

[leepowelldev]

@flotwig Requests to external assets over http fail with a timeout. Operating system is OS X. Changing to lowercase and it works for me.

[flotwig]

@leepowellcouk This is the line of code that decides if your request gets sent through an HTTP proxy or just sent normally:

cypress/packages/network/lib/agent.ts

Line 153 in c1a345d

if (process.env.HTTP_PROXY) {

All HTTP requests go through this as of #3531. It checks for HTTP_PROXY, not http_proxy - this is actually a bug, and I filed an issue to fix it at #3865, but it also means that the behavior you're seeing should not be possible in develop. Only uppercase environment variable names should send traffic through the proxy, so this is weird that it's only working with lowercase for you.

Are you sure you've pulled down the latest develop branch? How are you invoking Cypress?

[leepowelldev]

This is one of the assets we are trying to access:
http://cdn.aviva.com/static/core/v.4.7.0/css/customer/core-split.css

As you can see network requests fail when using uppercase:

However, lowercase works as expected:

[leepowelldev]

Yes I'm definitely on develop - I cloned your repo, built and ran. Then ran my tests from the resulting instance.

[flotwig]

Can you share the debug logs from that failing run? It might help to shed some light on the issue. Just run your Cypress command like this:

DEBUG=cypress:* npm run cypress:open

[leepowelldev]

Sure - do you want me to post it here, or email you?

[flotwig]

Here should be fine.

[flotwig]

@leepowellcouk Got your log, it looked you might have accidentally included some identifying information in it so I deleted your comment. Looking over it now. Thank you for helping to debug this issue!

[flotwig]

@leepowellcouk I think that there's an http_proxy environment variable set which is causing this bug. Do you also have an http_proxy variable set to http://127.0.0.1:8080, in addition to the HTTP_PROXY you're setting? Can you try unsetting any environment variables that might be interfering?

unset http_proxy https_proxy no_proxy HTTP_PROXY HTTPS_PROXY NO_PROXY
HTTP_PROXY=... HTTPS_PROXY=... NO_PROXY=... npm run cypress:open

[leepowelldev]

Ah sorry, I had all those disabled this morning when I was trying to get this to work - give me a sec and I'll run it again with all those disabled - do you have an email I can send the new log to?

[leepowelldev]

Ah, that's it ... I could of sworn I had that disabled this morning! Working as expected now. Thanks for taking the time to look at.

[flotwig]

Great! Glad you got it working. #3866 will make this less confusing, as it will always use the lowercase variables if they're available before using the uppercase ones.

[jennifer-shehane]

@flotwig Would be great to maybe add this as part of the new Proxy Configuration docs - just how to list the environment variables that are set on your machine.

[cypress-bot]

Released in 3.3.0.

[xlenz]

Quite useless with no support of automatic configuration.

Proxy Auto-Configuration (PAC) files are not currently supported.

but is really a good starting point for sure

[xlenz]

Btw, this fix doesn't work for me at all. It is even worse than before, loading of site is not even started, getting ERR_CONNECTION_CLOSED

Running tests like this HTTP_PROXY=http://****:3128 cypress run

Before it was working bad, slowly but at least somehow by running http_proxy=http://****:3128 cypress run

[leepowelldev]

Same here - this fix made things worse for us. We get a websocket connection error when trying to connect to a StoryBook instance:

cypress_runner.js:176209 WebSocket connection to 'ws://localhost:9001/__socket.io/?EIO=3&transport=websocket' failed: Connection closed before receiving a handshake response

[flotwig]

@xlenz We decided not to add PAC support as it would greatly increase the complexity of the implementation, and as you've discovered, it's trivial to just use your proxy URL directly.

@leepowellcouk @xlenz If you could share your full DEBUG logs as described here: https://docs.cypress.io/guides/guides/debugging.html#Print-DEBUG-logs ...that would help to debug the issue and improve proxy support in the next release.

Right now, I'm guessing you're both experiencing #4257, but there could be something else at play.

[leepowelldev]

@flotwig - Thanks! That was exactly the issue for me. Adding undefined to no_proxy works.

[flotwig]

@leepowellcouk Great, glad to hear that the workaround worked for you. Don't worry about the DEBUG logs then. We'll have a bugfix release out soon that will fix #4257.

[catalin7mar]

Hi!
I work in an organization that uses PAC. From the cypress specifications I have to use:

If your organization uses a PAC file, contact a network administrator to ask what HTTP proxy you should be using to access the general Internet, then use that proxy with Cypress.

However, I can't use a proxy for my project, because "internal" calls should be called just without a proxy.

This is the response from the support team (from my organization):

Please clarify with cypress team, if no PAC can be used, then tool must be adjusted so that no proxy is used, because "internal" calls should be called just without a proxy.

My question is: it is possible to adjust so that no proxy is used? Thanks! I tried with

"test:no_proxy": "set HTTP_PROXY= && set HTTPS_PROXY= && set NO_PROXY=https://int..../portal/web && npm run test:open",

and
"test:no_proxy1": "set NO_PROXY=https://int..../portal/web && npm run test:open",
but no one wroks.

[flotwig]

@catalin7mar You can disable proxy usage by passing HTTP_PROXY=, that is correct.

You can view your current, resolved proxy configuration by following these instructions: https://docs.cypress.io/guides/references/proxy-configuration.html#View-proxy-settings-in-Cypress

There is also more info about setting up a proxy on that page.

If it's not working as expected, please open up a new issue so we can figure out what's not working for you.