Allow Persona login to work behind a (corp) proxy

[GrmpCerber]

Beeing behind a corporate firewall I need be able to pass proxy configuration to each application.

frontend sign-in makes use of mozilla's personas but it doesn't seem to handle proxy env variable

Bottom Note : nothing is logged in console but I never get connected while on corporate network. With a direct connexion I got connected 1/3 of the time

Regards

[yeukhon]

I am curious, since I don't have to deal with proxy myself, are we having two separate issues?

1. Persona doesn't handle proxy? or Minion cannot communicate with Persona?
2. With direction connection, no proxy env, you still have login issue?

Can you give a bit more detail how to reproduce. I can try connect to proxy.

[GrmpCerber]

It seems that minion cannot communicate with personna but I can't tell it for sure : no error message is logged.

I took a couple of minutes yesterday evening to boot it on my home network with a direct internet access and I managed to be connected and got a default (almost empty) screen but I had no time to go further for the moment.
(By the way can you point me out the getting started guide so that I can add sites and ... get things started ;) )

What I see while behind the corporate firewall is the following (in develop mode) :

1. Firefox open http://127.0.0.1:8080
2. I click the big blue personna button
3. personna popup opens
4. I fill it with my credential
5. I'm logged in personna
6. popup disappears ... THE END

• initial login window doesn't refresh
• manual refresh show the same login window

All I get in log is :

127.0.0.1 - - [02/Aug/2013 16:36:29] "GET / HTTP/1.1" 200 -
[... static stuff ... ]
127.0.0.1 - - [02/Aug/2013 16:36:40] "POST /api/login HTTP/1.1" 500 -

[yeukhon]

You are not the only one. Someone else on IRC #websectools have the same exact problem. This is a big issue: many of our users are going to be corporate developers and analyst. It would be a pity if minion cannot communicate behind a proxy. I will try to sort this out today for you and I will try the proxy today (again, I don't really use proxy so it might take a little while for a slowpoke like me hehehe )

[yeukhon]

And on the second issue, yes, I will actually get the documentation up and running in the next few days (have to be done because I am getting request from internal as well).

If you have time come to #websectools (im yeukhon) later (i am actually walking to office soon) and I can be your question guy if u need quick access. or just ask on github.

[ygjb]

I suspect that what is happening here is that the minion-frontend needs to talk to the Persona verifier to complete the authentication process [1].

In order to get this working you need to set up a proxy for the minion-frontend by setting the appropriate environment variables for the minion-frontend service [2]

Can you let me know if this sorts things out for you? If so, we can update the documentation!

[1] https://github.com/mozilla/minion-frontend/blob/master/minion/frontend/views.py#L373
[2] http://docs.python-requests.org/en/latest/user/advanced/#proxies

----- Original Message -----
From: "Thomas" [email protected]
To: "mozilla/minion-frontend" [email protected]
Sent: Friday, August 2, 2013 7:41:17 AM
Subject: Re: [minion-frontend] Proxy support (#95)

It seems that minion cannot communicate with personna but I can't tell it for sure : no error message is logged.

I took a couple of minutes yesterday evening to boot it on my home network with a direct internet access and I managed to be connected and got a default (almost empty) screen but I had no time to go further for the moment.
(By the way can you point me out the getting started guide so that I can add sites and ... get things started ;) )

What I see while behind the corporate firewall is the following (in develop mode) :

1. Firefox open http://127.0.0.1:8080
2. I click the big blue personna button
3. personna popup opens
4. I fill it with my credential
5. I'm logged in personna
6. popup disappears ... THE END

• initial login window doesn't refresh
• manual refresh show the same login window

All I get in log is :

127.0.0.1 - - [02/Aug/2013 16:36:29] "GET / HTTP/1.1" 200 -
[... static stuff ... ]
127.0.0.1 - - [02/Aug/2013 16:36:40] "POST /api/login HTTP/1.1" 500 -

---

Reply to this email directly or view it on GitHub:
#95 (comment)

[GrmpCerber]

Done, but it's odd : your second link states it clearly : You can also configure proxies by environment variables HTTP_PROXY and HTTPS_PROXY.
But this doesn't work for me ...

So I modified the code there : https://github.com/mozilla/minion-frontend/blob/master/minion/frontend/persona.py#L10
But still all I have is : 127.0.0.1 - - [02/Aug/2013 16:36:40] "POST /api/login HTTP/1.1" 500 -

I'm a novice in python but I'll try to add a few logs to be able to analyze personna's server response

[GrmpCerber]

DEAD END : minion fetches a valid personna response. The problem is somewhere else ...
No need to change persona.py it does make use of proxy env vars

{u'audience': u'127.0.0.1:8080',
 u'email': u'XXX@YYY',
 u'expires': 1375457878492L,
 u'issuer': u'login.persona.org',
 u'status': u'okay'}

[ygjb]

can you run the individual components from an interactive terminal (for example, using screen), and report the errors that show up there?

----- Original Message -----
From: "Thomas" [email protected]
To: "mozilla/minion-frontend" [email protected]
Cc: "Yvan Boily" [email protected]
Sent: Friday, August 2, 2013 8:41:23 AM
Subject: Re: [minion-frontend] Proxy support (#95)

DEAD END : minion fetches a valid personna response. The problem is somewhere else ...
No need to change persona.py it does make use of proxy env vars

{u'audience': u'127.0.0.1:8080',
 u'email': u'XXX@YYY',
 u'expires': 1375457878492L,
 u'issuer': u'login.persona.org',
 u'status': u'okay'}

---

Reply to this email directly or view it on GitHub:
#95 (comment)

[GrmpCerber]

Hi ygjb.
The thing is I do run them separately like explained here : https://github.com/mozilla/minion#running-minion-in-development-mode
Unfortunately there is absolutly no errors in any of my terminals.
The only thing I've got is this silly error 500 without any explaination ...

[st3fan]

@GrmpCerber I made the following change, which should enable debug mode and show stack traces in case of a 500.

mozilla/minion@b691db1

Can you pull that in and post the stack trace when things fail?

[GrmpCerber]

1. Thanks
2. Done
3. (Almost) the end

walktrough

Frontend fails in minion-frontend/minion/frontend/views.py line 29 while trying to pass login to the backend (in function login_user while calling r.raise_for_status())
It fails with an error 503 coming from the backend side : HTTPError: 503 Server Error: Service Unavailable
On the backend side, nothing is logged but I got the new line you added with -r :

./setup.sh run-backend
 * Running on http://127.0.0.1:8383/
 * Restarting with reloader`

I've also tried to login while the backend was offline and got the same message on the frontend side
This lead me to belive the request was ending on the wrong server ... of course : I've set a global proxy so the requests frontend to backend are trying to go through this proxy too.

solution

I've unset all proxy settings in each terminals, then added a few lines as suggested earlier by @ygjb in his second link in comment #95 (comment) and I got connected !

I'm not fluent in python nor in minion (yet ;) ) so I don't think I could figure out a good pullrequest but what I would do is to add a config varibale for "internet proxy settings"

anyway thanks to the three of you

[yeukhon]

@GrmpCerber If you unset proxy settings and leave minion as it is, you should be able to connect, right? When you are not behind any firewall or proxy, minion should be able to communicate with Persona without any additional enhancement.

[GrmpCerber]

@yeukhon No,

I did a fresh install without global proxy settings.
Both initial git clone and ./setup.py clone require proxy to be set.
But after if you start frontend / backend and workers without proxy you can't log in :
you can connect to personna's website (popup) and connect there. But when the frontend tries to validate the assertion it needs to connect to personna's website too which it can't do.

[yeukhon]

@GrmpCerber so as I understand it, you are still connecting to a proxy. You access the Internet using proxy. Is this a corporate proxy or a free proxy? I didn't get to play with proxy this weekend. I am interested in getting this fix at some point, figuring what parts have to be changed. As far as I understand you can use @ygjb's suggested method. If you are using a public proxy I probably can try it really quickly once I am done with my documentation. (Yes, we are writing docs :) )

We probably can add a configuration in minion such that if proxy is set, the request objects when doing the verification will use the proxy settings, otherwise it remains as ``None`..

[GrmpCerber]

It's corporate.
And since I assume this will be a frequent use case, I'd rather share this with you and get it done right.

By the way, since I went further with minion I noticed that some or all of the workers might need proxy setup too.
I haven't been able to scan ( / figure out a way to scan) yet. But I suppose I might have proxy issues there too.
So your configuration might be a little bit more complex than expected : in my company I must deal with several different proxy (Eg. internet proxy / High security zone proxy / Dev zone proxy / ....). Most of the time this is transparent to us because of PAC / WPAD.
But in the case of minion I think this sould be a per scan / per site manual setup ...

[st3fan]

Yeah the workers and plugins also need proxy support if you want to run this internally but scan an outside host. This is more complicated but I will file some bugs for it and see if we can plan it for a future release.

[st3fan]

I am simplifying this issue a bit. First lets allow people to configure a proxy and make sure the Persona login works over it. After that we can think of a strategy for the other bits of Minion that are using HTTP.

[neoCrimeSecurity]

I've been thinking on this. To make this work you would need the following:

1. A proxy setting for internet authentication
2. A proxy setting for talking to minion components (needed?)
3. A proxy setting for scanning - configured via UI.

3.a) Proxy Settings
3.a.1) Create a table for proxies.
3.a.2) Each proxy can have associated hosts/ports/protocols/credentials

3.b) Sites Settings
3.b.1) Be able to associate if a site uses a proxy

I'm probably missing something. Been working with the code a whole day now. :-)