Authentication runs in a loop. 'Gateway session active but not authenticated' #152
Comments
dabrazhe
changed the title
|
I suggest try the new version #147. For many this has solved the authentication loop. Please do share your results. |
|
voyz/ibeam:0.5.0-rc2 is running now on my real account and it works istead of voyz/ibeam |
|
works with --env IBEAM_AUTHENTICATION_STRATEGY=B |
|
Hey everyone 👋 As much as I'd like to suggest a fix, this authentication loop doesn't seem to be something that is in our control. Ever since I remember IBKR servers will just sporadically refuse to allow you to authenticate successfully over a large period of time despite successfully completing the authentication flow, and cause IBeam to get into this loop. Remember that what happens is:
And so we're back where we started and we end up with the famous authentication loop. What we've observed so far is:
Honestly, this, along with other IBKR-side issues (eg. 'Invalid username and password' error) is what keeps me from upgrading IBeam to version 1.0.0, despite it being used in production by various users. I think everyone here should be aware that until the service is reliable, we're using this system acknowledging that there are and will be long and unexpected drops in connectivity to the broker. There's a good chance that for you it will work for months on end without interruption. There's a good chance that for you it will break every other day. Best we can do is to:
I've turned this response into an entry in the Troubleshooting Wiki. |
|
I tried again with the 0.5.0-rc3 version. The authentication is still in the loop. |
|
The loop is gone and authentication had succeeded upon using |
|
I created a script to request the tickle endpoint every minute and run it overnight and day. About 7% of tickle requests fail, which looks like it drops the auth session, below @Voyz Is this expected, and how to remediate it? |
|
@dabrazhe have a look at my reply above in this post in case you missed it: #152 (comment) In short, as far as I know there's little we can do about the authentication loop. |
|
Alas, the issue is back again. Tried with a newly opened IBKR paper account and IBeam won't log in with any of the strategies. |
|
This is the message I got from the IBKR Client Services @Voyz ==== While you may see a message in response stating "Client login succeeds" after logging in to the Client Portal Gateway, this is not always the full story. In order to confirm authentication, you will typically need to make a call to the /iserver/auth/status endpoint in order to confirm authentication status. If the response message includes "authenticated":false, then you have not been authenticated properly. Sometimes this is just a matter of logging in to the client portal gateway once again from the localhost page. While we are working towards a more consistent result, it can sometimes be beneficial to try calling the /logout endpoint, then proceeding through the login procedure once again. This will clear out any stale or competing brokerage sessions that may be halting the login process for a fully authenticated brokerage session. |
|
@dabrazhe thanks for reporting what you heard from IBKR. Let me address these points:
Yupp, we do that to confirm.
We do that too, and interpret it in the same way.
In Strategy A we do that exactly, in Strategy B we first try to reauthenticate before re-logging in.
We do this in Strategy A, to no avail. In Strategy B we reauthenticate instead, then if that fails repeatedly we kill the Gateway and log in. Note that in both cases we check for competing sessions, and logout if we find one. Unless we've gotten something wrong or are missing something, none of these suggestions help unfortunately. |
Would it be worth it to try a logout even if a competing session isn't found? |
|
For my use case, and I suspect for many others', I don't actually need the gateway connected 24/7. I only need it connected when I need it to be. Is there a way to use that detail to limit the impact of this problem? For example, would it be worthwhile to add something like "quiet hours"? That is, if the connection drops during those hours, don't bother looping until the "quiet" period ends. That wouldn't fix the problem, but would at least limit the notification storm on my phone. Alternatively, don't immediately re-authenticate if the connection drops after the API hasn't been used within some timeout. Instead, hold off on re-authenticating until an API call comes in. Again, this isn't a true fix, and would probably require the addition of a proxy server, but it would limit the amount of looping. Obviously these are both bad ideas, but it seems like we're out of good ideas. Maybe it's time to try some bad ideas until IB sees fit to join the modern era. |
|
@bfoz thanks for all these suggestions and questions 👍
Yes, we do that in Strategy A, but it doesn't seem to alleviate the problem 😔
I wouldn't know what impact would this have on the authentication loop problem, but that's an interesting hypothesis.
That sounds like you want to create an scheduler for either your IBeam container or VM (or wherever you're running it on) - something that would start and stop the service during the 'quiet hours'. Cloud providers have some variations of pipelines that allow to do this. On Google Cloud Platform, for instance, you could use Cloud Scheduler or Instance Schedule. Or do I misunderstand something? I remember some users reporting that restarting IBeam container daily seemed to help.
Yupp, you'd need a proxy that would intercept the calls to the Gateway and administer IBeam accordingly. Actually my own setup used a proxy like this before (though for completely different reasons) - let me know if you'd find it useful and I could see if I can extract some code out of it to make it public.
Hahah, sorry this is just a fantastic quote, they should put it in a film someday. Or is it a quote from a film? Anyway, I don't think these are bad ideas. However, I think they require a peripheral setup that is independent from IBeam itself. Nevertheless, if you find that doing so helps with the authentication loop, then definitely report back and let us know, as this could be very useful to know. |
|
I think this problem is in the GW and not at api.ibkr.com. TradingView's IBKR integration (which uses api.ibkr.com) never has this issue. It doesn't let you log in at busy times though. I haven't been able to authenticate for 20 hours now with iBeam and I've tried everything. Contacted IBKR also but they stopped replying. The GW is java app so you can decompile it easily to readable code (I did it once, you just drag the .jar file into the decompiler). Maybe somebody could fix it or add some debugger stuff. Also I'm not sure if the GW is really needed if you copy the certs etc. The cert pass is "mywebapi". You could probably connect directly to api.ibkr.com. The GW doesn't seem to do much if you check the code... They seem to have at least three different APIs but only one is documented (3rd party): |
|
I sent these .har files for them today, let's see. |
|
hey @jussirantala thanks for sharing your info 👍 As for decompiling the Gateway or going around it - interesting observations, though make sure you're not breaking IBKR's ToS by doing so.
We tried that, to no avail unfortunately. Strategy A does exactly that, Strategy B will only logout if it sees a conflicting session. |
|
It's weird that I'm not having any problems with the CPWEBAPI itself. Only if I use with IBeam. Do you think they can somehow detect it's inside docker container? Or maybe this is something that could be fixed with some docker configs or something. |
Are sure that your local client is exactly the same as the one inside IBeam? Apart from, it should be more or less impossible to detect that something runs inside a container. (I'm not able to login for about 1 month already, both with and without Docker) |
|
@CaptainTrunky Looks like it's not the same. My version: @Voyz Seems |
@CaptainTrunky Could you please send your GW logs to [email protected] ? |
|
@jussirantala I appreciate your thorough effort trying to figure this out 👍 The Gateway version inside of IBeam gets updated sporadically and doesn't follow the release cycle of IBKR. In the past I, and other users, tried seeing if updating it would solve any issues, but it never produced any observable improvements. That being said, if you believe it could be the version that is the problem, we can update it no problem - let me know. As for why it works for you outside of IBeam - there could be a number of reasons. For example, if you're deploying the IBeam instance on a cloud provider, it has been observed that just communicating from a different IP can bring different results. A VPN has been shown to solve some problems. As another example, some users' country of registration affected their ability to connect. I'm bringing these up to illustrate that there could be some diverse reasons for why you could be observing different results with your standalone Gateway. Again, the recommended way would be to try to figure these reasons out with IBKR support. |
|
Weird. IBeam can authenticate locally now but still not on my server. I deployed a new server which is located in the same country I am instead of New York but no luck... Maybe there's some information about the operating system in the headers (see HttpProxy.java) 🤔 Also I noticed it's sending your network adapter MAC address as an URL param when using the SSO (see SsoService.java and TstTokenUtils.java).
🤔 @Voyz Btw is it possible that you would publish 0.5.0 image? I've been trying to build it but I'm having some issues... Might be related to M1 processor. EDIT: Tried to change my container MAC to something random but didn't help. |
|
Sorry for the slow reply.
Is it worth it to try doing the same in Strategy B?
Actually, I'm thinking of the opposite of that. The scheduled start/stop solution potentially stops a perfectly connected and functioning gateway, which seems less than ideal. So instead, I'm imaging a mechanism that does nothing during the "quiet hours", and only does the authentication dance If the connection goes "bad" during the "non-quiet hours".
Creating a proxy seems like a heavy lift, especially if you've already tried it and decided that it didn't work. But if you ever feel like posting the code somewhere I'd be happy to take a look at it.
It probably is a quote, but I have no idea what it's from if it is.
The "quiet hours" idea might need some support, since it would require disabling the authentication loop at certain times. |
I've tried to do this, but I wasn't able to open an issue - support services were down for a while - ticket creation fails with an unknown issue. |
Sure. I was travelling for the past few weeks so didn't want to roll it out yet. But I should have some time to do it now. |
My initial thoughts would be that no, I wouldn't do it. The reasoning here, is that Strategy B attempts to perform authentication in a different way to Strategy A, in order to see if that will be more reliable. I don't think making them very similar is the right way to go. If there was some strong indication that
I'm not sure if I'm following you here, sorry 😔 Currently we don't do anything during the 'quiet hours' and only 'do the authentication dance' if the connection goes bad. If the connection is fine we don't do anything. Could you open a separate issue and describe in detail how would you see this work?
I created the proxy for a completely different task, unrelated to the authentication. |
|
FYI: Strategy B with voyz/ibeam-0.5rc4 seems to work for me as well. |
|
Does this issue affect more paper trade accounts? Was working fine until today. I also noticed seems to not like 2FA accounts. Seems to loop as well. Getting version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400 This is the Client Portal Gateway https://www.interactivebrokers.com/api/doc.html Open https://localhost:5007 to login version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400 This is the Client Portal Gateway https://www.interactivebrokers.com/api/doc.html Open https://localhost:5007 to login version: 485dc2d762781c4ff3954ac4fb66a9469a1405f7 Mon, 20 Mar 2023 14:39:35 -0400 This is the Client Portal Gateway https://www.interactivebrokers.com/api/doc.html Open https://localhost:5007 to login |
|
@greybox1 we haven't noticed whether this would affect paper accounts more than live. And yes, this seems to be an IBKR issue. |
|
FYI, in August, I sent all the logs etc which IBKR asked. After that I've asked two times how they are progressing. The first reply was "Still investigating". And they didn't reply at all the second time. |
|
@bfoz @jussirantala @dabrazhe and others - there's a new attempt at fixing the authentication loop: It's based on some intel from @greybox1 talking to the IBKR support that casts some light on the auth loop. According to the support agents, even though you may be authenticated, the I've increased the If this doesn't help, the next step would be to stop relying on |
|
@Voyz I'm running into what I think is a similar issue here. I'm running version Here are my startup logs: I'm not sure if this is helpful but, it appears that it detects the SSO Token is invalid but never tries to revalidate it. When I call the {
"authenticated": false,
"competing": false,
"connected": false,
"MAC": "F4:03:43:CD:BD:C0"
}When I call an endpoint that requires a valid session, I get a response that says to initiate a session first. For example here is the response from trying to place an order with this endpoint {
"error": "Init session first",
"statusCode": 400
}I think your suggestion of trying another endpoint to determine auth status could work. I'm happy to give this a try if you want to point me to the spot in the code to make the change and maybe suggest an endpoint to use. |
|
Hey @jcity thanks for reporting all that back.
To be honest, if you get that As for the log you added, not sure if that's a complete log or if it misses something? These lines are confusing: So it confirms being authenticated, works fine, continues working fine, and the suddenly starts a new IBeam instance? |
|
Hmmmm, so this is strange and may not end up being unrelated to the issue being discussed here. When I run ibeam locally, I'm able to establish a session but when I run as a docker container in AWS Fargate, a session is never able to be established and I continue to the the I'm wondering if I've potantially hit the rate limit and they have blocked the IP associated with my Internet Gateway |
|
Interesting @jcity - could be also that your AWS server is hitting a different IBKR server which causes the issues. |
|
@Voyz Does TWS also suffer the auth issue? |
|
@pinkfrog9 TWS is a completely different interface, and as such I don't think this particular issue occurs, at least not in the exact same form. However, TWS has its own set of issues, and some of them are also connectivity related. |
|
I want to chime in here and close the thread on my end. I could never get this stable enough where I felt comfortable it would work in production. At no fault to @Voyz and all the contributors -- thank you all. This is the best project I've seen to get something somewhat working. I ended up switching to the 1st party OAuth integration and that has been relatively stable. Unfortunately, I don't think it's available for personal accounts at this time. You do still have to maintain a session as well but that piece has been relatively stable. We also intermittently get |
|
I'm going to close this issue due to inactivity. Thanks for your contribution and please feel free to request a reopen if you'd like to continue the discussion 👍 |
|
One last thing. I played with various types of IB accounts and noticed that it works best with a paper account coupled with a Live account. The live account does not need to be funded. |
|
@dabrazhe many thanks for adding all that! Could I ask you to elaborate a tiny bit? This could be very useful to a lot of us here.
When you say 'coupled' - what do you mean? I though that all paper accounts need to have at least one live account?
What's a 'standalone paper account'? From what I understand when you open an account with them you get a live account, and you can add a paper credential to it. What would 'standalone' be?
I'm honestly really happy to read this. Great job! 👏 Is that a live server? If so, how do you handle 2FA? |
|
Hi @Voyz
The credit is all yours! Thank you for your work and the hours you put in.
A live account is useless for my purposes because it requires MFA authentication every day. IB won’t budge on this. This would solve my problems because the things I need from the IB API are the margins for my account and real-time pricing subscription. Which the paper acc. does not have. (
You can choose to open a paper account without the Live one, but this does not work for me. After back and forth with support, it was clear that Live is needed for the stable API server run. |
|
The container can run for months, but if you are interested it crashed with this kind of errors. 
|
|
@dabrazhe thanks for reporting back on your interaction with IBKR support. I've added a section to troubleshooting based off of it. Also thanks for sharing that log. Indeed, Invalid Username and Password is a frequent out-of-nowhere error. Hope it fixed itself 🤞 |
|
Hi @Voyz and all, I came across your repo and this particular issue as I've built a very similar tool (not knowing that iBeam was already out there in the wild) that occasionally has a very similar login loop issue. I tried many of the same things in this thread including a full The three thing I've done which appear to have helped (but not 100% solved) the login issue in the IBKR gateway conf.yaml file are:
If I'm not mistaken, most or all of these settings are available in the iBeam docker env list. Hopefully some or all of these help with your login issues. |
|
Hey @rileyvdl16 many thanks for sharing all that info! And glad to hear you've already found IBeam. Btw. check out IBind if you'd be looking for a Python client for the IBKR Web API.
There is no way of knowing, really. Occam's razon would suggest that it is not deliberate but accidental though.
Many thanks for suggesting this!
Almost - you indeed can modify these, yet not through env vars but by providing a custom |
|
Just wanted to comment that I was having problems running on Apple silicon (M2 Pro). I just got a new Raspberry Pi to self host some things and deployed the docker container just to check things and it's incredibly stable running in Docker using Portainer. |
Describe the bug
The service keeps authenticating over again
See logs
Environment
IBeam version: the latest
Docker image or standalone: Docker
Additional context
The login info is correct, can login on the IB web site.
The text was updated successfully, but these errors were encountered: