Login Occuring but Still Unauthenticated Per Status Code

[ntilley905]

I've been noticing that the sensor has been unknown for a while, probably the same issue as #8. I made sure that there isn't a security check by logging in via a browser and then enabled debug logs and restarted. It appears that the component is logging in, but then the actual data is returning a 401 Unauthenticated error code. Logs are below.

2020-06-29 08:56:25 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for xfinity which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant. 2020-06-29 08:56:42 DEBUG (SyncWorker_12) [custom_components.xfinity.sensor] Finding reqId for login... 2020-06-29 08:56:43 DEBUG (SyncWorker_12) [custom_components.xfinity.sensor] Found reqId = 5664b480-3509-48e1-884e-25ab3a225389 2020-06-29 08:56:43 DEBUG (SyncWorker_12) [custom_components.xfinity.sensor] Posting to login... 2020-06-29 08:56:44 DEBUG (SyncWorker_12) [custom_components.xfinity.sensor] Logged in successfully, status_code: 200 2020-06-29 08:56:44 DEBUG (SyncWorker_12) [custom_components.xfinity.sensor] Fetching internet usage AJAX... 2020-06-29 08:56:44 ERROR (SyncWorker_12) [custom_components.xfinity.sensor] Failed to fetch data, status_code:401, resp: {'error': 'unauthenticated'}

[ntilley905]

Not sure why the logs above formatted strangely, the line breaks are in the right place. Regardless, some additional info. I tried getting the API data through a few other sources and it appears that it's returning a 401 Unauthorized. Not sure why Requests throws that as Unauthenticated, but it's just a run of the mill 401. I'm wondering if this API endpoint is no longer available? I'm having trouble finding any data as to what is made available via API access, it doesn't look like Xfinity makes any of that publicly available, at least not easily.

[ntilley905]

Disregard the above, when I logged in via Firefox and pulled the API data through there, it returned successfully. When I dug in a little more to the response being provided through requests, it is returning an error of "errorMessage" : ["authn.unsupported_browser.error"] and further clarifying that the browser is not supported. Not really sure how to get around that one...

This is also causing the debug logs to be incorrect. The response provided after the login action does have a response code of 200, but it is not actually logging you in.

[ntilley905]

Some more info (sorry for not consolidating into one comment): the unsupported browser is easily fixed by adding headers. Once I got past that issue, however, it is saying that I am using an invalid user/password. I've confirmed that I'm not, it just seems like something is getting tripped up in the post process. It's still returning a 200 on the login post action, but within that response there is an error that says the info is incorrect (thanks for using the correct status codes, Comcast). To be continued...

[robert-alfaro]

Thanks for debugging! I'll be looking into this soon

[kendrome]

@robert-alfaro Any luck? Would be really helpful now that caps are back.

[cszucko]

I stumbled upon this thread the other day while looking for a fix for a similar tool of mine. I haven't come up with a solution yet but, to share what I've seen, I believe the source of the trouble is that the login page is now using some obfuscated JavaScript to generate a series of fields (see below) that need to be present in the POST request.

X-hzfdeCEGvt-f
X-hzfdeCEGvt-b
X-hzfdeCEGvt-c
X-hzfdeCEGvt-d
X-hzfdeCEGvt-z
X-hzfdeCEGvt-a

Without those, the response comes back HTTP 200 but it just dumps you back to the login page and shows a "The Xfinity ID or password you entered was incorrect. Please try again." error.

[migitcheetah]

I am also seeing this issue. Login in from a browser then hitting the api from the same browser seems to work. Maybe a cookie or session issue? Its definitely per browser, as hitting the api from different machines gives me the same error as HA until I log in from that machine.

[jasii]

Is it even possible to get around those generated fields necessary in the POST request? If you're able to log in using a browser then hit the API, maybe it will be necessary to use a headless browser to log in first? That will significantly increase the size of this script and increased processing power though.

[gomsnothereman]

Any updates on this? spent a few hours trying to get this to work on home assistant before i checked the issues tab :). Plugin looks great/handy with so many working from home now and comsucks being comsucks and having a ridiculous low limit for 2020...

[apippin]

It appears that the xfinity login page has started rejecting authentication attempts by automation engines like selenium or python based session requests. Even though I am using a valid username and password, and can login from the same computer and browser into my xfinity account, xfinity returns an invalid username and password error when attempting to do the same thing via this script.

The work around I found was with another homeassistant project that uses Selenium to login and get this data. I posted an issue with a solution to fix the Selenium based approach here: jantman/xfinity-usage#30. I use that approach to get the JSON needed for homeassistant.

[robert-alfaro]

No luck on my attempts to get this working. I've very occupied with work and have little to no time to give this the attention is deserves. I used to use the headless approach in the past but it stopped working. Great that's still a go.

[Vendo232]

It appears that the xfinity login page has started rejecting authentication attempts by automation engines like selenium or python based session requests. Even though I am using a valid username and password, and can login from the same computer and browser into my xfinity account, xfinity returns an invalid username and password error when attempting to do the same thing via this script.

The work around I found was with another homeassistant project that uses Selenium to login and get this data. I posted an issue with a solution to fix the Selenium based approach here: jantman/xfinity-usage#30. I use that approach to get the JSON needed for homeassistant.

would you be able to share the working json for HA? maybe the whole XFINITY custome folder? thnak you very much

[apippin]

This is still working great for me. I have compressed my comcast custom config dir and pointed to the url at the end of this post that you can download. It contains a comcast.yamls file with all of the different places I changed in the hass yaml files to support this flow. You can uncompress this into your hass/config/ dir. You can look through the files and figure out what to change to make it work for you. I’ve posted it on my website since I couldn’t attach it here. You can download it from here: comcast.tar.gz

[Vendo232]

I`m getting this error when I try to run compast.py, any idea what could be wrong?

…

________________________________ From: Alan Pippin <[email protected]> Sent: Friday, October 2, 2020 13:43 To: robert-alfaro/xfinity-usage <[email protected]> Cc: Vendo232 <[email protected]>; Comment <[email protected]> Subject: Re: [robert-alfaro/xfinity-usage] Login Occuring but Still Unauthenticated Per Status Code (#9) This is still working great for me. I have compressed my comcast custom config dir and pointed to the url at the end of this post that you can download. It contains a comcast.yamls file with all of the different places I changed in the hass yaml files to support this flow. You can uncompress this into your hass/config/ dir. You can look through the files and figure out what to change to make it work for you. I’ve posted it on my website since I couldn’t attach it here. You can download it from here: comcast.tar.gz<https://cloud.pippins.net/s/ej85TopaqGfajTF/download?path=%2Fhass&files=comcast.tar.gz> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#9 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AKQOFNRWYILPYGU3KH3PXY3SIYUOBANCNFSM4OLF7LIA>.

[migitcheetah]

This is still working great for me. I have compressed my comcast custom config dir and pointed to the url at the end of this post that you can download. It contains a comcast.yamls file with all of the different places I changed in the hass yaml files to support this flow. You can uncompress this into your hass/config/ dir. You can look through the files and figure out what to change to make it work for you. I’ve posted it on my website since I couldn’t attach it here. You can download it from here: comcast.tar.gz

I have not attempted you fix yet, but I was looking through the files on my phone, and saw references to comcast.py in the readme, but the comcast.py supplied labeled old.broken.
Also there are references to comcast_usage.json and xfinity-usage.data that I am not seeing in the package.

I was just wondering if there was something missing from the zip file?

edit:
I do see xfinity-usage file, but not .data, so that might be it, just want to make sure.

[apippin]

Sorry for the confusion. I had 2 different methods/scripts in that dir. I reposted an updated .tar that takes out the confusing README and .py script that was in there. The json file is something the xfinity-usage script produces. The supported solution is to use the xfinity-usage python package with the wrapper scripts in that tar file.

[migitcheetah]

So do we move all your files into the xfinity haccs add on folder, then modify the confin.yaml to whitelist that dir?
Or do we set the XFINITY_USAGE=$HASS_PATH/xfinity-usage path to be the dir for the haccs addon dir?
In my case it is /config/custom_components/xfinity.

Also what is this in comcast.sh PATH=<path to xfinity-usage script>:$PATH
I don't see any usage of it.

Sorry for so many questions, still fairly new to Home assistant and its scripting and automations.

[apippin]

I did put these files under a subdir under /config. The $HASS_PATH is the path to your homeassistant base dir. The <path to xfinity-usage script> is just the path to the files in the tar file (where ever you uncompress them to). It adds it to the PATH so that you can just call the command later in the script without having to specify the full path to it (which is why you don't see any usage of it).

[migitcheetah]

So I figured out that the issue I am seeing now is on the xfinity-usage file. that path to python is my issue currently. I am using a hyper-v image of home assistant, so I dont know how the setup is, but I cant seem to see a python version from the command line.
I can see from the UI that I am running python 3.8.5, but the command line cant seem to find it. so I don't know my path.

hopefully I will be able to figure that out. But it looks like everything else so far is connected and working. fingers crossed.

[apippin]

I don't run homeassistant in a hyper-visor like you do, so I'm not sure of the paths either. My homeassistant runs in a python virtualenv, so the python3 it uses is under a /bin/ dir under the homeassistant directory. If you do python3 --version is one in your path? If so, great. That is probably what homeasistant is using. If not, hopefully you can find the one it is.

[jasii]

I'm posting this here even though it's slightly unrelated becuase I think it may be able to help some of you out. Basically I've taken a different approach in monitoring my internet usage. I figured since it's so difficult to get the data from xfinity, why not monitor what is going in and out of my network and throw that into a HA sensor.

This is only for those of you running an USG router

I found this guide by @tynick @https://tynick.com/blog/12-30-2019/internet-data-usage-monitoring-so-you-dont-hit-your-data-cap/

I slightly modified it to use Home Assistant REST API to post the USG TX and RX and Total data to a sensor in Home Assistant. This isn't perfect and I suck at sensor templating, but I got the usage sensor onto my Home Assistant! I hope someone here can take this as a start and build upon it and maybe fix up my sensor template.

https://github.com/jasii/unifi_usage_home_assistant_status

Big thanks to @tynick for his guide and code

[cashmore1]

Has anyone come up with a solution for this that doesn't require selenium. This has been confirmed to still be an issue as of today.

[JSylvia007]

Has anyone come up with a solution for this that doesn't require selenium. This has been confirmed to still be an issue as of today.

Followingggggg... Freaking Comcast.

[robert-alfaro]

I have zero time to look into this...I too am missing out on my this sensor.

I'd reallllly appreciate it if anyone has time to provide a PR. I'm guessing that the other comments mentioning working api can be used as guidance to fix this component.

[jasii]

Yesterday I was digging into the Android app using mitmproxy and the apps API requests. I was able to get the information but the authorization code would time out after 60 minutes. I'm new to this stuff. I seem to be missing one piece of the puzzle. I think it's from some JavaScript that runs when you load the page that generates a special url with unique parameters for code and state and I'm not sure how to get those values using requests.

[McGuire00]

I found a fix to login to Xfinity's site using selenium.

https://piprogramming.org/articles/How-to-make-Selenium-undetectable-and-stealth--7-Ways-to-hide-your-Bot-Automation-from-Detection-0000000017.html

"If you open your ChromeDriver / GeckoDriver using a Text editor and go to approximately line 4000+ you will find some JavaScript that will be run when you are using Selenium.

That is why Bot detection software like FingerprintJS, Imperva (former Distil Networks), or Google’s Captcha will look for this JavaScript Code.

But luckily you can just edit this JavaScript right in the executable - just change up variable names with ones of the SAME LENGTH (otherwise Selenium will just crash).

If you are using the ChromeDriver you're going to find something like $cdc_asdjflasutopfhvcZLmcfl_. Replace the entire section just after $ with another string of the very same length. This is the variable most detectors are searching for. That being done, a lot of gates are already wide open to you.

Save it. From now on, use this executable as your driver."

[mattlward]

I just installed this in my HA install... getting the 401 error as well. I am running a healty/supported supervised install.

2021-03-02 10:28:46 ERROR (SyncWorker_4) [custom_components.xfinity.sensor] Failed to fetch data, status_code:401, resp: {'error': 'unauthenticated'}

[Scylla2020]

Anyone found a solution with requests yet? I tried to reverse the javascript creating the fingerprints but ran out of time as its too long.