Swale council producing an error

[Chewbacca222222]

Name of Council

Swale

Issue Information

When setting up the integration I get this error:

This error originated from a custom integration.

Logger: custom_components.uk_bin_collection.sensor
Source: helpers/update_coordinator.py:315
integration: UK Bin Collection Data (documentation, issues)
First occurred: 11:54:50 (1 occurrences)
Last logged: 11:54:50

Unexpected error fetching Home data: Could not parse council website.
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/uk_bin_collection/sensor.py", line 133, in _async_update_data
    data = await self.hass.async_add_executor_job(self.ukbcd.run)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 96, in run
    return self.client_code(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/collect_data.py", line 115, in client_code
    return get_bin_data_class.template_method(address_url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/get_bin_data.py", line 78, in template_method
    bin_data_dict = self.parse_data(
                    ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/uk_bin_collection/uk_bin_collection/councils/SwaleBoroughCouncil.py", line 34, in parse_data
    raise ConnectionAbortedError("Could not parse council website.")
ConnectionAbortedError: Could not parse council website.

Verification

• I searched for similar issues at https://github.com/robbrad/UKBinCollectionData/issues?q=is:issue and found no duplicates
• I have checked my address/postcode/UPRN works on the council's website
• I have provided a detailed explanation of the issue as well as steps to replicate the issue
• I understand that this project is run by volunteer contributors therefore completion of this issue cannot be guaranteed

[dp247]

Looks like the council website has been updated since the parser was written

[Chewbacca222222]

@dp247 @robbrad Is someone able to walk me through the project setup? I'm happy to give a go fixing the issue but I can't get it set up on my machine, even after multiple attempts!

[dp247]

This is the way I set it up:

1. Forked the repo to my GitHub account.
2. Run the environment setup in Windows Terminal, then open the clone in Pycharm (I don't use venv, so my packages are installed to the system's version of Python, which I then use in Pycharm)
3. In Pycharm, go to Run > Edit Configurations. I run Python 3.11 with the script set to the collect_data,py script. The run parameters (in the box below) can be found on the wiki - just don't include the python collect_data.py part.
4. Open the council that I set run parameters for, then I can set breakpoints and debug.

[dolce08]

Were you able to get this working for swale?

[Chewbacca222222]

No sorry, got way to busy!

…

On Sat, 24 Aug 2024, 12:20 dolce08, ***@***.***> wrote: Were you able to get this working for swale? — Reply to this email directly, view it on GitHub <#665 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGNOQV5PNVUQYVEZMO4LQ2DZTBUBHAVCNFSM6AAAAABFXBCVHCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMBYGM2TSNBSG4> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[dolce08]

I've noted the council web address has changed slightly to https://swale.gov.uk/bins-littering-and-the-environment/bins/my-collection-day. Please could this be updated to see if it fixes the issue?

[Chewbacca222222]

@dolce08 Hi yeah I noticed this too but it's a completely different process so simply updating the URL won't fix it!

[dolce08]

Thanks for letting me know. I'll try to find the time to test in PyCharm

[robbrad]

Thanks for letting me know. I'll try to find the time to test in PyCharm

Just a heads up if you want: there is a devcontainer that works in vscode

[Chewbacca222222]

So I've taken a good look at the website and I don't think it's going to be easy to replicate the request (I tried many things and couldn't get it to work). To that end I suggest switching over to Selenium, I have very little experience using devcontainers and Selenium but I'm happy to give it a go. However I'm not sure about anyone else but I get a fair few errors when launching that dev container, tried a few things to get it working but it can't seem to access some endpoints:

38.69 Err:2 http://deb.debian.org/debian bullseye InRelease
38.69   Connection failed [IP: 199.232.58.132 80]
39.77 Err:3 http://dl.google.com/linux/chrome/deb stable InRelease
39.77   Connection failed [IP: 142.250.187.206 80]
77.02 Err:4 http://deb.debian.org/debian-security bullseye-security InRelease
77.02   Connection failed [IP: 199.232.58.132 80]
115.4 Err:5 http://deb.debian.org/debian bullseye-updates InRelease
115.4   Connection failed [IP: 199.232.58.132 80]

[dp247]

So I've taken a good look at the website and I don't think it's going to be easy to replicate the request (I tried many things and couldn't get it to work). To that end I suggest switching over to Selenium, I have very little experience using devcontainers and Selenium but I'm happy to give it a go. However I'm not sure about anyone else but I get a fair few errors when launching that dev container, tried a few things to get it working but it can't seem to access some endpoints:

38.69 Err:2 http://deb.debian.org/debian bullseye InRelease
38.69   Connection failed [IP: 199.232.58.132 80]
39.77 Err:3 http://dl.google.com/linux/chrome/deb stable InRelease
39.77   Connection failed [IP: 142.250.187.206 80]
77.02 Err:4 http://deb.debian.org/debian-security bullseye-security InRelease
77.02   Connection failed [IP: 199.232.58.132 80]
115.4 Err:5 http://deb.debian.org/debian bullseye-updates InRelease
115.4   Connection failed [IP: 199.232.58.132 80]

You don't need the dev container per se, it just saves setting everything up manually

[m26dvd]

I've fixed the logic on this one however Swale have introduced a Cloudflare check which sometimes means the check will fail.

I've not managed to find a way round this unfortunately

[Chewbacca222222]

I've fixed the logic on this one however Swale have introduced a Cloudflare check which sometimes means the check will fail.

I've not managed to find a way round this unfortunately

Thank you for fixing this!