-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Massive increase of DutyCycle in a connected CCU/RaspberryMatic when used in combination with custom_homematic #328
Comments
Ok Jens, let's see where this journey ends. From my personal experience i cannot confirm this dramatic increase of the RMs duty cycle. My base DC is around 25% and a restart raises this value by 10-20%. I have around 120 devices connected which are in majority HMIP devices. We use the getValue method to fetch device data. I also tested On an initial startup we load all paramset descriptions with getParamsetDescription and cache them in a file, to avoid future api calls on restarts. Also the device descriptions are cached in a file. Rooms and names are loaded by a call to the json api.
Please come back, if you need more information. |
I created a version without in the manifest in "HA config folder/custom_components/hahm" |
Well, in my case I have about 200 devices with a baseline DC of about 30% and a restart of HACS/custom_integration raises the DutyCycle by about 30-35% which IMHO is quite substantial given the fact that other external applications like ioBroker or even Smartphone apps like pocketControl cause a 0% increase of DC due to how they query the current parameters.
Well, the question here is really if hahomematic/custom_homematic should really exclusivly rely on xmlrpc communication with the CCU. The point is, that by using
Why don't you also try to load the current values and the paramset descriptions using the json api? As said, this should then allow to use the internally cached values rather than freshly querying them from the rf devices.
Well, thanks for these first information. Perhaps you can point me more to the actualy code passages that are doing the xmlrpc queries and all the other different data queries during startup of hahomematic/custom_homematic. So perhaps explain the query procedure more in a step-by-step guide here with reference to the actual code so that I can review and debug them in my own test environment so that I can identify the actual query code that causes most of the DC increase. Or perhaps you already debug this and find out which queries are responsible for the DC increase. But as it stands, IMHO the current way of querying / initialising the device parameters is definitly not perfect yet, because - as said - other external applications like ioBroker, etc. are also not causing any noteable DC increase. |
Ok, before writing lots of step-by-step guides, i will try using the json methods. @danielperna84 Any arguments against json methods? @jens-maus I'll come back in short. |
Well, it will break compatibility with Homegear, as there we only have XML-RPC. If it's reasonable to implement both mechanisms and use one or the other depending on the detected backend (like we're doing with the metadate already), then we could use JSON-RPC if it's not Homegear. I suspect Homegear installations not to be big as CCU users have them. As mentioned above, So essentially this is what hahomematic is doing during startup: from xmlrpc.client import ServerProxy
for host in ["http://1.2.3.4:2001", "http://1.2.3.4:2010"]:
client = ServerProxy(host)
devices = client.listDevices()
for device in devices:
for channel in device:
for param in channel:
if not param.blacklisted:
# This is the problematic read-access to the XML-RPC API
value = client.getValue(device + channel, param) Note: this is not code that actually works in this form. It's just to illustrate what is happening during startup. |
I created a beta release 0.37.0b1 that should address the following things for CCU/RM users:
@jens-maus I have some questions:
If the cache is empty after a CCU restart, and HA is restarted directly afterwards, then the values are empty for a while. |
Would it be an option to use the |
Well, this would then of course end up in the same situation as before: an increased DutyCycle because all values are directly acquired from listening RF devices. And a smart home centrale (thus CCU) is nothing that you should reguarly reboot, etc. Thus in mid/long-term you will always have valid values in the cache. So I would definitely not advice to do that, but simply do what the other external engines like ioBroker do: Avoid racing the dutycycle under all circumstances. |
This sounds great. I will try to test it ASAP and report back if it really improves the DC handling. Do you have any easy description on how to force an update to 0.37.0b1 in an already installed HACS/cutsom_homematic installation?
Well, I am not 100% sure and would have to check it. As the
Not it is not.
This should be essentially the very same like having the CCU restarted and looking into the WebUI. There, also in some cases (e.g. for homematicIP devices) values have default values until a battery driven homematic devices sends an update after some cyclic time. Thus, I would not overestimate the importance of that "limitation". More important is, that the DutyCycle of a CCU/RM will not be increased just because a restart of an external engine results in massive direct XMLRPC requests that end up in massive RF communication and thus an overload. As said, a smart home central is something that you would not restart daily or even regularly and "stablizes" only in mid/long term uptime. |
If you use hacs you could click on the integration and press "erneut herunterladen". Then you could also select the beta. |
I agree, but to understand me: In the last months, power outages are happening here quite regularly. So when a reboot happens, it's not always wanted. |
Simply buy a UPS to prevent power outages, I would suggest as a valid solution ;) Really: it is totally valid and normal that battery driven RF devices should and often cannot be actively queried. Thus, you have to make sure that your automations can deal with the default values as well until sich battery RF devices send new values with their next cycle! |
I'm playing around with |
So, Now I could test the latest 0.37.0b1 beta version you referenced with my production environment at home here. While I could immediately spot an improvement in terms of raised dutycycle due to a restart of HomeAssistant or HACS or even HA core, I still spot an increase of about 15% of DutyCycle here (compared to 20-25% previously)... So can you please perhaps elaborate on the exact API methods you are using one after another to query all the different device information on a restart of hahomematic/custom_homematic so that we can perhaps try to further optimize the data queries up to a level that would definitly not increase the DC like other external applications like ioBroker or pocketControl (smartPhone app)? |
Updates are received by XmlRPC callbacks. |
I created a new beta (0.37.0b2) that should further reduce the no of api calls. |
Thx. Any of these kind of changes are appreciated. However, I still notice a DC increase of about 18-20% upon restart of the integration. So no real change compared to 0.37.0b1 but definitly better than 0.36.x And thanks for the technical details on the API calls used. I will try to invest some time in future to look more detailed into that in trying to identify the concrete API calls that are responsible for the rest of the DC increase. My current guess is, that we might try to think about replacing some of the JSONRPC or XMLRPC calls by queries to the ReGa-Skripting API on Port 8181 as the ReGaHss is more or less the central db for all homematic devices in a CCU and it uses massive caching to reduce RF communication to a minimum. |
I created a new beta (0.37.0b3) as an alternative to 0.37.0b2 with With these json implementations i cannot see a HA restart on the DC diagram anymore. Would is your mixture of HmIP/BidCos/Virtual devices? |
Thx for another beta version. However, this version is again back to an increase of DC of about 30-35% compared to only 18-20% with 0.37.0b2. Thus, I would propose to revert the getValue() change as this seems to definitly increase the DC by a similar amount like the 0.36.x versions.
Thanks for the hint. I will try to test this during the next days and report back as well as provide some more statistics on my device setup. But I definitly have a whole bund of BidCos-RF devices and not so much HmIP devices. So lets see where this might end up in the end when I switch of different interfaces. I will also try to log the amount of API calls to the central in some way and then try to see which API calls make the DC raising even worse. But at least 0.37.0b2 definitly performed better/nicer than 0.37.0b3, I am afraid. |
0.37.0b3 was just a quick test to sort out if the problem might be related to getParamset, which it is obviously not as much as getValue. |
0.37.1 adds a semaphore(1) to improve cache usage (less api calls / one api call per paramset). Add this to configuration.yaml to enable api logging:
|
Installed 0.37.0 (which comes with hahomematic 0.37.1), but noticed the same slight DC increase like 0.37.0b2. Tried your suggestion to identify the interface which actually causes the DC increase and found out that as soon as I disable the BidCos-RF interface the 18-20% DC increase vanishes and that with just HmIP and VirtualDevices enabled I also cannot see any DC increase at all. So I will need to debug this further in seeing why exactly only with the BidCos-RF interface (rfd) a DC increase is happening and why this is only happening for hahomematic and not e.g. with ioBroker. Still my current guess is, that this could also be some slight limitation of the rfd XMLRPC interface to some extend and that we would probably need to perform some data queries with ReGa scripting instead (ioBroker is actually doing this) - which in fact we could also do with the JSONRPC interface function And thanks for the debugging possibilities. This really helps and I was also thinking about requesting improved debugging capabilities for hahomematic :). Only thing that you might probably improve/change in this respect is to make sure that the password for |
From my past experience i expected the problem with the BidCos devices. I'm curious what you find We could give ReGa scripting a try, if this is the only solution, alltought i have no clue how to do it. So it is up to you to deliver the required code pieces. Is iobroker really loading all data at startup? |
Haven't invested a lot of time yet (more to come) I just walked through the Homematic XMLRPC documentation and stumbled on the And still I don't understand yet, why only the BidCos-RF interface seems to be affected by the DutyCycle increase if
Definitly. If we really would need rega scripting here to further reduce the DC I think I can definitly assist here by writing/providing the necessary rega scripts for the data queries.
That's actually a very good question, which I would have to find out first (this is nothing I regularly have on my table). But perhaps @foxriver76 can assist on that question or provide the necessary links to iobroker code passages of the xmlrpc or rega scripting based data queries in ioBroker.hm-rpc or ioBroker.hm-rega? At least in a quick search I could not spot any |
I think i already found something interesting https://github.com/ioBroker/ioBroker.hm-rega/blob/master/regascripts/datapoints.fn |
Funny. This is exactly what I would have suggested next, but first wanted to share my thoughts in my last comment. :) So yes, this kind of rega code snippet is able to retrieve all current datapoint values of all devices in the rega db. So you could simply throw that kind of code into the |
@jens-maus
|
Perfect. I just did it and now there is no DutyCycle increase at all upon restart of HomeAssistant. I can also spot the In addition, I can see that you execute that ReGa script for every interface. But AFAICS this script has to be executed only once for all interfaces and not for every interface startup again. So perhaps you can change your internal logic in some way that it will only be executed once before all interface xmlrpc registrations. Furthermore, using this But seeing that using ReGa scripting seems to work and does not raise the DutyCycle anymore is really great. Thanks for that and I hope you/we can make it even more smooth until this is available to the public. BTW: Where can I see your 0.37.1b1 code modifications in hahomematic because I can't see these changes in the hahomematic respository yet?!?! |
I don't want to add the script result to the debug output, but if a device has initial data then it is from the script.
It's currently a dirty hack. It's not executed per interface. It should be executed once initially and when newDevices is called.
Good idea, but there might be an issue if people use different ports.
Thanks for pointing me in that direction.
The code is currently a fast and dirty hack. Now i will clean it up and push it to the repo. |
Code can now be seen here |
Can this issue be closed? |
Since changes for this issue i got a warning in the RM-Logs every time i use "Integration neu laden" or "Homeassistant neu starten"
|
Regarding this issue, I can indeed report that the increase of DutyCycle due to the use of However, perhaps you can indeed have a look at what @Baxxy13 stated here regarding warning and error messages which are appearing now in the |
I noticed that on restart of Home Assistant the Duty Cycle still goes up for me on 1.0.4. Also I can see my HM Doorbell (HM-OU-CFM-TW) blinking red once on reboot of HA. Since it is not used in any automation or any other means in Home Assistant (spare device...) I assume it is because some command gets sent to it on HA reboot. |
Which CCU device are you using? External or using the RaspberryMatic Addon within HA? |
Original CCU3 (=Raspi 3) flashed with RaspberryMatic (latest). |
custom_component/hahomematic version (if applicable):
0.36.3
Home Assistant version (if applicable):
core-2022.3.3
CCU version:
Problem-relevant configuration:
Do you use tls?
No
Do you use callback?
No
Do you use username and password?
Yes
Which interfaces do you use?
BidCos-RF, homematicIP, homematicIP-Wired
Describe the bug
In case of a restart of HomeAssistant core or the whole HomeAssistantOS, or even in case of an update of the
custom_homematic
version the queried CCU/RaspberryMatic instance receives quite a tremendous amount of device / data queries in a way that causes the CCU / RaspberryMatic device to generate real RF traffic so that the DutyCycle of the connected CCU / RaspberryMatic device is raised quite tremendously. Dependent on the actual number of RF devices connected to the CCU / RaspberryMatic device, this could easily result in a DutyCycle of 100% (e.g. here in case of two consecutive restarts ofcustom_homematic
) so that no further RF communication is possible for that CCU / RaspberryMatic deviceTo Reproduce
Steps to reproduce the behavior:
custom_homematic
is querying all necessary device information.Relevant HA log entries
Expected behavior
custom_homematic
should use device query pathes / APIs that won't result in actual RF communication to be forced on the connected CCU / RaspberryMatic device so that the DutyCycle won't be increased at all.Screenshots
n/a
Additional context
If wanted, I could personally have a look at the actual mechanisms which are currently used to query device information from within
hahomematic
and/orcustom_homematic
and then try to work out a solution which won't increase the DutyCycle anymore. However, for this to happen it would be great if someone (@SukramJ ?) could actually point his finger at the actual code passages inhahomematic
orcustom_homematic
which are doing the device queries, so that alternative APIs or access pathes could be considered by me.The text was updated successfully, but these errors were encountered: