Add Tuya v2 quirk builder

[prairiesnpr]

Proposed change

This extends QuirkBuilder to add add_tuya_dp this allows DPs and converters to be added in the following manner.

.tuya_dp(
  3,
  TuyaSoilMoisture.ep_attribute,
  "measured_value",
  converter=lambda x: x * 100,
)

Or if the if it's known attribute, such as battery, temperature, or soil_moisture.

.tuya_temperature(dp_id=5, scale=10)

I've implemented replacement V2 Quirks for the existing soil quirks and tested with a _TZE284_aao3yzhs.

Additional information

Full Quirk Example:
Orginal V2: #3411
With TuyaQuirkBuilder: https://gist.github.com/prairiesnpr/e32e10b86a0736f5ff09693f6006ab3b

Checklist

• The changes are tested and work correctly
• pre-commit checks pass / the code has been formatted using Black
• Tests have been added to verify that the new code works

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.36%. Comparing base (a29ad2f) to head (f1323ac).
Report is 9 commits behind head on dev.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev    #3417      +/-   ##
==========================================
+ Coverage   88.71%   89.36%   +0.64%     
==========================================
  Files         306      308       +2     
  Lines        9820     9985     +165     
==========================================
+ Hits         8712     8923     +211     
+ Misses       1108     1062      -46     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheJulianJES]

Nice work!

Currently, we have a bit of an issue with duplicating parts of a v2 quirk. There's no proper way to share them, so a downside with this approach is that there'll be somewhat of a duplication for basic datapoints like temperature and so on.
Personally, I'd probably keep the Tuya datapoint mappings in the temperature/soil clusters and rather only define Tuya datapoints in a v2 quirk (with this approach) for specific/custom entities that are unlikely to be common across devices.

That would allow us to easily re-use common clusters like temperature/soil (in v1 and v2 quirks), but adds the flexibility to just tack on custom v2 entities with Tuya datapoints in a v2 quirk if needed.

In the long term, we should still take a look at how we can better share parts of a v2 quirk across different quirks, but I think common datapoint mappings just make more sense to have in a cluster compared to a in a device/quirk. What do you think?
EDIT: See my newer comment.

[dmulcahey]

Could we add methods to this for the known datapoints / clusters that are shared? Either by device type (I think they have required dp’s for each device type) or by functionality? Maybe something like .temperature, .onoff, .fan etc or if device type specific something like .smart_switch, .relay, .plug, etc? And have that pull in the dp’s (or clusters) depending on which way we decide to go?

[TheJulianJES]

Ah, regarding my previous comment. I somewhat forgot that the Tuya datapoints are obviously defined on the Tuya manufacturer cluster and not on the virtual "temperature cluster". I had in my mind that you just do .replaces(TuyaTempCluster) (and have the datapoints defined in that cluster), but that's obviously not the case.
So, you can kind of disregard my previous comment (except the duplication part).

Some methods like .tuya_temperature (with optional multiplication and optional other datapoint kwargs) in the TuyaQuirkBuilder would make sense. It could add a virtual ZCL temperature cluster and create the datapoint mapping.
I think I really like that approach. Should be easy to implement as well with the current TuyaQuirkBuilder.

This would also cause a lot less duplication across devices, as it's only like a single line (maybe with a differing datapoint or multiplication value). I really like this idea 😄

[prairiesnpr]

I think I prefer methods by functionality but still pass the DP. We could even have them go ahead and add the appropriate entity, a switch if it's onoff etc.

I'd like to be a bit careful that we don't abstract too much. A typical user building a quirk will go pull a list of DPs using something like https://www.zigbee2mqtt.io/advanced/support-new-devices/03_find_tuya_data_points.html. I'd like to keep it straightforward so that they can quickly scan a quirk and see that the quirk matches their device and that they can just add the manufacturer and model. I think having the list of DPs in the quirk will help with that even at the cost of some duplication.

[dmulcahey]

I think I prefer methods by functionality but still pass the DP. We could even have them go ahead and add the appropriate entity, a switch if it's onoff etc.

I'd like to be a bit careful that we don't abstract too much. A typical user building a quirk will go pull a list of DPs using something like https://www.zigbee2mqtt.io/advanced/support-new-devices/03_find_tuya_data_points.html. I'd like to keep it straightforward so that they can quickly scan a quirk and see that the quirk matches their device and that they can just add the manufacturer and model. I think having the list of DPs in the quirk will help with that even at the cost of some duplication.

Couldn’t we have our own version of this that also determines if the quirk exists and maybe even creates the PR if it doesn’t 🤔

[TheJulianJES]

I'm not set on this yet, but would it make sense to append a tuya_ prefix to all Tuya related methods?
Just so there aren't any conflicts down the road and it's clear this is for Tuya datapoints?

[prairiesnpr]

I'm leaning towards adding the tuya prefix. I'm working on adding support for adding attributes, switches, and enums now, and if we add the prefix, then you can still call the original methods if needed. I think it adds clarity and keeps functionality that we may need in the future if we run into something that doesn't fit the normal approach.

[prairiesnpr]

b83c29f adds methods corresponding to switch, sensor, binary_sensor, enum, and number from QuirkBuilder. This allows for a quirks such as this #3411 to be simplified to https://gist.github.com/prairiesnpr/e32e10b86a0736f5ff09693f6006ab3b

[prairiesnpr]

Will need a zigpy release for tests to pass, zigpy/zigpy@0707ff3 added entity_type for switches.

[TheJulianJES]

Also, would it make sense to move everything Tuya quirk builder related (including the small local clusters) into something like tuya/builder.py, tuya/mcu/builder.py, or tuya/builder/__init__.py?

[prairiesnpr]

Do we have any other Tuya methods we would want added to this initial PR?

[TheJulianJES]

I think this is almost good to go. Nice work 😄

[TheJulianJES]

The Metering cluster is most often used for tracking energy consumption in devices/quirks.
Most Tuya devices seem to support the ZCL cluster for that at least (just with a manual/different multiplier/divisor), so they do not send it via Tuya datapoints.
I thought about renaming this to tuya_water_metering, but that doesn't make a lot of sense, as we can change the metering_cfg. So, this is just a note.. It might not be obvious that tuya_metering is used for tracking water consumption by default.

[TheJulianJES]

The LocalDataCluster should return this as unsupported by default, right? (and thus prevent entity creation for that)
I mean, it doesn't hurt to explicitly add it as unsupported again, but is there a reason I'm missing?

[prairiesnpr]

Maybe this is a bug upstream? If you don't add it to unsupported attributes, HA will create the entity.

[TheJulianJES]

Hmm, I'll try and check on this in the future. The attribute is in the REPORT_CONFIG for the smart energy cluster, so it should be read upon pairing: https://github.com/zigpy/zha/blob/3d34a30499a80ad1802fbc72d36ad5cd5dc89ac0/zha/zigbee/cluster_handlers/smartenergy.py#L80-L84
When the attribute is read from the LocalDataCluster, it should return "unsupported attribute", as there's no value in the attribute cache for it.
Let's leave this in then.

[TheJulianJES]

Is the AttributeDefs stuff done like this, so we don't accidentally modify the "global class"?
If so, couldn't we create an AttributeDefs instance class in the __init__ of the quirk builder and just set the attributes there? Or am I missing something?

[dmulcahey]

AttributeDefs are just classes… there are no instances

[TheJulianJES]

We're "dynamically modifying" AttributeDefs in the tuya_attribute method. I was trying to say/ask if this affects "all AttributeDefs" used in quirks created with the builder (which it should not) or if there's a specific AttributeDefs class (which is what I meant with "instance" above) assigned to every quirk built with the builder (which should be the case).

[TheJulianJES]

I was thinking if we need something similar to where types.new_class is used here in zigpy, as we don't create an AttributeDefs class with attributes that are the same for all quirks created with the Tuya builder: https://github.com/zigpy/zigpy/blob/dfeb958b1f737b0a6c9dedbdb7fc169a188a516e/zigpy/zcl/__init__.py#L136-L143

[dmulcahey]

I was thinking if we need something similar to where types.new_class is used here in zigpy, as we don't create an AttributeDefs class with attributes that are the same for all quirks created with the Tuya builder: https://github.com/zigpy/zigpy/blob/dfeb958b1f737b0a6c9dedbdb7fc169a188a516e/zigpy/zcl/__init__.py#L136-L143

I think this is the correct way to do it

[dmulcahey]

That said… these really are for making code clearer / easier to write…. Do we need the AttributeDefs here or is the dict syntax sufficient?

[prairiesnpr]

I was in a hurry and missed a commit, this should be resolved now. While this seems odd, it's the only way I found that actually worked. Setting the attribute on TuyaReplacementCluster directly or replacing AttributeDefs similar to zigpy results in the attributes being dropped when passed to replace.

[TheJulianJES]

Do we need the AttributeDefs here or is the dict syntax sufficient?

(assuming "dict syntax" is just talking about the old-style zigpy attributes definition with a dict)
I was under the impression we no longer wanted to include them if possible. Although it's unlikely zigpy will drop them in the near future.. 😄

If it's easily possible, I'd prefer to use the new-style AttributeDefs. We can also fall back to the old-style attributes otherwise (at least with the new ZCLAttributeDef objects).
I guess for a case like this, using AttributeDefs is more difficult than just using the dict style.

---

If we want to use new-style AttributeDefs, mind trying to see if something like this works? (replace the whole add_to_registry registry method with below).
Not saying this is the best solution, just curious if this works..

    def add_to_registry(self) -> QuirksV2RegistryEntry:
        """Build the quirks v2 registry entry."""

        class TuyaReplacementCluster(TuyaMCUCluster):
            """Replacement Tuya Cluster."""

            data_point_handlers: dict[int, str]
            dp_to_attribute: dict[int, DPToAttributeMapping]

            def __init_subclass__(cls) -> None:
                cls.AttributeDefs = types.new_class(
                    name="AttributeDefs",
                    bases=(TuyaMCUCluster.AttributeDefs,),
                )
    
                for attr in self.new_attributes:
                    setattr(cls.AttributeDefs, attr.name, attr)

                super().__init_subclass__()

            async def write_attributes(self, attributes, manufacturer=None):
                """Overwrite to force manufacturer code."""
                return await super().write_attributes(
                    attributes, manufacturer=foundation.ZCLHeader.NO_MANUFACTURER_ID
                )

[TheJulianJES]

Ok, so, apparently that doesn't seem to work. The current way in this PR works at least.

I've thought about it again and it's preferable to use AttributeDefs if possible.
If the base clusters (TuyaMCUCluster and so on) use AttributeDefs, we also have to. If they use the attributes dict, we also have to.
Since AttributeDefs is the preferred way to go, we should use that here and convert the other clusters to AttributeDefs.

(EDIT: Actually, it looks like zigpy's conversion might accept both AttributeDefs and attributes dict in the same cluster at the same time?)

I'll try and take another stab at maybe slightly changing/improving what we have now (regarding AttributeDefs), but the current implementation works already..

[TheJulianJES]

Looks good 😄