Integration Quality Scale: Chapter 3

[joostlek]

Integration Quality Scale: Chapter 3

Link to the previous chapters

So after reading and taking in all feedback on the first two chapters, we've come to a proposal for the integration quality scale.
In this discussion, I will explain the proposed scale and the reasoning behind it.
I will also shed some light on what happens to the current scale.

The current scale

Currently, we have 4 scaled tiers: no score, silver, gold and platinum.
Like we've stated in the first chapter, those tiers have rules that are outdated and do not reflect the current state of Home Assistant.
We also have one non-scaled tier, internal, which is used for system integrations, like helpers, entity platforms, and the like.

The description of the current tiers is as follows:

No score

This integration passes the bare minimum requirements to become part of the index.

• Satisfy all requirements for creating components and creating platforms.
• Configurable via configuration.yaml

Silver 🥈

This integration is able to cope when things go wrong. It will not print any exceptions nor will it fill the log with retry attempts.

• Satisfying all No score level requirements.
• Connection/configuration is handled via a component.
• Set an appropriate SCAN_INTERVAL (if a polling integration)
• Raise PlatformNotReady if unable to connect during platform setup (if appropriate)
• Handles expiration of auth credentials. Refresh if possible or print correct error and fail setup. If based on a config entry, should trigger a new config entry flow to re-authorize.
• Handles internet unavailable. Log a warning once when unavailable, log once when reconnected.
• Handles device/service unavailable. Log a warning once when unavailable, log once when reconnected.
• Operations like service action calls and entity methods (e.g. Set HVAC Mode) have proper exception handling. Raise ServiceValidationError on invalid user input and raise HomeAssistantError for other failures such as a problem communicating with a device.
• Set available property to False if appropriate
• Entities have unique ID (if available)

Gold 🥇

This is a solid integration that is able to survive poor conditions and can be configured via the user interface.

• Satisfying all Silver level requirements.
• Configurable via config entries.
  • Don't allow configuring already configured device/service (example: no 2 entries for same hub)
  • Discoverable (if available)
  • Set unique ID in config flow (if available)
  • Raise ConfigEntryNotReady if unable to connect during entry setup (if appropriate)
• Entities have device info (if available)
• Tests
  • Full test coverage for the config flow
  • Above average test coverage for all integration modules
  • Tests for fetching data from the integration and controlling it
• Has a code owner
• Entities only subscribe to updates inside async_added_to_hass and unsubscribe inside async_will_remove_from_hass
• Entities have correct device classes where appropriate
• Supports entities being disabled and leverages Entity.entity_registry_enabled_default to disable less popular entities
• If the device/service API can remove entities, the integration should make sure to clean up the entity and device registry.
• When communicating with a device or service, the integration implements the diagnostics platform which redacts sensitive information.

Platinum 🏆

Best of the best. The integration is completely async, meaning it's super fast. Integrations that reach platinum level will require approval by the code owner for each PR.

• Satisfying all Gold level requirements.
• Set appropriate PARALLEL_UPDATES constant
• Support config entry unloading (called when config entry is removed)
• Integration + dependency are async
• Uses aiohttp or httpx and allows passing in websession (if making HTTP requests)
• Handles expired credentials (if appropriate)

As can be seen, the old scale is outdated.
It still heavy relies on configuration via YAML, whilst the majority of the integrations are now configured via the UI.

The proposed scale

So the scale I want to propose has a few endpoints we want to address:

• We want to strive to get every integration to have a place on the scale. This way we can explain end users what to expect from an integration.
• We want the integration quality scale to act as a guideline for developers to improve their integration.
• We want to make sure that the scale is easy to understand and easy to use.
• We want to provide a transparent and reasonable bar for new integrations.
• We want to expand the scope of the scale to also focus on additional things like documentation and user experience.

The scaled tiers

The proposed scale has 4 scaled tiers, bronze, silver, gold, and platinum.
Below are the proposed characteristics for each tier.

🥉 Bronze

The bronze tier is the baseline standard and requirement for all new integrations. It meets the minimum requirements in code quality, functionality, and user experience. It complies with the fundamental expectations and provides a reliable foundation for users to interact with their devices and services.

The documentation provides guidelines for setting up the integration directly from the Home Assistant user interface.

From a technical perspective, this integration has been reviewed to comply with all baseline standards, which we require for all new integrations, including automated tests for setting up the integration.

Characteristics:

• Can be easily set up through the UI.
• The source code adheres to basic coding standards and development guidelines.
• Automated tests that guard this integration can be configured correctly.
• Offers basic end-user documentation that is enough to get users started step-by-step easily.

🥈 Silver

The silver tier builds upon the “Bronze” level by improving the reliability and robustness of integrations, ensuring a solid runtime experience. It ensures an integration handles errors properly, such as when authentication to a device or service fails, handles offline devices, and other errors.

The documentation for these integrations provides information on what is available in Home Assistant when this integration is used, as well as troubleshooting information when issues occur.

This integration has one or more active code owners who help maintain it to ensure the experience on this level lasts now and in the future.

Characteristics:

• Provides everything “Bronze” has.
• Provides a stable user experience under various conditions.
• Has one or more active code owners who help maintain the integration.
• Correctly and automatically recover from connection errors or offline devices, without filling log files and without unnecessary messages.
• Automatically triggers re-authentication if authentication with the device or service fails.
• Offers detailed documentation detailing what the integration provides and instructions on troubleshooting issues.

🥇 Gold

The gold standard in integration user experience, providing extensive and comprehensive support for the integrated devices & services. A gold-tier integration aims to be user-friendly, fully featured, and accessible to a wider audience.

When possible, devices are automatically discovered for an easy and seamless setup, and their firmware/software can be directly updated from Home Assistant.

All provided devices and entities are named logically and fully translatable, and they have been properly categorized and enabled for long-term statistical use.

The documentation for these integrations is extensive, and primarily aimed toward end-users and understandable by non-technical consumers. Besides providing general information on the integration, the documentation provides possible example use cases, a list of compatible devices, a list of described entities the integration provides, and extensive descriptions and usage examples of available actions provided by the integration. The use of example automations, dashboards, available Blueprints, and links to additional external resources, is highly encouraged as well.

The integration provides means for debugging issues, including downloading diagnostic information and documenting troubleshooting instructions. If needed, the integration can be reconfigured via the UI.

From a technical perspective, the integration needs to have full automated test coverage of its codebase to ensure the set integration quality is maintained now and in the future.

All integrations that have devices in the Works with Home Assistant program are at least required to have this tier.

Characteristics:

• Provides everything “Silver” has.
• Has the best end-user experience an integration can offer; streamlined and intuitive.
• Can be automatically discovered, simplifying the integration setup.
• Integration can be reconfigured and adjusted.
• Supports translations.
• Extensive documentation, aimed at non-technical users.
• It supports updating the software/firmware of devices through Home Assistant when possible.
• Provides the tooling and documentation troubleshooting issues.
• The integration has automated tests covering the entire integration.
• Required level for integrations providing devices in the Works with Home Assistant program.

🏆 Platinum

Platinum is the highest tier an integration can reach, the epitome of quality within Home Assistant. It not only provides the best user experience but also achieves technical excellence by adhering to the highest standards, supreme code quality, and well-optimized performance and efficiency.

Characteristics:

• Provides everything “Gold” has.
• All source code follows all coding and Home Assistant integration standards and best practices and is fully typed with type annotations and clear code comments for better code clarity and maintenance.
• A fully asynchronous integration code base ensures efficient operation.
• Implements efficient data handling, reducing network and CPU usage.

The non-scaled tiers

In addition to the new 4 scaled tiers, we will also have 4 non-scaled tiers.
These tiers are used for integrations that do not fit the scaled tiers.

❓ No score

These integrations can be set up through the Home Assistant user interface. The “No score” designation doesn’t imply that they are bad or buggy, instead, it indicates that they haven’t been assessed according to the quality scale or that they need some maintenance to reach the now-considered minimum “Bronze” standard.

The “No score” tier cannot be assigned to new integrations, as they are required to have at least a “Bronze” level when introduced. The Home Assistant project encourages the community to help updating these integrations without a score to meet at least the “Bronze” level requirements.

Characteristics:

• Not yet scored or lacks sufficient information for scoring.
• Can be set up via the UI, but may need enhancements for a better experience.
• May function correctly, but hasn’t been verified against current standards.
• Documentation most often provides only basic setup steps.

🏠 Internal

This tier is assigned to integrations used internally by Home Assistant. These integrations provide basic components and building blocks for Home Assistant's core program or for other integrations to build on top of it.

Everything in Home Assistant is extendable using integrations. In fact, Home Assistant uses its own integration logic to extend base functionality. For example, the automation engine of Home Assistant is an integration. Those internal parts are thus marked with this “Internal” tier.

Internal integrations are maintained by the Home Assistant project and subjected to strict architectural design procedures.

Characteristics:

• Internal, built-in building blocks of the Home Assistant core program.
• Provides building blocks for other integrations to use and build on top of.
• Maintained by the Home Assistant project.

💾 Legacy

Legacy integrations are older integrations that have been part of Home Assistant for many years, possibly since its inception. They can only be configured through YAML files and often lack active maintainers (code owners). These integrations might be complex to set up and do not adhere to current/modern end-user expectations in their use and features.

The Home Assistant project encourages the community to help migrate these integrations to the UI and update them to meet modern standards, making these integrations accessible to everyone.

Characteristics:

• Complex setup process; only configurable via YAML, without UI-based setup.
• May lack active code ownership and maintenance.
• Could be missing recent updates or bug fixes.
• Documentation may still be aimed at developers.

📦 Custom

Custom integrations are developed and distributed by the community, and offer additional functionalities and support for devices and services to Home Assistant. These integrations are not included in the official Home Assistant releases and can be installed manually or via third-party tools like HACS (Home Assistant Community Store).

The Home Assistant project does not review, security audit, maintain, or support third-party custom integrations. Users are encouraged to exercise caution and review the custom integration’s source and community feedback before installation.

Developers are encouraged and invited to contribute their custom integration to the Home Assistant project by aligning them with the integration quality scale and submitting them for inclusion.

Characteristics:

• Not included in the official Home Assistant releases.
• Manual installable or via community tools, like HACS.
• Maintained by individual developers or community members.
• User experience may vary widely.
• Functionality, security, and stability can vary widely.
• Documentation may be limited.

Rationale behind the newly added tiers

The new bronze tier

We introduced the bronze tier as the baseline standard for all new integrations.
This tier would act as the minimum requirement for all new integrations and ensures that all integrations meet the minimum requirements in code quality, functionality, and user experience.
It provides a reliable foundation for users to interact with their devices and services.

The legacy tier

We introduced the legacy tier to differentiate between integrations that are not graded and ones that lie far below the current standards.
This way we want to let the end users know that these integrations might need some more work to install and operate, compared with what they are used to with the integrations configurable through the UI.

The custom tier

We introduced the custom tier to make sure that every integration has a place on the scale.
The benefit of this, is that the end users know what to expect from an integration, when comparing it to a core integration.
This tier will be automatically added by core.

Works with Home Assistant

Since Home Assistant is getting more and more popular, more manufacturers are interested in integrating their devices with Home Assistant.
The Open Home Foundation has a program called the Works with Home Assistant program.
The updated scale will require all integrations that have devices in the Works with Home Assistant program to be at least at the gold tier.
This way we can ensure that the end users get the best experience possible when using devices from the Works with Home Assistant program.

The rulebook

The above clearly defines the purpose of each tier. The scaled tiers are bound to a set of rules, that need to be satisfied in order to fit into that tier. This chapter is aimed a listing all rules tied to each tier.

🥉 Bronze

• 001 - Integration needs to be able to be set up via the UI
• 002 - Test a connection in the config flow
• 003 - Don't allow the same device or service to be able to be set up twice
• 004 - Full test coverage for the config flow
• 006 - Use ConfigEntry.runtime_data to store runtime data
• 007 - Check during integration setup if we are able to set it up correctly
• 008 - If it's a polling integration, set an appropriate polling interval
• 011 - Entities have a unique ID
• 012 - Entities use has_entity_name = True
• 022 - Entities event setup
• 034 - Dependency transparency
• 035 - Services are added in async_setup
• 036 - Place common patterns in common modules
• 040 - The documentation includes a high-level description of the brand, product, or service it integrates with; if available, with links to the brand, product, or service website.
• 042 - The documentation provides step-by-step installation instructions for the integration, including, if needed, prerequisites.
• 043 - The documentation provides removal instructions.
• 048 - The documentation described the provided actions that can be used.
• 053 - Has branding assets available for the integration

🥈 Silver

• 005 - Support config entry unloading
• 009 - If internet/device/service unavailable, log a warning once unavailable and once when back connected
• 010 - Mark entity unavailable if appropriate
• 017 - Services raise errors when encountering failures
• 020 - Reauthentication
• 023 - Set Parallel updates
• 024 - Above average test coverage for all integration module
• 025 - Has a code owner
• 037 - Keep the config flow understandable and validate where needed
• 044 - The documentation describes all integration installation parameters.
• 045 - The documentation describes all integration configuration options.

🥇 Gold

• 013 - Entities have translated names
• 014 - Entities use device classes where possible
• 015 - Entities use DeviceInfo correctly
• 016 - Entities are assigned an appropriate EntityCategory
• 018 - Integration disables less popular (or noisy) entities
• 019 - Is discoverable when possible
• 021 - Clean up stale devices
• 026 - Implements diagnostics
• 030 - Errors are translatable
• 031 - Icon translations
• 032 - Reconfigure flow
• 033 - Dynamic devices
• 038 - Make effective use of the config entry data and options
• 039 - issues and repairs are used effectively
• 041 - The documentation describes use cases to inspire the end-user in using this integration.
• 046 - The documentation provides a list of known supported / unsupported devices.
• 047 - The documentation describes the supported functionality, including entities, and platforms
• 049 - The documentation describes how data is updated and, if applicable, the polling interval used or how it is determined.
• 050 - The documentation describes known limitations of the integrations (not to be confused with bugs).
• 051 - The documentation provides troubleshooting information.
• 052 - The documentation provides examples the user can use. For example (snippets of) configuration, automations, blueprints, scripts, or dashboards.

🏆 Platinum

• 027 - Dependency is async
• 028 - If making HTTP requests, support passing in websession
• 029 - Strict typing

Adjusting the scale

Since we never really had rules around adjusting the scale or an integration tier, I would like to propose the following rules for adjusting the scale.

Adjusting the tier of an integration

Home Assistant encourages our contributors to get their integrations to the highest possible tier, to provide an excellent coding experience for our contributors and the best experience for our users.

When an integration reaches the minimum requirements for a certain tier, a contributor can open a pull request to adjust the scale for the integration. This request needs to be accompanied by the full checklist for each rule of scale (including all rules of lower tiers), demonstrating that it has met those requirements.

Once the Home Assistant core team reviews and approves it, the integration will display the new tier as of the next major release of Home Assistant.

Besides upgrading integration to a higher tier on the scale, it is also possible for an integration to be downgraded to a lower tier. This can, for example, happen when there is no longer an active integration code owner. In this specific example, the integration will be downgraded to “Bronze,” even if it otherwise fully complies with the “Platinum” tier.

Adjustments to rules contained in each tier

The world of IoT and all technologies used by Home Assistant are changing at a fast pace. Not just in terms of what Home Assistant can support or do, but also the software Home Assistant is built upon is constantly innovated. Home Assistant is truly pioneering the technology in the industry at a fast pace.

This also means that new insights and newly developed and adopted best practices will occur over time, resulting in new additions and improvements to the individual integration quality scale rules.

If a tier is adjusted, all integrations in that tier need to be re-evaluated and adjusted accordingly. One exception to this is integrations that have devices that are part of the Works with Home Assistant program. Those integrations will be flagged as grandfathered into their existing tier.

So what now?

Once this proposal is accepted, a few things will happen:

• The developer documentation will get extensive explanation per rule, with the rationale and examples. (To give an example of what I have in mind for this, the ruff linter rules are a good example).
• This proposal (excluding the rulebook) will be added as a new Architecture Decision Record in the ADR folder. To ensure that this is a change for long term and that the process is transparent.
• All current graded integrations will be re-evaluated and moved to the appropriate tier.
• All non-config flow integrations will move to the legacy tier

I would love to hear if you agree with this proposal; as we would love to move forward with all the rest of the implementation steps 🙏

[frenck]

As I've been there while it was written, I've reviewed it at this point 😄

I agree with the current proposal and framework. I think it is a major leap forward, and it sets clear goals on each of the scaled tiers but also provides tiers for the ones not fitting the scale. More importantly, it is making the user experience more central.

👍 ✅ Approved from my end.

[bdraco]

Overall looks solid. 👍 from me.

Consider moving 028 to gold or lower as not doing so has been the cause of blocking I/O in the event loop to create SSL context, and some other major performance problems with creating a new session every time.

[starkillerOG]

Then there is almost nothing left in platinum ;)

[joostlek]

I like to see this also as a gateway drug. I see the bronze level as a bare minimum, but I will most likely be still advocating a bigger goal for new integrations like a "hey you use entity descriptions with non translatable names, with a few extra lines you can make your integration even better and use translated names". Yes, there will be people that are only willing to do the bare minimum, but most people will cooperate if you advice them to go for a certain solution.

I think moving the 028 to lower, will also give the impression that we already want you to use aiohttp or httpx, which we do like, but we do not require.

[balloob]

For 015 - Entities use DeviceInfo correctly, I think that we can drop the word "Correctly". Everything needs to be correct. (I would love to highlight model_id but it's too specific implementation detail)

Otherwise, no further comments. ✅ 👍 from me.

[frenck]

I would love to highlight model_id but it's too specific implementation detail

We discussed this actually, we decided to make that part of the DeviceInfo rule details.

[zweckj]

I like it, though I find 041 a bit strange.

• it is not really measurable when this is fulfilled (like what is enough)
• imo users install integrations because they know what they want to achieve already and „unusual“ usage of the integration might not even be known to the code owner, so will rather end up in forums than anywhere else.

Besides that, I'd like a copyable checklist/template, that is in a format we desire, so that quality scale PRs have a standardized format as well.

[frenck]

The goal behind 41 is to have a high-level overview in the introduction on how one could/would generally use the integration. (E.g., "Can pull in your energy data to track your energy usage in the Home Assistant energy dashboard").

Details on what it entails should go, just like all other rules, into the specific dev docs for each rule.

Besides that, I'd like a copyable checklist/template

Yes, that is on the list. There are also other ideas, like storing meta data with the integration. That said, this is not the goal of this proposal, the proposal is to get a definition of the IQS.

[zweckj]

The goal behind 41 is to have a high-level overview in the introduction on how one could/would generally use the integration. (E.g., "Can pull in your energy data to track your energy usage in the Home Assistant energy dashboard").

Details on what it entails should go, just like all other rules, into the specific dev docs for each rule.

It currently sounds pretty much like 052, so I think clearer differentiation is needed. Same reasoning applies to 052 then. While I think it is nice to include that for more obscure integrations for lots of integrations that control devices the intended use cases are pretty clear.
Maybe merge 401 and 052 and add that as optional, so people are checking for this while grading, but it’s not a hard requirement. I don’t like it since it's hard to measure and I don’t believe it’s generally applicable / adding value.

Besides that, I'd like a copyable checklist/template

Yes, that is on the list. There are also other ideas, like storing meta data with the integration. That said, this is not the goal of this proposal, the proposal is to get a definition of the IQS.

This is not for automation but to make sure, it's easy for codeowners to re-apply for a iqs and to ensure the PRs are consistent when having to regrade 100s of integrations.

[andrew-codechimp]

• All current graded integrations will be re-evaluated and moved to the appropriate tier.
• All non-config flow integrations will move to the legacy tier

Is there documentation of the process on how to get an integration graded?

[joostlek]

When an integration reaches the minimum requirements for a certain tier, a contributor can open a pull request to adjust the scale for the integration. This request needs to be accompanied by the full checklist for each rule of scale (including all rules of lower tiers), demonstrating that it has met those requirements.

Once the Home Assistant core team reviews and approves it, the integration will display the new tier as of the next major release of Home Assistant.

[andrew-codechimp]

Read it twice and still missed that bit 😞
A template style checklist will be good for these if you're expecting a lot of submissions.

[joostlek]

It's still friday afternoon after all

[mib1185]

Should we still use the term "code owner" or maybe to lately introduced "integration owner" 🤔 🙈

[frenck]

Good comment 👍

[gjohansson-ST]

If I get it right it seems all helpers should be marked internal but it's not entirely clear if that's the case?

Otherwise 👍 from me.

[ScottG489]

Maybe this is rare case, but where would the alert integrate fall? It's definitely legacy since it's YAML-only, but from a few discussions I've had it's also not something we want to move to the UI or work on further.

The word "legacy" to me say that it isn't something that's going to be maintained further. However, it sounds like how legacy is defined here is mostly meant for YAML-only integrations?

Perhaps another category like "deprecated" could be useful if there are enough other integrations like this?

[frenck]

It isn't deprecated, so that isn't correct. It will be marked legacy; from a developer perspective, we have frozen it in this moment in time.

While you do have a point, I think it also isn't needed to adjust the whole framework to accommodate a few edge cases :)

[ScottG489]

Maybe I'm missing something but I don't think you'd have to adjust anything, would just be another category added on.

Regardless, it sounds like there aren't really enough integrations in this situation to deem another category anyways.

Still, from a user perspective it might be nice to know that, hey you probably shouldn't use this because it isn't going to be maintained further. I use alerts and I regret it haha. Perhaps just some kind of warning in the docs would suffice?

In any case, we can continue this discussion elsewhere since this is more relevant to an integration's "status" rather than "quality" I suppose.

[IceBotYT]

If accepted, would all open PRs to adjust quality scale have to be redone as well? See home-assistant/core#125932

[frenck]

Once we enforce the ADR, it will be applied to all (new) integrations; so yes, that would affect all open PRs.

[rytilahti]

Great work @joostlek, just a couple of suggestions which may or may have not been discussed previously:

• It would be great if (at least some of) the checks could be automated at some point. This would greatly reduce the review efforts and make it also visible to integration developers where they could improve. I realize that such progress would take time and is not feasible for all requirements, but enabling compliance check automation would make everyone's life easier in the long run.
• Please consider including the (machine-readable) checklist for quality scale as part of the integration meta data. Probably not in manifest.json, but in a similar manner next to the integration as a single source of truth. When a PR implements some rule, it can be ticked off as part of the change. As a plus, if this file is automatically updated when the "root" rule template / definition file changes, the codeowners would get notified to adjust their integrations accordingly. Likewise, this file could be used to adjust the level automatically.
• Having the rules named after their scale would be helpful Perhaps something as simple as an enumeration BRONZE-x instead of plain integer identifiers could do the trick?

[frenck]

It would be great if (at least some of) the checks could be automated at some point.

Yes, but that is a next step.

Please consider including the (machine-readable) checklist for quality scale

Same, but for now, this is about getting a ruleset to start with. One of the ideas is to add a iqs.json next to the manifest file (for example).

Having the rules named after their scale would be helpful Perhaps something as simple as an enumeration

I would disagree on that, as changing rules changes their name. We probably end up with IQS-001 or something.

[rytilahti]

Fair enough, just wanted to chime in as I didn't see anything in the otherwise concrete descriptions related to adjustments 👍

[starkillerOG]

Looks very nice, well done.

A few small remarks:

• 041 and 052 are basically the same thing: I would drop 041, keep 052 and possibly add the "inspiring" to 052.

• I feel like the platinum quality scale is a bit empty so personally I would shift some of the requirements from gold to platinum (and maybe from silver to gold and bronze to silver). It feels like getting the first step on the quality scale is really hard, which may discourage devolopers, while the last step (platinum) is easy when you already have gold.

• a bit off topic: "047 - The documentation describes the supported functionality, including entities, and platforms", it would be really nice if we could also show this info right at the entities in HomeAssistant itself, like the description of the config flow input fields. (Like when you hover over an entity or so).

[starkillerOG]

041 and 052 are basically the same thing: I would drop 041, keep 052 and possibly add the "inspiring" to 052.

They are not. 41 tells how it could be used (e.g., "This integration allows you to monitor your energy meter and monitor your energy usage in Home Assistant"). While 52 is about actual implementation examples and details like automations and whatnot.

In that case I misunderstood 41, I would maybe suggest rephrasing 41 a bit to make that more clear.

At the same time, in that case 41 is very simular to 40: a general description of what the integration provides.

[frenck]

In that case I misunderstood 41, I would maybe suggest rephrasing 41 a bit to make that more clear.

The next step after this is details pages for each rule to clearly define those. The rulebook isn't set in stone either, we can improve along the way, including changing/extending. This proposal sets out to have a good definition and a starting point, allowing us to dive deeper into the next step.

[joostlek]

The main goal would probably be to just give an answer to the "why would someone integrate that?" question. For example, nyt_games would be an integration which would raise such response at people. In that case it would be nice to inspire people that they can use that for alerting when they haven't done their wordle. (This part can be highly subjective as my use case isn't your use case). While 40 is just a general intro about what this service or device does in the first place

[starkillerOG]

It feels like getting the first step on the quality scale is really hard, which may discourage devolopers,

I mean I would probably have way more in the bronze tier as I like to review a lot of new integrations and I think there are a lot of quick wins you can add to make your integration nice. I think the bronze scale looks intimidating, but I'd rather be transparent upfront than drag them through a review process of a few months. Now we can be more transparent about what we expect from a new integration. I think with the added bonus of better docs we can make sure this is feasable and reachable.

But yea, it may look like a long list, but in the end, I think this is the bare minimum to also provide a good starting point to make your integration nice in the future.

I indeed think the bar for a starting integration is really high.

If I would have started to program today instead of a few years back, I don't know if I would have become a integration-owner with a starting bar that is this high. It's a very steep learning curve, and back in those days I needed to learn a lot. As a starting programmer you also need some successes to keep you motivated to continue the learning curve.

I guess I am just afraid we scare off starting developers.

[frenck]

To be fair, the bar is (at this moment) higher, so the proposed scale is lowering it. But, if you have a concrete proposal, feel free to share it.

[rytilahti]

On the platinum being rather empty, just a new thread to avoid mixing too many things together. One idea about the code quality, perhaps higher scale levels could also have some quality requirements for the test code? Not necessary from the get go, but something that could push towards the best practices (parametrize, mock properly), avoid c&p tests, use facilities provided by the core, etc. given maintaining test code is also part of good code hygiene.

[joostlek]

I like this idea, but for the topic on test code, I never really had the idea we had a consensus on what is good test code. I think this is worth exploring and maybe able to be added on a later stage. I think test code also varies between the type of integration, so we would have to collect general rules.

A few things that come to mind:

• Use fixture mocks instead of inline patching (not that there is a consensus on this, but I find this way cleaner than having all the with patch() all over the place)
• Cache fixture files (I think we do cache a lot already, but sometimes we don't)

[autinerd]

One thing could also be the usage of snapshots where applicable.

[mback2k]

I would like to suggest to add an "if possible" to "003 - Don't allow the same device or service to be able to be set up twice". Some services or hubs (eg. the one integrated via wmspro integration) do not support or provide a unique identifier besides the MAC address which may not be discoverable if the device is on another network. As hostnames and IP addresses can be influenced by the user, it would for example be possible to setup the same device at least twice right now: once with a name-based config entry and once with an IP-based entry.

[joostlek]

I think there still should be a way to make wmspro able to detect if its already setup, since the unique_ids you use for the entities are identifiers coming from the device, so if you setup the same device twice, it would raise all kinda of issues because of duplicate entity ids

[mback2k]

We could compare the available devices/entities and abort if is the same setup. Or do this via a hash over the internal WMS destination IDs, but then we would need to keep that one updated. But good point, I guess I will implement the basis for this in the pywmspro library.

Other than that, I think this new framework makes the quality scale much more approachable and transparent.

[joostlek]

But to get back to the point, I think "if possible" applies to more rules, but I'd rather put that in the explanation so people at least know the consequences of not applying it before calling it a day. Because yes, there are integrations where it's just not possible.

[catsmanac]

If you all are ready and want to test run this, then I'm offering an unrated integration, Enphase-envoy, to go through the process and identify the practical side of this.

[davet2001]

Looks good to me!

Minor point: I think 048 should be present tense ('describes' rather than 'described').
Minor point: If being published in an official way, the use of capitalisation and '.' should be consistent between each bullet.

Suggestion: The only other thing I would consider is potentially adding some expectations on the dependency library at the gold/platinum level, e.g. whether it is actively maintained, pinned dependencies, API strict typing and perhaps test coverage. We rightly push for business logic to be in these libraries rather than HA, but beyond that, the bar is quite low.

Observation: I think there is a task management/responsibility challenge on the upgrading/downgrading of tier ratings by PR. Since there is no obvious tangible advantage to having an integration labelled as e.g. gold vs silver, some will probably be left in the wrong category when they should change. Additionally with stale integration owners, there is no real incentive or person to notice inactivity and downgrade them. We may have to accept that the reported quality tier will lag the reality to some extent. Automation would help with this.

The ideal for me would be for the scale to motivate developers automatically (e.g. "you are 78% of the way to achieving silver status"), rather than being an additional thing we are asking them to do by self-assessment.

But those are just side points, and do not affect the proposed scale/rules which I 100% agree with. Great job @joostlek!

[rytilahti]

👍 on documenting the library expectations. I know there are some checks related to typing as I got bitten by it myself, and I think I've seen license checks being enforced by the CI. The new scale should, in my opinion, definitely also codify such requirements.

Regarding to your observation on the scale adjustments, I very much agree on that and I think that will also happen due time. :-)

[marcelveldt]

Looks good, super nice work @joostlek !
My only comment is the "legacy level" - I find it super weird to have that as level and especially confront users with it.
My proposal would be to set back all integrations to "no score", regardless of its origin. So these advanced/legacy/advanced integrations will maybe stay in "no score" forever or until someone picks up the glove to convert it to something which at least matches bronze.

[frenck]

The proposal, as provided in the current OP, has been accepted. Meaning we can continue with setting up everything that is needed to get this new scale out the door 👍

../Frenck

[frenck]

The scale is live!

https://developers.home-assistant.io/docs/core/integration-quality-scale/