RFC REST-API v2

[manup]

Why

I think it's time that we need to figure out a proper v2 REST-API to support a wider range of devices which are available today or might arrive in the future. I'm increasingly unhappy about the /lights hack which is currently used in some places to add devices which don't fit in the provided categories.

A first humble attempt to address more complex devices was the /devices endpoint which allows querying all sub-devices behind one physical device referenced by its MAC address. dresden-elektronik/deconz-rest-plugin@96daf83

Request For Comments

To proceed I invite everyone to share ideas and wishes how a new interface may look like.
Beside this GH issue the RFC REST-API v2 wiki page should be used to develop the new API.

Initial thoughts

• The current API v1 is used by several home automation projects, therefore v2 should be implemented as a parallel interface in order to not break any v1 client.
• The v2 API should be considered completely independent from v1, it should be the best possible API and not a workaround or compatibility compromise.
• A proper security scheme should be integrated. HTTPS can't be used due certificate complexity. However using widely available libsodium we can have end-to-end encryption, authentication and identity management without much hassle.
• The internal Resource and ResourceItem C++ classes are a good common ground to build a generic API on top. These are already used in many places and simplify the implementation of the rules engine. For v2 they are likely to be extended to be more versatile.
• deCONZ and the REST-API is now nearly 10 years old with some historically grown code. As a by-product of v2 new code must be designed and implemented with a proper architecture to escape spaghetti land. The code and many processes like joining new devices could be such much better than it is right now. By now we have so much knowledge about various devices, what can go wrong and how-to workaround issues manually, that's punishment of the user and should be handled by code. The related architecture and design decision should be thoughtful developed with RFC documents.
• The implementation of various switches have shown that a general notion of capabilities might be desirable.
• The v2 interface should be as high level as possible, clients shall not be required to understand any Zigbee specific knowledge. However an optional low-level v2 interface can be offered in parallel to provide full access to Zigbee world.
• Currently Websockets are used to provide real time events, MQTT can be offered as well. Qt has native support for it so the implementation is simple (in fact there exists already a very basic prototype for it).

[ebaauw]

Love where this is going! My initial thoughts below.

The current API v1 is used by several home automation projects, therefore v2 should be implemented as a parallel interface in order to not break any v1 client.

Agreed. Probably provide /api2/... or /api/v2/... series of endpoints next to the current /api/.... We should manage the apiversion properly so clients can know what features are available, especially while the v2 API is still being developed.

The v2 API should be considered completely independent from v1, it should be the best possible API and not a workaround or compatibility compromise.

Agreed. This does mean no more compatibility with the Hue API, making it hard(er) for clients to support both the deCONZ gateway and the Hue bridge. I'll probably create a separate homebridge-deconz plugin for the v2 API.
It probably also means that the (almost) Hue-compatible v1 API needs to be maintained indefinitely, or at least as long as the Hue API is relevant. We could eventually "dumb down" the v1 API to be fully Hue compatible.

A proper security scheme should be integrated. HTTPS can't be used due certificate complexity. However using widely available libsodium we can have end-to-end encryption, authentication and identity management without much hassle.

I particularly like the "without much hassle" part. I'm not sure adding a dependency to yet another package qualifies as that. There seems to be a NodeJS implementation, or at least a wrapper, of libsodium. I'm not sure if it needs an external component, which might exclude some more exotic homebridge installations like in a docker container on a Synology DiskStation. Then again, a separate homebridge-deconz plugin would likely run on the same system as deCONZ itself.
I managed to use HTTPS to the Hue bridge, including pinning the certificate for ph, but haven't yet implemented that in homebridge-hue (due to time constraints - so much to do, so little time). Some hassle alright, but no additional packages needed.

The internal Resource and ResourceItem C++ classes are a good common ground to build a generic API on top. These are already used in many places and simplify the implementation of the rules engine. For v2 they are likely to be extended to be more versatile.

I haven't yet given that much thought, but I'm not sure I would agree. Looking at the next and previous points, it might actually be better to start a new deCONZ REST API v2 plugin from scratch, rather than gradually changing the current plugin.

deCONZ and the REST-API is now nearly 10 years old with some historically grown code. As a by-product of v2 new code must be designed and implemented with a proper architecture to escape spaghetti land. The code and many processes like joining new devices could be such much better than it is right now. By now we have so much knowledge about various devices, what can go wrong and how-to workaround issues manually, that's punishment of the user and should be handled by code. The related architecture and design decision should be thoughtful developed with RFC documents.

I couldn't agree more. I think a key ingredient would be a proper class hierarchy with subclasses to handle exceptions instead of lengthy methods with dozens of if statements.

The implementation of various switches have shown that a general notion of capabilities might be desirable.

For me capabilities are needed at two levels:

• For API clients (e.g. to know which button events a particular switch supports). These need to be exposed trough the v2 API;
• For device support (e.g. to know what devices make creative use of the ZigBee standard). Preferably these are specified in a JSON (or even XML) file instead of in code, so adding support for more similar devices would be much easier by not requiring to compile the plugin. Cf. general.xml to configure the deCONZ GUI.

The v2 interface should be as high level as possible, clients shall not be required to understand any Zigbee specific knowledge. However an optional low-level v2 interface can be offered in parallel to provide full access to Zigbee world.

I like the idea of a layered architecture, where the "raw" low-level v2 API would be RESTifying the ZCL commands and attributes (cf. the level of the GUI interface and the CLI plugin), on top of which which the high-level v2 API would provide the "cooked" home automation features (caching values, rules, ...). We might even consider developing these layers as two separate repositories. On the long run, the high-level API v2 might be used with implementations of the low-level API v2, e.g. using the RaspBee/ConBee serial interface to the radio device, or using other hardware (e.g. XBee). And the Hue-compatible v1 API might eventually be based on the same low-level v2 API.

Currently Websockets are used to provide real time events, MQTT can be offered as well. Qt has native support for it so the implementation is simple (in fact there exists already a very basic prototype for it).

What about the "without much hassle" part? Afaik, MQTT requires an additional server component where web sockets do not.

[dergreg]

I‘m not able to contribute anything useful due to a lack of knowledge of the inner workings of deConz/ZCL (I‘m merely an API user). But I suggest taking a look at the Z-Way (Devices) API which I‘m using to control some ZWave powered devices on my RPi, such as blinds and outlets.
Devices are grouped into a set of types, which in turn offer a few properties to query and update via a REST API , which is rather convenient to use.

[titanous]

A proper security scheme should be integrated. HTTPS can't be used due certificate complexity. However using widely available libsodium we can have end-to-end encryption, authentication and identity management without much hassle.

I strongly recommend not rolling a new transport security protocol. That's a huge amount of implementation work, and making something new implies a bunch of work for all clients (TLS is available everywhere, your custom protocol will not be) and almost certain security vulnerabilities.

I think for this application, bearer tokens are sufficient for authorization and authentication. Transport security should definitely be TLS, you could use QSslSocket to provide this, or use an external proxy like Ghostunnel or Caddy to handle the TLS termination. A tool like mkcert could be used to generate certificates, or if necessary a custom tool could be developed and integrated (I'd be happy to help explore this option if needed).

[titanous]

Oh, and I should note that this isn't just a drive-by suggestion 😄

I'm a happy deConz user, and I'm quite happy to help you through any questions and issues around implementing TLS and bearer tokens properly.

[manup]

Hi all, thanks a lot for sharing the first ideas. I've been researching the topic of software architecture since a while now and found some stuff in the Internet which might be useful to proceed with the REST-API v2. Albeit this GH issue is quite young there are already a lot ideas and comments on ideas. Perhaps it's time to get these "on paper", before they get lost in the growing comment stream, see (2) below.

As a side note and small update, the Phoscon internal software developer team (we are 6 now) is cooking various new features and refactoring parts of the system, there is a notable a rush to get more thought in the craft of solid, maintainable code, compared to the past we're using way more paper and white-boards now :)

Back to the architecture topic, I think the basic idea of the following might be useful:

1. The C4 Model to visualize the software and its parts in a rather simple way without going too crazy with detailed uml diagrams and notions.

   I don't think we should pickup the C4 Model or any other model lightly. But having diagrams similar to that, which provide a simplistic visual view of the whole system, might help to develop the architecture and moreover could lower the entry barrier to new developers which wan't to get involved.

2. Architectural Decision Records (ADR) to discuss multiple approaches to solve a task and based on that make decisions where to go. The following example very quickly shows how it works: https://adr.github.io/madr/docs/adr/

   It could help to structure and divide aspects of the new REST-API v2 like the HTTPS vs. alternative encryption solutions into their own wiki sections, or as ADR suggests own markdown pages like 0008-support-https.md and 0009-support-custom-encryption.md. These can be referenced from higher level wiki pages and can also be cross linked.

   In a nutshell:

   • Common structure/template which isn't too cluttered
   • It's just Markdown (MADR), one wiki page per ADR
   • Allows discussion of alternatives
   • Documents everything which was considered, accepted, proposed or rejected
   • Can be cross linked: from wiki pages, GH issues, other ADRs and even in code comments
   • Keeps the main architecture wiki pages nice and clean
   • Of course not everything needs an ADR, but I recon it could be an asset here and there

What do you think, should we bring the first ideas in the wiki and would you consider something like ADRs to be useful? As alternative we could write aspects into own sections, but I'm afraid this might get too cluttered over time and thoughts which were part of decisions easily get lost. Perhaps there are better alternatives?

[titanous]

I like the idea of using ADRs, but I think it'd be good if they went through Pull Requests into either this repo or separate one. That way comments and additions could be discussed around each ADR on PRs. I don't think putting them in the wiki would facilitate discussions well.

[KodeCR]

First of all, I think this is a great initiative.

At this point some more general comments are probably most useful. These are of course my personal opinions so feel free to ignore them, but:

• I'm all behind a well thought out and agreed software architecture. I'd say the methodology to use for that is less important. It could just be the whiteboard and the wiki, but some of the models might help give some structure.
• Maybe even more importantly is to have a plan to transition towards it (I'm not sure of what you're currently intending to do). Starting from the ground up with the plan to replace looks to be of high risk to get bogged down in my experience.
• The REST api by design and being based on JSON should allow arbitrary extensions which should be ignored by clients not supporting them. I see the issues and workarounds for the current api mostly as a failure of the app clients.
• That said, supporting a basic compatible api and an advanced api I guess is the next best thing. Since we are going for compatibility for the first, making it fully compliant to the Hue api would be an idea I'm strongly behind.
• I'm no expert, but HTTPS does seem to be the easiest. What do you mean when you say "due to certificate complexity"?
• Zigbee(v3) looks like a well thought out architecture, at least there is plenty of experience behind it, so basing the low level api on that sounds sensible to me, in some ways it already is, right? Then again there is no shame in the new software architecture just being a cleaned up and refactored version of the old.

Hope this helps.

[ebaauw]

+1 for ADRs. I don't have any particular preference how to implement them (Wiki or PR).

I think using C4 or something similar would be beneficial, especially when starting with an overview of the various containers:

• Device firmware (providing the serial protocol, https://github.com/dresden-elektronik/deconz-serial-protocol);
• deCONZ core, providing the C++ interface, https://www.dresden-elektronik.de/fileadmin/Downloads/Dokumente/Produkte/6_Software/deconz-cpp-doc/index.html and the GUI;
• deCONZ REST API plugin, providing the new REST API;
• Phoscon web app, providing the web UI.

Preferably a with separate GitHub repository for each container. Now we get a lot of end-user support issues for Phoscon in the REST API repository, were none of the REST API contributors can do anything, as Phoscon isn't open source.

[Kane610]

I don't see a reason to go away from web sockets.

If anything we should allow all communication to go over web sockets when not emulating the hue api.

I think the main thing to improve is how to model the data, right now there are lights, sensors, groups and config endpoints. With all the different device types supported maybe there should be one endpoint for all physical devices and then other endpoints that are purely virtual.

One thing Im missing is to know what attributes of a device are optional and what are always available. Things like that should be documented.

All events should be signalled to api users, creation, update, removal

When you start developing the full api it would be nice if you could share a micro server emulating a full deconz server with all different devices available

[manup]

https://github.com/dresden-elektronik/deconz-rest-plugin/issues/1688#issuecomment-519979187

Preferably a with separate GitHub repository for each container. Now we get a lot of end-user support issues for Phoscon in the REST API repository, were none of the REST API contributors can do anything, as Phoscon isn't open source.

I like that, the current issues and topics are pretty broad and slightly chaotic. Organising them should clarify and help users and contributors.

Here's a rough sketch of a proposal:

• deCONZ Core and GUI
  To be honest, the existing deCONZ core, firmware and GUI won't be open sourced any time soon and will be kept restricted to dresden elektronik hardware. For the company there is currently no benefit to support 3rd party Zigbee modules since we only charge for the hardware. It is of course always possible to write a new core, after all the REST-API is just a plugin which implements a rather simple interface. I hope this is understandable, the commercial aspect here is tricky to keep everything running.

• REST-API v1
  The existing repository deconz-rest-plugin just continues and only hosts issues, the wiki and code directly related to the REST-API.

• REST-API v2
  A deconz-rest-plugin-v2 can host the ADRs in form of markdown files (to support PRs for them), source code and issues.

  I was thinking of keeping this in the v1 repository, but this might lead to unnecessary confusion. Refactoring v1 and also supporting v2 in one repository might be very tricky. However I can imagine that the code of the new API will be much much slimmer and produce reusable parts which can later be integrated into v1, replacing most of the spaghetti code without changing the public API. For example robust pairing process of devices and reading/writing of ZCL attributes. With a proper architecture this is possible. Therefore the main difference between v1 and v2 will be the public API but the real hard Zigbee stuff can be shared in future.

• Phoscon App
  Currently there is the no longer maintained phoscon-app-beta repository. A new repository phoscon-app can host issues and a wiki related only to to the Phoscon App.

• Firmware
  A deconz-firmware repository can host issues and links related to the firmware and serial protocol behind it. The currently private repository of the serial protocol will be moved into it. It can also contain links to projects which implement the serial protocol like Zigpy and the Mozilla IoT gateway.

• Phoscon Homepage
  Currently the sources are not public, bringing it into a repository would allow accepting PRs for compatibility list and various other pages. The Phoscon homepage is a static site mainly build with markdown and pandoc.

  There is no decision made yet but I like the idea.

  • The compatibility list is a structured JSON document which can be extended with meta data per device like: supported in REST-API v1/v2, Phoscon App, Homebridge, HA, Hue Essentials, etc. and links to images, documentation and known issues of the device. Since it can be queried via GET, it can be embedded in apps and project pages.

  • The release notes currently are only available via the Github. There is a new Atom/RSS feed in the work which can be queried by feed readers or apps (since it is just an XML file). From our side it will be translated in english and german, with the open source Phoscon App repository PRs can be accepted for further translations.
    https://phoscon.de/en/changelog/changelog.en.xml

  • A not yet public part of the homepage is the new REST-API documentation which surely would also benefit being open source.

As for topics like Websockets, MQTT and HTTPS or other crypto mechanisms, I think we are not restricted :) There can be multiple providers which just implement a certain high level interface and just forward events and actions to and from the desired format with encryption on top. The lower level code doesn't need to know about REST, JSON or crypto it only cares about commands with parameters and fires events with related data, therefore the API on this level is basically just C++ data structures passed between methods and implemented interfaces (which are registered and activated, like Websockets).
This should also result in testable code by default.

[lbschenkel]

One suggestion that I have is that it would be great if deCONZ core exposed an API to the higher layers which does not use Qt. The reason being that this would make it much easier to do independent development on top of it (which is nice, you'll sell even more hardware) and do experimentation.

I really wanted for example to build a proof-of-concept implementation of more-or-less the same REST API as the one existing now but using Rust instead. What I wanted to experiment with is trying to refactor the existing device-specific logic from being all over the place to an encapsulated quirks "database". The reason I don't do that today is because my knowledge of C++ is very rusty (from 2 decades ago, a different time) and I'm not used to Qt and Qt does not have too many bindings. If I could demonstrate the idea as a proof-of-concept, even if the code itself does not end up being used it's more useful to have working code reimplemented in C++ than a high-level description.

Regarding the new API, if the core API was not Qt-based but had pure C bindings then anybody with wild ideas would be able to sit down and implement an alternative REST layer with their stack of choice, which would be useful at the very least to exchange ideas for V2.

P.S.: I may not have been clear enough above, so I'm not suggesting that the core should not be implemented in Qt (this is not realistic, it would require a rewrite). Just that it would be nice if there was an exported API that did not use Qt, in order to make it easier to build something on top of the core in a different programming language than C++.

[SwoopX]

I'd also like to add my 2 cents. First of all, I'd suggest to treat every end device as what it is: a device or thing. That would mean wrapping it in such a meta class and read its endpoints, profiles, attributes and what not generically. I guess this is not far from what the core currently does and presents in the GUI. While having that, a device whould be extensible with device or vendor specific clusters, command or what is required to have it fully functional.

It's pretty much what zigbee2mqtt does with zigbee-herdsman. You do not have to touch the core functionality but can write an own extension for a (new) device. Really love this approach.

Another thing really great to have would be a revised debugging. For me, it would be of help to be able to see e.g. raw hex values in requests and responses (as well as relevant parts of the packet structure) when I'm after something. In that sense, wireshark often helped me to better understand what might have gone right or wrong if you know what I mean.

Lastly, as a sort of premium feature, it would be awesome to have the capability to send customized packets to force certain (or vendor specific) commands for example. Probably nothing for the API but maybe for a layer below.

I'm also fully with the other suggestions. Especially, I strongly support the HTTPS intruduction. Security should play an important role here, even if it's just inside ones personal network. As of now, I don't see such a difficulty using openssl. I'd generate a self signed cert during installation for basic security. However, it should be possible to also exchange them by your own (PKI-based) certs. This would knid of eliminate the necessity of using a reverse proxy.

[duffbeer2000]

Is there already a beta version of the REST-API v2?

[merdok]

I also think that the API should be rewritten and improved. Adding new devices is really a pain compared to zigbee2mqtt.
The current code is not really clear any more and for new people it is extremely hard to understand what is happening there.

[Smanar]

BTW, there is some news about this API V2 ?

[MattL0]

up !

any news for us @manup ?

[MattL0]

A mqtt api would be a nice add!

[filoor]

Any news?

[Kane610]

I'd like to expand on my previous comment.

I think there should be a list of all devices on a high level giving attributes such as name (there is no need to allow each endpoint to be named separately), model, serial, manufacturer etc... and then all endpoints fitting to be presented separately from that device should have its own entry as well.

Another important thing is that everything should have events over web socket. Right now it seems about 50% coverage :)

I don't see a reason to go away from web sockets.

If anything we should allow all communication to go over web sockets when not emulating the hue api.

I think the main thing to improve is how to model the data, right now there are lights, sensors, groups and config endpoints. With all the different device types supported maybe there should be one endpoint for all physical devices and then other endpoints that are purely virtual.

One thing Im missing is to know what attributes of a device are optional and what are always available. Things like that should be documented.

All events should be signalled to api users, creation, update, removal

When you start developing the full api it would be nice if you could share a micro server emulating a full deconz server with all different devices available

[Mimiix]

As per announcement on Discord, this Friday there will be a dev meeting again. Please share anything related. Concerns, wishes, feedback: All is welcome.

[Digital-Thor]

First of all, I'm a huge fan and user of deconz, so please take this all constructively!

It would be great to have a parameter available for each device where I could add some application state/data related to each device. Since I use the API continually, this feature would be convenient and simplify the design of browser applications.

Regarding websockets, please consider a way to configure which fields are included in event messages. If that's not possible, please at least include the:

• device's name, and
• model info.
  

Finally, please consider adding more explanations and E X A M P L E S. It took a long time to figure out how to pair devices using the api. I'm still not sure how "reachable" is determined. Regarding "Creating" sensors, is this for when we have unknown hardware?

[Smanar]

But with the type and the id, it's easy to find the name and the model ?
It increase the data size, and I don't see in wich one situation it can be usefull. ATM the websocket send only the minimal data, or worst, only the changed one.

[Digital-Thor]

But with the type and the id, it's easy to find the name and the model ?
It increase the data size, and I don't see in wich one situation it can be usefull. ATM the websocket send only the minimal data, or worst, only the changed one.

A slightly larger data size doesn't bother me since it's a websocket link (not a Zigbee RF link where data size is critical to preserving battery). Without those fields, I need to maintain a local data model for 80 devices in my app, make an API request every time a websocket event comes in, error-handle it, and when successful, merge with the local data model.

It's just a suggestion which I hope the Friday meeting can consider. If no one sees the value of this, the first part of my websocket suggestion is to let the developer configure which fields are included in event messages. Thanks for your consideration. :-)

[Smanar]

On my side I m making like you "maintain a local data model", But I don't need "make an API request every time a websocket event comes in", I just update the local data.

Some device like power mesurement plug can be realy talkative with websocket ^^.

[Kane610]

@Digital-Thor IIRC you can actually configure the web socket to send all data, so that should already be supported. Just like @Smanar Im doing a web request to deconz on start up and then I just add the changes to that data when changes are sent over web socket.

[Kane610]

For tonights discussion I'd like to point out since the improvements are gonna be incremental I propose that we focus on the device level API forward