Letting the user extend the semantics model

[lolodomo]

PR #3519 introduced a technical way to add dynamically new tags in the semantics model. This was an important part but this is only a small part of what must be achieved to let the final user create and control the tags he would like.

I try to list all the missing steps:

1. Enhance what @jimtng did to also support removing tag classes
2. Add a new tag manager that will store all user defined tags in a JSON file (id, label, description, synonyms). This DB is loaded at startup and the corresponding tags are added in the model (class injection).
3. Extend the REST API to add/remove a tag and its synonyms that will call the new tag manager to update the DB
4. Create a new textual configuration file to let the user define a list of tags through a clear syntax
5. Extend MainUI to manage tags (add/remove tags) using the extended REST API

Please let me kn ow if I am missing something.

The first 3 items are the minimum requirements and I will try to provide items 2 and 3.

[spacemanspiff2007]

• Create a new textual configuration file to let the user define a list of tags through a clear syntax

When I first thought about how I can integrate it into HABApp I thought about adding a .yml file format where the user can define the tags. Instead of having yet another proprietary file format maybe that would be a good idea for openHAB, too?
From the structure it shouldn't be too complex so yml could fit nicely ...

[lolodomo]

When I first thought about how I can integrate it into HABApp I thought about adding a .yml file format where the user can define the tags. Instead of having yet another proprietary file format maybe that would be a good idea for openHAB, too?
From the structure it shouldn't be too complex so yml could fit nicely ...

Yaml is not very user friendly IMHO and all our configuration files use a dedicated syntax that is more user friendly.
But as the required structure for tags is very simple, that could be an easier solution than implementing all the XText stuff.
I will see that later.

Implementing managed tags (with its JSON DB) is already something not so trivial if you don't have a good knowledge of the OH core storage stuff. I am sure that I will need more time than a core maintainer but I can take examples from how it is done for items or things.

[J-N-K]

Please don't introduce new XText. XText is a PITA to work with and currently blocks us from upgrading GSON. Upgrading XText this time is even more difficult than the last time because they introduced dependencies on equinox runtime.

[lolodomo]

And @jimtng added the ability to add tag class but not to remove them. It is also something else missing. Added in my first message.

[lolodomo]

Regarding the textual config file, one option could be use a simple CSV file, similar to what we already have in code to generate the predefined model.

[rkoshak]

Add a new tag manager that will store all user defined tags in a JSON file (id, label, description, synonyms). This DB is loaded at startup and the corresponding tags are added in the model (class injection).

I know you are just listing examples, but the hierarchy of the tags is important too. Don't forget to capture the parent tag which can be any existing tag, not just Location, Equipment, Point or Property.

Extend the REST API to add/remove a tag and its synonyms that will call the new tag manager to update the DB

And return the full list. If MainUI is going to support this (and there's no reason it shouldn't) it needs to get the full list from the API too. Showing them in a hierarchy would be useful and informative so maybe return the list as a JSON hierarchy.

Yaml is not very user friendly IMHO and all our configuration files use a dedicated syntax that is more user friendly.
But as the required structure for tags is very simple, that could be an easier solution than implementing all the XText stuff.

Ephemeris uses XML so having a different format from some custom XTex based format or JSON has precedent. Compared to XML and JSON, YAML is way more user friendly. But in addition to CSV, .ini and .properties files could fit here too. I agree, we should try to move away from inventing our own file formats for file based stuff in OH where possible, even if XText didn't have the problems @J-N-K mentions.

The one advantage I can see to YAML over CSV or some other format is that in MainUI these tag definitions will be shown as YAML on the Code tab. It would be kind of nice if we could inch towards consistency between what you see in the UI and what you do in files. I know we can never get there entirely but even baby steps would be an improvement.

[spacemanspiff2007]

Yaml is not very user friendly IMHO and all our configuration files use a dedicated syntax that is more user friendly.

From all the file formats (csv, json, xml) in this threads yaml is without doubt the most user friendly.
Csv and json don't even allow comments which is a must have for a user editable file format.
And as @rkoshak mentioned we already show and use yaml in the UI so it would only be consistent to use it more.

The custom XText files might have shorter syntax, but it's still impossible to e.g. define nested item metadata because nobody knows how to define it in XText. I'd rather have a feature complete but more verbose file format which I can read with a somewhat standard file parser than another format which only benefit is that it's shorter.

[J-N-K]

There seems to be a well-maintained library for YAML which is also an OSGi bundle: SnakeYAML. It also supports serialization, so in the long-term we could switch from XText to YAML because we could parse the model and write a converted YAML for it.

[spacemanspiff2007]

@J-N-K just a heads up:
snakeyaml-engine seems to be a yaml 1.2 parser so it might be an alternative. I don't know about the osgi parts though.
It takes some rough edges out of the yaml file format and is a little bit more "robust".
Yaml1.2 is also a superset of json, so @lolodomo could write json files and it would still work. ;-)

[J-N-K]

It seems that snakeyaml-engine is not able to parse to custom types and always returns Lists/Maps. This is terrible, we already have that with XStream as XML parser and I'm very happy that we recently started to use JAXB.

[spacemanspiff2007]

I think it would be really good to have a 1.2 parser because the changes make it work much more straight forward and there are less traps for new users.
In python I typically use the yaml parser which returns the Lists / Maps and then I use a data validation library to parse this data into custom types. Maybe something like this will work for Java, too?
Non the less I'll definitely prefer a yaml 1.1 over the other file formats, however yaml 1.2 is more user friendly than 1.1.

---

It seems that snakeyaml-engine is not able to parse to custom types

It's possible to directly and arbitrarily embed custom type directives in yaml, e.g.

my_key: !!my.custom.data.type
   - value1

Doing that is normally discouraged and most yaml parsers provide a "save load" which ignores this directive.
From what I understand this is was snakeyaml-engine does by default so imho this is not a problem.
But I'm no java programmer and if snakeyaml is the better fit then so be it 😄

[lolodomo]

And @jimtng added the ability to add tag class but not to remove them. It is also something else missing. Added in my first message.

It looks like unloading a Java class is not really possible.
But we can probably remove the tag class from the semantic model and then let Java do Garbage Collector.

[J-N-K]

@spacemanspiff2007 I just found that we have YAML support already integrated:

    <dependency>
      <groupId>com.fasterxml.jackson.dataformat</groupId>
      <artifactId>jackson-dataformat-yaml</artifactId>
      <version>${jackson.version}</version>
      <scope>compile</scope>
    </dependency>

This seems to use SnakeYAML (but 2.0, so only YAML 1.1 support). I don't think we should add another YAML engine in that case, it is already confusing that we have

• JSON: GSon, Jackson (and others in openhab-addons)
• XML: XStream, JAXB, Jackson
• YAML: Jackson

[florian-h05]

@rkoshak

And return the full list. If MainUI is going to support this (and there's no reason it shouldn't) it needs to get the full list from the API too.

There already is a PR to load the tags from REST, but it is waiting for review and merge: openhab/openhab-webui#1882.
What probably postpones merging is, that the PR will remove the translations from MainUI, but not all are at core. They however cannot he migrated, because core wants plural as well. I think we shouldn’t block progress because of 7 languages that are already partly translated — I would contact the translators over CrowdIn and ask to translate at core.

[lolodomo]

I believe I should be able to submit a PR soon, at least a partial PR.

[lolodomo]

The main part is now implemented.
I just have to finish the custom tag update (class annotation to be updated at runtime).

Next step (and final step for me) will then be the adding of a provider taking custom tags from a configuration file.

[lolodomo]

There already is a PR to load the tags from REST

In my PR (not yet merged), I have enhanced with two new fields the data returned by the API. I hope this will not break your PR.

[florian-h05]

I can adjust my PR if needed.

[lolodomo]

Any idea how to show/manage the tags tree in MainUI to allow adding/removing custom tags?
Note that the new REST API will only allow removing/updating part of the tree with custom tags, and forbid the update/deletion oif tags from the pre-defined model.

[lolodomo]

We have the model built by the user by assigning tags to items, this one is called the semantic model in MainUI.

How do you call the model with the predefined hierarchy of tags that can now be extended with custom tags ? Isn't it also the semantic model ? Or the tags model ?

[lolodomo]

There already is a PR to load the tags from REST, but it is waiting for review and merge: openhab/openhab-webui#1882.
What probably postpones merging is, that the PR will remove the translations from MainUI, but not all are at core. They however cannot he migrated, because core wants plural as well. I think we shouldn’t block progress because of 7 languages that are already partly translated — I would contact the translators over CrowdIn and ask to translate at core.

I agree, especially when you are not talking about the most used language all over the world.

@florian-h05 : with your UI PR, will the custom tags be automatically considered in the locations/equipments/points/properties tabs of the UI ?

[lolodomo]

Any idea how to show/manage the tags tree in MainUI to allow adding/removing custom tags?

I imagine we could have a new page "Tags" in addition to pages "Things", "Items", "Model", ...

With my PR in progress, you will receive all tags (predefined + custom) but there is currently no way to distinguish them in the result of the API call. I guess you will need this information. I can add a new boolean field "custom" ?

[lolodomo]

@J-N-K : for the configuration files, I will have to handle the external updates of files triggering the update of the tags registry. Can you please tell me where I can find a good entry point where it is already done in core framework ?

I already know that I need to add a new custom tag provider that will read the file(s) to return a list of custom tags.
But how to monitor the update of these files ?

Regarding the starting process, I am not yet sure we need the custom tags being loaded before the items ? If custom tags are used on items and custom tag is not yet loaded, I am not sure of the consequence. And if the load happens after, is it ok ?

[florian-h05]

Any idea how to show/manage the tags tree in MainUI to allow adding/removing custom tags?

I‘d propose a tree view such as for the semantic model on a separate page, see the next paragraph regarding naming.

How do you call the model with the predefined hierarchy of tags that can now be extended with custom tags ? Isn't it also the semantic model ? Or the tags model ?

I‘d name it‘s menu point „Semantic Tags“ with description „Manage tags for the semantic model“.

I agree, especially when you are not talking about the most used language all over the world.

Great, can you probably comment on my UI PR to ask Yannick to review it?

will the custom tags be automatically considered in the locations/equipments/points/properties tabs of the UI ?

Exactly, yes. Everywhere UI uses the semantic tags, the ones loaded from the API will be used as they are not hard-coded to the UI anymore.

With my PR in progress, you will receive all tags (predefined + custom) but there is currently no way to distinguish them in the result of the API call. I guess you will need this information. I can add a new boolean field "custom" ?

I‘d rather name it editable, this is what REST API provides for Items, Things, Transformations etc.

[lolodomo]

I‘d rather name it editable, this is what REST API provides for Items, Things, Transformations etc.

You're right.
It is now added.

[lolodomo]

I have now a file provider (YAML) working. I will submit a new PR probably tomorrow after the initial PR is merged.
The only point I have to think about is the case you have depending tags defined in different files and you remove the file containing a parent tag.

[lolodomo]

MainUI is now correctly handling semantic tags added through the REST API, that is a very good point. I will do more tests, I have done only a basic tests with the addition of a location and an equipment.

I believe this new feature will be very appreciated by a lot of users and I hope we can make it usable in a more convenient way before OH4 is released.

I can easily understand that adding a new page in MainUI to show and manage semantic tags is not something trivial and I am not sure that the remaining time before OH4 is released will be sufficient.

So I would like to provide at least the config file option. I was asked not to use XText to create a new file format but rather YAML format. I think my proposal in PR #3659 is already ready at 95% and I even considered @J-N-K general concept proposed in #3666. Note that there is not yet any UI tab showing YAML code for a semantic tag and the YAML code for a semantic tag is very simple.
@J-N-K : as you were already involved to expose your vision about YAML configuration, I would really appreciate a lot if we could comment my PR and help me to solve the few open points (bug with WatchService, where to put certain class in core and how to make existing integration tests compile).

[lolodomo]

I close this issue as we have everything in 4.1 except the UI part for managed custom tags.