RFC: YAML configuration

[J-N-K]

As discussed in #3636 we need configuration for user-defined tags. It was discussed that an easy to understand standard configuration format would be preferable over creating a new DSL. It was also suggested in different issues that it would be nice to be able to copy&paste from UI to files. Main UI uses a YAML representation for things and items, so this would be good choice. We already have a feature for Jackson's YAML parser, so no additional dependencies are needed for that.

However, there are a few things to discuss:

• Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?
• Should there be a versioning? This would allow us to better handle future upgrades.

I would suggest to add a YamlModelRepository (similar to our XText ModelRepository) that is an OSGi service that encapsulates all YAML processing and file processing. Consumer are OSGi services that implement YamlModelListener<T> where T is a DTO that describes the format. The listeners are injected into the YamlModelRepository and whenever a matching model (identified by the root name) is found the listener is notified.

public interface YamlModelListener<T> {
    void addedModel(String modelName, List<? extends T> elements);
    void updatedModel(String modelName, List<T> elements);
    void removedModel(String modelName, List<T> elements);
    Class<T> getElementClass();
    String getRootName();
}

Example listener (not fully implemented):

@Component(service=YamlModelListener.class)
public static class TestListener implements YamlModelListener<YamlItem> {
    private final Logger logger = LoggerFactory.getLogger(TestListener.class);

    @Override
    public void addedModel(String modelName, List<? extends YamlItem> elements) {
        logger.info("Added model {} with elements {}", modelName, elements);
    }

    @Override
    public void updatedModel(String modelName, List<YamlItem> elements) {
        logger.info("Updated model {} with elements {}", modelName, elements);
    }

    @Override
    public void removedModel(String modelName, List<YamlItem> elements) {
        logger.info("Removed model {} with elements {}", modelName, elements); 
    }

    @Override
    public Class<YamlItem> getElementClass() {
        return YamlItem.class;
    }

    @Override
    public String getRootName() {
        return "items";
    }
}

Example DTO:

public class YamlItem {
    private String name = "";
    private String type = "";
    private @Nullable String label;
    private @Nullable String category;
    private Set<String> tags = Set.of();
    private List<String> groupNames = List.of();
    private List<YamlLink> links = List.of();

    public String getName() {
        return name;
    }

    public String getType() {
        return type;
    }

    public @Nullable String getLabel() {
        return label;
    }

    public @Nullable String getCategory() {
        return category;
    }

    public Set<String> getTags() {
        return tags;
    }

    public List<String> getGroupNames() {
        return groupNames;
    }

    public List<YamlLink> getLinks() {
        return links;
    }

    @Override
    public String toString() {
        return "YamlItem{" + "name='" + name + "'" + ", type='" + type + "'" + ", label='" + label + "'"
                + ", category='" + category + "'" + ", tags=" + tags + ", groupNames=" + groupNames + ", links=" + links
                + "}";
    }
}

and the file:

version: 1
items:
  - name: Outdoor_temp
      type: Number:Temperature
      label: "Outdoor [%.1f °C]"
      groupNames:               
        - gTemperatures
        - gOutdoor
        - gPersist5
      links:
        - channel: "deconz:thing1:channel1"
        - channel: "deconz:thing2:channel2"
          profile: "MyScope:MyProfile

[rkoshak]

Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?

Thinking from the perspective of rule templates and sharing I think allowing it all to be configured in one file would be a nice step towards supporting a bundle of configs (e.g. a collection of Items, Rules, and Widgets). I know the marketplace can't handle that (yet?) but even with git repos and forum posts that would make sharing a little easier.

If we could support variables (like Ansible for an example) that could be very powerful from a sharing perspective. But I know, we need to walk before we run.

Should there be a versioning? This would allow us to better handle future upgrades.

I don't know the full implication from the developer perspective, but from an end user perspective it sounds like a really good idea. Upgrades are a pain point.

I would suggest to add a YamlModelRepository (similar to our XText ModelRepository) that is an OSGi service that encapsulates all YAML processing and file processing. Consumer are OSGi services that implement YamlModelListener where T is a DTO that describes the format. The listeners are injected into the YamlModelRepository and whenever a matching model (identified by the root name) is found the listener is notified.

Again, this might be jumping too far ahead, but I've always found it awkward that the only way to develop/test a rule template is to publish it in the marketplace. I would hope that this YamlModelRepository might provide a way to load a rule template locally without going through the marketplace. It would greatly improve the rule template development experience.

[lolodomo]

Ok, I understand what you want to achieve. All the stuff monitorting the folders/files and YAML files loading will then be in one unique place in the YamlModelRepository.

Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?

I would suggest to keep different files and different folders, like we have today.

Should there be a versioning? This would allow us to better handle future upgrades.

That looks like a good idea. That would mean a different listener (different DTO class) for each version ?

I can try to adapt my PR to add the class YamlModelRepository and have my existing class implementing YamlModelListener.

[spacemanspiff2007]

• Should different things be configured in one file (e.g. items and things) or would be prefer to have different files?

It would be nice to be able to put multiple things into one configuration file. You already put the item configuration in items, so why not allow things as a mapping, too? I currently have a shelly_flood.things where I have the shelly thing configuration and a shelly_flood.items where I have the items. It would be nice if I could merge these into one file. If subfolders will be allowed I can still use the legacy items and things folders, so that could be a solution that fits for everyone.

But one thing that needs to be improved is how things get loaded. If I define 100 items in the yaml and only change one then the whole 100 items get recreated and loose their state. There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

• Should there be a versioning? This would allow us to better handle future upgrades.

Imho this is not needed and should at least be optional defaulting to the newest version.
To support multiple versions you can try to parse the content with V3, if it fails with V2 and if that fails with V1. If that fails show the error message from parsing V3.
Also most of the time you need the new information and if it can't be derived from the default the use has to migrate the file anyway. Our data is mostly static, because things / items etc. are already defined so I don't expect much changes.

---

Concerning your example format - I think there is an easier way to model the links.
It all depends how much logic we are willing to put into the parser of the returned structure.
I think it's almost always worth it to provide a simpler way because it adds clairity

items:
  - name: Outdoor_temp
    type: Number:Temperature
    label: "Outdoor [%.1f °C]"
    groupNames:               
        - gTemperatures
        - gOutdoor
        - gPersist5

    # Option 1: allow link as a shortcut if it's only one link without a profile which makes "link" an optional entry
    # User has to either use "link" or "links" but may never use both at the same time
    link: "deconz:thing1:channel1"

    # Option 2: links takes the channel name as a key and is a mapping
    links:
        # these two are equivalent
        deconz:thing1:channel1: 
        deconz:thing1:channel1: null
        # simple profile is key: value
        deconz:thing2:channel2: MyScope:MyProfile
        # parametrized profile is key: mapping
        deconz:thing2:channel2: 
          name: MyScope:MyProfile
          lower:15

Suggested Metadata example:

Note that I reused value to allow for a more compact data structure in metadata.
I'm aware that value currently is an allowed keyword for the config but imho it makes sense
to disallow value so we don't have a structure that is too deep.

Rollershutter window_covering "Window Rollershutter" {homekit = "WindowCovering"}
Rollershutter window_covering "Window Rollershutter" {homekit = "WindowCovering"  [stop=true, stopSameDirection=true]}

items:
  - name: window_covering
    type: Rollershutter
    label: Window Rollershutter"
    metadata:
      homekit: "WindowCovering"   # first line of *.items
      homekit:                    # second line of *.items
         value: "WindowCovering"
         stop: "true"
         stopSameDirection: "true"

[rkoshak]

There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

I think that's a little more challenging than it might seem because the Items get unloaded/deleted when the original .items file gets unloaded. Then when the file gets reloaded the Items get created anew.

In order to support this use case:

• OH would have to be able to recognize the difference between a config file permanently being deleted instead of having been changed.
• New checks will have to be added to support recognition that an Item was removed from the file.

I believe, with my limited knowledge that would require changes to generic openHAB core stuff far above and beyond just adding a new model listener.

That's not an argument for or against it, just pointing out it might be outside the scope of this issue as it will probably impact all the other file formats too.

Note that I reused value to allow for a more compact data structure in metadata.
I'm aware that value currently is an allowed keyword for the config but imho it makes sense
to disallow value so we don't have a structure that is too deep.

Shouldn't the config match how the metadata is stored and used and even more importantly how it's shown in the Code tab of MainUI? If the proposal is to make the formatting changed in both places I'm good with that, though it probably means also changing the underlying JSON to support it which widens the impact of the change. Otherwise I think differences between what we see in MainUI and the YAML config files would be a net negative, even if the overall change is a simplification.

[spacemanspiff2007]

I believe, with my limited knowledge that would require changes to generic openHAB core stuff far above and beyond just adding a new model listener.

Yes - it's more than just adding another listener but not as much as it seems.
We already remove the items if the file gets deleted so there is already a mapping between file and containing item names.
The other step would just e.g. to wait one or two seconds after a file event and if the file is still there reload it, otherwise remove the items. I believe denouncing of events is already happening so most of the code is already there, too.

I agree that it's easier to just attach things to the existing code but if the goal is that in the long term yaml replaces the custom dsl formats then imho it makes sense to invest a little bit more time to remove the existing quirks and limitations.

That's not an argument for or against it, just pointing out it might be outside the scope of this issue as it will probably impact all the other file formats too.

Why not keep the existing code as is, freeze it and phase it out e.g. in OH5 or OH6?

Shouldn't the config match how the metadata is stored and used and even more importantly how it's shown in the Code tab of MainUI? If the proposal is to make the formatting changed in both places I'm good with that, though it probably means also changing the underlying JSON to support it which widens the impact of the change. Otherwise I think differences between what we see in MainUI and the YAML config files would be a net negative, even if the overall change is a simplification.

Yes - showing the metadata the same way should be the goal.
My suggestion is to just modify the parser/serializer so the underlying structure is not touched which would make it a small change.

E.g. this

homekit: 
  value: "WindowCovering"
  stop: "true"
  stopSameDirection: "true"

will still be saved as

{
  "value": "WindowCovering",
  "config": {
    "stop": "true",
    "stopSameDirection": "true"
  }
}

[lolodomo]

But one thing that needs to be improved is how things get loaded. If I define 100 items in the yaml and only change one then the whole 100 items get recreated and loose their state. There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

This is not yet included in the implementation I proposed in PR #3659 but it is very easy to add, I just have to compare old and new object. I am going to add this check.

[lolodomo]

I think that's a little more challenging than it might seem because the Items get unloaded/deleted when the original .items file gets unloaded. Then when the file gets reloaded the Items get created anew.

In my proposed implementation (coming from @J-N-K initial POC), this is no more a problem for the items in one file. The code detects the removed, added and updated items.
In case you remove an item from a file and then add it in another file (so move it from one file to another), in that case, the item will be first removed and then recreated.

[lolodomo]

One thing I have not yet implemented but could be done easily is to add a check (and reject) if several "things" with the same id are present either in the same file or different files.

[spacemanspiff2007]

I am going to add this check.

Awesome! 🚀 🚀 🚀

In case you remove an item from a file and then add it in another file (so move it from one file to another), in that case, the item will be first removed and then recreated.

That's how I would have expected it.

One thing I have not yet implemented but could be done easily is to add a check (and reject) if several "things" with the same id are present either in the same file or different files.

Along with a proper error message this would also be a nice feature

[rkoshak]

will still be saved as

My understanding though is that the JSON to YAML in MainUI is generic. Therefore what you see in the MainUI code tab would be

  value: "WindowCovering"
  config:
      stop: "true"
      stopSameDirection: "true"

which would be different from the YAML in the file.

So either there would have to be a special branch of code in MainUI to transform the JSON first or manipulate the YAML just for the metadata code page or the JSON would have change to become

{
  "value": "WindowCovering",
  "stop": "true",
  "stopSameDirection": "true"
}

Both are significant amounts of work and the latter would be a breaking change.

[lolodomo]

But one thing that needs to be improved is how things get loaded. If I define 100 items in the yaml and only change one then the whole 100 items get recreated and loose their state. There should be a very basic check:
Does the configuration already match the existing object, if yes skip if no delete and recreate.

This is not yet included in the implementation I proposed in PR #3659 but it is very easy to add, I just have to compare old and new object. I am going to add this check.

It is now implemented.

[lolodomo]

One thing I have not yet implemented but could be done easily is to add a check (and reject) if several "things" with the same id are present either in the same file or different files.

Along with a proper error message this would also be a nice feature

It is now implemented for objects in the same file. I kept the code simple and the warning log only mentions that "X elements elements with same ids".
Detection of objects with same ids in different files is not yet implemented.

[spacemanspiff2007]

It is now implemented for objects in the same file. I kept the code simple and the warning log only mentions that "X elements elements with same ids".

What will happen? Will the last definition overwrite all previous ones?
We could also use a mapping instead of a dedicated name attribute, then duplicate entries would be an invalid yaml:

items:
  item_name:
    type: Rollershutter
    label: Window Rollershutter"

I like this format significantly better than the obj with the name attribute.

Both are significant amounts of work and the latter would be a breaking change.

It's a one-liner in python and I suspect even with proper error handling it'll be less than 15 lines of (java) code.

But aligning the GUI with files is not the only way this can be done:
Another one could be "export to file" (which would align with the *.items import)
So we have two yamls but it's clear which is which because one is an export and the other one is code

[rkoshak]

It's a one-liner in python and I suspect even with proper error handling it'll be less than 15 lines of (java) code.

If you are talking about coding the MainUI page to show it differently, it's not Python nor Java but JavaScript and it means introducing logic to detect when the metadata is being shown and going down a different path than the standard path used everywhere else in MainUI. That's significantly more than just a one liner.

If you are talking about changing the JSON representation of metadata, it may be a one liner, but it's a one liner that impacts everything in OH that reads and uses metadata today, including potentially end user's rules.

But aligning the GUI with files is not the only way this can be done:
Another one could be "export to file" (which would align with the *.items import)
So we have two yamls but it's clear which is which because one is an export and the other one is code

And users will see a YAML posted to the forum and blindly copy and paste it into the code tab somewhere and cry when it doesn't work because it's different for some unapparent reason. And yes, shame on them. But it still takes time from those who provide support on the forum dealing with the problem.

I'm all for changing it either way or not changing it at all (it's just not that big of a deal). I am vehemently against there being two different YAML representations of metadata (or any other entity) though. Tiny little differences like these become land mines for users.

[spacemanspiff2007]

it means introducing logic to detect when the metadata is being shown and going down a different path than the standard path used everywhere else in MainUI.

No - the only place where it is shown is in the code tab, because that's the only place in the MainUI where the user sees the yaml.
All other code can stay the same.
Exactly two places have to be modified: during the creation of the yaml (in the code tab) and the parsing of the yaml (in the code tab).

If you are talking about changing the JSON representation of metadata,

This is not what I am suggesting

And users will see a YAML posted to the forum and blindly copy and paste it into the code tab somewhere and cry when it doesn't work because it's different for some unapparent reason. And yes, shame on them. But it still takes time from those who provide support on the forum dealing with the problem.

If the file format is (significantly?) different than the format code tab it'll become clear because not only the metadata representation is different but also the whole item representation.
Additionally I think it's easy to adapt the MainUI to the file format, once the file format is specified.

(it's just not that big of a deal)

As someone who works with lots of files it is a big deal.

---

I can only appeal to all the hard working maintainers to not rush the new file format and not make it blindly a yaml representation of the json. At least evaluate some structures and let's think about what is nice to edit, share and work with.
Just think about how long the *.items file has been around - I think it's worth to invest additional effort.

[lolodomo]

It is now implemented for objects in the same file. I kept the code simple and the warning log only mentions that "X elements elements with same ids".

What will happen? Will the last definition overwrite all previous ones?

Such file containing objects with same ids is considered as invalid and all the file is ignored.
Adding this check was only 3 lines of code. I can add few more lines to have a better warning log.
By the way, I realized that the number I am providing could be wrong in certain cases.

[lolodomo]

Just think about how long the *.items file has been around - I think it's worth to invest additional effort.

IMHO, what is urgent for OH4.0 is to have the core feature for the future and the YAML file syntax for the semantic tags. That is the perimeter of my PR.
Defining YAML syntax for other objects (items, things, ...) could be done in a next step and released in OH 4.1. We have time, the current XText files are there and work well.

[spacemanspiff2007]

Such file containing objects with same ids is considered as invalid and all the file is ignored.
Adding this check was only 3 lines of code. I can add few more lines to have a better warning log.

Sounds good. I think it's enough to show the duplicate ids, the number doesn't matter.

Defining YAML syntax for other objects (items, things, ...) could be done in a next step and released in OH 4.1

Sounds like a plan 👍

[rkoshak]

No - the only place where it is shown is in the code tab, because that's the only place in the MainUI where the user sees the yaml.

And that code calls a generic library to convert the JSON to YAML. That'd no longer a generic call.

As someone who works with lots of files it is a big deal.

And as someone who spends probably as much time helping users on the forum that you do writing HABApp I still maintain that I have concerns. If the YAML is different in the two places it's going to be a big impact on my time and UI users' time.

But I'll state it again, I'm not against changing the format. But I've seen no one chime in here and say "I'll make sure the YAML in these files and the YAML on the code tab match if we decide to differ." It's not even clear whether @ghys would allow such a custom set of logic in MainUI in the first place no matter how easy it is (he's rejected similar proposals for Blockly and his reasons are valid).

All I request is that we ensure it is the same in both places. How or why that is enabled is not my concern.

I think it's worth to invest additional effort.

I agree. But it's also worth the additional effort to consider the impacts to UI users, support on the forums, and maintainers. Decisions made here do have impacts on all these other people too. To reprise my statement when I've had this same discussion on openhab-js, it's not fair to impose on our least technically able users to advantage our most capable users. A balance needs to be reached and compromises on both sides need to be accommodated.

Having two different YAML representations, be they similar or very different, is an imposition on those least capable users as it adds yet another thing they need to be aware of and figure out. It's an imposition on those who want to learn how to create file based configs because they can't just take what they see in MainUI and map that to the YAML file. It's na imposition to the documentation writers to have to deal with two different YAML syntaxes and document that there is a difference in the first place. And it's an imposition on the helpers on the forum who will have to deal with problems when users just blindly assume that the formats would be the same. Why wouldn't they? It's a reasonable assumption.

[florian-h05]

All I request is that we ensure it is the same in both places. How or why that is enabled is not my concern.

Absolutely agreed! When we introduce a new YAML file format, the overall structure of the entities should match with the code tab of the UI to allow copy-and-paste.

But I'll state it again, I'm not against changing the format. But I've seen no one chime in here and say "I'll make sure the YAML in these files and the YAML on the code tab match if we decide to differ." It's not even clear whether @ghys would allow such a custom set of logic in MainUI in the first place no matter how easy it is (he's rejected similar proposals for Blockly and his reasons are valid).

I — as an UI maintainer together with Yannick — don‘t see a large problem with changing the format of the UIs code tab, even though that would invalidate examples from the forum (I have to admit that I haven‘t seen that much YAML code examples on the forum for sharing stuff with other users). However we really should keep in mind while designing a YAML file format that it also needs to be implemented in the UI that way, preferably without a ton of extra code. I also want to hear Yannick’s opinion before deciding anything on this side.
So before merging any new YAML file PR for Things, Items etc. into core, I‘d propose that we also get a UI PR in place to ensure UI and file have the same YAML structure.

Feel free to ping me and ask for help and advice for the UI side.

[spacemanspiff2007]

And that code calls a generic library to convert the JSON to YAML. That'd no longer a generic call.

That call will still be generic. The steps would be:

• get mapping for metadata (same as today)
• pull attribute one level up/down (new)
• transform to yaml (same as today)

---

All I request is that we ensure it is the same in both places. How or why that is enabled is not my concern.

I agree with you - that would be ideal and should be the goal.
But your argument is:
"We have something that is (very seldomly) used for copy paste so the file format should be the same".
And my argument is:
"We will have something that will be used by very much people very often so we should make it easy and comfortable to use."

These aren't mutually exclusive and again I agree with you and the points you make.
But we draw the different conclusion.
Yours is "The code changes are probably difficult and maybe the UI maintainers won't do it so we have to align the file format with what we have in the GUI now" (very exaggerated to make a point).
Mine is "Make the format nice and easy to use because it affects much more people and then (try to) make the UI follow that format".
And imho it's a reasonable assumption that the UI will follow: We currently have an *.items file parser in the UI which is magnitudes harder and way more complex than pulling up/down some attributes in a possible yaml file format.

---

However we really should keep in mind while designing a YAML file format that it also needs to be implemented in the UI that way, preferably without a ton of extra code.

If we decide on a new format and we want to use it in the UI why not expose it through the rest api.
That way the code tab always is in sync with the file format.
E.g. we could have a yaml/parse and a yaml/serialize or a .../yaml endpoint on the item / thing

[rkoshak]

"We have something that is (very seldomly) used for copy paste so the file format should be the same".

That is not my argument. My argument is very much that making the formats different increases complexity, adds confusion, and increases the work for lots and lots of people (users, maintainers, documenters, and helpers) outside this discussion.

And I take issue with the "very seldomly". It's only very seldomly used today because it's awkward and not well supported. If the formats are the same, I suspect it would become very frequently.

And my argument is:
"We will have something that will be used by very much people very often so we should make it easy and comfortable to use."

And your argument and mine are not against each other unless the format just changes to benefit only those who only use text files at the expense of everyone else.

Again, I don't care what the format becomes. I only care that it's the same in the files and in the UI. That's it. Having two different YAML representations for stuff in OH would be a huge mistake in my somewhat informed opinion.

And note: I'm really only talking about the stuff that you'd actually see in MainUI's code tabs. So, for example, all you see today for metadata is the "value" and the "config" so I only care that those are represented the same. The stuff that comes above those in the YAML hierarchy in the text file is not part of what I'm talking about. That stuff isn't shown in MainUI as YAML so I have no concerns about how that's formatted.

[ghys]

• You have those fromYaml () / toYaml () methods throughout the UI's code which perform the serialization/deserialization as YAML (from the JS objects that are meant to be posted as JSON to the API when clicking Save). What happens in them is open for debate as long as it makes sense and they remain reliable. Copying the server-side syntax looks sensible to me.

• Currently the YAML format is usually close to the target JS object for simplicity's sake, but can differ (e.g. omitting properties that aren't part of the object's model but are enriched, for instance statuses etc.; for scenes you also have a completely new items dictionary which doesn't exist in the rule specification and is transformed back to rule modules).

• Also note that those code editors may have CodeMirror "hints" for autocompletion (for example, thing configuration) which would have to be adjusted if the syntax changes.

• Finally, instead of (or in complement to) file watchers you might also consider the Kubernetes approach:

1. https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#organizing-resource-configurations

   Management of multiple resources can be simplified by grouping them together in the same file (separated by `---` in YAML).

2. https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#in-place-updates-of-resources (operation to "apply" the specified file(s) to the registry, or delete them in bulk)

   So you would define multiple related objects like:

kind: Thing
apiVersion: v1
UID: exec:command:MyThing1
name: MyThing1
thingTypeUID: exec:command ...
---
kind: Item
apiVersion: v1
name: MyItem1
type: Number ...
---
apiVersion: v1
kind: Link ...
---
apiVersion: v1
kind: Rule
uid: MyRule1 ...

3. Some REST endpoint accepting this kind of input would mean these could be created from a "Create from YAML input (or file") UI developer tool similar to the "create" features of the Kubernetes dashboard).

   

   https://adamtheautomator.com/kubernetes-dashboard/#Ensuring_Resources_Show_up_in_the_Dashboard

4. The same logic could be also used in the marketplace services to import these kind of multi-object YAML files.

5. This would also possibilities for new tools like an OH equivalent to Helm charts (packages of templated resources).

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/oh4-transformationservice-of-type-js-is-unavailable/148786/16

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/what-is-the-most-common-ui-you-are-using/149096/31

[lolodomo]

@rkoshak : as I understand, @J-N-K considers my PR #3659 cannot be merged due to one of your suggestion here.

In case you would like to have one unique file containing potentially any kind of OH data (is that really your request ?), that means it should contain many optional parts. When parsing the YAML file with a reference class, how do you specify in the reference class that each kind of objects is optional ? Is it technically possible with the YAML library we are relying on at your opinion @J-N-K ? I remember I tried to make the version field optional and I did not succeed.
That would require to have a reference class containing an optional list of items, an optional list of things, an optional list of sitemaps, an option list of tags, an optional list of rules, ... Then each time we would like to enhance one of these elements, we should increase the version. Then all users should update all their YAML files, even those that are not directly impacted by the change, at least to upgrade the version ?!

Today, we have files of each kind in different folders, something well structured. I understand you would like to allow a file containing items even in the folder "things" for example ?! That would require to scan all folders to find items, all folders to find sitemaps, ... etc

Would you agree to keep things simple and avoid adding a lot of complexity for a very low user profit ? Then we could maybe merge my PR.

One time again, I would also really appreciate @J-N-K if you confirm or not if it is technically doable with the YAML library we are using.

[lolodomo]

Maybe I was not clear in what I believe does not work. Imagine I have a YAML containing a list of tags and I try to parse it with a class defining a list of tags and a list of items, the parsing will fail. The error will tell me that my file does not contain any item.

[J-N-K]

I'm not 100% sure, but looking at FasterXML/jackson-dataformats-text#66 it should be possible. The file should be parsed to an array of ObjectNodes and these can be processed by the specialized parsers for items/things/tags.

This is very close to my original idea, because it allows examples containing items things for a binding together in a single file that can be used for both, textual configuration and import in the UI.

[spacemanspiff2007]

kind: Thing
apiVersion: v1
UID: exec:command:MyThing1
name: MyThing1
thingTypeUID: exec:command ...
---
kind: Item
apiVersion: v1
name: MyItem1
type: Number ...
---
kind: Item
apiVersion: v1
name: MyItem2
type: Number ...
kind: Item
apiVersion: v1
name: MyItem3
type: Number 

Imho the multi document approach is not as good as having a node in the document especially e.g. for items.
It quickly leads to a very verbose file with lots of repetition.
Since items and things have a unique identifier why not use those to create a map in the configuration file.
This should work for most of the entries consistently, e.g.:

things:
  exec:command:MyThing1:
    thingTypeUID: exec:command ...
    apiVersion: v1

items:
  MyItem1:
    apiVersion: v1
    type: Number
  MyItem2:
    apiVersion: v1
    type: Number
  MyItem3:
    apiVersion: v1
    type: Number

transformations:
  de.map:     # an idea could also be to specify the type with a dot
    type: map # or as a dedicated type entry, but then it would make sense to push the values one node down to have no reserved keys
    open: offen

  # or type of transformation and then name, e.g.
  map:
    de:
      open: offen

[jimtng]

This is a slightly different idea, but it might be integrated into this topic somehow? #3808

[spacemanspiff2007]

This is a slightly different idea, but it might be integrated into this topic somehow? #3808

HABApp offers something similar, bit you can automatically generate items and set parameters based on your things.

However I think the most easy way would be to provide a generator which then creates the new file format.
Creating a new good file format is already hard enough and I'm afraid something like automatic generation makes it unnecessarily harder.

[rkoshak]

In case you would like to have one unique file containing potentially any kind of OH data (is that really your request ?), that means it should contain many optional parts.

That is close to what I'm asking for I think. The opportunities I see if this were possible include:

• ability to keep the entire config for a subsystem together (Things, Items, Rules, Widgets, etc.)
• a first step towards rule templates that include more than one rule or even more than just rules (e.g. a template that also comes with the required Items and UI Widgets). Look at Justin's Scene Control postings. He has to have 4 different posts to the marketplace across two separate categories because we can't do this today.
• streamlined sharing on the forum since Items, Things, and rules can be posted as one document
• new opportunities for ways end users can modularize their configs based on their preferences.

When parsing the YAML file with a reference class, how do you specify in the reference class that each kind of objects is optional ?

I am no expert on defining scemas (or what ever they are called in YAML) and frankly wasn't aware YAML even has them. But, based on my experience writing Ansible playbooks and tasks as well as my own sensorReporter, I know this is at least possible from the YAML side of things.

You'd define a set of "root" tags: thing, item, widget, rule, metadata (perhaps). Then you'd have schemas to define what's allowed and required under each of this root tags like @spacemanspiff2007 describes.

I don't know what library you are using nor how it works. I don't know if what Ansible does is odd or not. But my idea comes from my experience with Ansible.

There has to be a way to make fields optional though. Every config format I've ever encountered, from XML to JSON to INI /Properties, etc. all support optional fields/elements. I cannot believe YAML as a format doesn't. It'd be next to useless for a lot of things.

Then all users should update all their YAML files, even those that are not directly impacted by the change, at least to upgrade the version ?!

Isn't the whole point of the version field to allow old configs to still work unchanged? I thought the whole version proposal was so that the configs could be marked with what version they are and it won't fail if it doesn't conform to the new version because the parser is smart enough to know the differences.

I understand you would like to allow a file containing items even in the folder "things" for example ?

I would do away with the separate folders entirely, come up with a new folder structure, or leave it to the end users to decide what folder structure (or not) they want. There would be one config parser that would load all the files and kick out to the right subparser based on that first element.

At least that's how I would design it and how I've seen it design in the past on other projects.

Would you agree to keep things simple and avoid adding a lot of complexity for a very low user profit ? Then we could maybe merge my PR.

I don't think it's low profit. But I also don't want to stand in the way of PRs as long as the door is open to readdress this later. I do think it's important and valuable.

[ThaDaVos]

It is now implemented for objects in the same file. I kept the code simple and the warning log only mentions that "X elements elements with same ids".

What will happen? Will the last definition overwrite all previous ones? We could also use a mapping instead of a dedicated name attribute, then duplicate entries would be an invalid yaml:

items:
  item_name:
    type: Rollershutter
    label: Window Rollershutter"

I like this format significantly better than the obj with the name attribute.

Both are significant amounts of work and the latter would be a breaking change.

It's a one-liner in python and I suspect even with proper error handling it'll be less than 15 lines of (java) code.

But aligning the GUI with files is not the only way this can be done: Another one could be "export to file" (which would align with the *.items import) So we have two yamls but it's clear which is which because one is an export and the other one is code

Just to chime in, by adding the mapping with items - it will also be easier to support the single file idea - as each type has its own base key - for example items, things etc:

items:
   item_name:
     type: Rollershutter
     label: Window Rollershutter"
............
things:
   thing_name:
     type: Rollershutter
     label: Window Rollershutter"
............

[spacemanspiff2007]

I've thought about this some more and have created some examples for items and persistence.
For transformation I only looked at map because I am not aware of other transformations that are defined in files.

For some values there are two forms how things can be expressed:

• a short form for convenience if it's just a simple value
• an extended form if it's more than a simple value

After loading the yaml all short forms should be converted to the extended form in a post processing step.
So this is just a convenience which makes reading/writing the file easier.

I tried to apply the following design principles:

• Prevent the use of comma separated values in strings and use a yaml list or key-value pairs instead.
• Prevent the use of string magic and use key-value pairs instead (e.g > % 10)
• Names of objects should always be the key in a key-value mapping.
• If there is the possibility to define a single or a multiple value it's always two keys, one singular and one plural e.g.
  value: asdf and values: ['asdf', 'asdf2']. The user can only specify one or the other. The singular form should be converted to the plural for in post processing

Notes:
Thing definition is very hard, so I am not 100% percent happy with it. I've eliminated the dedicated type bridge which now gets set through a key. This makes it much easier with no penalty for the user.
For items I've changed how the label transformation is applied (Example 7). It's okayish but still better than what we have now.

Persistence, transformations and items is already very good. The syntax is very easy to learn and comes natural after writing a couple of definitions.

Suggestion:

# ----------------------------------------------------------------------------------------------------------------------
# things
# ----------------------------------------------------------------------------------------------------------------------
things:
  # --------------------------------------------------------------------------------------------------------------------
  # Example 1:
  # Thing network:device:webcam "Webcam" @ "Living Room" [ hostname="192.168.0.2", timeout="5000", ... ]
  # --------------------------------------------------------------------------------------------------------------------
  network:device:webcam:
    label: "Webcam"
    location: "Living Room"
    config:
      hostname: "192.168.0.2"
      timeout: 5000

    # settings should be used if we finally separate the thing configuration from device configuration
    # settings change something on the device, config defines how the thing connects to the device
    settings:
      ...

  # --------------------------------------------------------------------------------------------------------------------
  # Example 2:
  #
  # Bridge hue:bridge:mybridge [ ipAddress="192.168.3.123" ] {
  #   Thing 0210 bulb1 [ lightId="1" ]
  #   Thing 0210 bulb2 [ lightId="2" ]
  # }
  # --------------------------------------------------------------------------------------------------------------------

  hue:bridge:mybridge:
    bridge: True  # optional key, default is False or empty. "true or yes" (case insensitive) defines this as a bridge
    config:
      ipAddress: "192.168.3.123"

  hue:0210:mybridge:bulb1:
    bridge: hue:bridge:mybridge # references the defined bridge
    config:
      lightId: 1

  hue:0210:mybridge:bulb2:
    bridge: hue:bridge:mybridge
    config:
      lightId: 2

  # --------------------------------------------------------------------------------------------------------------------
  # Example 3:
  #
  # Bridge mqtt:broker:MyMQTTBroker [ host="192.168.178.50", secure=false, username="MyUserName", password="MyPassword"] {
  #  Thing topic sonoff_Dual_Thing "Light_Dual" @ "Sonoff" {
  #    Channels:
  #      Type switch : PowerSwitch1  [ stateTopic="stat/sonoff_dual/POWER1" , commandTopic="cmnd/sonoff_dual/POWER1", on="ON", off="OFF"]
  #      Trigger String : customChannel1 [ configParameter="Value" ]
  #      State Number : customChannel2 []
  #      }
  # }
  # --------------------------------------------------------------------------------------------------------------------

  mqtt:broker:MyMQTTBroker:
    bridge: True  # optional key, default is False
    config:
      host: "192.168.178.50"
      secure: false
      username: MyUserName
      password: MyPassword

  sonoff_Dual_Thing:
    bridge: mqtt:broker:MyMQTTBroker
    label: "Light_Dual"
    location: "Sonoff"
    channels:
      - type: switch
        kind: type  # optional, can be one of "trigger", "state" and "type", default is "state"
        name: PowerSwitch1
        config:
          stateTopic: 'stat/sonoff_dual/POWER1'
          commandTopic: 'cmnd/sonoff_dual/POWER1'
          on: 'ON'
          off: 'OFF'

      - type: String
        kind: trigger   # optional, can be one of "trigger", "state" and "type", default is "state"
        name: customChannel1
        config:
          configParameter: "value"

      - type: Number
        name: customChannel2

  # --------------------------------------------------------------------------------------------------------------------
  # Example 4:
  #
  # Thing yahooweather:weather:losangeles [ location=2442047, unit="us", refresh=120 ] {
  #   Channels:
  #     Type temperature: my_yesterday_temperature "Yesterday's Temperature"
  # }
  # --------------------------------------------------------------------------------------------------------------------
  yahooweather:weather:losangeles:
    config:
      location: 2442047
      unit: us
      refresh: 120
    channels:
      - type: temperature
        kind: type
        name: my_yesterday_temperature
        label: Yesterday's Temperature



# ----------------------------------------------------------------------------------------------------------------------
# items
# ----------------------------------------------------------------------------------------------------------------------
items:
  # --------------------------------------------------------------------------------------------------------------------
  # Example 1:
  # Number Livingroom_Temperature "Temperature [%.1f °C]" <switch> (Livingroom, Temperatures)
  # --------------------------------------------------------------------------------------------------------------------
  Livingroom_Temperature:
    type: Number
    label: "Temperature [%.1f °C]"
    icon: switch  # todo: REST reports this as category, the docs refer to it as icon. Choose one and stick to it
    groups:
      - Livingroom
      - Temperatures

  # --------------------------------------------------------------------------------------------------------------------
  # Example 2:
  # Switch Garage_Gate {channel="xxx", autoupdate="false", expire="1h,command=STOP"}
  # --------------------------------------------------------------------------------------------------------------------
  Garage_Gate:
    type: Switch
    channel: xxx
    # Both autoupdate and expire are core functionalities, so they should be set directly on the item even though
    # they are implemented through metadata
    autoupdate: false
    expire: "1h"

    # two forms for expire, one a single string for posting UNDEF and one extended form for posting arbitrary things
    # This is something that has to be checked by the parser, e.g. the short form with only a string should automatically
    # be transformed to the extended form like this (with the time key-value and one of command/update key values)
    expire:
      time: "1h"
      command: "STOP" # sends a command
      update: 'STOP'  # posts an update

    # another option would to always require the extended form which could be written like that
    expire: {time: '1h'}

  # --------------------------------------------------------------------------------------------------------------------
  # Example 3:
  # Number:Temperature Outdoor_Temperature { channel="openweathermap:weather-and-forecast:api:local:current#temperature" }
  # --------------------------------------------------------------------------------------------------------------------
  Outdoor_Temperature:
    type: Number
    measurement: Temperature  # alternative to measurement could be "kind"
    # short form if it's only one channel with no profile
    channel: "openweathermap:weather-and-forecast:api:local:current#temperature"

    # extended form which allows multiple channels with profiles
    channels:
      openweathermap:weather-and-forecast:api:local:current#temperature: {}


  # --------------------------------------------------------------------------------------------------------------------
  # Example 4:
  # Switch Outdoor_Temperature_Low_Alert { channel="openweathermap:weather-and-forecast:api:local:current#temperature"
  #                                        [profile="system:hysteresis", lower="0 °C", inverted="true"] }
  # --------------------------------------------------------------------------------------------------------------------
  Outdoor_Temperature_Low_Alert:
    type: Switch
    channels:
      openweathermap:weather-and-forecast:api:local:current#temperature:
        # multiple profiles can be specified by name
        system:hysteresis:
          lower: "0 °C"
          inverted: "true"

  # --------------------------------------------------------------------------------------------------------------------
  # Example 5:
  # Switch MyFan "My Fan" { homekit="Fan.v2", alexa="Fan" }
  # --------------------------------------------------------------------------------------------------------------------
  MyFan:
    type: Switch
    label: My Fan
    metadata:
      homekit: Fan.v2
      alexa: Fan  # short form of metadata

  # --------------------------------------------------------------------------------------------------------------------
  # Example 6:
  # Switch MyFan "My Fan" { homekit="Fan.v2", alexa="Fan" [ type="oscillating", speedSteps=3 ] }
  # --------------------------------------------------------------------------------------------------------------------
  MyFan2:
    type: Switch
    label: My Fan2
    metadata:
      homekit: Fan.v2 # short form of metadata
      alexa:  # extended form of metadata
        Fan:
          type: oscillating
          speedSteps: 3


  # --------------------------------------------------------------------------------------------------------------------
  # Example 7:
  # String Livingroom_Light_Connection "Livingroom Ceiling Light [MAP(error.map):%s]" <myerror>
  # --------------------------------------------------------------------------------------------------------------------
  Livingroom_Light_Connection:
    type: String
    label: # extended form to use the transformation
      label: "Livingroom Ceiling Light [%s]"
      transformation: error.map
    icon: myerror  # todo: REST reports this as category, the docs refer to it as icon. Choose one and stick to it



# ----------------------------------------------------------------------------------------------------------------------
# transformations
# ----------------------------------------------------------------------------------------------------------------------
transformations:
  de:
    type: map
    default: 'default value'  # It makes sense to define the default here and not in the values with an empty key which is very confusing
    transformation:
      open: open



# ----------------------------------------------------------------------------------------------------------------------
# persistence
# ----------------------------------------------------------------------------------------------------------------------
persistence:
  # note that strategies and filter is pulled up above the used persistence service
  strategies:
    # user given name
    every30s: "0/30 * * * * ?"


# Filters {
#   exactlySomeState : = "ARMED", "UNARMED"
#        fivepercent : > % 5
# }
  filters:
    exactlySomeState:
      type: equals
      values: # the key name is "values" here
        - ARMED
        - UNARMED

    fivepercent:
      type: relative threshold
      value: 10 # key name is "value" for single value


  # normal nodes
  rrd4j:
    - strategies:
        - everyChange
        - restoreOnStartup
      filters:
        - exactlySomeState
      groups: # we use a dedicated group field instead of the *gRRD4j notation
        - gRRD4j
      items:
        - MyItem1


  # collapsed nodes
  other_service:
    - strategies: [everyChange, every30s]
      groups: [gOtherService]
      items: [MyItem2, MyItem2]

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/add-an-include-option-to-the-jsondb-files-for-file-based-configurations/152509/3

[rkoshak]

For transformation I only looked at map because I am not aware of other transformations that are defined in files.

• SCRIPT transforms are in files (JS, Rules DSL, jRuby, etc.). These should look and work similar to rules.
• Scale also can have a file configuration. These work a whole lot like Map.

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/add-an-include-option-to-the-jsondb-files-for-file-based-configurations/152509/10

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/changing-items-metadata-very-frequently/155172/19

[openhab-bot]

This issue has been mentioned on openHAB Community. There might be relevant details there:

https://community.openhab.org/t/changing-items-metadata-very-frequently/155172/22