New software configuration and provisioning mechanism

[carlescufi]

Introduction

This issue deals with a topic that has been the center of multiple previous discussions and existing issues and PRs. This is an attempt at briefly summarizing the problem in order to solve it in a way that is agreeable to all potential users and developers alike.

IMPORTANT: Please keep the discussion around this particular feature in this GitHub issue, so that we do not have to collect feedback from multiple sources.

Problem description

Zephyr currently lacks a mechanism to configure multi-instance software components at compile-time well above the hardware. Instead, as described multiple times, we have:

• Devicetree: Multi-instance, but currently restricted to represent (mostly) actual hardware
• Kconfig: Single-instance, mostly restricted to software enablement and configuration

A third mechanism is therefore required. This issue describes the use cases and requirements in order to end up concluding with a final proposal.

Zephyr also lacks a provisioning mechanism, which would be able to generate a pre-compiled "image" of the data that is required in order to provide the required data for certain subsystems to execute.

Use cases

The following use cases have so far been identified:

• Network stack interface description: Handle boot time configuration of network stack #68127
• Configuration of init priorities: [RFC] Init entries: Automatically computing priorities #73836
• Multi-instance software (non-driver) configuration (e.g. buffer sizes in CHAT/CMUX)

We foresee additional users of this mechanism in the future, but the two use cases above exist today and require a solution.

Requirements (WIP)

Requirements document

Proposed diagrams

@nashif:

@fgrandel:

[bjarki-andreasen]

An additional use case is buffer sizes for instances of CHAT (at command handling), CMUX (uart multiplexing), UARTs etc. A CMUX instance can have multiple channels, where only one needs a huge buffer (for network traffic for example) and others require small buffers for typically AT commands.

[fgrandel]

Initial Use Cases:

• Provisioning a PSA secure storage target (Introduction of a subsystem that implements the PSA Secure Storage API #75275). This needs to be sourced from a secure container at prod time. If we don't want to complicate things at dev and test time then we should integrate this with a simple dev-time config approach for testing.
• I also suggest looking at the use cases I collected from community contributions. Ignoring those would mean ignoring those contributions IMO (none of those comes from me, they've all been collected from previous discussions or asking stakeholders on Discord). Few need to be supported initially but any proposal should be aware of them and include a credible migration path.
• It was mentioned in the arch WG that the sensing subsystem has an immediate stake in the config subsys.

Sources:

• Several people have asked for support of alternative source formats (not now but via a migration path) as input which they generate from internal systems. The requirement here is not that we need multi source at once but it should be clear how we want to migrate there.
• UX for first-time users, this ideally includes a single source for single- and multi-instance configuration.
• A way to cross-reference DT based on a common data model as the entities are mostly the same (required by both, init and net conf)
• A well-defined distinction of Kconfig/DT/sw config that can be reviewed and enforced based on easy-to-observe criteria.
• We need a method to map properties in Kconfig, DT and the new config approach for backwards compatibility as many properties will have to be migrated from both DT and Kconfig over time (as in net config).
• Overlays (required by init right away and probably net conf, too)

Targets:

• type bindings (obvious - just mentioning it for completeness), ideally kept close to usage sites and (as many have suggested) split up based on frequency of updates (maintainability)
• Several people have asked for integration with the settings subsystem (and other NVS). Not required right away, but for net config this is on the migration path.
• Specific configuration properties should be updatable at runtime (net config is an example).
• zero footprint by default (core Zephyr standard for all kinds of config systems)

Documentation:

• Several people have asked for self-documenting bindings, ideally integrated with the doc build system.

[decsny]

my comment copy pasted from the Arch WG chat as requested:

"can we be clear that provisioning requirement is obvious when talking about networking IP address but most multi instance configuration needed to develop software is probably not about provisioning, ie buffer sizes and stack sizes of drivers or properties of how parts of the system will act that doesnt change between serial numbers"

and

"i agree with brix about separating the provisioning problem space from the static multi instance configuration problem space"

[nordicjm]

For network configuration, without having run-time configuration, the whole thing can be done in Kconfig, e.g.

# subsys/Kconfig.network_interfaces file:
# total_interfaces would need to come from some sort of DTS processing or macro for counting number of "okay" devices"
if "$(interface)" <= "$(total_interfaces)"

config NETWORK_$(interface)_IP_ADDRESS
    string "IP address of network interface $(interface)"

endif

# Kconfig file in test:
interface = 0
source "subsys/Kconfig.network_interfaces"
interface = 1
source "subsys/Kconfig.network_interfaces"
interface = 2
source "subsys/Kconfig.network_interfaces"
interface = 3
source "subsys/Kconfig.network_interfaces"
interface = 4
source "subsys/Kconfig.network_interfaces"

Where total_interfaces would need to be set in dts directly or using macros to count the number of devices of a certain type

[carlescufi]

Arch WG:

• Devicetree vs this new mechanism:
  • @henrikbrixandersen mentions that if there's a tie to a struct device when the property is in Devicetree
  • @fgrandel explains how Devicetree is currently being used in many instances to configure "software"
• @nashif states that we should be able to serialize this in a format that can be used at runtime, independent of the image (i.e. re-provisioning without recompiling)
• @nordicjm states that multi-instance configuration could be done with Kconfig, no need for an additional mechanism to configure this
• @jfischer-no suggests using CBOR as a serialization language
• @henrikbrixandersen states that we may be mixing up different concepts with this: provisioning (serialized data that can be used across multiple instances of a single image) and build-time configuration that is different in every build. @henrikbrixandersen uses the setting subsystem downstream in order to handle this
• @fgrandel reminds us of the provisioning use case for Bluetooth Mesh, which is a very common one. Differentiating between provisioning and multi-instance configuration in terms of language and infrastructure may complicate things, so we may want to have the ability to generate either serialized data or just macros that can be used by the code directly.
• @nashif talks about the developer experience: language, syntax, tooling can all be shared.
• @jfischer-no mentions that anything that resembles a hardware device belongs in Devicetree, and does not require a new mechanism

[de-nordic]

Note on images build for MCUboot:
when planning provisioning mechanism, note that it may be hard to generate some hex and add something directly to image (as part of binary), as outcome of building image for MCUboot may have it already signed and encrypted. So in case when some blob is inserted as binary section, where you could directly address constants, that blob would require signing with the image.

Note on using external, out of image, blob:
such blob will require careful check, the same level as any user input as it may be used as attack vector on firmware. And what to do when these are tampered with? What to revert to? Note that something that may look like tampering may be just a problem that should not brick device, but rather, with annoying effect for user, default to factory setting - the factory settings that provisioning provides.

I have also a question: how to avoid using RAM for things that always be constant for a device? For example you may not change embedded MAC or serial number of device, yet when accessing these, from what has been written during provisioning, the device will have to load them to RAM, at least temporarily. This also brings cost of enabling Flash API and other subsystems required, for device to get the factory provisioned info, even if every unique instance of a device only reads the info and never attempts to store updates, or chooses other subsystem of storage (like FS).

[fgrandel]

Thanks @carlescufi for subsuming our discussion. A few corrections to avoid misunderstandings:

• @fgrandel explains how Devicetree is currently being used in many instances to configure "software"

Not exactly. I don't use the software/hardware distinction myself. I was rather saying that almost all properties that we introduce change at the same frequency as driver software and not at the same frequency as vendor hardware. From a well-established encapsulation perspective this indicates a requirement to keep driver-specific properties apart from vendor-specific properties and de-facto standardized stable Linux properties (as different files, not as different formats).

I was citing others who had laid out alternative heuristics to define the sw/hw line:

• "Everything that ties to a struct device is 'hardware'" as mentioned by @henrikbrixandersen and others. This heuristic breaks down in both directions: 1. By normalization rules, this definition would include almost all of the network interface configuration which is 1-to-1 to drivers. 2. We have a considerable number of DT properties that do not tie to struct device instances.
• "DT should contain everything that changes at the frequency of hardware" as @decsny (and myself) have been informed by several community members (but does not find himself being practicable nor do I): This heuristic also breaks down in practice. As pointed out above by that definition almost all many Zephyr-specific properties in DT would be "software".

I pointed out that as long as we do not have an intersubjectively observable heuristic that defines the line between sw and hw, each maintainer will define their own personal criteria as we observe in practice. I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

• @jfischer-no suggests using CBOR as a serialization language

I mentioned that this is part of the solution space. Others have voted for protobuf or Thrift.

• @henrikbrixandersen states that we may be mixing up different concepts with this: provisioning (serialized data that can be used across multiple instances of a single image) and build-time configuration that is different in every build.

I responded, that we cannot cleanly define per property whether they are "provisioning only" or "build only". On a per-app basis this line will change considerably in practice.

Maybe I misunderstood what was proposed, but if it means that we'd introduce another "a priory" (ontological) distinction then I'd strongly oppose such a strategy. Here a few examples others mentioned in earlier discussions:

• build-time "hw" property (e.g. gain of ADC channels)
• provisioning-time "hw" property (e.g. CDC ECM MAC address)
• build-time "sw" property (e.g. buffer sizes)
• provisioning-time "sw" property (e.g. IPv6 address)

I could immediately think of products (apps) or use cases (tests, debugging) where those properties would not fit those categories.

I therefore argued that any attempt to bind properties to a specific configuration source or target is conceptually flawed and will not work in practice. The fix is: Define a common abstract data model for everything closely tied by encapsulation rules to software programming models (which includes drivers) based on well-established normalization rules (i.e. shared entities and properties) independently from any specific representation or usage context. This can be done w/o any ontological assumptions with mathematical precision and has therefore been well-established best practice in application configuration for decades. This model then needs to be serializable from/to multiple files on a per use-case and encapsulation basis and/or overlayed at any level.

The settings subsystem is part of the solution space but was introduced to the discussion. That's why I voted against having to depend on the settings subsystem for all configuration as it severly limits extensibility of the config subsystem and introduces unnecessary complexity and resource usage to certain use cases like tests or simple custom applications that otherwise won't need it. I vote for technology that allows us to flexibly merge/split serialization from/to any sources/targets depending on specific application requirements. Several technologies with excellent UX exist for all proposed serialization formats to implement such a merge/split from/to a common intermediate representation.

I mentioned that as a bonus, this will make it easier to switch between alternative source and target representations at dev-time (e.g. for easy reproduction of bugs and debugging with minimal dependencies).

• @fgrandel reminds us of the provisioning use case for Bluetooth Mesh

Not only B. Mesh, my comment was including BLE proper as both seem to support the settings subsystem as config target according to @jhedberg. This becomes even more interesting when adding secure storage or large-scale provisioning systems as possible config sources. I also mentioned several other immedately relevant use cases introduced to the discussion by others and that are not yet part of the use case list.

• @nashif talks about the developer experience: language, syntax, tooling can all be shared.

I added to this that IMO tooling will be 90% of the maintenance effort and will be mostly shared between all use cases discussed so far (including init and net config). Assuming a common intermediate representation, sourcing from different formats and serializing into different targets can be added flexibly at very low cost. A common intermediate representation will help to debug the configuration space (similarly to .config and merged DTS) no matter whether done in YAML, JSON or DT.

I therefore vote for extracting and re-using existing DT tooling (not format) and re-use it for whatever configuration system and hopefully common intermediate format we decide upon. Any addition to the DT and/or config world will then equally profit both sides, ie. we can expect that DT tooling will benefit from improvements in the config area, too (e.g. UX improvements, validations, bindings, documentation, etc.). This also has obvious advantages from the user perspective as a lot of target-side and tooling knowledge could be re-used, independently of context.

UPDATE: To clarify as I was probably misunderstood here - I didn't argue for any specific tooling solution here although I admit that I can be read that way. I was just arguing to share tooling among different configuration requirements which is in the problem space, not in the implementation space as I'm not saying anything specific about how such shared tooling should look in the end. OTOH let's be realistic: Any shared tooling will need to build on top of what we have but I admit that such a statement is in the solution space and not required right now. So sorry if I was misread in that sense.

As long as those use cases and basic requirements (including those mentioned in my first comment) are not explicitly made part of the RFC I'll find it very hard to agree to any configuration system. They are all immediately relevant to the initial use cases mentioned by the OP and are therefore part of a "minimal set" as required by this issue. And please note that none of those requirements favors DT over YAML (or whatever other solution space technology) in any way.

UPDATE: Edited to ensure feedback given further down has been included in the argument.

[nashif]

I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

[nashif]

I therefore vote for extracting and re-using existing DT tooling (not format) and re-use it for whatever configuration system and hopefully common intermediate format we decide upon.

why would you use a tooling for a format if you are not using the format itself? Also, we established that runtime is important, most of the DT tooling is build time and based on macros.

[fgrandel]

@nordicjm @de-nordic I think your comments are really relevant but they both rather belong to rather specific solution spaces unless we extract abstract requirements from those examples. I find it hard myself to cleanly distinguish between problem and solution w/o loosing sight of feasibility. But it would be nice if we tried to keep very specific implementation problems always linked to more abstract requirements as I understood @carlescufi 's approach.

@nordicjm In more abstract terms you're making the case - if I understand correctly - that almost any source format will be able to represent an abstract data model (including Kconfig) and that we should not introduce additional formats unless required for a good reason (i.e. simplicity and UX as a requirement).

@de-nordic If I understand you correctly you're requiring at a more abstract level that configuration needs not only to integrate with the settings subsystem alone but also with other subsystems (e.g. MCUBoot signing in your case). From a requirements perspective this confirms that we cannot tie the configuration subsystem target to a single subsystem alone. It must be sufficiently open to integrate with whatever target and process we already have or invent in the future.

[fgrandel]

@nashif

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

Sure: #76902 (comment)

[nordicjm]

@nordicjm In more abstract terms you're making the case - if I understand correctly - that almost any source format will be able to represent an abstract data model (including Kconfig) and that we should not introduce additional formats unless required for a good reason (i.e. simplicity and UX as a requirement).

No, I'm stating that without runtime configurability (i.e. provisioning and able to change it) the entire network configuration PR is moot because no tooling is required, static configuration can be wholly done in Kconfig, thus the prime feature that this must support is provisioning

[jfischer-no]

• @jfischer-no suggests using CBOR as a serialization language

I mentioned that this is part of the solution space. Others have voted for protobuf or Thrift.

Not introducing new dependencies or minimizing dependencies should be a requirement. IIRC it was mentioned somewhere, but I cannot find it here. (Side note, we already use CBOR implementation and IMO it fits better to use it as a DTB alternative).

[fgrandel]

@nashif

why would you use a tooling for a format if you are not using the format itself? Also, we established that runtime is important, most of the DT tooling is build time and based on macros.

Considerable parts of DT tooling are not specific to DT as a source format (which as such I'm not questioning). Re-using those parts not only means less maintenance cost but also considerable re-use of existing community knowledge.

Just a few examples off of the top of my head w/o trying to be complete:

• Bindings are in YAML, not in DT. The binding format, all validation and generator infrastructure around that is re-usable. This tooling could benefit from a common approach as the existing binding approach has several modeling and usability flaws that could at the same time be fixed for everyone. I even suspect that the fact that many seem to find DT "awkward" is at least partly due to bindings rather than DT itself. This alone is a very important investment if we have to re-invent the wheel and maintain it on both sides.
• Many low-level generic DT macros are not tied to a specific source format. They only tie to an abstract hierarchical node (aka entity) vs property (aka attribute) distinction that we'll most probably need in the config area, too.
• In the config area we'll need a zero-footprint (probably macro) default target, too. Code that generates macros from typed entities and nodes (bindings plus intermediate representation) can be re-used among DT and config.
• Nordic and others have built visualization and maintenance tooling around DT that from a structural viewpoint could be (partly) re-used for for any entity-relationship graph including config which would make it easier to learn and maintain tooling.
• We need to represent and validate links between DT and config. This is much easier in a common intermediate infrastructure.
• I forgot to mention overlay infrastructure and rules. We already have two different overlay systems, do we really need a third one? We could certainly share logic between those systems to make them all easier to maintain and to enforce common rules re what overrides what. I remember that when the hardware model was changed recently there already was quite some redundancy in the change set due to DTS vs. Kconfig overlays.

I'm sure I could find more examples if looking at the code. But I think this is enough to make the point.

[nashif]

@nashif

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

Sure: #76902 (comment)

@decsny : "From being on the (recently removed, by me) DT binding maintainer area, I saw many things being added to DT that aroused this type of argument over HW/SW distinction."

This is very vague and a general obvervation which might be valid and to the point but this needs to be supported with examples that show whether this is going from bad to worse or something manageable that can be tolerated depending on the use case and context. If someone thinks this is a serious problem that needs attendion, someone needs to open an issue with examples and bring this to the attention of everyone.

[decsny]

I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

I think florian is referring to my engagement of him on #76902, he mentioned that I was taking a definition for a software and hardware distinction as for granted, I just responded to him that since I have been on the DT reviewer list for a while, I actually did see that it was quite a gray area and everybody seems to have different opinions.

As far as the point that florian is trying to make, I have not created a compiled list of detailed examples to support his argument, which IMO sounds a bit zealous but I think his spirit is going in the right place regardless, it's an important issue (as everybody interested in this thread would agree I think). And to be clear, I would not say that almost all newly introduced properties are contended, that is a bit of an exaggeration, but there are multiple cases:

• First of all, yes, I don't think it's uncommon to see disagreements and change requests on PRs from somebody who thinks something being added to devicetree doesn't belong there or vice versa. I think everybody who has been involved in the zephyr upstream has observed this plenty of times, and it's even discussed almost regularly in the working groups and discord forums as well as github. I do not think this claim needs to be proved to anybody who is a regular here.
• On the other hand, I couldn't tell you an exact proportion, but I think a good amount of DT changes and additions are not controversial and rather routine.
  • As far as how much we are adhering to Linux I'm not sure and don't know a good way to measure it, but I think a lot of MCU stuff being added to Zephyr is not found in Linux and a lot of bindings being added for platforms added to zephyr are new territory as far as relevant inter-project DT most of the time.
  • As far as how much we are respecting the various flavors of hardware vs software distinction, like I said above there can be disagreements, but for the most part, I think most changes and additions to DT are due to drivers being expanded or added to account for more features or aspects of the hardware and enable awareness of them through a more thorough DT description. I think there is a high frequency of changes, as florian pointed out, but this is not necessarily because DT is being mapped 1:1 to software constructs and is being forced to change with the same frequency, but I think most of the cases are just because there is a rapid amount of new hardware enablement development work going on in Zephyr due to it being a growing project with a lot of interest from silicon vendors who are shifting resources to support it. There might be some cases in DT where there is a problem with strong software mapping, and we should avoid it, but I don't think it's this dire as I am being quoted saying.
• Finally, there is the case where there is something being changed in DT that is in my opinion questionable, but nobody questions it or contends it. And to be clear, this case is not about my personal interpretation of the hardware config vs software config debate, but I think it's a different issue, where a lot of people (including myself) just don't want to put in the energy to fight against the tidal wave of interest from whoever is adding the thing when there is no alternative to suggest to them anyways since we don't have a good multi instance static software configuration scheme (which is the point of this issue, I suppose).

[nashif]

just a nother note here, can we please STOP talking about implementation and formats and DTS please? thought we had a good discussion yesterday while remaining at a high level and now we are going down the path again talking tooling and formats, this is very disruptive.

[decsny]

just a nother note here, can we please STOP talking about implementation and formats and DTS please? thought we had a good discussion yesterday while remaining at a high level and now we are going down the path again talking tooling and formats, this is very disruptive.

you should quote reply whoever you're talking to with what specifically you think is not relevant.

[nashif]

you should quote reply whoever you're talking to with what specifically you think is not relevant.

too much to quote reply. hint: I am not referring to you, just seen your message :)

[decsny]

you should quote reply whoever you're talking to with what specifically you think is not relevant.

too much to quote reply. hint: I am not referring to you, just seen your message :)

I do think it's not productive to make vague statements about people's discussion being irrelevant, it doesn't help not to be specific, you want to avoid discouraging the participation you actually do want by being too broad with your meta-criticism of the discussion

[nashif]

I do think it's not productive to make vague statements about people's discussion being irrelevant, it doesn't help not to be specific, you want to avoid discouraging the participation you actually do want by being too broad with your meta-criticism of the discussion

where am I being vague?

can we please STOP talking about implementation and formats and DTS please?

I am saying we need to first collect requirements before we go into implementation details and the discussion is shifting toward what was already discussed in the issues that were abrubtly closed. Why at this stage people are "voting" for implementations and formats and tooling without first agreeing on the requirements? Why is this discussion shifting toward how DTS is being used or mis-used?

[decsny]

I do think it's not productive to make vague statements about people's discussion being irrelevant, it doesn't help not to be specific, you want to avoid discouraging the participation you actually do want by being too broad with your meta-criticism of the discussion

where am I being vague?

can we please STOP talking about implementation and formats and DTS please?

I am saying we need to first collect requirements before we go into implementation details and the discussion is shifting toward what was already discussed in the issues that were abrubtly closed. Why at this stage people are "voting" for implementations and formats and tooling without first agreeing on the requirements? Why is this discussion shifting toward how DTS is being used or mis-used?

I'm not disagreeing with you that we need to talk about the requirements before proposing myriads of solutions, but a lot of these requirements are going to come from observations of weaknesses about the current implementation we have, so saying not to talk about it at all is too vague. Clearly there is some direction of the discussion above that you didn't like and I think it would be more useful to be more specific about what direction it went that you didn't like and by who. On the other hand, if by your quoted request you really flatly think that any mention of DT or implementation (especially the current) is irrelevant, then I tend to disagree for the reason I just gave.

[nashif]

I'm not disagreeing with you that we need to talk about the requirements before proposing myriads of solutions, but a lot of these requirements are going to come from observations of weaknesses about the current implementation we have, so saying not to talk about it at all is too vague.

and how does a "vote"" for an implementation in a discussion about requirements and as you say, weaknesses in such implementations help here? We will get there, sooner or later, but when this forum goes into the same rabbit hole about the same topic and the same arguments, most of us will just give up and lose interest.

Some DTS topics being touched on here deservee their own issues and tracks.

[fgrandel]

I would not say that almost all newly introduced properties are contended, that is a bit of an exaggeration, but there are multiple cases.

Yes, I admit this was an exaggeration. Sorry for that. :-( It's enough that there are many instances to make my argument.

but I think most of the cases are just because there is a rapid amount of new hardware enablement development work going on in Zephyr due to it being a growing project

IMO a sufficiently large part of changes are made due to driver-specific features for existing hardware to support the argument that those properties should be packaged with drivers rather than hardware. This is established encapsulation practice, why deviate? Does not invalidate your argument, though.

I acknowlege that we try hard to make our configuration models independent of specific vendors and Zephyr driver programming models. But for custom out-of-tree drivers or custom hardware this model often won't match project requirements.

just another note here, can we please STOP talking about implementation and formats and DTS please?

I said the same thing above - so why re-iterate? There was nothing deviating being brought to the discussion in the meantime except for your own questions re DT and tooling, right?

I therefore have to say that it's at least partly your fault if the discussion has been digressing again from the problem space. You just got what you asked for, IMO.

Arguing about the software/hardware distinction is in the problem space, focussing on DT as a format is not. Also feasibility cannot be completely excluded from our discussion.

thought we had a good discussion yesterday

I don't agree. Yesterday's discussion brought no new arguments, only repetitions of what had been said earlier (except for the truly original proposal to forget about config altogether and do everything in Kconfig). It's a pity that so many contributions were ignored by most participants. We need to do our homework before we attend arch WG meetings.

There are many who have participated in the meta-discussion but have not provided much own substantial contributions or comments on arguments that are already on the table.

too much to quote reply

IMO this amounts to weaseling. Everybody might implicitly feel indirectly accused unless you make this specific.

This is very vague and a general obvervation which might be valid and to the point but this needs to be supported with examples that show whether this is going from bad to worse

I personally find this comment a bit unfair. Many others (including myself) have made this point over and over including specific examples. @decsny's comment was embedded in an RFC that pointed to specific examples and only confirmed the observations made there. And his new comment is equally balanced and well argued.

But here some specific examples then as you asked for those:

• I pointed to the USB maintenance area yesterday to make the point.
• I can also point to my own maintenance area. IEEE 802.15.4 configuration as I found it established when I arrived has always made a very liberal interpretation of the hw/sw split, too. We even got a distinction between ieee802154 and ieee802154g accepted upstream some years ago without anyone mentioning its ambiguity.
• You'll hopefully not deny that in the past there have been many PRs where @gmarull repeatedly mentioned questionable use of the hw/sw distinction - and I totally agree with him on all accounts if one takes DTSpec's OS independency requirements seriously. I'm sure @gmarull will easily be able to name a few of those examples right away if challenged.
• It is documented practice in Zephyr to mix multi instance software configuration into the hardware container.
• Update: I can provide many more examples on demand after reviewing existing bindings.

If someone thinks this is a serious problem that needs attendion, someone needs to open an issue with examples and bring this to the attention of everyone.

There are at least two PRs and one RFC where this argument has been brought forward in the context of this topic alone not counting all the unrelated PRs where the hardware/software line has been questioned by commenters in other contexts.

@nashif It would also be nice to get your personal requirements list or at least some confirmation of existing requirements that others have mentioned. This would add tangible content to the discussions rather than just focussing on the meta-discussion alone (which I otherwise find worthwile, too btw, but not exclusively so).

Edit: Updated to the latest state of the discussion include arguments given by others.

[decsny]

I'm not disagreeing with you that we need to talk about the requirements before proposing myriads of solutions, but a lot of these requirements are going to come from observations of weaknesses about the current implementation we have, so saying not to talk about it at all is too vague.

and how does a "vote"" for an implementation in a discussion about requirements and as you say, weaknesses in such implementations help here? We will get there, sooner or later, but when this forum goes into the same rabbit hole about the same topic and the same arguments, most of us will just give up and lose interest.

Some DTS topics being touched on here deservee their own issues and tracks.

I think we are having a misunderstanding here, I am not saying that I think we should be voting on implementations or even proposing them, what I'm saying is that observation of the current implementation is going to be an important source of deriving requirements since most of the motivation for talking about this issue seems like it stems from the limitations people are discovering with the current state of things. That's why I asked you to clarify what you were talking about, since your meta-criticism sounded to broad to me since I personally think discussion of the current implementation is relevant to requirement gathering, but your brief request to filter discussion if taken completely as worded would exclude discussion of that. As for what anybody else in this thread is talking about, I'm not defending the direction anybody is taking, I honestly only skimmed what's going on because I got pinged like 5 times, and it sounded like I needed to clarify my quotation.

edit: late proofread reword

[nashif]

I personally find this comment a bit unfair - at least targeting @decsny specifically.

@decsny do you really feel targetted here? I emphasized that all of this might be true and valid, but we need specific examples and we need to discuss this on a different track if it is really that bad.

@nashif You haven't given any specific examples either to support your own argument, right?

right, and I will not provide examples because I am not making any claims or arguments, you are the one making them.

[fgrandel]

most of us will just give up

Before most of us give up they'll have to make their first substantial contribution or at least refer explicitly to what others have brought to the table. ;-) We have a lack of people listening to each others' argument and a lack of original content.

If you think others are contributing the wrong things to the discussion why don't you simply say the right things rather than making such a fuss here.

[decsny]

@decsny do you really feel targetted here? I emphasized that all of this might be true and valid, but we need specific examples and we need to discuss this on a different track if it is really that bad.

no, i don't feel targeted, from the comment he's referring to that was before I even came to this thread, and you're not wrong that specific examples are needed. Which is why in my original response to all the pings I specifically clarified in response to the claim that I knew about all this that I do not have a compiled list of examples, that would take time and effort which I have not had the luxury to spend. And I want to be clear also based on some comments in this thread that I personally don't desire to be the vehicle for an appeal to authority about what's being discussed here and don't completely see what value it adds anyways, since another point of what I was trying to say above is that those experiences I've seen a lot from being a reviewer, I would expect should also to have probably been commonly seen also by everybody else interested in this issue.

[fgrandel]

Now that we all seem to converge on a common problem statement (pending a few more examples I trust will be easy to find) we can hopefully see what the essence of this back and forth would be wrt the concern of this issue.

Requirements:

• We have to make any conceptual distinction between different "a priori" (=application/use case independent) config containers (Kconfig vs. DT vs. X) so precise that they could ideally be enforced mostly automatically to avoid further conflict and confusion.
• The more such "a priori" containers we propose the more it is important to make those distinctions objectively observable, otherwise users will find it very hard to implement and understand those concepts in practice. Imprecise distinctions and overlap in the config area have cost us as a community and even more our users too much energy already - as was again exemplified by all those comments here re sw vs hw and Kconfig vs DT.

What I mean by precise has been explained in this comment, see the definition of "axiomatic" vs. "ontological": Our configuration space must be accompanied by easy-to-observe (axiomatic) heuristics that refer to intrinsic properties of our software rather than relying on extrinsic "real world" concepts that cannot be mapped to code artifacts without contextual interpretations that we will by definition (and backed by a lot of research) never be able to agree upon objectively.

I've shown that the distinctions and concepts I'm proposing (feature selection vs. configuration, normalization, encapsulation) can be mostly automated or at the very least be defined with mathematical precision. The same can not be said about distinctions like hw vs sw or provisioning vs build time properties. At least not until someone has come up with a heuristic that really represents what we actually do. Note that I've shown that heuristics that have been proposed so far are quite obviously not good enough.

I'm of course open to being proved wrong or convinced otherwise. But certainly not by "hand wavy" concepts that no one has been able to define precisely so far.

[fgrandel]

@nashif

and where is this need documented?

Chris and I spoke up yesterday to explain this requirement once again. I've not heard myself withdrawing it, so it remains on the list, like yours and those of everyone else. We can prioritize it down into the future, but I don't see the slightest reason why we should eliminate requirements just because you feel like it.

What are you accusing me of here?

I think I was rather clear about that. I'll certainly remind you if it happens again.

[benediktibk]

No, it is not missing and should not be there. This system is neither about configuring and selecting OS/HW features for a firmware image nor it is about configuring hardware resources.

@nashif I've got the feeling this could be a new requirement, or even two. Could they be described like this?

• CR6: As a contributor I want to have a separate system for selecting available features.
  • Basically what Kconfig currently is mostly used for, build the UART driver if an instance is enabled
• CR7: As a contributor I want to have a separate system for configuring hardware resources.
  • using DMA stream x for device y

[nashif]

@nashif

and where is this need documented?

Chris and I spoke up yesterday to explain this requirement once again. I've not heard myself withdrawing it, so it remains on the list, like yours and those of everyone else. We can prioritize it down into the future, but I don't see the slightest reason why we should eliminate requirements just because you feel like it.

Chris or anyone else with input and requirements should add them directly and should not need a proxy or a mediator. I am not, and have not requested "elimination" of anything. It is fully OKAY to question requirements and their significance to the ongoing discussion of the high level requirements and determinging where we are going with this shall not be derailed because something can be done or is feasible or sounds interesting.

What are you accusing me of here?

I think I was rather clear about that. I'll certainly remind you if it happens again.

If you think I have been disrespectful, abused some power/authority or anything else, file a complaint please. You seem to be accusing me of something and if I am at fault, it needs to be dealt with immediately. I do not want to wait until "next time".

[nashif]

No, it is not missing and should not be there. This system is neither about configuring and selecting OS/HW features for a firmware image nor it is about configuring hardware resources.

@nashif I've got the feeling this could be a new requirement, or even two. Could they be described like this?

• CR6: As a contributor I want to have a separate system for selecting available features.

  • Basically what Kconfig currently is mostly used for, build the UART driver if an instance is enabled

• CR7: As a contributor I want to have a separate system for configuring hardware resources.

  • using DMA stream x for device y

right, but those are already fullfilled requirements, so are we trying to change how we do configuration in general, for everything?

[benediktibk]

right, but those are already fullfilled requirements, so are we trying to change how we do configuration in general, for everything?

The general area of discussion here is configuration, which of course also includes the already existing methods for configuration. I am at this point not saying we should change something, I am only collecting all the requirements we have for this area. What we then keep as it is or change will depend on how we prioritize possibly conflicting requirements. And we will of course also have to weigh in the necessary implementation effort and the possible benefits.

[fgrandel]

file a complaint please

Please calm down now - I'm not trying to prove you wrong, I'm just preserving my own interest. The complaint is dealt with at the appropriate level - immediately and openly where it originated rather than behind your back.

should not need a proxy or a mediator

Please note, that I'm behind this requirement, too.

[nashif]

file a complaint please

Please calm down now - I'm not trying to prove you wrong, I'm just preserving my own interest. The complaint is dealt with at the appropriate level - immediately and openly where it originated rather than behind your back.

no, I am not calming down. I take this very seriously. You are making very serious accusations here and that can't be let go like this. You can go behind my back, you have my blessing.

should not need a proxy or a mediator

Please note, that I'm behind this requirement, too.

Ok. To quote you verbatim in the meeting yesterday:

Florian Grandel: Around, and to make it really clear. This is not my requirements right.

[carlescufi]

I’d also like to point out that during the conversation, you had ample time to speak uninterrupted, while others who had raised their hands were not given the same opportunity. I understand it’s easy to critique from a position of authority, but it's much harder for community contributors without organizational support to make their voices heard and respected. I believe this is something we should keep in mind moving forward, as we've seen several examples of this imbalance in our current discussion.

@fgrandel as the chair of the Architecture WG, I must emphasize that my goal is to make all voices heard. This is not always easy, and even less so in the context of a cross-continent, cross-company audio conversation over the internet. Requesting your turn to speak is generally mandatory in these meetings, and for the most part always respected. I am always trying to improve the flow of the meeting, so all suggestions welcome, but in no case have I given priority to specific participants because of reasons that could be considered non-technical, including, as you put it, "authority". Some people extend themselves more, and are more proactive to request their speaking turn multiple times during the call, but the Architecture WG call is open to all and considers all contributions under the exact same light and importance.

[fgrandel]

Florian Grandel: Around, and to make it really clear. This is not my requirements right.

Yes, I remember. I've changed my mind, and you know why.

[fgrandel]

I must emphasize that my goal is to make all voices heard.

Yes I know, and I really appreciate it. You're not at all the target of my critique. It's the people talking down others aggressively and those giving heart emojis to each and every statement that serves their personal interest, no matter whether it has merit for the question at hand or not.

[fgrandel]

You can go behind my back, you have my blessing.

I take my own decisions as I see fit. In the meantime I perceive your statement as an open menace. Not the kind of behavior that convinces me that you're willing to talk to me at eye's height.

Meta: I prefer to confront the people concerned in the open rather than escalating this anywhere. Like that everyone else is free to observe what is happening and build their own opinion. I think our respective behaviors speak for themselves. Ever heard of the likability vs. respect/competence dilemma? - I currently very much feel caught in it.

That being said, I'm certainly not free of weak points, so please, feel free to point them out with respect. Maybe you can come out of the defense and tell me what you think are good team working behaviors? The behaviors you currently show are not among those on my personal list. Remember that point on the rule list that you like to cite? Accept critique gracefully...

In any case I think this meta-topic has rightfully accompanied our discussion and I hope we can address it more seriously in the future in the same way we usually address technical problems. This is not only about you @nashif , several others have shown similar behaviors. I've been equally outspoken about those. In that sense I've consistently stated a few rules that I think will help us to work more constructively as a team:

• All comments must either strive to establish community requirements or refer to requirements brought forward by others before.
• All suggested design improvements shall refer to well-established state-of-the-art design criteria including keywords that empower everyone to read up on them if they are not aware of them.
• Comments must show that the author is fully aware of all questions to be decided upon and all arguments that were brought forward previously and explicitly refer to those if they contradict own proposals.

These are not meant to replace the rules that you and others like to cite so frequently. They are meant to make them work in practice.

By these rules @benediktibk and I have assembled a full list of requirements in a few hours that beats everything else that has been contributed so far in this discussion. Your diagram was a great contribution, too, but the way you dealt with how I tried to build upon it constructively (!) to achieve a similar team result illustrates the differences in behaviors and outcome quite well.

[fgrandel]

@Laczen @henrikbrixandersen

the static configuration generates a default const struct (e.g. net0_cfg = {}) for the subsystem (or for a part of the subsystem). At the same time the "backend" is constructed:

const struct static_backend net0_backend[] = {
        .name = "net/net_cfg",
        .data = &net0_cfg,
        .data_size = sizeof(net0_cfg),

The "net/net_cfg" handler would copy the constant struct to the runtime struct.

For the static case this works nicely. :-) Now a few more detailed design questions:

1. In case we have an independently provisioned settings image from which we want to re-construct the net0_cfg object:

Am I assuming correctly that the settings frontend won't care where the data comes from, so basically the backend would check whether there is an image with updated data and uses that to instantiate the configuration dynamically? If the backend finds one, it distributes that data, otherwise it falls back to the statically configured default...

2. How would we scale this approach across multiple subsystems w/o breaking subsystem encapsulation?

a) For the case that we don't need to write back to the store we could have multiple sources, each encapsulating the knowledge of one subsystem. No problem.

b) For the case that we need to write back: As far as I understand we currently can only have a single destination (i.e. writable) backend. Am I assuming correctly that in this case we'd either have to extend the settings subsystem to support multiple destinations or we'd have to implement something like a "delegating backend", i.e. interested subsystems would have to register themselves with the backend so that it can save data on their behalf w/o the backend having to "know" about all the subsystem details.

Is my understanding correct? Have I overlooked something? Any more ideas how to design this elegantly in practice?

Advantages of this approach I discovered while thinking about it:

• This approach solves the static memory allocation problem elegantly, as long as we won't change the number of instances at provisioning or run time. In the latter case we'd have to work with dynamic memory allocation, not a problem but it's nice that we have a static allocation option.

• It's easy to upgrade (or migrate) from a "build only/no settings subsys" approach to this approach as code required to instantiate the default during build can be re-used.

• If the backend (delegate) is co-located with the submodule then we can implement more type safety as the submodule has knowledge of the expected (de)serialization schema other than a generic backend.

So overall I'm quite convinced that this is the right way to go to cover all requirements we currently know of.

@nashif If this is what you mean by a "configuration subsystem abstraction layer", then we're on the same page.

[Laczen]

@fgrandel ,

1. In case we have an independently provisioned settings image from which we want to re-construct the net0_cfg object:

Am I assuming correctly that the settings frontend won't care where the data comes from, so basically the backend would check whether there is an image with updated data and uses that to instantiate the configuration dynamically? If the backend finds one, it distributes that data, otherwise it falls back to the statically configured default...

Yes, the data could even come from an "online" source. As soon as the subsystem struct or part of it would need to be changeable, a memcpy to a non const struct could be used.

2. How would we scale this approach across multiple subsystems w/o breaking subsystem encapsulation?

a) For the case that we don't need to write back to the store we could have multiple sources, each encapsulating the knowledge of one subsystem. No problem.

b) For the case that we need to write back: As far as I understand we currently can only have a single destination (i.e. writable) backend. Am I assuming correctly that in this case we'd either have to extend the settings subsystem to support multiple destinations or we'd have to implement something like a "delegating backend", i.e. interested subsystems would have to register themselves with the backend so that it can save data on their behalf w/o the backend having to "know" about all t he subsystem details.

There are several ways to tackle this: the dst (destination) backend is used by the settings system when a settings_save is used. It is a "convenience" solution to store changes so that they can be reused. A subsystem does not have the obligation to use the settings_save method, it could just as well have its own way to store changes directly in the proper src system. Another way could be to use the "convenience" solution but to encrypt the data so it can't be used by any other subsystem. A delegating backend could also be used.

Is my understanding correct? Have I overlooked something? Any more ideas how to design this elegantly in practice?

IMHO the settings system might require some changes to simplify the switching between backends (registering and unregistering srcs and dst), reworking some of the naming (e.g. settings_name_steq, settings_name_next, ...), adding namespace support, reducing the stack usage for multipart data, ... But most important, start using the features.

The method of how the const structs are generated can be based on macro's or yaml, I would prefer the latter (even as a non-yaml user) because writing and debugging FOR_EACH_INST() macros is tedious and error-prone when it is needed to handle setups where some part of the data is missing or validation needs to be done. But again, this is a personal preference and both methods can be used.

Another advantage is that direct support of multiple static configurations can be done. There are just multiple "const structs" defined and switching between them is just a change of a pointer.

[fgrandel]

Thanks @Laczen, all your suggestions make sense to me. Very helpful.

Just one question to confirm my understanding:

The method of how the const structs are generated can be based on [...] yaml,

When you say "based on yaml", you mean something similar to what Jukka did? I.e. a python script that takes in a YAML configuration file and generates C code files or snippets?

[Laczen]

Thanks @Laczen, all your suggestions make sense to me. Very helpful.

Just one question to confirm my understanding:

The method of how the const structs are generated can be based on [...] yaml,

When you say "based on yaml", you mean something similar to what Jukka did? I.e. a python script that takes in a YAML configuration file and generates C code files or snippets?

Yes.

As a personal note I really like the "configuration" method used in esphome, where it is even possible to add code to the yaml files. Leaving devicetree for the hardware setup and just enable parts in the overlay (or have them enabled based upon an entry in the yaml). Move all non-hardware setups (also e.g. partitions as well as buffers, ...) out of dts and governed by a subsystem that is configured by yaml files. For easy configuration of subsystems (that as standard should be multi-instance) they could all provide a macro to create a subsystem item (passing any configuration option trough the macro) so that the translation from yaml to config is just filling up a(multiple) macro call(s). Access to hardware should be done through a subsystem that can also be used as the layer to provide buffers,... As a downside this would probably make much of the tooling that has been added on top of dts obsolete, it might not result in the smallest image and I am certain this is not inline with the idea's of some maintainers and developers. But then again, this is a personal note, all of the formulated ideas can be selected or rejected and I am sure there will be areas where they do not apply.

[fgrandel]

@Laczen

Thanks for those considerations and proposals! IMO there cannot be enough ideas on the table (personal or not). I mean, you clearly do not express them at the expense of other people's ideas and arguments.

We are deep into solution space here, but it helps me to think more clearly about higher level ideas, when I have a few potential implementation options in mind.

Access to hardware should be done through a subsystem that can also be used as the layer to provide buffers,...

That's the one sentence I didn't understand. All the rest seems clear to me.

where it is even possible to add code to the yaml files

Can you point me to a docs page where this is explained?

[Laczen]

@fgrandel

We are deep into solution space here, but it helps me to think more clearly about higher level ideas, when I have a few potential implementation options in mind.

I am in the same position, small solutions proposals help me figure out the global approach

Access to hardware should be done through a subsystem that can also be used as the layer to provide buffers,...

That's the one sentence I didn't understand. All the rest seems clear to me.

Let's take the example of a uart. This is clearly a device. To use it with a circular buffer to store data this circular buffer can be added to the driver (needing configuration in dts/Kconfig) or it could be added to a layer above the driver (e.g. a serial subsystem), with configuration in the new config system. Now for a uart this might be overkill, but it is only the idea that I'm trying to explain.

where it is even possible to add code to the yaml files

Can you point me to a docs page where this is explained?

https://esphome.io/cookbook/lambda_magic

[nashif]

@Laczen

As a personal note I really like the "configuration" method used in esphome, where it is even possible to add code to the yaml files.

I Like this example, I personally use esphome and get what you are saying here and that is basically the direction i was thinking as well.

To use it with a circular buffer to store data this circular buffer can be added to the driver (needing configuration in dts/Kconfig) or it could be added to a layer above the driver (e.g. a serial subsystem), with configuration in the new config system.

This is what I refer to above in a previous comment:

"Each subsystem or software component, where applicable, should be configurable both using Kconfig for static and initial
use and at run-time using a subsystem specific configuration ."

so we are on the same page, i.e. to configure some LED, you need a LED service that supports configuration, to toggle some GPIOs you will need a GPIO service etc...

[fgrandel]

to configure some LED, you need a LED service that supports configuration, to toggle some GPIOs you will need a GPIO service etc...

@nashif Do I understand you correctly that you're speaking about application facing services here? Something like the existing net mgmt service - just more generic so that it can be used to configure any subsystem?

If that assumption is correct: Would you also migrate, say, net mgmt, to this same approach?

[nashif]

@nashif Do I understand you correctly that you're speaking about application facing services here? Something like the existing net mgmt service - just more generic so that it can be used to configure any subsystem?

more like net_config

[Laczen]

This is what I refer to above in a previous comment:

"Each subsystem or software component, where applicable, should be configurable both using Kconfig for static and initial use and at run-time using a subsystem specific configuration ."

I wanted to refer to this statement, but could quickly find it :).

[carlescufi]

And I agree, I think it would be best if @carlescufi would put some version of this list into the original post and maintain it there for visibility.

I've moved the requirements to a separate document that I link from the PR description

[nordicjm]

Comments on list of requirements:

UR1: Not relevant to software configuration, these are hardware configuration, these go in dts, not relevant to topic.
UR2: High importance
UR3: Point 1 valid, points 2-3 not valid as per UR1.
UR4: Low-medium importance
UR5: Don't see how devicetree syntax macros of bindings syntax is relevant to this topic, no importance
UR6: Mentions things like data compression, outside scope of any initial implementation thus no importance
UR7: Again, refers to drivers and bindings, no relevance to topic, no importance
UR8: Essential requirement
UR9: No opinion
UR10: No opinion
UR11: Low to no importance
UR12: Medium importance
UR13: No importance
UR14: No relevance to topic
UR15: Does not state if this is default/static data or user-changed data, if default/static data then medium importance, if user-changed data then does not make sense with the first part of the question, if clarified then could be idea for after initial implementation
CR1: Can't see relevance of this question or how it is related, dts is dts, it should not be used in this topic
CR2: Medium importance
CR3: Duplicate of UR12
CR4: Too many unrelated things grouped together, no importance
CR5: Too many unrelated things grouped together, no importance

Not really many of the requirements even talk about the topic of discussions, there's not enough relevant requirements to draw any conclusion from

[fgrandel]

Not really many of the requirements even talk about the topic of discussions, there's not enough relevant requirements to draw any conclusion from.

No problem. Feel free to add the really relevant ones yourself. That's why we have that list. :-) Dropping requirements later or prioritizing them down is easy. Overlooking a relevant one would be bad, though.

[benediktibk]

UR15: Does not state if this is default/static data or user-changed data, if default/static data then medium importance, if user-changed data then does not make sense with the first part of the question, if clarified then could be idea for after initial implementation

I agree with @nordicjm, we should clarify this requirement. My intention while writing was that configuration data, which is loaded during runtime for instance from an SD card, should be verified. This is important in combination with the requirements for the signed images.

CR3: Duplicate of UR12

I would not consider this as a duplicate. Fulfilling CR3 will probably help UR12, but one can still think of an unnecessary complex configuration system which is hard to understand, although it has automatically generated documentation.

For the comments from @nordicjm on CR4 and CR5 I kind of agree, it would really help if we could maybe split them up into a few smaller ones? @fgrandel

Overall I understand the comments from @nordicjm that he prefers not to change anything devicetree related at all. This is definitely a valid requirement, which I have tried to reflect with UR5. But this might contradict other requirements, like for instance CR1 and CR2. Therefore, I ask you to keep your mind open at this stage. And of course feel free to refine the requirements or add new ones, if you think some are missing.

[carlescufi]

Architecture WG meeting

• Discussion about what in Devicetree can or cannot be provisioned. There seems to be an agreement that some values in Devicetree make sense to be provisioned (MAC address for example)
• @fgrandel suggests a new configuration data model that is able to then selectively choose which "properties" (i.e. items) are to be provisioned or not
• In general there is clearly a lack of understanding among the participants as to what we want to achieve. The analysis of the requirements made it clear that we need to define the scope better in order to make progress