How to manage device and non-device initialization order

[pabigot]

I like the general path of #23407 which decouples non-device initialization from the device structure, but a major follow-on task is coordinating the different types of initializations.

Currently Zephyr uses five init levels: two pre-kernel, one post-kernel, one application, one SMP. These levels identify the state of the kernel and application. Within each level there are 100 priorities to enforce dependencies. These get sorted at link time into arrays that are processed to enforce the order.

I believe it's agreed that we can determine dependencies among device initializations based on devicetree relationships (ignoring non-device dependencies in the device init function, e.g. on kernel state). So theoretically we could generate an ordered sequence of devices which if executed would satsify the device relationships, ignoring the init level. This was the original, too-simplistic, plan that breaks as described later.

I also believe it's agreed that we also need to be able to have non-device initializations a la SYS_INIT, and that these initializations should be associated with kernel initialization stages. (I don't personally see a need for two pre-kernel stages since that could be handled by priority within a single stage, but that's a side argument.)

The problem is that we have no way to express dependencies between device and non-device initialization. For example SYS_INIT function to select a default antenna may require that a GPIO driver be functional; or a driver initialization may use a shared data structure that depends on SYS_INIT.

We can certainly continue to use the hard-coded init level and Kconfig-influenced priority, but (a) it's fragile in the face of application customization, and (b) doesn't currently allow for instance-specific priority differences.

If we can associate an external symbol with each device and non-device init object, and require that init registration includes an explicit list of things the developer believes it depends on, then we could encode the dependencies into a data structure that's walked at runtime to ensure execution is in the correct order.

Or we could use that same technology to generate the data structure in a pre-link step and process it during the build and replace it with an ordered sequence that includes markers where the kernel state transitions (e.g. pre-kernel to post-kernel).

That's a bit complicated but is the only general solution I've come up with.

How do other people think this should be addressed?

[pabigot]

@mbolivar-nordic @tbursztyka could you take a look at this and provide your perspectives?

[pabigot]

Marked dev-review to increase exposure.

[mbolivar-nordic]

If we can associate an external symbol with each device and non-device init object, and require that init registration includes an explicit list of things the developer believes it depends on, then we could encode the dependencies into a data structure that's walked at runtime to ensure execution is in the correct order.

I think that's basically the systemd approach instead of the runlevel approach. And I think it's probably the right way to go if it can be done without breaking the bank in terms of resources.

[tbursztyka]

Or we could use that same technology to generate the data structure in a pre-link step and process it during the build and replace it with an ordered sequence that includes markers where the kernel state transitions (e.g. pre-kernel to post-kernel).

You will definitely need to keep some of the existing markers. First because it is "readable" and second because obviously the kernel needs to know when to initialize stuff device/services before doing things on its own and initialize another batch of devices/services. (We could of course declare these "in between" duties as services but it would render kernel/init.c cryptic)

Merging both pre_kernel labels would just require to raise the priority from 0 to beyond 99 (I don't think it's an ld limitation).

That being said I'd still try to keep the core init structure as the base for ordering the initialization, all at build time. Adding another data structure, require more runtime processing, would be a regression.

[pabigot]

@andrewboie @galak please provide your perspective, and invite others who have a stake in this. Until we know how device and non-device initialization will be integrated given new information about device dependencies we can't move forward on some devicetree and device rework.

[andrewboie]

You need to talk to @tbursztyka

[pabigot]

@galak Leaving in dev review for 2020-04-23.

Unless there is significant interest in and consensus support for a plan to change things I expect Zephyr will continue to use level and priority to determine initialization order. I don't see a path to taking devicetree order constraints into consideration for initialization with this model since level cannot be inferred.

[tautologyclub]

This may be a naive idea but would it be horrible if each init held responsibility of calling the inits of its dependencies? It would be a major change I reckon but it would feel pretty natural imo - expose the init function as a public runtime api and let init calls propagate upwards in JIT fashion, until an already_inited checks reports true

[tautologyclub]

This would ofc solve the problem of a delayed init api as well, which is sought after by power management and other random use cases, see #24318 for instance

[pabigot]

Dev-review 2020-04-23: Continue to use level and priority as-is for now. Come up with a way to name both devices and sys-init values with unique values in a way that can represent a set of dependencies (e.g. the ordinal of the entry for the corresponding struct init_entry from #23407). Evaluate the expressibility of that for power management, then figure out how to expose the sys-init values and associate dependencies between them and devices.

[ceolin]

#34518