Device initialization order that respects devicetree dependencies

[mbolivar]

I wasn't able to find an existing issue (neither #20348 nor #6293 cover it exactly). Happy to close if I missed something.

Is your feature request related to a problem? Please describe.

Currently, device drivers must specify the initialization level (PRE_KERNEL_1, etc.) and priority (usually something like CONFIG_KERNEL_INIT_PRIORITY_DEVICE) within that initialization level for each device. These determine the order in which device init functions are called.

This mechanism can make it awkward to capture more complex dependencies, e.g. a GPIO expander that controls a SPI device CS line, which should therefore be initialized before the SPI device.

It's always possible to do this by manually messing around with priorities. However, that's cumbersome, especially since the dependency information is already available in the devicetree.

Describe the solution you'd like

It should be possible to perform device initialization based on the dependency order implied by the "dependency ordinal" assigned to each node in the devicetree (and available in edtlib.Node.dep_ordinal).

To avoid breaking things too much, I think this could be added under a new Kconfig bool, CONFIG_DEVICE_DT_INIT. When enabled, this would allow devices to pass their dependency ordinal from devicetree to a new DEVICE_INIT-type macro (DEVICE_AND_API_INIT_DT or so as prototyped in #20253) that assigns a priority based on this value.

To avoid breakage, I think this also means we need to make sure the following when CONFIG_DEVICE_DT_INIT=y:

• CONFIG_KERNEL_INIT_PRIORITY_DEFAULT is higher priority than the highest DT-based priority
• CONFIG_KERNEL_INIT_PRIORITY_DEVICE is lower priority than the lowest DT-based priority

We'll also probably need to increase the number of .init_<LEVEL><PRIO> linker sections.

Open questions

• How should initialization levels fit into this world? Example: does it still make sense to have a PRE_KERNEL_1 and PRE_KERNEL_2, or can we merge them into one PRE_KERNEL init level and just let DT ordinals sort it out?
• If we do keep separate init levels, what should happen if actual initialization order is the reverse of DT dependency order? For example, suppose deviceA is in PRE_KERNEL_1 and deviceB is in PRE_KERNEL_2, but deviceA depends on deviceB in the DT. That seems like it should be a build-time error, or at least a warning.
• How do "pseudo-devices" created using SYS_INIT() fit into this picture? For example, should we ensure their priorities always lie outside of the DT ordinal priority range?

[pabigot]

Yeah, this gets messy due to existing usage. Rambling thoughts below that do not lead to a coherent proposal.

I think it's fair to specify that device initialization is done either in the existing Kconfig + INIT_LEVEL order, or in the order defined by devicetree dependencies, depending on CONFIG_DEVICE_DT_INIT. If it were possible to verify that Kconfig+INIT_LEVEL mis-orders a dependency that'd be great, but since we don't do that now I'd be willing to say "if you screw up initialization order that's your fault".

The devicetree dependencies only impose a partial order, while the ordinals impose a total order. The documentation specifies that PRE_KERNEL_1 is for devices that have no dependencies. We could easily post-process the existing order with a stable sort where nodes with no dependencies are moved before nodes with at least one dependency. Then we can provide at build-time the ordinal of the first device that should be part of PRE_KERNEL_2.

Because z_sys_device_do_config_level() operates on non-device things like SYS_INIT, I think we should try to preserve the two-stage pre-kernel distinction when initializing in devicetree order. I can imagine a use for PRE_KERNEL_2 priority 0 meaning "first thing after the SOC is initialized", with the intent that it precedes specific device initialization.

We could then perhaps use a SYS_INIT at CONFIG_KERNEL_INIT_PRIORITY_DEVICE to run the initializations for the devices for the PRE_KERNEL_1 and PRE_KERNEL_2 parts of the ordered list using z_sys_device_do_config_level().

This doesn't address what happens if somebody needs to use SYS_INIT() after one device is initialized but before another is initialized.

Looking at nxp_mcu.c I see code that selects between PRE_KERNEL_1 and PRE_KERNEL_2 depending on whether logging is enabled. So perhaps we need to do two passes through the list:

• in PRE_KERNEL_1 do the initialization of all devices that have no dependencies and are configured for PRE_KERNEL_1.
• in PRE_KERNEL_2 do the initialization of everything that wasn't initialized in the first pass.

Or in some other way continue to respect the level parameter of the initialization regardless of dependency order. Because some of these devices need to be initialized in POST_KERNEL.

And at this point it seems there's no clear path to supporting dependency-based initialization without reworking how initialization priorities are specified. Syntactically it's difficult to deal with the separated level and priority elements, so maybe it's right that we continue to respect them.

Could we do something like a macro SYS_DT_INIT_PRIORITY(_dt_path) which expands to either CONFIG_KERNEL_INIT_PRIORITY_DEVICE or DT_PATH_##_dt_path##_INIT_PRIORITY_ (where that's an integer value derived from the dependency partial order, and the trailing underscore is a feeble attempt to prevent clashing with an init-priority property)?

[mbolivar]

Thanks for the feedback!

Yeah, this gets messy due to existing usage.

Agreed.

We could easily post-process the existing order with a stable sort where nodes with no dependencies are moved before nodes with at least one dependency. Then we can provide at build-time the ordinal of the first device that should be part of PRE_KERNEL_2.

This would take some zephyr-specifics like filtering out trivial or useless dependencies such as /soc, but that could definitely be done.

I'd be willing to say "if you screw up initialization order that's your fault".

👍

Could we do something like a macro SYS_DT_INIT_PRIORITY(_dt_path) which expands to either CONFIG_KERNEL_INIT_PRIORITY_DEVICE or DT_PATH_##_dt_path##_INIT_PRIORITY_ (where that's an integer value derived from the dependency partial order, and the trailing underscore is a feeble attempt to prevent clashing with an init-priority property)?

I'll look into that. For now, to make things concrete, here is an early prototype that at least has a skeleton in place, and is opt-in based on the config option discussed in the proposal:

https://github.com/mbolivar/zephyr/tree/dt-dep-order

From the commit log for the final patch I'm using for testing:

To test the previous commits, simply build and run
samples/subsys/shell/shell_module for e.g. qemu_cortex_m3. The
initialization order will be printed, and you can use 'device levels'
to see where the test devices defined in main.c fit.

[nordic-krch]

Note that i'm not very familiar with DT and all scripts around but

I was wondering if it wouldn't be possible to describe all dependencies in DT and let the script figure out best order of initialization. It could create a static array with indexes of entities to initialize (or maybe somehow reorder items in sections before linking to avoid array?). That approach would require to have all dependencies somehow instantiated in the DT (even kernel).

Dependencies like:

• i2c0: kernel
• gpio_exp: i2c0 (derived from the tree), gpio_p0, power_rail
• kernel: clock_control, soc
• clock_control: soc
• soc: none

Derived order: soc, clock_control, kernel, i2c0, gpio_exp

[mbolivar]

I was wondering if it wouldn't be possible to describe all dependencies in DT

We already have this, thanks to @pabigot

and let the script figure out best order of initialization.

Yup, that's exactly the purpose of this issue. Take a look at this PR for a rough idea of how the dependency info is in devicetree (though the code is just throwaway stuff): #22802

[mbolivar]

@tbursztyka I saw your #22941 and I wonder if you'd be open to completely getting rid of PRE_KERNEL_1, PRE_KERNEL_2, and friends and just letting DT do all the work?

[mbolivar]

@tbursztyka I saw your #22941 and I wonder if you'd be open to completely getting rid of PRE_KERNEL_1, PRE_KERNEL_2, and friends and just letting DT do all the work?

The more I think about this and look at the docs, the more I wonder if PRE_KERNEL_1 and friends should be replaced entirely as APIs if we're doing DT-based init. Their documentation feels like hard-coding dependencies:

"Used for devices that have no dependencies, such as those that rely solely on hardware present in the processor/SOC."

This is a contradiction, of course. Even SoC-internal hardware devices have dependencies on one another.

"The interrupt subsystem will be configured however so it’s OK to set up interrupts"

Which interrupts, exactly? When is the "interrupt subsystem" "configured" here?

[tbursztyka]

@mbolivar Let's do it iteratively. First let's move level/prio setting to DTS, and only then we will be able to rework the level names etc... Much easier that way.

[tbursztyka]

To avoid breaking things too much, I think this could be added under a new Kconfig bool, CONFIG_DEVICE_DT_INIT. When enabled, this would allow devices to pass their dependency ordinal from devicetree to a new DEVICE_INIT-type macro (DEVICE_AND_API_INIT_DT or so as prototyped in #20253) that assigns a priority based on this value.

Too complicated. It would be just easier to always let DTS generate the right label/prio for each and every device. And up to maintainers to switch to these DT generated macros instead of hardcoded/Kconfig values.

OR even better: do nothing until we get DTS based device instantiation (via code generation etc...). That will be way much easier at it will require only to mess a bit with DTS and that's all. Removing pre_kernel_<1/2> would be also way much easier then.

I was wondering if it wouldn't be possible to describe all dependencies in DT and let the script figure out best order of initialization. It could create a static array with indexes of entities to initialize (or maybe somehow reorder items in sections before linking to avoid array?). That approach would require to have all dependencies somehow instantiated in the DT (even kernel).

All dependencies: so something like #23394 (if the idea gets accepted) + added SYS_INIT() for kernel (and all other missing parts). That get rid of all the level concept though, not sure we want to get rid of it entirely.

[mbolivar]

I'm happy to get rid of levels and just let hardware dependencies get managed by DT.

I think software dependencies could be managed entirely in software, though, e.g. by letting system initialization functions declare the devices they need set up before they run, rather than mixing software into DT.