RFC — DtSh, shell-like interface with Devicetree

[dottspina]

RFC: DTSh: DTS file viewer with a shell-like command line interface

Introduction

Problem description

Prior to Zephyr, I've only approached Devicetree while looking through LKML or The Linux Kernel documentation.

And, quoting @mbolivar-nordic (October 2020):

Is your enhancement proposal related to a problem? Please
describe.

Devicetree is widely used by Zephyr, but it lacks convenient tooling
for development.

Describe the solution you'd like

A graphical devicetree viewer and editor, like 'guiconfig' for DTS.

I can agree that, when learning Zephyr use of Devicetree, I've personally felt the lack of a quick, simple tool to:

• see the final devicetree generated at build-time
• find nodes based on supported bus protocols, generated IRQs, or keywords like "BME680" or "PWM"
• access the relevant binding files
• generate simple figures to illustrate my note-taking

References:

• Improved user interface for devicetree (like guiconfig) #29295
• RFC - DTV (Device Tree Visualiser)

Proposed change

The proposed change is a new West command that opens Devicetree Source files (DTS) in a shell-like command line interface:

• navigate and visualize the devicetree through a hierarchical file-system metaphor
• search for devices, bindings, buses or interrupts with flexible criteria
• redirect command outputs to files (text, HTML, SVG) to document hardware configurations or illustrate notes
• rich CLI, command line auto-completion, command history, user themes

$ west build
$ west dtsh
dtsh (4.0.99): Shell-like interface with Devicetree
How to exit: q, or quit, or exit, or press Ctrl-D

/
> cd &flash_controller

/soc/flash-controller@4001e000
> find -E --also-known-as (image|storage).* --format NKd -T
                             Also Known As               Description
                             ───────────────────────────────────────────────────────────────────────────────────
flash-controller@4001e000    flash_controller            Nordic NVMC (Non-Volatile Memory Controller)
└── flash@0                  flash0                      Flash node
    └── partitions                                       This binding is used to describe fixed partitions…
        ├── partition@c000   image-0, slot0_partition    Each child node of the fixed-partitions node represents…
        ├── partition@82000  image-1, slot1_partition    Each child node of the fixed-partitions node represents…
        └── partition@f8000  storage, storage_partition  Each child node of the fixed-partitions node represents…

where:

• west dtsh: called without argument, the command will open the DTS file build/zephyr/zephyr.dts, and retrieve the bindings that Zephyr used at build-time from the CMake cache file build/CMakeCache.txt
• cd &flash_controller changes the current working branch from the devicetree's root to the node at /soc/flash-controller@4001e000, using its DTS label flash_controller
• find is with no surprise a shell command that will search for devices, bindings, buses, interrupts, etc
• here -E --also-known-as (image|storage).* will match nodes with a label or alias starting with image or storage; predicates like --with-reg-size >4k are also supported
• --format NKd: set the node output format to the columns "node Name,", "Also Known As" (all labels and aliases), and "description" (D is the "Depends-on" column)
• the descriptions, e.g. "Flash node", are actually hyperlinks that, when clicked, will open the corresponding binding files
• -T: list found nodes in tree-like format
• appending > doc/partitions.svg to the last command line would save the partitions tree to the file doc/partitions.svg, in SVG format

The considered use cases include:

• to help getting started with Devicetree: hierarchically or semantically explore a devicetree, contextually access binding files or the Devicetree specification, save figures to illustrate notes
• to have on hand a simple DTS file viewer: quickly check the enabled buses and connected devices or the RAM and Flash memory, get a first insight when debugging a Devicetree issue
• automatic generation of hardware documentation in non-interactive mode (batch)
• an extensible framework for prototyping CLI tools based on Zephyr's python-devicetree library

This proposal is based on DTSh - A Devicetree Shell: please also refer to the Handbook (also included in this PR) to find out more about its features and how it works.

Detailed RFC

This RFC includes:

• DTSh implementation and unit tests (Python), version v0.2.4
• the companion West command
• documentation in reStructuredText format

Proposed change (Detailed)

This PR comes with 36 commits:

• [1]: Python building and packaging, including ruff configuration and DTSh requirements
• [2..33]: DTSh implementation
• [34]: DTSh unit tests
• [35]: West command implementation and integration
• [36]: Documentation

They can of course be rearranged.

DTSh

DTSh is implemented in Python and is a Devicetree tool: it seems its natural location is zephyr/scripts/dts/dtsh, sibling to the python-devicetree library with which it's tightly coupled.

Source files are located in dtsh/src, and dtsh is the root Python package.

Bellow is a summary of the software architecture.

Devicetree model

DTSh introduces its own model layer above edtlib.EDT:

• to factorize DTSh coupling with the edtlib API
• to support the hierarchical file system metaphor at the model layer
• to support matching and sorting nodes at the model layer

Module		Unit tests
dtsh.hwm	Zephyr hardware models	test_dtsh_hwm
dtsh.dts	devicetree source definition	test_dtsh_dts
dtsh.model	devicetree model	test_dtsh_model
dtsh.modelutils	devicetree model helpers (e.g. sorters, criteria)	test_dtsh_modelutils

GNU Readline integration

Responsibilities:

• provide DTSh client code with a single entry-point API for GNU Readline integration
• isolate the completion logic (find matches) and view providers (display matches) from the Readline hooks machinery

Module		Unit tests
dtsh.rl	GNU Readline integration
dtsh.autocomp	completion logic and base display callback	test_dtsh_autocomp

Devicetree shell

Responsibilities:

• command line parser
• devicetree shell sessions setup
• helpers for implementing shell commands

Module		Unit tests
dtsh.io	base I/O streams for the devicetree shell	test_dtsh_io
dtsh.config	devicetree shell configuration and user preferences	test_dtsh_config
dtsh.shell	devicetree shell	test_dtsh_shell
dtsh.shellutils	devicetree shell helpers (e.g. common options)	test_dtsh_shellutils
dtsh.session	base devicetree shell session

Rich CLI

DTSh's Command Line Interface is built on top of the Textualize's rich API.

Module		Unit tests
dtsh.rich.theme	colors and styles, user themes	test_dtsh_theme
dtsh.rich.text	text view factories
dtsh.rich.tui	base layout definitions
dtsh.rich.modelview	devicetree model to views
dtsh.rich.html	command outputs redirection to HTML files	test_dtsh_rich_html
dtsh.rich.svg	command outputs redirection to SVG files	test_dtsh_rich_svg
dtsh.rich.io	rich I/O streams for the devicetree shell
dtsh.rich.autocomp	styled contextual display callbacks
dtsh.rich.shellutils	helpers for formatted command outputs	test_dtsh_rich_shellutils
dtsh.rich.session	rich devicetree shell session

Built-in Commands

Module		Unit tests
dtsh.builtins.pwd	pwd: print path of current working branch.	test_dtsh_builtin_pwd
dtsh.builtins.cd	cd: change the current working branch	test_dtsh_builtin_cd
dtsh.builtins.ls	ls: list information about nodes	test_dtsh_builtin_ls
dtsh.builtins.tree	tree: list nodes in tree-like format	test_dtsh_builtin_tree
dtsh.builtins.find	find: search branches for nodes	test_dtsh_builtin_find
dtsh.builtins.alias	alias: list aliased nodes	test_dtsh_builtin_alias
dtsh.builtins.chosen	chosen: list chosen nodes	test_dtsh_builtin_chosen
dtsh.builtins.cat	cat: concat and output information	test_dtsh_builtin_cat
dtsh.builtins.uname	uname: print system information

Please refer to Built-in Commands in the DTSh handbook for detailed command descriptions.

Unit tests

Unit tests are implemented with pytest, and located in dtsh/tests.
The res sub-directory contains test resource files.

The easiest way to run the tests is simply hatch test.

West extension

West command implementation:

• class DTShell in zephyr/scripts/west_commands/dtshell.py
• packages dtsh and devicetree made available with sys.path.append() statements

The West dtsh extension is added to zephyr/scripts/west-commands.yml.

Support for West completion is added to west-completion.bash and west-completion.zsh.

Documentation

The proposed change:

• adds a brief entry in zephyr-cmds.rst to document the West command
• that links to the DTSh handbook (handbook.rst), located in a dedicated directory along with some images

Dependencies

Beside a couple of extra Python requirements, DTSh does not affect other Zephyr components or needs changes in other areas.

It is however tightly coupled with Zephyr’s devicetree tooling, in particular the edtlib library.

Extra requirements

These requirements are added to zephyr/scripts/requirements-extra.txt.

Textualize's rich

DTSh's Command Line Interface is built on top of Textualize's rich library.

It's a mature and actively maintained project, upon which Textual, the company's flagship framework, is based.

rich itself installs a few dependencies:

• pygments: Pygments Python syntax highlighter
• markdown-it-py: Markdown parser done right

GNU Readline (macOS)

For auto-completion, command history and key bindings, DTSh relies on the readline module of the Python standard library.

On POSIX-like systems, the underlying Readline facilities are implemented by either:

• libreadline, the GNU Readline library, on GNU/Linux systems
• or libedit, the Command Line Editor Library, typically used on BSD-like systems

These libraries are not fully compatible, and DTSh currently supports only the GNU Readline implementation of the Python readline module.

On macOS, where libedit will likely be the default for most Python distributions, DTSh substitutes the shipped readline module with gnureadline, "bundling the standard Python readline module with the GNU Readline source code, which is compiled and statically linked to it".

Although gnureadline is added only to the macOS requirements, DTSh will prefer this stand-alone module wherever it is installed, assuming it's on purpose: this allows users of other POSIX-like systems to fix their Readline support with just pip install gnureadline.

On Windows, Readline support will very likely be disabled: see Readline on MS Windows.

Zephyr’s devicetree tooling

DTSh naturally relies on Zephyr’s python-devicetree for loading DTS and binding files into Devicetree models: API changes in edtlib may require changes in DTSh, and, conversely, DTSh may welcome API changes in edtlib.

This adds some maintenance, but may also help edtlib mature as an independent library: DTSh seems like a good eat your own dog food context for evaluating the use of edtlib outside of the Zephyr build-system.

This was more or less part of the initial goals:

• Zephyr’s devicetree tooling
• Design goals

Note

DTSh also relies on the contents of the CMake cache populated when the DTS was generated, if found.
Changes in the available (cached, external) variables or their semantic may therefore also affect DTSh.

For example, in the absence of more conclusive evidence, DTSh may consider the availability of SOC_FULL_DIR to be an indication of a version 2 hardware model (HWMv2).
This heuristic is actually based on the contents of soc_v1.cmake and soc_v2.cmake: only the latter will save SOC_FULL_DIR to the CMake cache.
I thought of this example because I don't believe this difference in cache files is due to a difference between hardware models: nothing actually prevents the relevant maintainers from changing their mind and deciding to cache SOC_FULL_DIR also when building an HWMv1 target. DTSh would then have to adapt (drop the broken rule, without regret).

Concerns and Unresolved Questions

Known Issues

Just linking to the DTSh project's known issues.

If you find any bugs while reviewing this RFC, please (also) report them here.

Maintenance

This is probably the main concern with this RFC:

• the code base is sizable
• DTSh must track changes in edtlib, and adapt when necessary
• DTSh may also be affected by (or interested in) other changes, e.g. in Zephyr's bindings or hardware model

Once the initial review is complete, day-to-day maintenance should be low, though:

• the Python module dtsh.model encapsulates all DTSh uses of the edtlib API: if breaking changes are introduced, DTSh will fail there
• the included unit tests should catch the most obvious regressions early, and help narrowing issues

More significant changes, like "Hardware model v2" or transitioning from YAML to JSON bindings (unlikely to happen anytime soon), can however involve a little more work to either fix DTSh or add support for the new concepts or resources.

Fortunately:

• such changes don't happen every full moon night
• the Zephyr project seems to operate this kind of transitions cautiously: changes are discussed in advance, and introduced incrementally with some backward compatibility

Finally, I'll gladly continue myself contributing support, features and bug fixing in the long run.

Testing

DTSh use cases include an educational tool for newcomers to Zephyr Devicetree: incorrect, erroneous results (command outputs) would do more harm than good.

Existing unit tests should provide a minimum level of correctness, but:

• they involve only a small subset of the possible configurations (boards, shields, peripherals)
• most test the model layer, while what the user interacts with is the controller (still reasonably covered), and what the user sees is the view

I expect we will improve this when reviewing this RFC, may be:

• identify a set of hardware configurations covering the most common use cases (from emulation to multi-core)
• identify a set of Zephyr samples covering the most common uses of these hardware (e.g. a sample with an I2C device, a sample with an SPI device, etc)
• with this matrix, test a scenario of successive DTSh commands with their expected outputs

Ideally, this should be automated.

Note

DTSh unit tests are currently not integrated with Zephyr CI.
Should I already try adding:

• a tox.ini file to DTSh Python building and packaging (for now tests are setup only for Hatch)
• a job entry in zephyr/.github/workflows/devicetree_checks.yml

Or is it just as well to wait until we have thought a bit more about what to test and how ?

Readline on MS Windows

The readline module of the Python standard library has a rather eventful history:

• AFAIK, it was first supported equally on all platforms, with (incidentally) the GNU Readline semantic
• at one point, Python distributions for macOS switched to libedit instead of libreadline to avoid the GPL license, and some users had to fix their configuration or workflows due to incompatibility issues
• starting with Python 3.7, distributions for MS Windows dropped the readline API, and users complained they lost TAB-completion in REPL
• both issues, the compatibility between the libedit and libreadline backends on POSIX-like systems, and the need for some readline support on MS Windows, are gradually being addressed by the Python community

In a nutshell, although the situation is improving, for now readline integration will very likely be disabled on MS Windows, resulting in a degraded user experience (neither auto-completion nor command history).

To get a fully cross-platform implementation, I've also considered the pure Python Prompt Toolkit:

• what I've been able to achieve was clearly disappointing compared to binding native GNU Readline hooks to DTSh display callbacks
• using both the Prompt Toolkit and Textualize's rich library was painful, and introduced other issues

My approach here is to follow changes in Python and incorporate any improvements into DTSh as they come.

Meanwhile, I'm quite optimistic that this will not be considered a merge blocker: Zephyr seems to admit this kind of little (though very unfortunate here) difference between POSIX and non POSIX systems, e.g. west itself also supports auto-completion only with POSIX shells.

Note

IIUC, Python 3.13 for MS Windows includes an almost complete readline module.

It seems the only relevant missing API is set_completion_display_matches_hook(), which DTSh uses for plugging in its own display of the completion candidates.
On MS windows, it could however fallback to the default display implementation provided by the readline module.

I can draft a patch if there is demand from Windows users with Python 3.13.

References:

• There's no readline module on Windows Python
• python's readline module not available for windows?
• Support GNU readline-like keybinds in the Windows REPL
• readline.set_history_length corrupts history files when used in a libedit build
• cmd.py command completion does not work with libedit
• Add readline.backend for the backend readline uses
• Python 3.13 readline — GNU readline interface

Security

The proposed change is not that hazardous:

• dtsh does not evaluate (eval()) any part of the user input
• dtsh does not pipe any part of the user input to the system interpreter (e.g. os.system())
• Devicetree source and binding files, CMake cache files, etc, are all opened read-only
• by default dtsh will not override user files, excepted when appending a redirected command output to an existing file
• the writable user specific files (command history, custom configuration and theme files) are written to (and read from) a per-user application data directory located under the appropriate platform-dependent root (e.g. $XDG_CONFIG_HOME or %LOCALAPPDATA%)
• the West command itself is rather simple

Handbook

The DTSh Handbook (handbook.rst) is quite long, and contains sections, subsections, sub-subsections, etc.

Its integration with Zephyr's documentation is almost fine in HTML.

Its integration when generating PDF is more debatable:

• since the Handbook's contents in the generated PDF document already starts at the "Second section heading level", proper styles quickly run out to highlight its own headings
• for the same reason, the TOC pane (typically shown at left in PDF viewers) will flatten the Handbook to a single heading level

The final PDF document is still legible, but not optimal: I will gladly try any ideas for improvement.

Personal biases

Why a command line application ? Well, when you've done west build, you're already at the command line, and I think continuing from there with west dtsh, getting a different prompt but the same user interface paradigms and even key bindings (the base ones used by Zsh, GNU Bash, Emacs, GDB), is more ditrsaction-free than opening a GUI.

Beyond this personal biases, things like PyGObject or Qt for Python have nonetheless been considered, but:

• these would be huge requirements to add to Zephyr, and to the users' workspaces
• I wanted something simple and fast, easy to extend and to prototype with
• at the end of the day, I'm not sure a full GUI will be more user-friendly or informative than what is possible with a CLI

Alternatives

There doesn't seem to be many DTS file viewers, and even fewer that support Zephyr's bindings.

Nordic Semiconductor's distributes the nRF DeviceTree extension for VS Code, which looks interesting but does not address the initial problem (as @hongshui3000 also pointed out):

• it's not Open Source, and seems to assume you're using the nRF Connect SDK, not Zephyr
• VS Code with the nRF Connect Extension Pack is not really a simple tool everyone is supposed to have on hand (or to work with)

[hongshui3000]

I think VS Code extension: nRF DeviceTree is a good graphical tool. But it's not open source. Not particularly friendly to non-nRF devices

[mbolivar-ampere]

Thanks for this. I think it's a great idea and I especially appreciate the attention to detail you put into the RFC as well as volunteering maintenance. Consider me onboard with this idea. I'll give it some testing and review.

[mbolivar-ampere]

Some unboxing comments:

Python provided by Homebrew or Fink: should also work OOTB

I can't get tab completion working as I expect using the homebrew python 3.11. From the prompt:

/
❭ 

I press TAB and expect commands to be completed. But instead I see a tab character is inserted. Is that expected? Here is my python info (using a virtual environment):

(devel) mbolivar@PDX-FY74HK3HFV zephyr % python -V
Python 3.11.4
(devel) mbolivar@PDX-FY74HK3HFV zephyr % echo $(readlink $(which python))
/opt/homebrew/opt/[email protected]/bin/python3.11

I can import readline from the python REPL with no error. I did a pip install -r zp/zephyr/scripts/dts/dtsh/requirements.txt before running west dtsh.

The command parser also doesn't seem to be stripping leading whitespace:

/
❭     q
dtsh: command not found: q

/
❭ q
bye.

this isn't like what you'd expect from a shell in my opinion -- I would expect both of those to quit.

[rruuaanng]

I'm looking forward to seeing this happen 💯

[kartben]

Hi @dottspina - would you be available to present this work at an upcoming Zephyr Architecture Working Group meeting? Given this adds quite a bit of functionality (in a positive way!) this is worth raising broader awareness about. Group meets every Tuesday at 5pm CET, and I think there would be room on the agenda on the 28th.

In the meantime, I wonder if @pdgendt or @mbolivar have feedback. One piece of feedback from my side is that this might need to be split across several PRs.

[pdgendt]

In the meantime, I wonder if @pdgendt or @mbolivar have feedback. One piece of feedback from my side is that this might need to be split across several PRs.

I didn't dive much in the code yet, mostly because of the sheer size TBH.
Wouldn't it also be opportune to have this as a module? It can evolve/be maintained separate from zephyr itself that way.

[dottspina]

Hi @dottspina - would you be available to present this work at an upcoming Zephyr Architecture Working Group meeting? Given this adds quite a bit of functionality (in a positive way!) this is worth raising broader awareness about. Group meets every Tuesday at 5pm CET, and I think there would be room on the agenda on the 28th.

Hi @kartben,

Yes, great, that might be helpful.

One piece of feedback from my side is that this might need to be split across several PRs.

I agree that the size of this PR is a concern (@pdgendt also seemed a little discouraged).

I tried to soften the effort required by organizing the changes into successive commits, each dedicated to a single task, and depending only on the previous ones. But, in the end, 36 commits proved no less overwhelming than 115 files ;-)

What kind of division into multiple PRs were you thinking of ?
One idea that comes to me would be to dedicate a PR to what I call the model layer: this could help to review all DTSh uses (and interpretations) of the edtlib API without being confused by the whole thing.

I'm willing to dig deeper, but it seems to me that we should first agree on the type of integration (Zephyr main tree or external module/project), as it means different PRs anyway (and may be a different review process).

Wouldn't it also be opportune to have this as a module? It can evolve/be maintained separate from zephyr itself that way.

Hi @pdgendt,

@decsny suggested a somewhat similar approach a few months ago, adding DTSh to Zephyr as an optional module [1].

I welcomed the idea, my only real concern was not figuring out how to install the missing Python requirements (rich, plus gnureadline on macOS) from there. This discussion ended here.

Before answering you, I reconsidered this type of approach, and I am still undecided.

An optional module integrates with the Zephyr build system, which could give us some hook to install the missing Python requirements: it seems a ugly hack to me, and I'm not even sure DTSh would qualify (make sense) as a module. Besides, users having to go through the west config manifest.project-filter dance is not really an out-of-the-box experience.

External tooling, "software that assists the compilation, testing or simulation processes but in no case ends up being part of the code compiled and linked into the final image" [2] sounds interesting.

IIUC, we then have three options:

1. the scripts/ folder in Zephyr main tree, e.g. scripts/dts/dtsh as in the current PR
2. a West project hosted within the zephyrproject-rtos GitHub organization
3. an optional West project hosted outside the zephyrproject-rtos GitHub organization

The second option seems to me a good compromise to have DTSh both available OOTB, and maintained separate from zephyr itself.

If we choose this approach, my only remaining concerns will be:

• I still can't see how to install the rich and gnureadline Python dependencies from an external West project; I may be missing something obvious, any ideas are welcome
• less worrying, but still an issue for which I don't know a clean solution: adding the edtlib library to the Python system path with some (convoluted) sys.path.append() is okish when running the West command itself, but hardly scale to an external project that e.g. should be able to run its unit tests (what CI path would go from e.g. zephyrproject-rtos/dtsh/tests/ to zephyrproject-rtos/zephyr/scripts/dts/python-devicetree/src/ ?)

To summarize my thoughts:

• I understand the size of the PR may be inappropriate for an integration into Zephyr main tree
• I'm fine with the idea of developing and maintaining DTSh as an external West project, preferably within the zephyrproject-rtos GitHub organization to make the tool available out-of-the-box (not optional)

If we choose this option, we must make it clear as soon as possible: this will lead to different PRs and may imply some infrastructure/organizational setup. I will also really appreciate any help in working through the above concerns.

As I understand it, feature freeze for zephyr 4.1 is planned for mid-February: I'll do my best to respond to any feedback promptly.

Thanks.

[1] https://docs.zephyrproject.org/latest/contribute/external.html#integration-as-optional-modules
[2] https://docs.zephyrproject.org/latest/contribute/external.html#contributing-external-tooling

[pdgendt]

There's already a west packages extension that Zephyr modules can use to list python dependencies!