dts: Add new DTS/binding parser

[ulfalizer]

dts: Add new DTS/binding parser

Add a new DTS/binding parser to scripts/dts/ for generating
generated_dts_board.conf and generated_dts_board_unfixed.h.

The old code is kept to generate some deprecated defines, using the
--deprecated-only flag. It will be removed later.

The new parser is implemented in three files in scripts/dts/:

dtlib.py:
  A low-level .dts parsing library. This is similar to devicetree.py in
  the old code, but is a general robust DTS parser that doesn't rely on
  preprocessing.

edtlib.py (e for extended):
  A library built on top of dtlib.py that brings together data from DTS
  files and bindings and creates Device instances with all the data for
  a device.

gen_defines.py:
  A script that uses edtlib.py to generate generated_dts_board.conf and
  generated_dts_board_unfixed.h. Corresponds to extract_dts_includes.py
  and the files in extract/ in the old code.

  Note that dtlib.py and edtlib.py are general libraries (similar to
  Kconfiglib) that could be used for other stuff besides gen_defines.py.

testdtlib.py:
  Test suite for dtlib.py. Can be run directly as a script.

testedtlib.py (uses test.dts and test-bindings/):
  Test suite for edtlib.py. Can be run directly as a script.

The test suites will be run automatically in CI.

The new code turns some things that were warnings (or not checked) in
the old code into errors, like missing properties specified with
'category: required' in the binding for the node.

The code includes lots of documentation and tries to give helpful error
messages instead of Python errors.

Co-authored-by: Kumar Gala <[email protected]>
Signed-off-by: Ulf Magnusson <[email protected]>

[zephyrbot]

Found the following issues, please fix and resubmit:

License issues

In most cases you do not need to do anything here, especially if the files
reported below are going into ext/ and if license was approved for inclusion
into ext/ already. Fix any missing license/copyright issues. The license
exception if a JFYI for the maintainers and can be overriden when merging the
pull request.

• scripts/dts/dtlib.py is not apache-2.0 licensed: bsd-new
• scripts/dts/edtlib.py is not apache-2.0 licensed: bsd-new
• scripts/dts/gen_defines.py is not apache-2.0 licensed: bsd-new
• scripts/dts/test.dts is not apache-2.0 licensed: bsd-new
• scripts/dts/testdtlib.py is not apache-2.0 licensed: bsd-new
• scripts/dts/testedtlib.py is not apache-2.0 licensed: bsd-new

[galak]

is this missing prefix?

[ulfalizer]

Oops... fixed

[erwango]

Can't build nucleo_f401re board (sample hello_world), due to missing symbol CONFIG_RAM_SIZE
It is defined in arch/Kconfig:

config SRAM_SIZE
	int "SRAM Size in kB"
	default $(dt_int_val,DT_SRAM_SIZE)

DT_SRAM_SIZE is not defined anymore

EDIT: Seems it has been replaced by:

#define DT_SRAM_SRAM_SIZE                           96

Cf fix in scripts/dts/gen_defines.py, line 318:

-    out("{}_SRAM_SIZE".format(prefix), dev.regs[0].size//1024)
+    out("{}_SIZE".format(prefix), dev.regs[0].size//1024)

[ulfalizer]

Can't build nucleo_f401re board (sample hello_world), due to missing symbol CONFIG_RAM_SIZE
It is defined in arch/Kconfig:

config SRAM_SIZE
	int "SRAM Size in kB"
	default $(dt_int_val,DT_SRAM_SIZE)

DT_SRAM_SIZE is not defined anymore

Fixed by the latest force push. Thought I could make a small change without diffing generated_dts_board.conf before and after, but I couldn't. :P

[erwango]

(Slack is broken to me so pasting my point here)
Would be nice to introduce nodes sorting at some step.
From experience, this was useful in previous script when trying to debug and comparing 2 versions of generated output

[ulfalizer]

@erwango
Since Slack is dead:

The order in the conf and header file matches the order in the .dts file now, so it should be consistent at least. Would be pretty easy to change though, if you have some case where that order is bad.

[erwango]

The order in the conf and header file matches the order in the .dts file now, so it should be consistent at least. Would be pretty easy to change though, if you have some case where that order is bad.

This is not that I see, actually. For instance nucleo_f401re board, gpio nodes order:
nucleo_f401re.dts_compiled: A, B, C, D, E, H
generated_dts_board_unfixed.h: A, E, B, D, C, H (Changing at each build)

[ulfalizer]

@erwango
Try this patch and see if it fixes it. set() iteration order can vary between runs in Python.

diff --git a/scripts/dts/gen_defines.py b/scripts/dts/gen_defines.py
index 720c297615..9d8c43a9c7 100755
--- a/scripts/dts/gen_defines.py
+++ b/scripts/dts/gen_defines.py
@@ -72,7 +72,7 @@ def main():
             active_compats.update(dev.compats)
 
     out_comment("Active compatibles (mentioned in DTS + binding found)")
-    for compat in active_compats:
+    for compat in sorted(active_compats):
         #define DT_COMPAT_<COMPAT> 1
         out("COMPAT_{}".format(str2ident(compat)), 1)

[ulfalizer]

@erwango
What's your python3 --version btw? Dictionary iteration order didn't match insertion order before Python 3.6.

[erwango]

$ python3 --version
Python 3.5.2

Tried your proposal, but it didn't work (didn't help sorting devices). But as I understand, this aims at sorting compatibles and not devices, so this is logical.

[ulfalizer]

@erwango
Yeah, the device-is-there flags use a set(), so they can move around between runs even on Python 3.6+, unless you sort them somehow.

The rest of it is consistent for me, so I'm guessing it's because you have Python 3.5.

Could you post a diff between runs? Would help with figuring out where it's coming from.

Issue fixed