script/dts: Generate Extended Device Tree database #9876

erwango · 2018-09-10T12:26:35Z

This PR is subset of #6762 (Codegen introduction, proposed by @b0661), focused on edts database generation.

Extended Device Tree database is a json file generated by extract_dts_includes.py script, from board device tree (dts) collated with yaml bindings.
Since it is post processed from dtc, information is displayed in an easily accessible 'dict' format:

 _edts dict(
   'devices':  dict(device-id :  device-struct),
   'compatibles':  dict(compatible : sorted list(device-id)),
   'device-types': dict(device-types : sorted list(compatible)),
   ...
 )

 device-struct dict(
   'device-id' : device-id,
   'compatible' : list(compatible) or compatible,
   'label' : label,
   property-name : property-value ...
 )

 device-id: opaque id for a device (do not use for other purposes),
 compatible: any of ['st,stm32-spi-fifo', ...] - 'compatible' from <binding>.yaml
 label: any of ['UART_0', 'SPI_11', ...] - label directive from DTS
 device-type: any of zephyr generic type: usb, spi, i2c, ...

Edit 09/11: Added device-type injection

codecov-io · 2018-09-10T13:01:59Z

Codecov Report

Merging #9876 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #9876   +/-   ##
=======================================
  Coverage   53.29%   53.29%           
=======================================
  Files         215      215           
  Lines       25843    25843           
  Branches     5694     5694           
=======================================
  Hits        13773    13773           
  Misses       9754     9754           
  Partials     2316     2316

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fad97ef...88155ce. Read the comment docs.

scripts/dts/edtsdatabase.py

scripts/dts/extract/gpios.py

b0661 · 2018-09-11T20:57:58Z

scripts/dts/edtsdatabase.py

+    # @param none
+    # @return list of supported device types
+    def get_device_types(self):
+        return list(self._edts['device_types'])


'device_types' -> 'device-types'

b0661 · 2018-09-11T21:02:42Z

scripts/dts/edtsdatabase.py

+            self._edts['device-types'][device_type].append(device_id)
+            self._edts['device-types'][device_type].sort()
+
+    def insert_device_type(self, compatible, device_type):


Remove - already accomplished by insert.

I've chosen a different device-type dict as the one you proposed.
I've chosen to display: dict(device-type : sorted list(compatible)) instead of a sorted list of device_id.
Aim was to make it easier to access to get device-type information per compatible (for DTS_ #define's generation), as well as device-type list for CI.
So I need to inject (device-type, compatible) couple, which I can't using insert.

b0661 · 2018-09-11T21:03:46Z

scripts/dts/extract/compatible.py

+            if not isinstance(device_types, list):
+                device_types = [device_types, ]
+            for j, dev_type in enumerate(device_types):
+                edts_insert_device_type(comp, dev_type)


Remove edts_insert_device_type - see above.

After thinking a bit - device type generation should be moved to heuristics.py.

See reasonning behind this choice above

My last answer was response to first comment.

After thinking a bit - device type generation should be moved to heuristics.py.

Yes, I hesitated indeed. Moved

b0661 · 2018-09-11T21:05:40Z

scripts/dts/extract/edts.py

@@ -23,6 +23,9 @@ def edts_insert_device_property(node_address, property_path, property_value):
    device_id = edts_device_id(node_address)
    edts.insert_device_property(device_id, property_path, property_value)

+def edts_insert_device_type(compatible, device_type):


Remove - use edts_insert_device_property with 'device-type/{}'.format(device_type) instead.

I'm already using edts_insert_device_property with 'device-type/{}'.format(device_type)
in heuristics class.
Aim of edts_insert_device_type is not to fill in device-type sub-node property but dedicated device-type dict. This new dict should be used at least for CI and potentially other purposes related to #9406.

SebastianBoe · 2018-09-12T10:55:21Z

scripts/dts/edtsdatabase.py

+        for compatible in compatibles:
+            for device_id in self._edts['compatibles'].get(compatible, []):
+                device_ids[device_id] = 1
+        return list(device_ids.keys())


We need to re-use the code between this function and the previous.

E.g. by having a private function that returns a dict. And then have each of these two call that function and extract either the keys or the values.

Done, thanks

galak · 2018-09-13T14:40:21Z

Can you update scripts/dts/edtsdatabase.py to do something similar to scripts/dts/devicetree.py w/regards to main, and being able to run it and have it print out the database.

Validate generated etds.json against a reference edts file. Script first checks edts structure then content of various elements and finally devices parameters values. This last step is done using a specific method for ref_edts parameter extraction so we can verify behavior of edtsdatabase class extraction function. Signed-off-by: Erwan Gouriou <[email protected]>

The Extended Device Tree Specification database collates device tree (dts) information with information taken from the device tree bindings. The EDTS database may be loaded from a json file, stored to a json file or extracted from the DTS files and the bindings yaml files. The database is integrated into cogeno as a module. The commit integrates database development done in zephyrproject-rtos#9876 which was based on zephyrproject-rtos#6762. Major differences/ improvements to zephyrproject-rtos#9876 are: - the database now has an own extraction function that can be used instead of e.g. extract_dts_includes. The extraction function follows the design of the extract_dts_includes script and the additions that were done in zephyrproject-rtos#9876. It is restructured and several globals are now classes and objects. All functionality of extract_dts_includes related to the generation of defines is not part of the database extract function. It's sole purpose is to fill the database directly from the compiled DTS file. - the database got itś own directory 'edtsdb' to structure all files related to the database. - The EDTSDevice class from zephyrproject-rtos#9876 was enhanced to allow devices to access the database they are taken from. Mayor compatibility issues to zephyrproject-rtos#9876. - The consumer, provider API and the internal structure of the database is copied from zephyrproject-rtos#9876. - API should be fully compatible. - Extraction of children was replaced as it broke the concept of the devices struct as a list of devices. The functions device.get_children() and device.get_parent() may be used to acess the parent<->child relation. - The EDTSDevice class is copied from zephyrproject-rtos#9876. - The device API should be compatible except for - the constructor which now needs the EDTS database and - the unique id feature. To ge an unique id the device.get_name() function can be used instead. Signed off from zephyrproject-rtos#9876 added to attribute for the changes done there and copied. Signed-off-by: Erwan Gouriou <[email protected]> Signed-off-by: Kumar Gala <[email protected]> Signed-off-by: Bobby Noelte <[email protected]>

galak · 2019-07-26T15:29:56Z

Closing, as this should be superceded by PR #17660

erwango added the area: Devicetree label Sep 10, 2018

erwango requested review from tbursztyka, b0661, carlescufi, galak and agross-oss September 10, 2018 12:26

erwango requested review from nashif and SebastianBoe as code owners September 10, 2018 12:26

erwango mentioned this pull request Sep 10, 2018

scripts: add Zephyr inline code generation with Python #6762

Closed

erwango force-pushed the push_extract_edts branch 2 times, most recently from 3311a99 to 7f61c4c Compare September 10, 2018 15:34

erwango added DNM This PR should not be merged (Do Not Merge) RFC Request For Comments: want input from the community labels Sep 11, 2018

erwango force-pushed the push_extract_edts branch 3 times, most recently from 372afa8 to 9c02e67 Compare September 11, 2018 11:38

This was referenced Sep 11, 2018

Device Tree support overhaul #8499

Closed

Generate DTS 'compatible' based compilation flags #9406

Closed

b0661 suggested changes Sep 11, 2018

View reviewed changes

erwango force-pushed the push_extract_edts branch 3 times, most recently from 4e3fe7a to 40434cf Compare September 12, 2018 09:59

SebastianBoe reviewed Sep 12, 2018

View reviewed changes

erwango force-pushed the push_extract_edts branch 4 times, most recently from 7c07936 to 0090121 Compare September 13, 2018 09:43

erwango force-pushed the push_extract_edts branch from 0090121 to 3fe947d Compare September 13, 2018 15:40

galak force-pushed the push_extract_edts branch from 3b0c866 to 88155ce Compare October 25, 2018 09:29

b0661 mentioned this pull request Oct 26, 2018

scripts: inline code generation with cogeno (and edts) #10885

Closed

erwango mentioned this pull request Oct 29, 2018

Requirements for driver devices generation using device tree #10904

Closed

erwango mentioned this pull request Nov 12, 2018

scripts: dts: Use DT_ prefix in generated labels #11180

Merged

pabigot mentioned this pull request Dec 4, 2018

[RFC]: moving to eliminate dts_fixup files #11737

Closed

galak closed this Jul 26, 2019

erwango deleted the push_extract_edts branch September 3, 2019 12:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

script/dts: Generate Extended Device Tree database #9876

script/dts: Generate Extended Device Tree database #9876

erwango commented Sep 10, 2018 •

edited

Loading

codecov-io commented Sep 10, 2018 •

edited

Loading

b0661 Sep 11, 2018

b0661 Sep 11, 2018

erwango Sep 12, 2018

b0661 Sep 11, 2018 •

edited

Loading

b0661 Sep 12, 2018

erwango Sep 12, 2018

erwango Sep 12, 2018

b0661 Sep 11, 2018

erwango Sep 25, 2018

SebastianBoe Sep 12, 2018

erwango Sep 12, 2018

galak commented Sep 13, 2018

galak commented Jul 26, 2019

script/dts: Generate Extended Device Tree database #9876

script/dts: Generate Extended Device Tree database #9876

Conversation

erwango commented Sep 10, 2018 • edited Loading

codecov-io commented Sep 10, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

b0661 Sep 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

galak commented Sep 13, 2018

galak commented Jul 26, 2019

erwango commented Sep 10, 2018 •

edited

Loading

codecov-io commented Sep 10, 2018 •

edited

Loading

b0661 Sep 11, 2018 •

edited

Loading