*.cbuild-index.yml + *.cbuild.yml as long term replacement for *.CPRJ + Proposal for use as lockfile + Generator input

[ReinhardKeil]

This proposal is to replace (over time) the *.CPRJ file(s) with a single *.cbuild.yml file that:

• contains the build instructions for the whole application. A new cbuild tool could selectively generate one project or all projects.
• uses controls that are identical with csolution.yml, but no build/target types (and the related for-type:)
• compiler is fixed for a project (therefore no for-compiler:)
• component files are explicitly stated (which removes the need to manage components at cbuild tool-level)
• additional information (such as from-pack:) is stated for each component
• clayer.yml/cdefault.yml information is included (no reference to other files - all self-contained in *.cbuild.yml file)
• lists all packs: with explicit version number (this can be used as loop-back file to csolution tool to lock the pack versions)

The file would be generated by the csolution tool.

Example:

# section that defines global setup for all projects (initally only packs: would be listed here)

  packs:  # all packs used in the projects (exception when a project uses a different pack version the packs: would be also under project)
    - pack: ARM::[email protected]
    - pack: ARM::[email protected]
    - pack: ARM::[email protected]
    - pack: ARM::[email protected]
    - pack: AWS::[email protected]
    - pack: AWS::[email protected]
    - pack: AWS::[email protected]
    - pack: AWS::[email protected]
    - pack: AWS::[email protected]
    - pack: AWS::[email protected]
    - pack: AWS::[email protected]
    - pack: Arm-Packs::[email protected]
    - pack: Keil::[email protected]
    - pack: MDK-Packs::[email protected]
    - pack: MDK-Packs::[email protected]
    - pack: ARM::[email protected]    # DFP and BSP packs might need some special handling
    - pack: Keil::[email protected]
    - pack: Keil::[email protected]

#project specific section; almost identical with cproject, except all types are explicit, components are expanded 

  project:  MQTT_MutualAuth_Demo.Debug+AVH_MPS4_Corstone-300
    compiler: [email protected]   

    board: B-U585I-IOT02A

    device: SSE-300-MPS3     # assuming that all projects use the same device

    add-paths:
      - ./config_files
      - ./amazon-freertos/demos/include
      - ./amazon-freertos/demos/network_manager
  
    defines:
      - CONFIG_CORE_MQTT_MUTUAL_AUTH_DEMO_ENABLED

    components:
      - component: ARM::CMSIS:RTOS2:FreeRTOS&Cortex-M
        from-pack: ARM::[email protected]
        files:
          - file: %CMSIS_PACK_ROOT%/ARM/CMSIS-FreeRTOS/10.4.6/CMSIS/RTOS2/FreeRTOS/Source/ARM/clib_arm.c
          - file: %CMSIS_PACK_ROOT%/ARM/CMSIS-FreeRTOS/10.4.6/CMSIS/RTOS2/FreeRTOS/Source/cmsis_os2.c
          - file: %CMSIS_PACK_ROOT%/ARM/CMSIS-FreeRTOS/10.4.6/CMSIS/RTOS2/FreeRTOS/Source/os_systick.c
      - component: ARM::RTOS&FreeRTOS:Config&CMSIS RTOS2
        from-pack: ARM::[email protected]
        files:
          - file: ./RTE/RTOS/FreeRTOSConfig.h
            config-version: 1.2.3                   # identifies local config files along with version number

      - component: ARM::Security:mbed TLS
        from-pack: ARM::[email protected]
        defines:
          - MBEDTLS_CONFIG_FILE="aws_mbedtls_config.h"
        files:
          ..

    groups:
      - group: Documentation
        files: 
          - file: ./README.md
      - group: main
        files:
          - file: ./app_main.c
          - file: ./tfm_ns_interface_freertos.c

  project: # next project

Note: this example is inspired by https://github.com/MDK-Packs/TrustZone

There are for sure things to be discussed such as:

• why a single file for all build/target-types: this makes it easier to use it as loop back file. Selection of projects to build could be at cbuild tool-level instead.
• should we add options to build just a selection of build/target-types: perhaps - not sure about this
• as this file is generated by the csolution tool, a back-up of the *.cbuild.yml could be used to identify the effect of changes in the csolution input file(s).

[slhultgren]

To connect the dots with #119 I'll try to give some feedback.

cbuild.yml and lockfile usage

• I think I agree that it might be a good thing to use a format closer to the csolution/cproject format, although the extreme case that was raised in 119 is that: If we include everything about build in this file, why not instead just generate a CMakeLists.txt directly that relies on $CMSIS_PACK_ROOT variables and relative paths, where important information is stored in CMake-variables that could be parsed and used by csolution tooling as loop back? Maybe we should consider this as well, effectively it would even eliminate the need for another tool.
• The packs list I suppose also can point to local packs with relative paths as in the csolution.yml today?
• For components and files I think we should also keep the conditionid stored since this helps with uniquely identifying which component of the PDSC was used. Can be helpful when creating packs to debug conditions, and during review to see which parts of the project actually changes. I would propose to extend the ComponentID format with a # separator for the conditionid part, basically extend current to
  • [Cvendor::] Cclass [&Cbundle] :Cgroup [:Csub] [&Cvariant] [@[~ | >=]Cversion] [#conditionid]
  • For files, the condition property could be added
• constructed files should somehow be visible here as well, (RTE_Components.h, RTE_PreInclude.h etc)
• I think we should also show the origin ComponentID reference so that it is trivial to see that e.g. CMSIS:RTOS2 from the YML resolved to ARM::CMSIS:RTOS2:FreeRTOS&Cortex-M

Generator input file usage

The format in this issue is also a good fit for the generator input file to replace the current *.generate.yml file if we do the following changes when using it for the generator input file specifically:

• Only a single project context would be listed
• All Packs should point to absolute paths, no CMSIS_PACK_ROOT lookup should be necessary
• Each component should also have the generator kept from the PDSC definition so that the generator can identify its' own components
• Files should keep all information from PDSC, so attr, category, path in pack, and RTE-Path. (Only conditionally true files should be included)

[ReinhardKeil]

@slhultgren I agree with most of your suggestions.

There are aspects that I suggest to discuss:

• CMSIS_PACK_ROOT is a simple environment variable. It should be fairly easy for tools to look it up. The reason why to use it: the build process might be on a different computer (i.e. CI in the cloud) and the CMSIS_PACK_ROOT is most likely different on each system. Also the explicit CMSIS_PACK_ROOT may contain user name information of a local computer which could cause a privacy issue.
• CMakeList.txt instead of YML seems somewhat possible, but the file could be complex and the CMake format is to my knowledge not a standard data format (would therefore require bespoke reader implementations).

The rest seams to be trivial, but I would suggest to use a separate key/value pair for "condition id" as it could be also applied to files.

I agree with the usage as Generator Input file, but don't fully understand why multiple project context would be an issue.

Q: The packs list I suppose also can point to local packs with relative paths as in the csolution.yml today?
A: Yes the path key would be preserved for local packs.

So it could be:

       - component: ARM::CMSIS:CORE    # .\Board\AVH_MPS3_Corstone-300\Board.clayer.yml: component: CMSIS:CORE
         from-pack: ARM::[email protected]
         condition: ARMv6_7_8-M Device 
         files: 
           - file: CMSIS/Core/Include/tz_context.h
             category: header
             condition: TrustZone

If we are all aligned, we should agree on a timeline and the steps of implementation along with ownership. Obviously the first step is a complete specification, followed by the implementation of the output file by the csolution tool (both owned by Arm). This could enable then already the generators flow (should owned by ST or NXP). The rest of the steps seem less time critical to me, but could be: implement the lockfile feature in csolution, implement a new cbuild tool based on cbuild.yml.

[jkrech]

Regarding CMSIS_PACK_ROOT my suggestion is to include a line in the lockfile somewhere at the beginning reporting the value of the variable at the time the file got processed. So at least the information is preserved and could become useful for tools (e.g. generators) to use the same CMSIS_PACK_ROOT value. I am not sure yet whether to make the storing optional due to your privacy concerns. How real are these?

[ReinhardKeil]

We discussed that in technical review meeting on 30. Aug.

• we will use *.yml format and not implement CMakeLists.txt output
• we will use %CMSIS_PACK_ROOT% as an abstract path to the global pack directory of the host computer, as otherwise two different users would generate a different file (see also Remove explicit time-stamps from generated files #449 where a similar issue exists with timestamp).

Problems with a generator will be reviewed once we have the first version of the file available.

[ReinhardKeil]

This is now reflected in https://github.com/Open-CMSIS-Pack/devtools/blob/main/tools/projmgr/docs/Manual/YML-CBuild-Format.md

I'm sure there are some problems, but would apricate if this could be raised in separate tickets.

[jkrech]

We are closing this issue now and plan to develop the lockfile writer based on the documentation here: https://github.com/Open-CMSIS-Pack/devtools/blob/main/tools/projmgr/docs/Manual/YML-CBuild-Format.md

[brondani]

Feedback concerning the current proposed format of the cbuild.yml file:

• The projects/project keywords refer actually to project contexts.
  Isn't it inconsistent to reuse these keywords since they are already used in csolution.yml?
• Each project context may require different packs, but in cbuild.yml there is only a common list of packs valid for the whole solution.

[ReinhardKeil]

The projects/project keywords refer actually to project contexts. Isn't it inconsistent to reuse these keywords since they are already used in csolution.yml?

Agreed, we should rename projects/project to contexts/context.

Each project context may require different packs, but in cbuild.yml there is only a common list of packs valid for the whole solution.

Under build: a list of common packs is presented (packs that can be used across the csolution). However, each context may have a separate pack list.

[jkrech]

Let's keep it open for capturing additional feedback. Once we have a PR we should move discussions there.

[slhultgren]

A few more comments on the current proposed format

• selected-by: MyProject.cproject.yml#55 A reference is a good thing, but line numbers are a bit too detailed I think and will likely be out-of-date/needing update a lot more frequently than keeping only the component reference i.e. CMSIS:CORE here, e.g.

    components:
    - component: ARM::CMSIS:RTOS2:FreeRTOS&Cortex-M
      from-pack: ARM::[email protected]
      selected-by: CMSIS:RTOS2:FreeRTOS   # What was written in the component list in cproject.yml

  so I would vote for removing the optional "file#num" part
• output-dirs should have the rtedir and gendir properties as well I believe
• local-repository if this is absolute path, then the .cbuild.yml will not easily be shareable. So would the recommendation be here that if you want to share your cbuild.yml, don't use local_repository.pidx?
• %CMSIS_PACK_ROOT% usage may be misleading for project local packs (not talking about local_repository.pidx). This seems a bit more generic:

component-files:
      - file: %PACK_LOCATION%/CMSIS/RTOS2/FreeRTOS/Source/ARM/clib_arm.c

Or we could allow both forms of strings

Generator input file version of cbuild.yml

It would be very nice if the following properties of the cbuild.yml would always be expanded to absolute paths:

• All packs in the pack list would have the path property set with absolute paths, both for local packs and for "installed" packs.
• All files use resolved values for %PACK_LOCATION% / %CMSIS_PACK_ROOT%
• path to cproject file should be there as well
• path to csolution file should be there as well
• destination property should be there (from output-dirs[gendir] from current context, so could be skipped if the full output-dirs is provided in the context already)
• the generator input file might not include all contexts (maybe only one at a time?)

[ReinhardKeil]

@slhultgren @brondani thanks for your feedback, I have incorporated some in #481

• selected-by: simplified as proposed
• added: csolution: node to reflect the project structure
• changed: project(s): -> context(s):
• added a remark that cbuild files with local-repository: might be not portable.

I will address the gendir: and rtedir: issue once it is clearer how we structure complex projects. There is a proposal regarding rte-path: that somewhat conflicts with rtedir:.

Some remarks about the other proposals:

1. %CMSIS_PACK_ROOT%`

%CMSIS_PACK_ROOT% is established and typically an environment variable in the system that points to the central pack directory. Shell scripts replace therefore this to the final directory path (full path). For example:

ECHO %CMSIS_PACK_ROOT%/ARM/CMSIS-FreeRTOS/10.4.6/CMSIS/RTOS2/FreeRTOS/Source/os_systick.c
outputs:
C:\Users\xxx\AppData\Local\Arm\Packs/ARM/CMSIS-FreeRTOS/10.4.6/CMSIS/RTOS2/FreeRTOS/Source/os_systick.c

which is the final absolute path in my system.

2. relative vs. absolute paths

Imagine that two people work together on the same project, one person responsible for software, another for hardware. The hardware person has not software tools installed (therefore he cannot recreate the *.cbuild.yml), but uses the generator to change device pinouts (to reflect his hardware design).

• The hardware person uses C:\work as his work directory
• The software person (is lazy like me) and just uses C:\w as his work directory.

Both work on the same repo.

I therefore recommend to use relative paths (relative to the ./*.cbuild.yml) file that start with ./

NOTE: similar problems arise when in the long run a web based and a desktop based tool is offered. The web based tool may use pathnames of the repo, whereas the desktop installation uses slightly different paths.

3. Visibility of generator to all project context

By providing a complete view to the whole solution, the generator may issue notifications to the user that the setup that is changed applies to other context setups. It should be fairly simple for the generator to select a specific data set that belongs to a context - could you check if this makes really a big difference in the implementation?

[jkrech]

We plan to have a first iteration of the *.cbuild.yml files in csolution 1.2.0 currently in development.

[brondani]

cbuild output implementation progress:

Feature	Status	Version
cbuild-idx.yml file generation	✔️	csolution 1.2.0
cbuild-idx.yml cdefault, csolution, cprojects, clayers and cbuilds nodes	✔️	csolution 1.2.0
cbuild-idx.yml licenses node	❌
cbuild.yml file generation for each context	✔️	csolution 1.2.0
cbuild.yml compiler, board, device, processor, packs, add-paths, defines, misc, optimize, debug, warnings, components, groups	✔️	csolution 1.2.0
cbuild.yml processing of del-paths and undefines at component, group, file levels	❌
cbuild.yml licenses node	❌
cbuild.yml messages node	❌
cbuild.yml constructed-files node	✔️	csolution 1.3.0
cbuild.yml output-dirs node	✔️	csolution 1.3.0
cbuild.yml component files attr	✔️	csolution 1.3.0
cbuild.yml pre-includes	✔️	csolution 1.3.0
cbuild.yml file category	✔️	csolution 1.3.0
cbuild.yml config files versions	✔️	csolution 1.3.0
cbuild.yml keep generatorID with components	✔️	csolution 1.3.0
cbuild.yml use Unix style ${CMSIS_PACK_ROOT} instead of access sequence	✔️	csolution 1.3.0
generators references (TBD)	❌
PLM files references, relative path to original file - equivalent to PLM base file? (TBD)	❌

[slhultgren]

@brondani for the last two me and @ErikLundinSTM have been doing a bit of work, maybe we can have a PR going next week

[slhultgren]

Generator usage with cbuild.yml

Our thinking is that for generated components (also known as components with the generator attribute), we would add a new section to the component in the cbuild.yml.
Lets say we have a component that uses a custom generator that only renames source files and places them into the generated folder, we imagine it should look like this. The usable path is still in the components > files section, but the generator input files are provided in another list generator-input-files

    - component: MyVendor::Component:WithGenerator
      files:
        - file: generated/source_file_renamed_by_generator_to_fit_project.c
          category: source
      generator-input-files:
        - file: ${CMSIS_PACK_ROOT}/MyVendor/CustomPack/1.0.0/source_file.c
          category: source
      generator: MyCustomGeneratorRenamer

By doing it this way we say that the component > files section is the usable section that we should use for building the project (So here it is coming from the gpdsc).

The component > generator-input-files section would be the list of files mentioned in the pdsc for the "generator component".

Also by doing it this way we should be able to use the cbuild.yml as an input file to the generator itself, since the generator can rely on the component > generator-input-files

About PLM file references

We are moving in the direction of adding a origin: property on each file (that has attr=config or attr=template) which would point into the pack as relative path.
E.g.

    - component: ARM::Device:[email protected]
      condition: ARMCM7 CMSIS
      from-pack: ARM::[email protected]
      selected-by: ARM::Device:Startup
      files:
        - file: RTE/Device/ARMCM7/gcc_arm.ld
          category: linkerScript
          attr: config
          version: 2.1.0
          origin: ${CMSIS_PACK_ROOT}/ARM/CMSIS/5.9.0/Device/ARM/ARMCM7/Source/GCC/gcc_arm.ld

[edriouk]

generator-input-files: such files can also be indicated by a dedicated file attribute, e.g.

  - file: generated/source_file_renamed_by_generator_to_fit_project.c
          category: source
          generator: output
  - file: ${CMSIS_PACK_ROOT}/MyVendor/CustomPack/1.0.0/source_file.c
          category: source
          generator: input

Besides, it might be useful to supply this information in pdsc file as well

[brondani]

@slhultgren

1. I believe there are more info missing for running a generator, for example the working directory and the gpdsc path and filename as well as command and arguments - see the spec.
   Maybe all generator related info should be placed as child nodes of a generator node?

2. Concerning PLM, referencing the origin (actually base in the concept) from the original pack is not correct because it may not even be installed. The base and update files come always alongside the current file, so they can be implicitly found. If you want to reference them explicitly I would suggest to add the following nodes instead of origin in your example:

base: RTE/Device/ARMCM7/[email protected]
update: RTE/Device/ARMCM7/[email protected]

[slhultgren]

After discussing with @brondani and @ErikLundinSTM this is an updated proposal

    - component: MyVendor::Component:WithGenerator
      files:
        - file: generated/source_file_renamed_by_generator_to_fit_project.c
          category: source
      generator:  # The generator node is defined once in the context as a base, then each component can override/extend properties
        - id: MyCustomGeneratorRenamer  # id[#Gvendor][::Gtool][@Gversion]
          working-dir: project/relative/folder/or/from/pack/workingDir
          command:
             - linux: ${CMSIS_PACK_ROOT}/doit.sh $D (but resolved to actual value)  $G (pointing to this file)
               windows: ${CMSIS_PACK_ROOT}/doit.sh $D (but resolved to actual value)  $G (pointing to this file)
               mac: ${CMSIS_PACK_ROOT}/doit.sh $D (but resolved to actual value)  $G (pointing to this file)
               all: ${CMSIS_PACK_ROOT}/doit.sh $D (but resolved to actual value)  $G (pointing to this file)
          gpdsc: ../relative/path/to/gpdsc/that/generator/creates.gpdsc
          files:
            - file: ${CMSIS_PACK_ROOT}/MyVendor/CustomPack/1.0.0/source_file.c
              category: source

[jkrech]

This continues into CMSIS-Toolbox 2.1.0 to complete cbuild.yml format.

[jkrech]

Move to at least 2.5.0 in order to keep cbuildgen build flow as fall back.

[brondani]

The completeness of *.cbuild.yml content for build purposes as a full replacement for CPRJ is being validated as part of the cbuild2cmake test strategy and CMSIS-Toolbox E2E integration tests
Open-CMSIS-Pack/cmsis-toolbox#101
#1508