Proposal: CPRJ Extensions

[brondani]

The CPRJ format shall be extended to cope with several add-ons introduced within the csolution concept:​

• Description of inter-context dependencies. All related CPRJs and their inter-dependencies should lead to the generation of a top-level CMakeLists.txt file. To keep the separation of concerns between Project Manager and Build Manager this should be described in the intermediate format.​
• Compiler options abstractions​
• Defines and Includes at group/file level​
• Pre/post-build steps (TBD)

Proposal: Multi CPRJ collection = MCPRJ​

<mcprj ...>​
  <contexts>​
    <context>​
      <path>./parentProject.buildType+targetType.cprj</path>​
    </context>​
    <context>​
      <path>./project.buildType+targetType.cprj</path>​
      <depends>parentProject.buildType+targetType</depends>​
    </context> ​
  </contexts>​
</mcprj>​

CPRJ schemas extension proposal: #170
​
After hearing @madchutney concerns I believe a good complementary information to add to this proposal could be the explicit back reference from a cprj to its parent mcprj. In such case the Build Manager can unequivocally recognize whether the cprj is part of a multi-project solution. For example:

<cprj ...>
  <info ...>
    <mcprj>path/to/.mcprj</mcprj>
  </info>
</cprj>

[madchutney]

I have been thinking about the original use case for the cprj file, which was to provide reusable build instructions (perhaps best thought of as a lock file) for situations such as running regression tests in CI. Part of the concept was that if the build succeeded once, it should always succeed and result in the same binary being generated.

The original solution was well designed for this use case and it has also been particularly suitable as an interface between services and tools. There is a single file that defines the build and that is the only data (a link tor path to the file) that needs to be provided. This means the build manager could be encapsulated in the service, with a simple interface that provides an excellent separation of concerns.

My understanding of the proposal:

• The single cprj lock file will be replaced by a single mcprj file and one or more (possibly interdependent) cprj files, which will roughly map to the csolution and cproject YAML files respectively?

Assuming I understand the proposal correctly, my concern is that this proposal threatens the efficacy of the build manager for both these use cases.

There will no longer be a single file that will provide a reproducible build.

• If one crpj file changes then the build of another project could be changed.
• The user will have to manage multiple files, this may not be as obvious as the current solution.

The interface to the build manager is more complex:

• What happens if the build manager is called with a cprj file that requires another project? In more general terms what is the relationship between cprj files and how will that be determined?
• If the build manager is called with the mcprj file, what will be built? Will additional parameters be required to identify the particular project to build?
• What happens if there is a cprj file and no mcprj file, but there is a dependency?
• What happens if a mcprj refers to a cprj that has been moved, renamed or doesn't exist for some other reason?

A potential alternative is to extend the cprj format to support multiple projects or single projects in the same schema. This does have its own issues, potentially around an increase in file size and complexity. But it is conceptually simpler and has a greater chance of ensuring the integrity of the build, rather than using a file hierarchy and multiple dependent files.

However, another concern is that we are creating a dependency system that already exists in systems such as CMake, which is already used by the Build Manager. Perhaps this is the time to revisit the decision to have a proprietary format as a lock file and instead use CMake.

[brondani]

@madchutney thanks for the comments.

The single cprj lock file will be replaced by a single mcprj file and one or more (possibly interdependent) cprj files, which will roughly map to the csolution and cproject YAML files respectively?

mcprj will not replace the single cprj, but it will be added to describe the multi-project-contexts use case that was never supported before.

Assuming I understand the proposal correctly, my concern is that this proposal threatens the efficacy of the build manager for both these use cases.

The multi-project-contexts use case was never supported by cprj. The proposal does not touch the normal single project use case.

There will no longer be a single file that will provide a reproducible build.
If one crpj file changes then the build of another project could be changed.
The user will have to manage multiple files, this may not be as obvious as the current solution.

A normal single project will continue to have a single cprj and new multi-project-contexts will have a cprj for each context plus the mcprj. In both cases it provides reproducible builds.

What happens if the build manager is called with a cprj file that requires another project? In more general terms what is the relationship between cprj files and how will that be determined?

With references in both directions mcprj<>cprj the build manager will be able to generate CMakeLists at project and at solution level and CMake will figure out what it needs to build for the given context.

If the build manager is called with the mcprj file, what will be built? Will additional parameters be required to identify the particular project to build?

If only the mcprj is specified it builds the whole solution. A cprj could be additionally specified and/or a parameter to specify a context can be introduced.

What happens if there is a cprj file and no mcprj file, but there is a dependency?
What happens if a mcprj refers to a cprj that has been moved, renamed or doesn't exist for some other reason?

Error messages.

A potential alternative is to extend the cprj format to support multiple projects or single projects in the same schema. This does have its own issues, potentially around an increase in file size and complexity. But it is conceptually simpler and has a greater chance of ensuring the integrity of the build, rather than using a file hierarchy and multiple dependent files.

This was considered when elaborating this proposal, in my opinion separated files are conceptually cleaner.

However, another concern is that we are creating a dependency system that already exists in systems such as CMake, which is already used by the Build Manager.

We don't create a dependency system, we inform CMake about the existing dependencies using standard CMake functions. CMake is fully responsible for the dependency handling.

Perhaps this is the time to revisit the decision to have a proprietary format as a lock file and instead use CMake.

CMake does not know about CMSIS versioning, alone it can not create environment-independent lock files.

[jkrech]

Conceptually we have a solution where the user selects an active build-type / target-type combination. csolution's convert command will then generate the mcprj + the cprj file for all resulting contexts at once. In other words we could also generate the content of multiple files into a single file, as they cannot be updated separately.
Obviously we could at some point also consider rather than first generating mcprj + cprj we could directly generate CMakeLists.txt files.

[VGRSTM]

Per Open-CMSIS weekly call Open-CMSIS-Pack+Technical+Meeting+2022-03-08 sounds we were getting some agreement *cprj is going to act as a lock file if build process. Sounds fine to me closing possibly [projmgr] Using existing .cprj as lockfile.

Would say no strong opinion / concerns about \<mcprj\> introduction as:

• Is obviously supporting a must have to handle inter projects dependencies if not aiming to come back to yml contributions
• Is kind of internal process artifact end user will not have to act on (yml remaining main project description end user has to act on a way or another). As end user *cprj stuff just has to be possibly part of its source controlled files aiming to handle build reproducibility.

Would just highlight two key concerns to me embracing foreseen way of working then:

• Do we agree yml->cprj devtools process have to rely by default on cbuildgen option allowing to get as output generate <CprjFile> with fixed versions for reproducing the current build
• What about if introducing local pack(s)/component(s) copy per proposal Handling of packs (Filter, Repositories, Local packs, etc.) ? yml description will be aware of local path but cprj not ... here comes question to propagate yml capabilities to cprj or ... to skip cprj to CMakelists.txt with describe resolved paths. But if CMakelists.txt I foresee some other troubles ... as CMakeLists.txt may promote absolute paths which are less flexible if project sharing ....

[brondani]

• Do we agree yml->cprj devtools process have to rely by default on cbuildgen option allowing to get as output generate <CprjFile> with fixed versions for reproducing the current build

The cprj generated by csolution already generates it with fixed versions for reproducible builds.

• What about if introducing local pack(s)/component(s) copy per proposal Handling of packs (Filter, Repositories, Local packs, etc.) ? yml description will be aware of local path but cprj not.

You are right, this has to be solved, probably by making the cprj aware of it and extending the build manager handling.

[tcsunhao]

Not sure if it is proper to raise this question/issue here, that is, is there any extension point in CPRJ to add vendor specific debug related information?
NXP IDE/tools plan to directly get everything about project from CPRJ, I don't see any element to hold debug related information, like link server script parameters(if this example needs a special one.)
In pdsc, environment is provided for vendor specific data.

[VGRSTM]

@tcsunhao is *.cprj suitable for ? Sounds we are all more moving to *.yml support as project metadata. *.cprj is kept at that time serving mainly build concerns.

Please be aware, just in cas, a vendor specific capability has been added to *.yml stuff: [projmgr] Open door to vendor specific data addon

[tcsunhao]

@VGRSTM , thanks. *.yml can support this, it should be ok.

[tcsunhao]

@jkrech , continue with last meeting's discussion about extension point.
MCUXpresso SDK/IDE wants to provide out-of-box debug experience for customers which means customers can directly import/copy project from pack and download/debug with one or 2 click without manually settings.
To achieve this, some debug configuration should be provided in packs.
The reality is for a device, most examples can share the same debug configuration data, we plan to place it under "environment name="mcux"" in pdsc of DFP/BSP. But some examples may need special configuration due to many reasons. For example specific debug configuration data, like

<debug_configuration id="com.crt.advproject.config.exe.release" name="Release Configuration" probe="LinkServer">
          <scripts>
            <script type="init" silent="false" emulators="${gdb.stub} -mi -info-emu">
              <script>
set remotetimeout 60000
##target_extended_remote##
set mem inaccessible-by-default ${mem.access}
mon ondisconnect ${ondisconnect}
set arm force-mode thumb
${load}</script>
            </script>
            <script type="run" silent="false" emulators="${gdb.stub} -mi -info-emu">
              <script>${run}</script>
            </script>
          </scripts>
          <params>
            <params id="vector.catch.release.mcuxpresso" name="vector.catch" value="false"/>
            <params id="internal.cache.release.mcuxpresso" name="internal.cache" value="Disable"/>
          </params>
        </debug_configuration>

I see 2 place for such example specific data:

1. pdsc project environment. I don't think it is good, because name="mcux" only contain debug configuration data.

      <project>
        <environment name="uv" load="mdk/freertos_event.uvprojx"/>
        <environment name="iar" load="iar/freertos_event.ewp"/>
        <environment name="mcux" load="special_debug_configuration.xml"/>
        <environment name="csolution" load="csolution.yml" />
      </project>

2. csolution/cproject extension point. We need to translate above debug data into extension point. Our IDE can consume it then.

I know csolution/cprj focus on build, but it should provide place for debug configuration data.

[ReinhardKeil]

As we will deprecate CPRJ in a future version of CMSIS-Toolbox, I suggest to close that.

Also it is somewhat a duplicate with #662