[projmgr] Using existing .cprj as lockfile

[slhultgren]

With the recent advances to the .yml format and projmgr/csolution tools I'd like to reopen the discussion around lockfiles.

It seems that the source of truth for the projects are now likely to be the .yml files.

Use-case: User adds a compiler flag to the project

1. Manually edit the cproject/csolution.yml file to add the defined symbol
2. Run csolution/projmgr tool to update .cprj files
3. Trigger build of the project using cbuildgen and the cprj file

During step 2, if there is already a cprj file existing, we should preserve/reuse as much information from that as possible.

For example, the yml files may not specify any packs, or without any version ranges, but the .cprj may be strictly locked down to a specific pack+version.
Same for components.

For the above use-case, the user expects only the define to be added in the build. If the 2. would result in a full recompute/resolving of components+pack-versions it would make the .yml files impossible to use over time or between colleagues, since the contents of the .cprj would very likely be different due to changing CMSIS_PACK_ROOT contents.

This is a followup on the discussion from the build gap

Also, working in this way means that the user should put commit the .yml and the .cprj files in their git repository to make sure that colleagues/CI are building the same thing.

[VGRSTM]

Same idea here thinking #82 cprj acting as lock file if end user resources.

[DavidJurajdaNXP]

Source of truth for the projects are now likely to be the .yml files

Agree, lock file should be designed on this level too.

During step 2, if there is already a cprj file existing, we should preserve/reuse as much information from that as possible.

Agree, but... I think decision whether certain change impact dependency resolution or not might be difficult. If we take in account configuration (which is currently not covered by scope of open cmsis pack), even the compiler flag could be in theory dependency of some component and could affect dependency resolution. I would postpone this and consider it as performance optimization task.

I would personally design lock file in similar way as python package manager pip:

$ pip freeze
certifi==2021.10.8
charset-normalizer==2.0.11
idna==3.3
pathspec==0.9.0
PyYAML==6.0
requests==2.27.1
urllib3==1.26.8 
yamllint==1.26.3

$ pip freeze > requirements.txt
$ pip install -r requirements.txt

In open cmis pack ecosystem it could be something like:

$ csolution freeze cfreeze.yml
$ cpackget pack add cfreeze.yml

I am using cfreeze.yml because clock.yml sounds confusing.
Yml format is used in order be uniform with other cproject/csolution yml files (from the implementation perspective => one loader for any data).

[slhultgren]

High-level, there are a few main requirements of a lockfile:

• Ensure the project is stable over the course of development. Principle of fewest surprises.
  • I should be able to add a new component to my project, and this should change as few things as possible
• Ensure the project is reproducible
  • Each time I resolve/build my project I expect the same result
• Ensure the project is shareable
  • I should be able to share my project with my colleagues and they should be able to use my locking information to build the same thing
• Ensure the project state is obvious
  • If something changed in my project, it should be visible and obvious in the lockfile so that I can see it in git diff

In #171 we see that the cprj-format grows and grows to support the use-cases supported by the yml formats. I think this is a problem since it means that we will always have two different formats that always needs to support the same things.
Why? Sounds like a maintainers nightmare with no clear gain?.

I would much prefer if we either go in the route of yml->CMakeLists directly as in #171 (comment) or improve and align on the Generator input file (*.cgen.json) format that could also possibly serve as a lockfile (https://github.com/Open-CMSIS-Pack/devtools/blob/main/tools/projmgr/docs/Manual/YML-Format.md#generator-proposal and Open-CMSIS-Pack/Open-CMSIS-Pack-Spec#104).

So we need:

• When going from YML, take any existing Pack list from the lock file and ONLY use this set of packs for resolving. This will make the project stable and reproducible
  • If YML pack list is missing or loosely defined (ranges), reuse from the lock file, or define a pack list if the lock file is missing
  • If YML pack list definition is incompatible with the lockfile, update the lock file
• When creating the lock file we should show at least these things to make it obvious:
  • Packs used (including checksum of the pack that could be optionally verified)
  • Target used
  • All conditions resolved in the project
  • Components used and from which packs they are used (including the condition used for the component)
    • All files used from the component (referencing project/pack in a shareable way)

The nice thing about this is that then we no longer need a tool to understand the complete state of the project, since everything has been made explicit in the lockfile.
We will also be able to detect non-obvious changes with git diff in the project due to condition resolving, e.g. adding ComponentB may make "ComponentA" resolve to a different "ComponentA"

[jkrech]

The benefit I am seeing in the representation in the 'cprj' format is, that it only describes a single context in a "fully specified" way.
The benefit of csolution is that it should allow to specify all contexts of a solution with the freedom to range from loosely specified all the way to precisely specified. E.g. you can now specify the packs to be used even reference a copy of a pack which is "private" to the solution.
Therefore I still find this two phase approach csolution -> cprj(s) useful.

[slhultgren]

@jkrech I agree about the single context, this is needed since a single csolution/cproject may resolve to several unique lockfiles, one per context.

To fullfil my needs, the current cprj-format would need to evolve abit, since it does not satisfy the obvious requirement at least.

Below is a small proposal of what a reproducible shareable and obvious lockfile could look like:
This is based/inspired by the cgen.json but not exactly the same:

{
    "device": "STM32F407IGHx",
    "board": "NucleoF407",
    "solution": "../MySolution.csolution.yml",
    "project": "../Blinky.cproject.yml",
    "build_type": "Debug",
    "target_type": "Nucleo",
    "packs": [
        "Keil::[email protected]",
        { "pack": "Keil::[email protected]", "sha256": "..." },
        { "pack": "MyPack::[email protected]", "path": "../Some/Path/toMyPack" }
    ],
    "components": [
        {
            "pack_id": "Keil::[email protected]",
            "name": "Device:STM32Cube HAL:[email protected]",
            "condition": "STM32F4 GCC",
            "pack_files": [
                "./path_inside_pack/file.c"
            ],
            "project_files": [
                "./path_inside_project/file.c"
            ]
        },
        {
            "pack_id": "Keil::[email protected]",
            "name": "Board:[email protected]",
            "condition": "STM32F4 GCC",
            "pack_files": [
                "./path_inside_pack/file.c"
            ],
            "project_files": [
                "./path_inside_project/file.c"
            ]
        },
        {
            "pack_id": "Keil::[email protected]",
            "name": "Device:HAL:[email protected]",
            "condition": "STM32F4 GCC",
            "pack_files": [
                "./path_inside_pack/file.c"
            ],
            "project_files": [
                "./path_inside_project/file.c"
            ],
            "instances": {
                "WiFi": {
                    "instance": 0,
                    "setup": "wifi-config.json",
                    "baudrate": 19200 
                },
                "Debug": {
                    "instance": 2,
                    "baudrate": 57600
                }
            }
        }
    ]
}

This lockfile could also be described like this using TypeScript types as "schema"

interface Lockfile {
    device: string,
    board: string,
    solution: string,   // Relative to lockfile directory
    project: string,    // Relative to lockfile directory
    packs: PackType[],
    build_type: string,
    target_type: string,
    components: ComponentType[],
}

type PackType = string | Pack;

interface Pack {
    pack: string,       // Full identifier of pack
    path?: string       // Optional path to where the pack is located, relative to the lockfile directory
    sha256?: string     // Optionally store sha256 of entire pack installation
}

interface ComponentType {
    pack_id: string,    // Identify which pack that brings this component
    name: string,       // Full name of the component
    condition: string,  // Condition used by the component
    pack_files: string[],     // Component files that stay in the pack installation location
    project_files: string[],  // Component files that end up in the project AKA "RTE files" AKA "attr=config" files
    instances?: InstancesType,
}

interface InstancesType {
    [instanceName: string]: InstancesType,
}

interface Instance {
    instance: number,
    setup?: string,     // Path to setup file
    baudrate: number,
}

Some open discussion points about this proposal:

• It does not list user files
• It does not show compiler flags
• It does not show dependencies to other projects (build order)
  All these three can still be found by parsing the referenced csolution/cproject files however.

The project_files from a component would be what we currently think of RTE files.
The pack_files from a component would be any other files that are coming from a component, but always stays in the pack-installation-location.

If we think we should go for a more detailed lockfile including the above three points, then I think we should consider moving directly from YML to CMakeLists, one CMakeList per context.

EDIT: To clarify, With the above proposed file I suggest that we have two separate flows for building the project:

1. YML-based
2. CPRJ-based

For the 2. CPRJ-based, we use what we have today.

For the 1. YML-based, we would have this flow:
1.1 User edits YML file
1.2 User uses tool to resolve the project, which may update the lockfile, taking existing locking information into account
1.3 User builds based on YML and Lockfile
So here there is a new function needed to take YML+Lockfile and produce a CMakeLists.txt.

[jkrech]

@slhultgren regarding 1.2 I think the devil is in the detail. I like the idea of "minimal" changes compared to previous lockfile but I doubt that this will always create the desired "update" behavior. E.g. if a newer version of the pack got installed but the previous version is no longer present, do we have to install the previous version in order to achieve minimal changes? What if the previous version is still installed, do we let the user know that there is a newer version available?
Maybe it is not for the tool to optimize for minimal changes, but allow the user to compare the previous and the latest lockfile to identify the changes. Restricting e.g. the pack selection in the csolution will allow the user to minimize the differences in the lock file. What am I missing?

[slhultgren]

@jkrech I agree there will be finer details to hash out indeed. for the ones you mention I would be very conservative:

• Newer pack is available in CMSIS_PACK_ROOT but lockfile is using an older one that also exists in CMSIS_PACK_ROOT
  -> Following the priniciple of least surprise and stability between colleagues who may very well have different contents in the CMSIS_PACK_ROOT, it means that we should indeed only use the version mentioned in the lockfile, regardless of what is available in the CMSIS_PACK_ROOT

• If newer is in CMSIS_PACK_ROOT but the lockfile-pointed one is not
  -> This should result in error (missing pack) with potential auto-suggestion, "would you like to install XXX@YYY"

This would be the minimum, then we could indeed add some more "sugar" on top, like suggesting possibly to move to a newer pack version. But moving to a newer pack version may change a lot of things in terms of component resolve state, so it needs to be a conscious choice by the user when and why this happens.

As you say, technically we could restrict the pack selection in the yml file, but this is a less nice user-experience IMO.
In case I later want to do a full "re-alignment" to the latest versions supported by my yml files I would simply remove the lockfile and recreate it from scratch. This would make the new lock file be created based on the latest information available in the CMSIS_PACK_ROOT combined with any version range or pack filter from the yml file.

Lastly however I also again want to highlight the value of having a file showing the complete state of the pack/component state of the project as the tools would understand it:

• Showing which packs are used
• Showing which components (exactly, meaning indicating both originating pack and condition)
  • Showing where RTE files ends up from these components (are some still left in the PACK_DIR, and are some in the project, and if so, exactly where in the project did they end up)
• and not mentioned before, also indicate any potential unresolved component in the project

All of these features also fits perfectly IMO with the generator input file mentioned previously, which would simplify implementing generators, so having a file format to explicitly describe the exact project pack/component state is very attractive here.

[slhultgren]

@ReinhardKeil I would like to get your feedback on this as well since my latest thoughts in #119 (comment) is basically a slightly expanded version of your cgen.json proposal.

High-level idea is that we should have a format that can explain the exact pack+component state, and this same format can then fill the needs of both a generator input and a lockfile.

[ReinhardKeil]

@slhultgren the *.CPRJ file has most of this information. I suggest we start there and make a gap analysis.
I understand that XML is not the best user-oriented format (and we may change that at a later point - which could be with a viewer or list utility), but fundamentally most of the information should already be there. The fact that a single file serves the need of a log file and build input file has IMHO the benefit that it cannot deviate. Files from a previous build can be easily compared.

We can of course add missing information once we have understood the usage. For example it does currently not show the invocation line of the csolution tool so that a convert process is cannot be replicated. Likewise the Environment variables and inherent settings (for example cdefault assumptions) is missing. However, these gaps are minor, and we should just list them as requirements for further development.

We should also discuss if some of the current information should be removed. For example timestamp (shown below) as this creates the effect that every convert run creates effectively a different file, even when the rest of the content is identical.

  <created timestamp="2022-04-27T15:31:27" tool="csolution 0.9.4"/>

We should check the behavior of git systems, so that the *.CPRJ file can be committed, but effectively does not change when csolution recreates the same content.

Note: the csolution tool as it is deployed right now has a bug as it does not fix the pack versions, but allows to use also later versions of that pack. This is fixed already in the next iteration.

[DavidJurajdaNXP]

Beside requirements described in
#119 (comment)
#119 (comment)
which I share, I would add following use cases for lock file:

1. Manual update of fixed pack versions in already existing project is problematic. See issue described in python ecosystem.
   https://pipenv.pypa.io/en/latest/
   "Managing a requirements.txt file can be problematic, so Pipenv uses Pipfile and Pipfile.lock to separate abstract dependency declarations from the last tested combination."
   Examples:
   https://realpython.com/pipenv-guide/#the-pipfile
   https://realpython.com/pipenv-guide/#the-pipfilelock

2. Record of pack versions which were successfully tested for particular example could be used as quality metric.
   Lock files could be saved after successful test in order to document history of tested pack versions for given project.

I would prefer yml format for lock file since it is already used for solution definition.

[ReinhardKeil]

As discussed yesterday, I got confused by the difference of:

• log file (a protocol that lists the current stated)
• lock file (a way to fix the selection - obviously a lot more complicated).

With *.CPRJ I believe only the "log file" requirement can be fulfilled. Are we aligned in this view? I would then split this issue into two.

[slhultgren]

We could discuss the format, (I have some changes in mind to what I've previously posted), but I think it is also interesting to align on what this could bring for the tooling.

In my view, having a file that clearly both locks the state (by locking on packs) and describes the resolved state, it then means that the user no longer need any runtime tool to understand what and why something is part of the project.

This would also help streamline cbuildgen since thanks to the lockfile, it will no longer have to do any component/condition/RTE resolution at all to be able to construct the CMakeLists.txt.
This means that we remove the creation of the RTE from the build step.

I'll take a few use-cases to explain the flow:

Add a component [YML] high level

1. User edits YML files
2. User runs csolution to create RTE files and lockfile
3. User runs cbuildgen to create CMakeLists.txt
4. User runs cmake to build

Add a component [CPRJ] high level

1. User edits CPRJ file
2. User runs cprj (TBD) to create RTE files and lockfile
3. User runs cbuildgen to create CMakeLists.txt
4. User runs cmake to build

Add a component [YML] detailed

1. User edits the .cproject.yml and adds component CMSIS:Core
2. User edits the .csolution.yml and adds pack ARM.CMSIS note no version number
3. User runs csolution tool
   3.1. csolution checks in the lockfile if there is a matching pack for ARM.CMSIS already being used
   3.1.a. If there is a match, use this version
   3.1.b. If there is no match, record the latest version of ARM.CMSIS pack found in CMSIS_PACK_ROOT in the lockfile
   3.2. csolution resolves the CMSIS:Core loose description to an exact match, e.g. ARM::CMSIS:[email protected]#ARMv6_7_8-M Device # is used as separator to include condition id as well
   3.2.1. csolution then creates all the RTE files for the component
   3.2.2. csolution records the state of the created ARM::CMSIS:[email protected]#ARMv6_7_8-M Device in the lockfile with
   • The full component id including condition id
   • The Pack id so that we know where it came from
   • Each file that the component is using is recorded with it's pack path and project path and condition id, category...
     • RTE files will have both a pack path and a project path
     • generated files will at least one of pack path and project path, possibly both, depends on the generator
     • other files will only have a pack path
4. User runs cbuildgen tool
   4.1. cbuildgen reads the lockfile, concatenates paths from component files and their associated pack paths and constructs a CMakeLists.txt
5. User runs cmake to build their project

Using a generated component [YML]

1. User edits YML files to include a generated component (AKA component that uses generators)
2. User runs csolution to create RTE files and lockfile (lockfile contents based on pdsc description of generated component)
3. User runs csolution to trigger generators with the lockfile as argument to the generator since this is the full current state
   3.1. generators can create their files by reading only the provided lockfile state, no condition resolving necessary
   3.2. csolution uses the returned gpdsc description and updates the lockfile entry for the component with the generated contents
4. User runs cbuildgen to create CMakeLists.txt
5. User runs cmake to build the generated code

[jkrech]

@slhultgren - in the meantime we have been discussing to disable the cbuildgen functionality regarding 'managing' RTE config files. Do you think that editing cprj files is still relevant?

[slhultgren]

@jkrech

in the meantime we have been discussing to disable the cbuildgen functionality regarding 'managing' RTE config files.

Perfect :) I very much agree with this. There should only be one point in the flow where files are added in the project and this should be well defined. Then the lockfile is used to describe what/how/where files where created in the project.
This makes the assumptions related to how all of this is done restricted to one place: csolution.
Then other consumers (cbuildgen) of the project info don't need try to come up with the same assumptions, they can just take the exact description of the result.

So long term, if we add some flexibility in YML on how certain components are placed for example, this would not change anything for cbuildgen since it would still rely on the "string value path" of the files described in the lockfile.
Only csolution would need to be updated to place RTE files correctly, and then store the value in the lockfile.

Do you think that editing cprj files is still relevant?

Honestly no, I would like us to consider the CPRJ format as deprecated. I kept it in the use-cases since it is still part of the OpenCMSIS standard, so we need to think of a way to handle it, but ideally I would like to avoid double-implementation of features.

In my view:

• CPRJ is a simple transport format for basic projects (not multicore, not trustzone etc)
• YML is the format where the user works day to day

So for me the main build path (ideally only build path) is

YML -> [csolution] -> { Lockfile, RTE files, generated files } -> [cbuildgen] -> CMakeLists.txt

In the case of CPRJ there would be a single one-time conversion to YML done, then back to the normal YML based workflow.

[madchutney]

Apologies for coming late to this conversation, but just recently I have been thinking about this issue from a slightly different perspective. Mostly due to the separation of concerns design principle, I have been thinking about how to manage CMSIS components in a project agnostic way.

This project agnostic approach is followed by most package managers, so in principle it should be relatively simple to replicate. However, CMSIS gives us an extra dimension, that package managers do not normally have to handle. Normally the package dependency being managed is the same as the library or component being used. With CMSIS we allow loosely defined component references, which can be resolved to a number of components that can be delivered by different packs. This increases flexibility, but makes both reproducibility and dependency management more tricky.

Another aspect that most package managers posses, is a separation between the "required" components (those that a developer explicitly wishes to use) and the dependencies of those components (e.g. Python's Pipfile vs Pipfile.lock). The reason this is important is dependency management over time. If a component is removed from a project, or new version of a component is released that has different dependencies, then all the irrelevant dependencies should be removed from the project. Unless the required components and dependencies are not stored separately, then it is not possible to automatically determine which components should be removed.

Conceptually we can used the csolution format to store the required components, assuming certain constraints are applied. For example, if the component references included in the cproject files are limiting to only those that are "required" and the pack list is not included in the csolution file, but instead deferred to a lock file.

Please note that the lock file does not include any project specific information, such as files, build settings etc. It solely contains the components being used and the packs that they come from.

A lock file could look something like the following, where the component reference is keyed from those included in the cproject file:

Components:
  - ComponentReference:
      Vendor: ARM
      Class: Device
      Bundle:
      Group: Startup
      Subgroup:
      Variant: C Startup
      Version:
    ComponentId:
      Vendor: ARM
      Class: Device
      Bundle:
      Group: Startup
      Subgroup:
      Variant: C Startup
      Version: 2.1.0
    PackId:
      Vendor: ARM
      Name: CMSIS
      Version: 5.9.0
    Dependencies:
      - ComponentId: {}
        PackId: {}
        Dependencies:
          - ComponentId: {}
            PackId: {}
            Dependencies: []

This results in a tree structure that makes it straight forward to manage component dependencies and packs. For reference the schema for this document is as follows:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
 
  "definitions": {
    "ComponentReference": {
      "type": "object",
      "properties": {
        "Vendor": {
          "type": "string"
        },
        "Class": {
          "type": "string"
        },
        "Bundle": {
          "type": "string"
        },
        "Group": {
          "type": "string"
        },
        "Subgroup": {
          "type": "string"
        },
        "Variant": {
          "type": "string"
        },
        "Version": {
          "type": "string"
        }
      },
      "required": [
        "Class",
        "Group"
      ]
    },
    "ComponentId": {
      "type": "object",
      "properties": {
        "Vendor": {
          "type": "string"
        },
        "Class": {
          "type": "string"
        },
        "Bundle": {
          "type": "string"
        },
        "Group": {
          "type": "string"
        },
        "Subgroup": {
          "type": "string"
        },
        "Variant": {
          "type": "string"
        },
        "Version": {
          "type": "string"
        }
      },
      "required": [
        "Vendor",
        "Class",
        "Group",
        "Version"
      ]
    },
    "PackId": {
      "type": "object",
      "properties": {
        "Vendor": {
          "type": "string"
        },
        "Name": {
          "type": "string"
        },
        "Version": {
          "type": "string"
        }
      },
      "required": [
        "Vendor",
        "Name",
        "Version"
      ]
    },
    "Dependencies": {
      "type": "array",
      "items": [
        {
          "type": "object",
          "properties": {
            "ComponentId": {"$ref": "#/definitions/ComponentId"},
            "PackId": {"$ref": "#/definitions/PackId"},
            "Dependencies": {"$ref": "#/definitions/Dependencies"}
          },
          "required": [
            "ComponentId",
            "PackId"
          ]
        }
      ]
    }
  },
 
  "type": "object",
  "properties": {
    "Components": {
      "type": "array",
      "items": [
        {
          "type": "object",
          "properties": {
            "ComponentReference": {
              "$ref": "#/definitions/ComponentReference"
            },
            "ComponentId": {
              "$ref": "#/definitions/ComponentId"
            },
            "PackId": {
              "$ref": "#/definitions/PackId"
            },
            "Dependencies": {
              "$ref": "#/definitions/Dependencies"
            }
          }
        }
      ]
    }
  },
  "required": [
    "Components"
  ]
}

[slhultgren]

@madchutney thanks for the input, very interesting

separation between the "required" components (those that a developer explicitly wishes to use) and the dependencies of those components

Very valid point, I agree totally with this. Dependencies are not always that interesting, and indeed, if you change the top-level component, you likely also want to change/remove the now unused dependencies.

Q: Do we now then say that the list of the components in cproject.yml is now no longer necessarily the full list of components used in the project?
i.e. full list of components would be guaranteed in the lockfile, but not guaranteed in cproject.yml

Conceptually we can used the csolution format to store the required components, assuming certain constraints are applied. For example, if the component references included in the cproject files are limiting to only those that are "required" and the pack list is not included in the csolution file, but instead deferred to a lock file

On this I'm not sure I fully understand what you mean.
Does it imply this:

1. No need to have a pack list in csolution or cproject? => if so, then this sounds interesting, since it opens up more flexibility of even allowing to use components from different versions of the same pack series. I.e. a single project can combine ComponentA from ARM.CMSIS.5.9.0, and ComponentB from ARM.CMSIS.5.6.0 in theory
2. cprojects are only allowed to select between required components specified in csolution? => This I don't think is a good idea, it restricts the cprojects too much. The lockfile needs to be one per "permutation", (one per $BUILD_TYPE*$TARGET_TYPE*$CPROJECT_COMPONENT_LIST)

Please note that the lock file does not include any project specific information, such as files, build settings etc. It solely contains the components being used and the packs that they come from.

For build settings, we could discuss, I'm not sure myself. I do think we should find a simple workflow so that our tooling can be simpler than what it is today. Implementing YML logic in csolution.exe and cbuildgen.exe is not very nice IMO.

For files, I would argue that in case of CMSIS it is actually necessary to have it as part of the lockfile.
For me, the purpose of a lockfile is to describe

• what you use in a project (so that colleagues can use the same)
• why you have it (so that you can detect/debug mistakes)
• where it comes from, and where it is placed (so that you can know and understand the project structure)

With a purely package based package manager like pip or NPM, the granularity is on the module/package level. If you know which module you have, then you implicitly also know all the files.

With CMSIS it is not that simple, since the file set is not static for a component (file-based conditions, RTE connection, Generated components).

To summarize my comments:

• Allowing the cproject component list to be more focused on the top level components that the user is interested in, and delegating the pure dependencies to the lockfile is a good idea
• ComponentID could be stored as a plain string using the official OpenCMSIS YML component ID format
  • ComponentID should also include condition id, otherwise it is not a unique match and requires more tooling logic to understand => Harder to understand for user when looking at the files
  • Every ID-part available for the selected component should be stored on the ComponentID. e.g. if the selected component specifices the optional Cvariant, then the lockfile should include the Cvariant in the ID string since we had this information during the resolution process.
• files should be listed for the components since they are also condition-based, and changes in the project that modifies how/what files are included in the project should be reflected in the lockfile
  • To make RTE connection and generated components understandable, both the path_in_pack and path_in_project should be recorded for each file

[slhultgren]

@madchutney I forgot to add, your ComponentReference as a reverse-link to be able to connect the dots of I wrote this in cproject.yml, and then I got this longer ID in the lockfile is a nice thing as well :)

[jkrech]

Differentiating between "user required" components and those being selected to satisfy dependencies resonates with me. Considering the situation that there maybe choices in resolving dependencies, these user decisions would be recorded in the lock file, which means that the lock file becomes a mandatory project file or am I missing something?

I am wondering whether we could record such components as "secondary components". Meaning they could be "excluded" from build in case they are no longer showing in the dependency list.
When the users selects a component, there is a choice whether the component is required by the user's code (e.g. a certain USART interface is called directly, or to satisfy an I/O component's requirement to print to a terminal).

[ReinhardKeil]

After reviewing this issue, there might be an intermix of different things:

• a "log" file that lists all important parameters about a csolution/cproject
• a "lock" mechanism that protects from accidently modifying properties

My suggestion: introduce a "log" file that lists all important information about a csolution based application in one file. The information could include:

• packs used
  • license information of these packs
• components used, with information about:
  • required components, which would indicated other components that are requested by conditions of that component
  • required-by components, which would indicated other components that request this component by conditions
  • pack indicates the software pack that defines the component

It should be also possible to list:

• overall information for all context configuration and context specific information.
• files that belong to the component along with path information.

This "log" file could be in YAML format (using the standard naming conventions) so that it is user digestible, but also serves as data source for analysis tools. Over time we could develop it into a "lock" mechanism (but I suggest this is not the initial priority).

After considering all feedback the *.CPRJ file seems not the right place to add above information, instead my proposal is a single "log" file per *.csolution.yml" that is generated by the csolution tool.

[madchutney]

@slhultgren Thanks for taking the time to review my suggestion.

Q: Do we now then say that the list of the components in cproject.yml is now no longer necessarily the full list of components used in the project? i.e. full list of components would be guaranteed in the lockfile, but not guaranteed in cproject.yml

That was my thinking, yes.

Conceptually we can used the csolution format to store the required components, assuming certain constraints are applied. For example, if the component references included in the cproject files are limiting to only those that are "required" and the pack list is not included in the csolution file, but instead deferred to a lock file

On this I'm not sure I fully understand what you mean. Does it imply this:

I trying to illustrate how the csolution or cproject files could be the source of the requirements, while a lock file used to provide the additional information on component resolution and dependencies. If the constraints were not applied it would mean that it was not possible to differentiate between required components and dependencies and the the pack list in the would have to be synchronised somehow.

1. No need to have a pack list in csolution or cproject? => if so, then this sounds interesting, since it opens up more flexibility of even allowing to use components from different versions of the same pack series. I.e. a single project can combine ComponentA from ARM.CMSIS.5.9.0, and ComponentB from ARM.CMSIS.5.6.0 in theory

That is what I was thinking. The csolution or cproject may add this constraint but in terms of a lock file format I was trying to be as flexible as possible.

2. cprojects are only allowed to select between required components specified in csolution? => This I don't think is a good idea, it restricts the cprojects too much. The lockfile needs to be one per "permutation", (one per $BUILD_TYPE*$TARGET_TYPE*$CPROJECT_COMPONENT_LIST)

I agree, the lock file should be per Project Context ($BUILD_TYPE*$TARGET_TYPE*$PROJECT). The lock file format could be structured such that there was a "lock" per Project Context or an individual lock file per Project Context (my example takes the latter approach as this seemed simpler).

Please note that the lock file does not include any project specific information, such as files, build settings etc. It solely contains the components being used and the packs that they come from.

For build settings, we could discuss, I'm not sure myself. I do think we should find a simple workflow so that our tooling can be simpler than what it is today. Implementing YML logic in csolution.exe and cbuildgen.exe is not very nice IMO.

For files, I would argue that in case of CMSIS it is actually necessary to have it as part of the lockfile. For me, the purpose of a lockfile is to describe

• what you use in a project (so that colleagues can use the same)
• why you have it (so that you can detect/debug mistakes)
• where it comes from, and where it is placed (so that you can know and understand the project structure)

With a purely package based package manager like pip or NPM, the granularity is on the module/package level. If you know which module you have, then you implicitly also know all the files.

With CMSIS it is not that simple, since the file set is not static for a component (file-based conditions, RTE connection, Generated components).

Great points, the lock file I was proposing is only for CMSIS Components and Packs. There is a lot of information missing that would need to be in Solution/ Project File and for that matter in the source files. I came at this from the perspective of what do we need for a Component Manager for CMSIS. I was hoping that breaking down the problem into the different concerns of Packaging and Project might make the tools more extensible and flexible.

[madchutney]

Differentiating between "user required" components and those being selected to satisfy dependencies resonates with me. Considering the situation that there maybe choices in resolving dependencies, these user decisions would be recorded in the lock file, which means that the lock file becomes a mandatory project file or am I missing something?

Yes, my suggestion was to record user decisions in the lock file and not in the csolution/cproject files. So while the lock file is not perhaps mandatory in all cases (e.g. a generic example project) it would be needed for most situations.

I am wondering whether we could record such components as "secondary components". Meaning they could be "excluded" from build in case they are no longer showing in the dependency list. When the users selects a component, there is a choice whether the component is required by the user's code (e.g. a certain USART interface is called directly, or to satisfy an I/O component's requirement to print to a terminal).

I think the "secondary components" is the same concept as the dependencies. If a component is being added to resolve a dependency then it would be a "secondary component", but there should also be an option to make it a "primary component" even if it happens to satisfy a dependency requirement.

From a component management perspective I think it would be clearer if unused components and packs were removed from the definition.

[madchutney]

My suggestion: introduce a "log" file that lists all important information about a csolution based application in one file. The information could include:

• packs used

• components used, with information about:

  • required components, which would indicated other components that are requested by conditions of that component
  • required-by components, which would indicated other components that request this component by conditions
  • pack indicates the software pack that defines the component

I think this information is very similar in content to the lock file format for components #119 (comment), although the organisation is different. Rather than using required and required-by a tree structure is used to convey the relationship between components.

[slhultgren]

@ReinhardKeil

After reviewing this issue, there might be an intermix of different things:

• a "log" file that lists all important parameters about a csolution/cproject
• a "lock" mechanism that protects from accidently modifying properties

Yes and no for me, all information that is put in the file is also functional, meaning that tools should use this data to understand the project.
Since it is functional information, IMO it is wrong to call it a log file.

My ideal view of the future is a simplified flow where cbuildgen doesn't have to understand:

• the various details on how to resolve a component
• the future flexibility options introduced for RTE file placement, or PLM in general

cbuildgen should simply by inspecting the lockfile be able to figure out which files it should pick and where they are located in the project.
For example, imagine that we in the future add a special property for the YML on where to place a specific component, today this flexibility also needs a reflection in CPRJ.
If cbuildgen only consumes the lockfile for component files, then it can trust that the files are already there.

So I would like to also streamline the lifecycle of the devtools to avoid double-implementing every flexibility feature we will introduce in the future in the standard.

Today the flow is

YML => csolution.exe => { PLM, RTE, GeneratedComponents, CPRJ } => cbuildgen.exe => { PLM, RTE, GeneratedComponents, CMakeLists.txt }

I would love it if it could be simplified to

YML => csolution.exe => { PLM, RTE, GeneratedComponents, Lockfile } => cbuildgen_new.exe => { CMakeLists.txt }

# with also a 
CPRJ => cprjconvert_new.exe => YML

Because this means that only one tool need to consider PLM and dealing with flexibility of file placement etc (csolution.exe).

This streamlining is a bit of work I agree, but I would be happy to work in this direction.

[ReinhardKeil]

@slhultgren

I my proposal the log file is indeed just a report file for now. It would be generated in parallel to the *.cprj files, but only one *.log file for the complete csolution.yml. Let's discuss today.

[ReinhardKeil]

I believe this is somewhat a duplicate with #434 -> suggest to close it.

[slhultgren]

I agree, lets close this one and continue in #434 , for me the cbuild.yml covers most of the discussion (if not all) so far