[projmgr] Source patterns have to be supported (as well as related excluded ones)

[VGRSTM]

Per current https://github.com/Open-CMSIS-Pack/devtools/blob/a9919425f6554425e8723aee5bf7a9b299387e95/tools/projmgr/docs/Manual/Overview.md#groups-and-files proposal source description is file based.
It sounds like a pain from an IDE context. Such assumption is fully valuable if build reproducibility but purpose of *.cprj then. Considering project manager, so *.yml contributions, I would be please to add source patterns and related exclude patterns addon. Up to project manager process building *.cprj file to compute a runtime such patterns leading to a file based proposal.

groups:
- group: main
files:
- file: ./src/main.c
source_patterns:
- pattern: ./src/**/*.c
- pattern: ./src/**/*.cpp
exclude_patterns:
- pattern: ./src/**/*.md

- group: func1
files:
- file: ./func1/src/func1.c

Note: Keywords / schema to be discussed just promoting high level requirement here

[DavidJurajdaNXP]

I personally prefer explicit list of files from two reasons:

• it is useful for content validation
• I like to read it as "Table of contents"

but I know also people who prefer reduced maintenance effort and use wildcards as much as possible.
Open CMSIS Pack should probably support wildcards and let user to decide.

[VGRSTM]

This topic has been identified as part of CMSIS-Build Gap Analysis

@DavidJurajdaNXP & all, already shared Thanks some other comments, I'm fully sensitive myself to build reproducibility but I would back promote my point which is OpenCMSIS proposal is moving and we have now 2 "containers" to support workflow:

• .cprj / .clayer
• csolution.yml /cproject.yml /clayer .yml

I foresee myself reproducibility concerns being addressed Thanks .cprj & project source control.
*.yml contributions may then support other stuff which is project description in the light of PLM.
This way I'm thinking *.yml data may allow mix of exact and more loose descriptions enabling various user and integrator flows. I'm only pushing here to get pattern support do not mean you're forced to use it but devtools have to support it in sync. with yml schema update if we agree on

[mark-edgeworth]

No problem with the concept of wildcards for files here. However, several observations and questions:

1. What is the standard that is in use for specifying file patterns. Need to identify and document the choice here (one of the standard file globbing standards I'm guessing, but which one?). What common conventions are supported **, [abc], a*b, a?b etc
2. What are the semantics surrounding inclusion and exclusion patterns? Does exclude take precedence over include or is it the opposite?
3. How does this interact with for-type and not-for-type.
4. Is it necessary to specify an additional field pattern? Could we just expand files instead? If not, how do files entries interact with all of the other inclusion and exclusion mechanisms?

This is all getting rather complex, and needs clear documentation to enable reliable implementations to be built.

[VGRSTM]

@mark-edgeworth considering we are in agreement on principle (good !) yes let's move to specifications refinement.
To my view:

1. Myself I like https://code.visualstudio.com/docs/editor/codebasics#_advanced-search-options especially because ** is nice discussing about file pattern here
   

2. I'm thinking prior being able to apply any exclusion pattern we have first to build a list to exclude against. So I would promote includes (aka. source_patterns) first to build resolved files list. Next I would process excludes (aka. exclude_patterns) to remove related resolved items from resolved includes list.

3. My guess is as to support for-type and not-for-type as file is supporting

4. I'm thinking adding source_patterns support to files is right way.
   Same I'm thinking to add exclude_patterns pattern support to files.
   Back to 2. file have to be added to includes resolved file list and exclusion pattern to applied on top of.

groups:
- group: main
files:
- file: ./src/main.c
- pattern: ./src//*.c
- pattern: ./src//.cpp
- exclude: ./src/**/.md

- group: func1
files:
- file: ./func1/src/func1.c

Even better to me as end user ... just get rid of pattern ... file supporting regular expression :-)

groups:
- group: main
files:
- file: ./src/main.c
- file: ./src//*.c
- file: ./src//.cpp
- exclude: ./src/**/.md

- group: func1
files:
- file: ./func1/src/func1.c

According my view sounds like not so complex. I'm pushing to get "only" files extension. Let's think this way:

• I guess today as part of devtools we have some API which is promoting getFiles(...) function returning collection of file items (ok possibly some per group filter or ...)
• Tomorrow sounds same to me adding some pattern resolution & list pruning based on exclusion one to getFiles(...) core process.

[jkrech]

While I like the idea to not distinguish file from pattern, do people care that a pattern potentially describes multiple files (singular vs. plural).

Do we need a precedence in processing or is it done sequentially?
a) process 'file' and 'pattern' entries before 'exclude'
b) process entries sequentially

Only process entry if for-type/not-for-type allow

optimize, debug, defines, undefines, add-paths, del-paths, misc are not supported for "exclude"

What other "dependencies/context" do people see need to be considered?

[VGRSTM]

@jkrech
To my view we have to update nothing to current devtools implementation what means to keep current type/not-for-type, misc. etc. processing related to file.

Based on current processing we are at a point building some internal lists.

Just consider:

• Adding pattern capability to file is same than adding multiple explicit file items got Thanks pattern resolution
• Adding exclusion capability is just removing related pattern resolution to internal lists
  Back lists building have to be completed prior exclusion application.

My first guess schema is:

We may end up with such usage: