Update target/variant/modifier logic in build_examples.py (#23054)

* Starting to add some build target logic and unit tests

* Restyle

* make basic unit tests pass

* More unit tests

* Restyle

* Update prefix logic: use the builder prefix as well

* Prepare to move build targets for linux

* More conversion tests

* more conversions

* Converted more targets

* Converted more targets

* Updted more targets

* Simplify target part

* Restyle

* All applications are converted

* Remove old target code - should not be needed anymore

* Restyle

* Building at least chip-tool works. No introspection though

* Fix typo in m5 builds

* A bit easier to parse logging

* More updates

* Add ability to output all variants of some target

* Restyle

* Add the ability to print all possible targets

* Add esp32 qemu support

* Restyle

* Fix typo

* Simplify target printout, make nrf tests work

* Restyle

* Remove some usages of target glob

* Replace glob in build scripts

* Restyle

* convert some globs into instructions, fix some target compile bugs

* make unit tests pass

* more unit tests

* Restyle

* Remove the default of "all" in build targets since that does not mean anything anymore

* Rename arm64-clang to arm64 in targets for cloudbuild smoke test

* For arm64, clang is a modifier that is required

* Remove some whitespace.Tested that targets are ok

* Fix ameba

* Fix tizen

* Fix infineon

* Fix qpg

* Fix infineon, cc13x2x7 and update android a bit

* Fix linux standalone

* Fix efr32

* Fix esp32

* Remove glob option from tasks.json

* Fix platform naming for targets - fixes darwin target naming

* Fix k32w targets - no more release

* Adjust arm64 cross compile target names

* Fix boufallolab app target

* All-clusters full thread device does not link (insufficient ram/flash) on cc13x2x7. Updating to minimal thread device only

* Update k32w example builds: nologs is mandatory for low-power builds

* Fix output dir for boufallolab: output should be all lowercase

* Add back the test/extra_tests variant to host builds

* Fix qpg size reports

* Update unit tests for available outpust now that test variant was added

* Fix expected paths for cc13x2x7 builds

* Fix typo in infineon build

* Fix BL702 path for output - duplicate boufallolab

* Allow rpc modifier for BL706

* Fix unit tests

* Move nltestrunner to esp32 test runner and include in cloudbuild

* Undo duplicate build target addition

* Fix typo: esp32 -> efr32

* Split out fake platform: only tests are really supported there

.github/workflows/examples-ameba.yaml

@@ -47,4 +47,12 @@ jobs:
             - name: Checkout submodules
               run: scripts/checkout_submodules.py --shallow --platform ameba
             - name: Build example Ameba All Clusters App
-              run: scripts/run_in_build_env.sh "./scripts/build/build_examples.py --target-glob 'ameba-*' build"
+              run: |
+                  ./scripts/run_in_build_env.sh \
+                     "./scripts/build/build_examples.py \
+                        --target ameba-amebad-all-clusters \
+                        --target ameba-amebad-all-clusters-minimal \
+                        --target ameba-amebad-light \
+                        --target ameba-amebad-pigweed \
+                        build \
+                     "

.github/workflows/examples-bouffalolab.yaml

@@ -56,17 +56,17 @@ jobs:
         timeout-minutes: 30
         run: |
           ./scripts/run_in_build_env.sh \
-            "./scripts/build/build_examples.py --target bouffalolab-BL706-IoT-DVK-light build"
+            "./scripts/build/build_examples.py --target bouffalolab-bl706-iot-dvk-light build"
           .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py bl702 bl702 lighting-app \
-            out/bouffalolab-BL706-IoT-DVK-light/chip-bl702-lighting-example.out /tmp/bloat_reports/
+            out/bouffalolab-bl706-iot-dvk-light/chip-bl702-lighting-example.out /tmp/bloat_reports/
 
       - name: Build example BL702 Lighting App with RPCs
         timeout-minutes: 30
         run: |
           ./scripts/run_in_build_env.sh \
-            "./scripts/build/build_examples.py --target bouffalolab-BL706-IoT-DVK-light-rpc build"
+            "./scripts/build/build_examples.py --target bouffalolab-bl706-iot-dvk-light-rpc build"
           .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py bl702 bl702+rpc lighting-app \
-            out/bouffalolab-BL706-IoT-DVK-light-rpc/chip-bl702-lighting-example.out /tmp/bloat_reports/
+            out/bouffalolab-bl706-iot-dvk-light-rpc/chip-bl702-lighting-example.out /tmp/bloat_reports/
 
       - name: Uploading Size Reports
         uses: actions/upload-artifact@v2

.github/workflows/examples-cc13x2x7_26x2x7.yaml

@@ -68,7 +68,15 @@ jobs:
               timeout-minutes: 60
               run: |
                   scripts/run_in_build_env.sh "\
-                      ./scripts/build/build_examples.py --target-glob cc13x2x7_26x2x7* build \
+                      ./scripts/build/build_examples.py \
+                      --target cc13x2x7_26x2x7-all-clusters-mtd \
+                      --target cc13x2x7_26x2x7-all-clusters-minimal \
+                      --target cc13x2x7_26x2x7-lock-ftd \
+                      --target cc13x2x7_26x2x7-lock-mtd \
+                      --target cc13x2x7_26x2x7-pump \
+                      --target cc13x2x7_26x2x7-pump-controller \
+                      --target cc13x2x7_26x2x7-shell \
+                      build \
                       --copy-artifacts-to out/artifacts \
                   "
             - name: Get lock FTD size stats
@@ -104,7 +112,7 @@ jobs:
               run: |
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     cc13x2_26x2 LP_CC2652R7 all-clusters-app \
-                    out/artifacts/cc13x2x7_26x2x7-all-clusters/chip-LP_CC2652R7-all-clusters-example.out \
+                    out/artifacts/cc13x2x7_26x2x7-all-clusters-mtd/chip-LP_CC2652R7-all-clusters-example.out \
                     /tmp/bloat_reports/
             - name: Get All Clusters Minimal App size stats
               timeout-minutes: 5

.github/workflows/examples-efr32.yaml

@@ -77,7 +77,12 @@ jobs:
           ./scripts/run_in_build_env.sh \
              "./scripts/build/build_examples.py \
                 --enable-flashbundle \
-                --target-glob '*-brd4187c-{lock,light,switch,window-covering,unit-test}' \
+                --target efr32-brd4187c-window-covering \
+                --target efr32-brd4187c-switch \
+                --target efr32-brd4187c-unit-test \
+                --target efr32-brd4187c-light \
+                --target efr32-brd4187c-light-rpc \
+                --target efr32-brd4187c-lock-rpc \
                 build \
                 --copy-artifacts-to out/artifacts \
              "

.github/workflows/examples-esp32.yaml

@@ -70,7 +70,9 @@ jobs:
                   ./scripts/run_in_build_env.sh \
                      "./scripts/build/build_examples.py \
                         --enable-flashbundle \
-                        --target-glob '*-m5stack-{all-clusters,all-clusters-minimal,all-clusters-rpc-ipv6only}' \
+                        --target esp32-m5stack-all-clusters \
+                        --target esp32-m5stack-all-clusters-minimal \
+                        --target esp32-m5stack-all-clusters-rpc-ipv6only \
                         build \
                         --copy-artifacts-to out/artifacts \
                      "

.github/workflows/examples-infineon.yaml

@@ -126,7 +126,9 @@ jobs:
               run: |
                   ./scripts/run_in_build_env.sh \
                     "./scripts/build/build_examples.py \
-                      --target-glob 'cyw30739-cyw930739m2evb_01-{light,lock,ota-requestor-no-progress-logging}' \
+                      --target cyw30739-cyw930739m2evb_01-light \
+                      --target cyw30739-cyw930739m2evb_01-lock \
+                      --target cyw30739-cyw930739m2evb_01-ota-requestor-no-progress-logging \
                       build \
                       --copy-artifacts-to out/artifacts \
                     "

.github/workflows/examples-k32w.yaml

@@ -71,11 +71,11 @@ jobs:
               run: |
                   scripts/run_in_build_env.sh "\
                       ./scripts/build/build_examples.py \
-                      --target k32w-light-ota-se \
-                      --target k32w-light-release-no-ota \
-                      --target k32w-lock-low-power-release \
-                      --target k32w-contact-low-power-release \
-                      --target k32w-shell-release \
+                      --target k32w-light \
+                      --target k32w-light-no-ota \
+                      --target k32w-lock-low-power-nologs \
+                      --target k32w-contact-low-power-nologs \
+                      --target k32w-shell \
                       build \
                       --copy-artifacts-to out/artifacts \
                   "
@@ -84,21 +84,21 @@ jobs:
               run: |
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     k32w k32w0+release light \
-                    out/artifacts/k32w-light-release-no-ota/chip-k32w0x-light-example \
+                    out/artifacts/k32w-light-no-ota/chip-k32w0x-light-example \
                     /tmp/bloat_reports/
             - name: Get lock size stats
               timeout-minutes: 5
               run: |
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     k32w k32w0+release lock \
-                    out/artifacts/k32w-lock-low-power-release/chip-k32w0x-lock-example \
+                    out/artifacts/k32w-lock-low-power-nologs/chip-k32w0x-lock-example \
                     /tmp/bloat_reports/
             - name: Get contact size stats
               timeout-minutes: 5
               run: |
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     k32w k32w0+release contact \
-                    out/artifacts/k32w-contact-low-power-release/chip-k32w0x-contact-example \
+                    out/artifacts/k32w-contact-low-power-nologs/chip-k32w0x-contact-example \
                     /tmp/bloat_reports/
             - name: Uploading Size Reports
               uses: actions/upload-artifact@v2

.github/workflows/examples-linux-arm.yaml

@@ -69,26 +69,26 @@ jobs:
               run: |
                   ./scripts/run_in_build_env.sh \
                      "./scripts/build/build_examples.py \
-                        --target linux-arm64-clang-all-clusters \
-                        --target linux-arm64-clang-chip-tool-ipv6only \
-                        --target linux-arm64-clang-lock \
-                        --target linux-arm64-clang-minmdns \
-                        --target linux-arm64-clang-thermostat-no-ble \
+                        --target linux-arm64-all-clusters-clang \
+                        --target linux-arm64-chip-tool-ipv6only-clang \
+                        --target linux-arm64-lock-clang \
+                        --target linux-arm64-minmdns-clang \
+                        --target linux-arm64-thermostat-no-ble-clang \
                         build \
                      "
             - name: Bloat report - chip-tool
               timeout-minutes: 5
               run: |
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     linux arm64 chip-tool-ipv6only \
-                    out/linux-arm64-clang-chip-tool-ipv6only/chip-tool \
+                    out/linux-arm64-chip-tool-ipv6only-clang/chip-tool \
                     /tmp/bloat_reports/
             - name: Bloat report - thermostat
               timeout-minutes: 5
               run: |
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     linux arm64 thermostat-no-ble \
-                    out/linux-arm64-clang-thermostat-no-ble/thermostat-app \
+                    out/linux-arm64-thermostat-no-ble-clang/thermostat-app \
                     /tmp/bloat_reports/
             - name: Uploading Size Reports
               uses: actions/upload-artifact@v2

.github/workflows/examples-linux-standalone.yaml

@@ -71,7 +71,11 @@ jobs:
               timeout-minutes: 10
               run: |
                   ./scripts/run_in_build_env.sh \
-                  "./scripts/build/build_examples.py --no-log-timestamps --target-glob '*-chip-cert' build"
+                  "./scripts/build/build_examples.py --no-log-timestamps \
+                      --target linux-x64-chip-cert \
+                      --target linux-arm64-chip-cert \
+                      build \
+                  "
             - name: Build minmdns example with platform dns
               timeout-minutes: 10
               run: |

.github/workflows/examples-qpg.yaml

@@ -72,7 +72,10 @@ jobs:
                   ./scripts/run_in_build_env.sh \
                      "./scripts/build/build_examples.py \
                         --enable-flashbundle \
-                        --target-glob 'qpg-{lock,light,shell,persistent-storage}' \
+                        --target qpg-qpg6105-lock \
+                        --target qpg-qpg6105-light \
+                        --target qpg-qpg6105-shell \
+                        --target qpg-qpg6105-persistent-storage \
                         build \
                         --copy-artifacts-to out/artifacts \
                      "
@@ -85,11 +88,11 @@ jobs:
               run: |
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     qpg qpg6105+debug lighting-app \
-                    out/qpg-light/chip-qpg6105-lighting-example.out \
+                    out/qpg-qpg6105-light/chip-qpg6105-lighting-example.out \
                     /tmp/bloat_reports/
                   .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
                     qpg qpg6105+debug lock-app \
-                    out/qpg-lock/chip-qpg6105-lock-example.out \
+                    out/qpg-qpg6105-lock/chip-qpg6105-lock-example.out \
                     /tmp/bloat_reports/
 
             - name: Uploading Size Reports

.github/workflows/examples-tizen.yaml

@@ -54,7 +54,11 @@ jobs:
                 ./scripts/run_in_build_env.sh \
                     "./scripts/build/build_examples.py \
                         --enable-flashbundle \
-                        --target-glob 'tizen-*' \
+                        --target tizen-arm-all-clusters \
+                        --target tizen-arm-all-clusters-minimal-no-wifi \
+                        --target chip-tool \
+                        --target light \
+                        --target light-no-ble-no-wifi \
                         build \
                         --copy-artifacts-to out/artifacts \
                     "

.github/workflows/full-android.yaml

@@ -111,4 +111,4 @@ jobs:
             # - name: Build Android Studio build (arm64 only)
             #   run: |
             #     ./scripts/run_in_build_env.sh \
-            #       "./scripts/build/build_examples.py --no-log-timestamps --target-glob 'android-androidstudio-arm64-chip-tool' build"
+            #       "./scripts/build/build_examples.py --no-log-timestamps --target android-androidstudio-arm64-chip-tool  build"

.restyled.yaml

@@ -75,6 +75,7 @@ exclude:
     - "src/darwin/Framework/CHIP/zap-generated/MTRBaseClusters.mm" # https://github.com/project-chip/connectedhomeip/issues/20236
     - "src/darwin/Framework/CHIP/zap-generated/MTRBaseClusters.h" # https://github.com/project-chip/connectedhomeip/issues/20236
     - "examples/platform/nxp/k32w/k32w0/scripts/demo_generated_certs/**/*"
+    - "integrations/cloudbuild/*.yaml" # uglier long command line content
 
 
 changed_paths:

.vscode/tasks.json

@@ -13,18 +13,6 @@
                 "base": "$gcc"
             }
         },
-        {
-            "label": "Build examples (glob)",
-            "type": "shell",
-            "command": "scripts/run_in_build_env.sh",
-            "args": [
-                "./scripts/build/build_examples.py --target-glob \"${input:exampleGlob}\" build"
-            ],
-            "group": "build",
-            "problemMatcher": {
-                "base": "$gcc"
-            }
-        },
         {
             "label": "Main Build",
             "type": "shell",

integrations/cloudbuild/build-all.yaml

@@ -27,8 +27,124 @@ steps:
       args:
           - >-
               ./scripts/build/build_examples.py --enable-flashbundle
-              --target-glob '*' --skip-target-glob
-              '{imx-*,tizen-*,*-androidstudio-*,*-tests*,*-chip-test}' build
+              --target ameba-amebad-light
+              --target ameba-amebad-pigweed
+              --target android-arm-chip-tool
+              --target android-arm64-chip-tool
+              --target android-arm64-tv-casting-app
+              --target android-arm64-tv-server
+              --target android-x64-chip-tool
+              --target bl602-light
+              --target bouffalolab-BL706-IoT-DVK-light-rpc
+              --target cc13x2x7_26x2x7-lock-ftd
+              --target cc13x2x7_26x2x7-lock-mtd
+              --target cc13x2x7_26x2x7-shell
+              --target cyw30739-cyw930739m2evb_01-ota-requestor-no-progress-logging
+              --target efr32-brd4161a-light-rpc
+              --target efr32-brd4161a-lock
+              --target efr32-brd4161a-switch
+              --target efr32-brd4161a-window-covering
+              --target efr32-brd4187c-window-covering
+              --target esp32-c3devkit-all-clusters
+              --target esp32-c3devkit-all-clusters-minimal
+              --target esp32-devkitc-all-clusters-ipv6only
+              --target esp32-devkitc-all-clusters-minimal-ipv6only
+              --target esp32-devkitc-bridge
+              --target esp32-devkitc-light-rpc
+              --target esp32-devkitc-lock
+              --target esp32-devkitc-ota-requestor-rpc
+              --target esp32-devkitc-shell
+              --target esp32-m5stack-all-clusters
+              --target esp32-m5stack-all-clusters-ipv6only
+              --target esp32-m5stack-all-clusters-minimal
+              --target esp32-m5stack-all-clusters-minimal-ipv6only
+              --target esp32-m5stack-all-clusters-minimal-rpc
+              --target esp32-m5stack-all-clusters-minimal-rpc-ipv6only
+              --target esp32-m5stack-ota-requestor
+              --target genio-lighting-app
+              --target imx-all-clusters-app
+              --target imx-chip-tool
+              --target imx-thermostat
+              --target infineon-psoc6-all-clusters
+              --target infineon-psoc6-all-clusters-minimal
+              --target infineon-psoc6-light
+              --target infineon-psoc6-lock
+              --target k32w-contact
+              --target k32w-light-no-ota
+              --target k32w-lock
+              --target k32w-shell
+              --target linux-arm64-all-clusters-clang
+              --target linux-arm64-all-clusters-app-nodeps
+              --target linux-arm64-all-clusters-app-nodeps-ipv6only
+              --target linux-arm64-all-clusters-ipv6only-clang
+              --target linux-arm64-all-clusters-minimal-clang
+              --target linux-arm64-all-clusters-minimal-ipv6only-clang
+              --target linux-arm64-bridge-ipv6only-clang
+              --target linux-arm64-chip-tool-clang
+              --target linux-arm64-chip-tool-ipv6only-clang
+              --target linux-arm64-chip-tool-nodeps
+              --target linux-arm64-chip-tool-nodeps-ipv6only
+              --target linux-arm64-dynamic-bridge-ipv6only-clang
+              --target linux-arm64-light-clang
+              --target linux-arm64-light-ipv6only-clang
+              --target linux-arm64-light-rpc-clang
+              --target linux-arm64-light-rpc-ipv6only-clang
+              --target linux-arm64-lock-clang
+              --target linux-arm64-lock-ipv6only-clang
+              --target linux-arm64-minmdns-clang
+              --target linux-arm64-ota-provider-nodeps-ipv6only
+              --target linux-arm64-ota-requestor-nodeps-ipv6only
+              --target linux-arm64-shell-ipv6only-clang
+              --target linux-arm64-thermostat-ipv6only-clang
+              --target linux-arm64-tv-app-ipv6only-clang
+              --target linux-arm64-tv-casting-app-ipv6only-clang
+              --target linux-x64-address-resolve-tool
+              --target linux-x64-all-clusters
+              --target linux-x64-all-clusters-nodeps
+              --target linux-x64-all-clusters-nodeps-ipv6only
+              --target linux-x64-all-clusters-coverage
+              --target linux-x64-all-clusters-ipv6only
+              --target linux-x64-all-clusters-minimal
+              --target linux-x64-all-clusters-minimal-ipv6only
+              --target linux-x64-bridge-ipv6only
+              --target linux-x64-chip-cert
+              --target linux-x64-chip-tool
+              --target linux-x64-chip-tool-coverage
+              --target linux-x64-chip-tool-nodeps-ipv6only
+              --target linux-x64-dynamic-bridge-ipv6only
+              --target linux-x64-efr32-test-runner
+              --target linux-x64-light-rpc-ipv6only
+              --target linux-x64-lock-ipv6only
+              --target linux-x64-minmdns
+              --target linux-x64-nl-test-runner
+              --target linux-x64-ota-provider
+              --target linux-x64-ota-provider-nodeps-ipv6only
+              --target linux-x64-ota-requestor-nodeps-ipv6only
+              --target linux-x64-rpc-console
+              --target linux-x64-shell-ipv6only
+              --target linux-x64-thermostat-ipv6only
+              --target linux-x64-tv-app-ipv6only
+              --target linux-x64-tv-casting-app-ipv6only
+              --target mbed-CY8CPROTO_062_4343W-light-release
+              --target mbed-CY8CPROTO_062_4343W-lock-release
+              --target mbed-CY8CPROTO_062_4343W-pigweed-release
+              --target mw320-all-clusters-app
+              --target nrf-nrf52840dk-all-clusters
+              --target nrf-nrf52840dk-all-clusters-minimal
+              --target nrf-nrf52840dk-light-rpc
+              --target nrf-nrf52840dk-lock
+              --target nrf-nrf52840dk-pump
+              --target nrf-nrf52840dk-pump-controller
+              --target nrf-nrf52840dk-shell
+              --target nrf-nrf52840dongle-light
+              --target nrf-nrf5340dk-all-clusters
+              --target nrf-nrf5340dk-all-clusters-minimal
+              --target nrf-nrf5340dk-shell
+              --target qpg-qpg6105-light
+              --target qpg-qpg6105-shell
+              --target telink-tlsr9518adk80d-light
+              --target tizen-arm-light
+              build
               --create-archives /workspace/artifacts/
       id: CompileAll
       waitFor: