Use zap-cli for zap codegen instead of using third_party/zap/repo (#…

…23664) * Switch zap workflows to use chip-build image * Remove zap third party submodule: we are using zap releases now * Switch "node zap-generate.js" with "zap-cli generate" * Undo openthread repo checkout * Remove reference to manufacturers.xml from zap repo * Remove references to third_party/zap * Add instructions to install zap from releases * Fix chip build version 09 is broken, need 10 after #24662 * Also add chip authors to manufacturers.xml copyright * update to image 0.6.11 so that we have npm in the image as well (should make javascript github actions work again) * Fix cmdargs arguments: run bootstrap not applicable anymore * Add safe.directory for git otherwise git operations fail. checkout_submodules used to do this, but without them we need a separate set * Provide options on how to execute zap using env variables * Update scripts to add back run_zaptool and zap_bootstrap.sh * Restyle and updated zaptool to be friendly with restyler * Update documentation to explain available environment variables * Restyle * Update docs/guides/BUILDING.md Co-authored-by: Boris Zbarsky <[email protected]> * Update docs/guides/BUILDING.md Co-authored-by: Boris Zbarsky <[email protected]> * Add back run_bootstrap command line option * Restyle * Fix cmdline args * Fix typo * Make sure zap convert is also aware of environment variables * Update convert.py workind directory. tested with local zap * Add back run_bootstrap to regen_all and convert_all * Restyle * Spelling fix * Update run_zaptool to support install for mac * Update zap argument logic * Restyle Co-authored-by: Boris Zbarsky <[email protected]>
project-chip · Jul 14, 2023 · 4672961 · 4672961
1 parent d405f08
commit 4672961
Show file tree

Hide file tree

Showing 17 changed files with 885 additions and 106 deletions.
diff --git a/.github/workflows/zap_regeneration.yaml b/.github/workflows/zap_regeneration.yaml
@@ -28,7 +28,7 @@ jobs:
 
         runs-on: ubuntu-20.04
         container:
-            image: connectedhomeip/chip-build-zap:0.6.06
+            image: connectedhomeip/chip-build:0.6.11
         defaults:
             run:
                 shell: sh
@@ -44,17 +44,11 @@ jobs:
                       token: ${{ github.token }}
                   attempt_limit: 3
                   attempt_delay: 2000
-            - name: Setup ZAP
-              timeout-minutes: 5
-              run: |
-                  cd third_party/zap/repo/
-                  npm ci
-                  npm run version-stamp
-                  npm rebuild canvas --update-binary
-                  npm run build-spa
             - name: Generate all
               timeout-minutes: 5
               run: scripts/tools/zap_regen_all.py
+            - name: Ensure git works in current working directory
+              run: git config --global --add safe.directory `pwd`
             - name: Add uncommitted changes
               run: git add .
             - name: Fix upstream

diff --git a/.github/workflows/zap_templates.yaml b/.github/workflows/zap_templates.yaml
@@ -29,7 +29,7 @@ jobs:
 
         runs-on: ubuntu-20.04
         container:
-            image: connectedhomeip/chip-build-zap:0.6.06
+            image: connectedhomeip/chip-build:0.6.11
         defaults:
             run:
                 shell: sh
@@ -44,18 +44,10 @@ jobs:
                       token: ${{ github.token }}
                   attempt_limit: 3
                   attempt_delay: 2000
-            - name: Checkout submodules
-              run: scripts/checkout_submodules.py --shallow --platform linux
-            - name: Setup ZAP
-              timeout-minutes: 10
-              run: |
-                  cd third_party/zap/repo/
-                  npm ci
-                  npm run version-stamp
-                  npm rebuild canvas --update-binary
-                  npm run build-spa
             - name: Generate all
               run: scripts/tools/zap_regen_all.py
+            - name: Ensure git works in current working directory
+              run: git config --global --add safe.directory `pwd`
             - name: Check for uncommited changes
               run: |
                   git add .

diff --git a/.gitmodules b/.gitmodules
@@ -54,10 +54,6 @@
 	url = https://github.com/Qorvo/QMatter
 	branch  = vlatest-libs
 	platforms = qpg
-[submodule "zap"]
-	path = third_party/zap/repo
-	url = https://github.com/project-chip/zap.git
-	branch  = master
 [submodule "freertos"]
 	path = third_party/freertos/repo
 	url = https://github.com/FreeRTOS/FreeRTOS-Kernel.git

diff --git a/docs/guides/BUILDING.md b/docs/guides/BUILDING.md
@@ -128,6 +128,35 @@ update_config=1
 
 Finally, reboot your RPi.
 
+## Installing ZAP
+
+`zap-cli` is already installed in pre-built docker images for chip-build, such
+as
+[chip-build-vscode](https://hub.docker.com/r/connectedhomeip/chip-build-vscode).
+
+Zap generation and tooling relies on `zap-cli` being available on the current
+system. You can install it from the zap project
+[Releases](https://github.com/project-chip/zap/releases).
+
+You should install a compatible release version, generally checking against the
+release set in
+[integrations/docker/images/chip-build/Dockerfile](../../../integrations/docker/images/chip-build/Dockerfile).
+
+On linux, installation from `zap-linux.zip` is recommended as it pulls fewer
+dependencies than the `.deb` package.
+
+### Which ZAP to use
+
+ZAP scripting uses the following detection, in order:
+
+-   `$ZAP_DEVELOPMENT_PATH` to point to a zap checkout. Use this if you are
+    developing zap locally and would like to run zap with your changes
+
+-   `$ZAP_INSTALL_PATH` to point to where `zap-linux.zip`/`zap-mac.zip` was
+    unpacked. This allows you to not need to place zap/zap-cli in `$PATH`
+
+-   Otherwise scripts assume `zap-cli` or `zap` is available in `$PATH`
+
 ## Prepare for building
 
 Before running any other build command, the `scripts/activate.sh` environment

diff --git a/examples/chef/chef.py b/examples/chef/chef.py
@@ -299,8 +299,6 @@ def main() -> int:
                       action="store_true", dest="do_interact")
     parser.add_option("-m", "--menuconfig", help="runs menuconfig on platforms that support it",
                       action="store_true", dest="do_menuconfig")
-    parser.add_option("", "--bootstrap_zap", help="installs zap dependencies",
-                      action="store_true", dest="do_bootstrap_zap")
     parser.add_option("-z", "--zap", help="runs zap to generate data model & interaction model artifacts",
                       action="store_true", dest="do_run_zap")
     parser.add_option("-g", "--zapgui", help="runs zap GUI display to allow editing of data model",
@@ -354,29 +352,6 @@ def main() -> int:
 
     splash()
 
-    #
-    # ZAP bootstrapping
-    #
-
-    if options.do_bootstrap_zap:
-        if sys.platform == "linux" or sys.platform == "linux2":
-            flush_print("Installing ZAP OS package dependencies")
-            install_deps_cmd = """\
-            sudo apt-get install nodejs node-yargs npm
-            libpixman-1-dev libcairo2-dev libpango1.0-dev node-pre-gyp
-            libjpeg9-dev libgif-dev node-typescript"""
-            shell.run_cmd(unwrap_cmd(install_deps_cmd))
-        if sys.platform == "darwin":
-            flush_print(
-                "Installation of ZAP OS packages not supported on MacOS")
-        if sys.platform == "win32":
-            flush_print(
-                "Installation of ZAP OS packages not supported on Windows")
-
-        flush_print("Running NPM to install ZAP Node.JS dependencies")
-        shell.run_cmd(
-            f"cd {_REPO_BASE_PATH}/third_party/zap/repo/ && npm install")
-
     #
     # CI
     #

diff --git a/integrations/docker/images/chip-cert-bins/Dockerfile b/integrations/docker/images/chip-cert-bins/Dockerfile
@@ -251,7 +251,6 @@ RUN case ${TARGETPLATFORM} in \
         *) ;; \
     esac
 
-RUN npm --prefix third_party/zap/repo/ ci
 RUN scripts/examples/gn_build_test_example.sh app1
 
 RUN source scripts/activate.sh && scripts/build_python.sh -m platform -d true -i no

diff --git a/scripts/helpers/pull_upstream_and_regenerate_zap.sh b/scripts/helpers/pull_upstream_and_regenerate_zap.sh
@@ -2,18 +2,9 @@
 set -x
 
 git pull upstream master
-git submodule update --init --recursive third_party/zap/
 
 git status
 
-cd third_party/zap/repo/
-npm ci
-npm run version-stamp
-npm rebuild canvas --update-binary
-npm run build-spa
-
-cd ../../../
-
 scripts/tools/zap_regen_all.py
 
 git status

diff --git a/scripts/tools/zap/convert.py b/scripts/tools/zap/convert.py
@@ -92,21 +92,46 @@ def runConversion(zap_file):
     templates_file = getFilePath('src/app/zap-templates/app-templates.json')
     zcl_file = detectZclFile(zap_file)
 
-    generator_dir = getDirPath('third_party/zap/repo')
-    os.chdir(generator_dir)
-    subprocess.check_call(['node', './src-script/zap-convert.js',
-                          '-z', zcl_file, '-g', templates_file, '-o', zap_file, zap_file])
-
-
-def runBootstrap():
-    subprocess.check_call(getFilePath("scripts/tools/zap/zap_bootstrap.sh"), shell=True)
+    # Accepted environment variables, in order:
+    #
+    # ZAP_DEVELOPMENT_PATH - the path to a zap development environment. This is
+    #                        a zap checkout, used for local development
+    # ZAP_INSTALL_PATH     - the path where zap-cli exists. This is if zap-cli
+    #                        is NOT in the current path
+
+    if 'ZAP_DEVELOPMENT_PATH' in os.environ:
+        convert_cmd = ['node', 'src-script/zap-start.js', 'convert']
+        working_directory = os.environ['ZAP_DEVELOPMENT_PATH']
+    elif 'ZAP_INSTALL_PATH' in os.environ:
+        convert_cmd = [os.path.join(os.environ['ZAP_INSTALL_PATH'], 'zap-cli'), 'convert']
+        working_directory = None
+    else:
+        convert_cmd = ['zap-cli', 'convert']
+        working_directory = None
+
+    try:
+        subprocess.check_call(convert_cmd + ['-z', zcl_file, '-g', templates_file, '-o', zap_file, zap_file], cwd=working_directory)
+    except FileNotFoundError as e:
+        print(f'FAILED TO EXECUTE ZAP CONVERSION: {e.strerror} - "{e.filename}"')
+        print('*'*80)
+        print('* You may need to install zap. Please ensure one of these applies:')
+        print('* - `zap-cli` is in $PATH. Install from https://github.com/project-chip/zap/releases')
+        print('*   see docs/guides/BUILDING.md for details')
+        print('* - `zap-cli` is in $ZAP_INSTALL_PATH. Use this option if you')
+        print('*   installed zap but do not want to update $PATH')
+        print('* - Point $ZAP_DEVELOPMENT_PATH to your local copy of zap that you')
+        print('*   develop on (to use a developer build of zap)')
+        print('*'*80)
+        sys.exit(1)
 
 
 def main():
     checkPythonVersion()
     zap_file, run_bootstrap = runArgumentsParser()
+
     if run_bootstrap:
-        runBootstrap()
+        subprocess.check_call(getFilePath("scripts/tools/zap/zap_bootstrap.sh"), shell=True)
+
     os.chdir(CHIP_ROOT_DIR)
 
     runConversion(zap_file)

diff --git a/scripts/tools/zap/generate.py b/scripts/tools/zap/generate.py
@@ -149,10 +149,38 @@ def extractGeneratedIdl(output_dir, zap_config_path):
 
 
 def runGeneration(zap_file, zcl_file, templates_file, output_dir):
-    generator_dir = os.getenv('ZAP_PATH', getDirPath('third_party/zap/repo'))
-    os.chdir(generator_dir)
-    subprocess.check_call(['node', './src-script/zap-generate.js', '-z',
-                          zcl_file, '-g', templates_file, '-i', zap_file, '-o', output_dir])
+    # Accepted environment variables, in order:
+    #
+    # ZAP_DEVELOPMENT_PATH - the path to a zap development environment. This is
+    #                        a zap checkout, used for local development
+    # ZAP_INSTALL_PATH     - the path where zap-cli exists. This is if zap-cli
+    #                        is NOT in the current path
+
+    if 'ZAP_DEVELOPMENT_PATH' in os.environ:
+        generate_cmd = ['node', 'src-script/zap-start.js', 'generate']
+        working_directory = os.environ['ZAP_DEVELOPMENT_PATH']
+    elif 'ZAP_INSTALL_PATH' in os.environ:
+        generate_cmd = [os.path.join(os.environ['ZAP_INSTALL_PATH'], 'zap-cli'), 'generate']
+        working_directory = None
+    else:
+        generate_cmd = ['zap-cli', 'generate']
+        working_directory = None
+
+    try:
+        subprocess.check_call(generate_cmd + ['-z', zcl_file, '-g', templates_file,
+                              '-i', zap_file, '-o', output_dir], cwd=working_directory)
+    except FileNotFoundError as e:
+        print(f'FAILED TO EXECUTE ZAP GENERATION: {e.strerror} - "{e.filename}"')
+        print('*'*80)
+        print('* You may need to install zap. Please ensure one of these applies:')
+        print('* - `zap-cli` is in $PATH. Install from https://github.com/project-chip/zap/releases')
+        print('*   see docs/guides/BUILDING.md for details')
+        print('* - `zap-cli` is in $ZAP_INSTALL_PATH. Use this option if you')
+        print('*   installed zap but do not want to update $PATH')
+        print('* - Point $ZAP_DEVELOPMENT_PATH to your local copy of zap that you')
+        print('*   develop on (to use a developer build of zap)')
+        print('*'*80)
+        sys.exit(1)
 
     extractGeneratedIdl(output_dir, zap_file)
 
@@ -222,15 +250,13 @@ def runJavaPrettifier(templates_file, output_dir):
         print('google-java-format error:', err)
 
 
-def runBootstrap():
-    subprocess.check_call(getFilePath("scripts/tools/zap/zap_bootstrap.sh"), shell=True)
-
-
 def main():
     checkPythonVersion()
     cmdLineArgs = runArgumentsParser()
+
     if cmdLineArgs.runBootstrap:
-        runBootstrap()
+        subprocess.check_call(getFilePath("scripts/tools/zap/zap_bootstrap.sh"), shell=True)
+
     # The maximum memory usage is over 4GB (#15620)
     os.environ["NODE_OPTIONS"] = "--max-old-space-size=8192"
     runGeneration(cmdLineArgs.zapFile, cmdLineArgs.zclFile, cmdLineArgs.templateFile, cmdLineArgs.outputDir)

diff --git a/scripts/tools/zap/run_zaptool.sh b/scripts/tools/zap/run_zaptool.sh
@@ -29,24 +29,41 @@ set -e
 
 SCRIPT_PATH="$(_get_fullpath "$0")"
 CHIP_ROOT="${SCRIPT_PATH%/scripts/tools/zap/run_zaptool.sh}"
-[[ -n "$1" ]] && ZAP_ARGS=(-i "$(_get_fullpath "$1")") || ZAP_ARGS=()
+[[ -n "$1" ]] && ZAP_ARGS="-i \"$(_get_fullpath "$1")\"" || ZAP_ARGS=""
 
-(
+if [ ! -z "$ZAP_DEVELOPMENT_PATH" ]; then
+    WORKING_DIR=$ZAP_DEVELOPMENT_PATH
+    ZAP_CMD="node src-script/zap-start.js"
 
     "$CHIP_ROOT"/scripts/tools/zap/zap_bootstrap.sh
+elif [ ! -z "$ZAP_INSTALL_PATH" ]; then
+    if [[ "$OSTYPE" == "darwin"* ]]; then
+        ZAP_CMD="$ZAP_INSTALL_PATH/zap.app/Contents/MacOS/zap"
+    else
+        ZAP_CMD="$ZAP_INSTALL_PATH/zap"
+    fi
+    WORKING_DIR="$CHIP_ROOT"
+else
+    ZAP_CMD="zap"
+    WORKING_DIR="$CHIP_ROOT"
+fi
 
-    cd "$CHIP_ROOT/third_party/zap/repo"
+(
+    cd "$WORKING_DIR"
 
-    echo "ARGS: ${ZAP_ARGS[@]}"
+    echo "ARGS: $ZAP_ARGS"
 
     if [[ "${ZAP_ARGS[@]}" == *"/all-clusters-app.zap"* ]]; then
         ZCL_FILE="$CHIP_ROOT/src/app/zap-templates/zcl/zcl-with-test-extensions.json"
     else
         ZCL_FILE="$CHIP_ROOT/src/app/zap-templates/zcl/zcl.json"
     fi
 
-    node src-script/zap-start.js --logToStdout \
-        --gen "$CHIP_ROOT/src/app/zap-templates/app-templates.json" \
-        --zcl "$ZCL_FILE" \
-        "${ZAP_ARGS[@]}"
+    bash -c " \
+        $ZAP_CMD \
+        --logToStdout \
+        --gen \"$CHIP_ROOT/src/app/zap-templates/app-templates.json\" \
+        --zcl \"$ZCL_FILE\" \
+        $ZAP_ARGS \
+    "
 )
diff --git a/scripts/tools/zap/zap_bootstrap.sh b/scripts/tools/zap/zap_bootstrap.sh
@@ -32,14 +32,13 @@ EOF
 
 set -e
 
-SCRIPT_PATH="$(_get_fullpath "$0")"
-CHIP_ROOT="${SCRIPT_PATH%/scripts/tools/zap/zap_bootstrap.sh}"
+if [ -z "$ZAP_DEVELOPMENT_PATH" ]; then
+    echo 'Please set $ZAP_DEVELOPMENT_PATH to your zap checkout'
+    exit 1
+fi
 
 (
-    cd "$CHIP_ROOT" &&
-        git submodule update --init third_party/zap/repo
-
-    cd "third_party/zap/repo"
+    cd "$ZAP_DEVELOPMENT_PATH"
 
     if [ $# -eq 0 ]; then
         echo "Running ZAP bootstrap"

diff --git a/scripts/tools/zap_convert_all.py b/scripts/tools/zap_convert_all.py
@@ -62,15 +62,13 @@ def runArgumentsParser():
     return parser.parse_args()
 
 
-def runBootstrap():
-    subprocess.check_call(os.path.join(CHIP_ROOT_DIR, "scripts/tools/zap/zap_bootstrap.sh"), shell=True)
-
-
 def main():
     args = runArgumentsParser()
     checkPythonVersion()
+
     if args.run_bootstrap:
-        runBootstrap()
+        subprocess.check_call(os.path.join(CHIP_ROOT_DIR, "scripts/tools/zap/zap_bootstrap.sh"), shell=True)
+
     os.chdir(CHIP_ROOT_DIR)
 
     targets = getTargets()
-Original file line number
+Diff line change
@@ Expand Up / @@ -251,7 +251,6 @@ RUN case ${TARGETPLATFORM} in \ @@
             *) ;; \
         esac
-    RUN npm --prefix third_party/zap/repo/ ci
     RUN scripts/examples/gn_build_test_example.sh app1
     RUN source scripts/activate.sh && scripts/build_python.sh -m platform -d true -i no
@@ Expand Down @@