Merge branch 'feature/pytest_build_system' into 'master'

build system: add initial version of pytest-based build system tests

See merge request espressif/esp-idf!19498

.gitlab/CODEOWNERS

@@ -197,6 +197,8 @@
 
 /tools/test_apps/**/*.py             @esp-idf-codeowners/ci @esp-idf-codeowners/tools
 
+/tools/test_build_system/     @esp-idf-codeowners/tools @esp-idf-codeowners/build-config
+
 /tools/unit-test-app/         @esp-idf-codeowners/system @esp-idf-codeowners/tools
 
 # sort-order-reset

.gitlab/ci/build.yml

@@ -573,6 +573,21 @@ test_build_system_spaces:
   variables:
     SHELL_TEST_SCRIPT: test_build_system_spaces.py
 
+pytest_build_system:
+  extends: .test_build_system_template
+  artifacts:
+    paths:
+      - XUNIT_RESULT.xml
+      - test_build_system
+    when: always
+    expire_in: 2 days
+    reports:
+      junit: XUNIT_RESULT.xml
+  script:
+    - ${IDF_PATH}/tools/ci/test_configure_ci_environment.sh
+    - cd ${IDF_PATH}/tools/test_build_system
+    - pytest --work-dir ${CI_PROJECT_DIR}/test_build_system --junitxml=${CI_PROJECT_DIR}/XUNIT_RESULT.xml
+
 build_docker:
   extends:
     - .before_script_minimal

.gitlab/ci/rules.yml

@@ -53,6 +53,7 @@
   - "tools/ci/test_build_system*.sh"
   - "tools/ci/test_build_system*.py"
   - "tools/ci/ci_build_apps.py"
+  - "tools/test_build_system/**/*"
 
 .patterns-custom_test: &patterns-custom_test
   - "components/espcoredump/**/*"

tools/test_build_system/MIGRATION.md

@@ -0,0 +1,107 @@
+# Migration from test_build_system_cmake.sh to pytest
+
+This table tracks migration of tests from [test_build_system_cmake.sh](../ci/test_build_system_cmake.sh) and [test_build_system_spaces.py](../ci/test_build_system_spaces.py) to pytest.
+
+When all tests are migrated to pytest, remove the original tests, corresponding CI jobs, and this file.
+
+Legacy test name | New test name | Comments
+-----------------|---------------|---------
+Initial clean build | test_rebuild::test_rebuild_no_changes |
+Updating component source file rebuilds component | test_rebuild::test_rebuild_source_files |
+Bootloader source file rebuilds bootloader | test_rebuild::test_rebuild_source_files |
+Partition CSV file rebuilds partitions | test_rebuild::test_rebuild_source_files |
+Partial build doesn't compile anything by default | test_rebuild::test_rebuild_no_changes |
+Rebuild when app version was changed | |
+Change app version | |
+Re-building does not change app.bin | |
+Get the version of app from git describe. Project is not inside IDF and do not have a tag only a hash commit. | |
+Get the version of app from Kconfig option | |
+Use IDF version variables in component CMakeLists.txt file | |
+Project is in ESP-IDF which has a custom tag | |
+Moving BUILD_DIR_BASE out of tree | |
+BUILD_DIR_BASE inside default build directory | |
+Can still clean build if all text files are CRLFs | |
+Updating rom ld file should re-link app and bootloader | test_rebuild::test_rebuild_linker |
+Updating app-only ld file should only re-link app | test_rebuild::test_rebuild_linker |
+Updating ld file should only re-link app | test_rebuild::test_rebuild_linker |
+Updating fragment file should only re-link app | test_rebuild::test_rebuild_linker |
+sdkconfig update triggers full recompile | test_rebuild::test_rebuild_source_files |
+Updating project CMakeLists.txt triggers full recompile | test_rebuild::test_rebuild_source_files |
+Can build with Ninja (no idf.py) | |
+Can build with GNU Make (no idf.py) | |
+idf.py can build with Ninja | |
+idf.py can build with Unix Makefiles | |
+Can build with IDF_PATH set via cmake cache not environment | |
+Can build with IDF_PATH unset and inferred by build system | |
+Can build with IDF_PATH unset and inferred by cmake when Kconfig needs it to be set | |
+can build with phy_init_data | |
+can build with ethernet component disabled | |
+Compiler flags on build command line are taken into account | |
+Compiler flags cannot be overwritten | |
+Can override IDF_TARGET from environment | |
+Can set target using idf.py -D | |
+Can set target using -D as subcommand parameter for idf.py | |
+Can set target using idf.py set-target | |
+idf.py understands alternative target names | |
+Can guess target from sdkconfig, if CMakeCache does not exist | |
+Can set the default target using sdkconfig.defaults | |
+IDF_TARGET takes precedence over the value of CONFIG_IDF_TARGET in sdkconfig.defaults | |
+idf.py fails if IDF_TARGET settings don't match in sdkconfig, CMakeCache.txt, and the environment | |
+Setting EXTRA_COMPONENT_DIRS works | |
+Non-existent paths in EXTRA_COMPONENT_DIRS are not allowed | |
+Component names may contain spaces | |
+sdkconfig should have contents of all files: sdkconfig, sdkconfig.defaults, sdkconfig.defaults.IDF_TARGET | |
+Test if it can build the example to run on host | |
+Test build ESP-IDF as a library to a custom CMake projects for all targets | |
+Building a project with CMake library imported and PSRAM workaround, all files compile with workaround | |
+Test for external libraries in custom CMake projects with ESP-IDF components linked | |
+Test for external libraries in custom CMake projects with PSRAM strategy $strat | |
+Cleaning Python bytecode | |
+Displays partition table when executing target partition_table | |
+Make sure a full build never runs '/usr/bin/env python' or similar | |
+Handling deprecated Kconfig options | |
+Handling deprecated Kconfig options in sdkconfig.defaults | |
+Confserver can be invoked by idf.py | |
+Check ccache is used to build | |
+Custom bootloader overrides original | |
+Empty directory not treated as a component | |
+If a component directory is added to COMPONENT_DIRS, its subdirectories are not added | |
+If a component directory is added to COMPONENT_DIRS, its sibling directories are not added | |
+toolchain prefix is set in project description file | |
+Can set options to subcommands: print_filter for monitor | |
+Fail on build time works | |
+Component properties are set | |
+should be able to specify multiple sdkconfig default files | |
+Supports git worktree | |
+idf.py fallback to build system target | |
+Build fails if partitions don't fit in flash | |
+Warning is given if smallest partition is nearly full | |
+Flash size is correctly set in the bootloader image header | |
+DFU build works | |
+UF2 build works | |
+Loadable ELF build works | |
+Defaults set properly for unspecified idf_build_process args | |
+Getting component overriden dir | |
+Overriding Kconfig | |
+Project components prioritized over EXTRA_COMPONENT_DIRS | |
+Components in EXCLUDE_COMPONENTS not passed to idf_component_manager | |
+Create project using idf.py and build it | |
+Create component using idf.py, create project using idf.py. | |
+Add the component to the created project and build the project. | |
+Check that command for creating new project will fail if the target folder is not empty. | |
+Check that command for creating new project will fail if the target path is file. | |
+Check docs command | |
+Deprecation warning check | |
+Save-defconfig checks | |
+test_build | |
+test_build_ulp_fsm | |
+test_build_ulp_riscv | |
+test_spiffsgen | |
+test_flash_encryption | |
+test_secure_boot_v1 | |
+test_secure_boot_v2 | |
+test_app_signing | |
+test_secure_boot_release_mode | |
+test_x509_cert_bundle | |
+test_dfu | |
+test_uf2 | |

tools/test_build_system/README.md

@@ -0,0 +1,166 @@
+# Tools & Build System Tests
+
+This directory contains tests for the build system and build-related tools. These tests are meant to be used both by developers and in CI. Please check the sections below for details on:
+
+- Running the tests locally
+- Debugging test failures
+- Adding new tests
+- Fixtures and helper functions
+
+## Running the tests locally
+
+1. Install pytest using `install.{sh,bat,ps1,fish} --enable-pytest`.
+1. Activate the IDF shell environment using `export.{sh,bat,ps1,fish}`.
+1. To run all the tests, go to `$IDF_PATH/tools/test_build_system` directory, then run:
+    ```
+    pytest
+    ```
+1. To run one specific test, use `-k` flag of pytest, for example
+    ```
+    pytest -k test_compile_commands_json_updated_by_reconfigure
+    ```
+1. To speed up the builds you can install Ccache and set the following environment variables:
+    ```
+    export IDF_CCACHE_ENABLE=1
+    export CCACHE_NOHASHDIR=1
+    ```
+
+## Debugging test failures
+
+If you are working on a bug fix or a feature and one of the tests starts to fail, you should try to reproduce the failure locally.
+
+1. Find the name of the failing test in the CI job log
+1. Follow the steps in the section above to run that one test
+1. By default, the fixtures which create temporary directories will remove them after the test. To prevent the directories from being removed, run `pytest` with `--work-dir /some/path` flag. The temporary directories will be created under `/some/path`, and you will be able to inspect them once the test fails.
+1. You can increase the logging level to see the commands being executed by the test by running `pytest` with `--log-cli-level DEBUG` argument.
+
+## Adding new tests
+
+1. When adding a new test, think of the developer who might have to run this test locally.
+   - Avoid adding tests which take a long time to run. Running the entire test suite should be possible!
+   - Remember that developers run these tests in their IDF work directories. Be careful with destructive actions, especially removing directories recursively. Developers might have untracked files in the directory you are removing! Prefer using the `idf_copy` fixture to make a copy of the IDF directory, when doing some modifications to IDF source. If this is too expensive, do the modification in place but make sure to clean up the changes you perform using a try/finally block. This especially applies to the newly created files.
+1. Read through the test cases and try to find a test which does something similar to what you need to test. This will usually be a good starting point. Also read through the section below, which explains fixtures and utility functions.
+1. The tests need to run on Windows, Linux and macOS. Avoid calling OS-specific programs such as `sed` or `awk` in tests. If you need to perform some complex file modification in the test case, consider writing a Python helper function for that.
+
+## Fixtures and helper functions
+
+If you aren't yet familiar with Pytest fixtures, please take a few moments and read a Pytest tutorial or watch the training, before moving on to the next section.
+
+### `test_app_copy` fixture
+
+This fixture selects the app (inside IDF) to be used by the test and copies this app to a temporary directory, recursively. The working directory is set to the root of the copied app. The directory is removed once the test is finished.
+
+```python
+def test_something(test_app_copy):
+    assert test_app_copy == os.getcwd()
+    # the current working directory now contains the copy of the test app
+```
+
+If the test case doesn't use the `test_app_copy` argument, pylint will typically warn about an unused argument, even if the fixture is actually used. To avoid the warning, use the following pattern:
+```python
+@pytest.mark.usefixtures('test_app_copy')
+def test_something(idf_py):
+    idf_py('build')
+```
+
+By default, the source app is `tools/test_build_system/build_test_app` and the destination directory name is derived from the test case name. (See more about this test app [here](#application-under-test).) This can be overridden using a `@pytest.mark.test_app_copy` decorator, as shown below. The first argument is the path of the source app. The second argument is the name of the temporary directory to create. The second argument is optional, it is mostly useful to test handling of special characters (such as spaces) in the path.
+
+```python
+@pytest.mark.test_app_copy('examples/get-started/blink', 'custom dir name')
+@pytest.mark.usefixtures('test_app_copy')
+def test_something():
+    pass
+```
+
+### `idf_py` fixture
+
+This fixture runs `idf.py` with IDF environment set up.
+
+```python
+@pytest.mark.usefixtures('test_app_copy')
+def test_something(idf_py)
+    # 1. 'test_app_copy' fixture has copied the test app into a temporary directory and
+    #    set the working directory there.
+    # 2. 'idf_py' fixture is a function which calls idf.py:
+    idf_py('fullclean')
+
+    # 3. It accepts multiple arguments and returns a subprocess.CompletedProcess
+    #    instance. It can be used to check the process output.
+    output = idf_py('-DIDF_TARGET=esp32c3', 'reconfigure')
+    assert 'CONFIG_IDF_TARGET="esp32c3"' in Path('sdkconfig').read_text()
+    assert 'Building ESP-IDF components for target esp32c3' in output.stdout
+
+    # 4. Raises subprocess.CalledProcessError on failure
+    with(pytest.raises(subprocess.CalledProcessError)) as exc_info:
+        idf_py('unknown_command')
+    assert 'command "unknown_command" is not known to idf.py' in exc_info.value.stderr
+```
+
+### `default_idf_env` fixture
+
+Returns a dictionary of environment variables required for the IDF build environment. It is similar to the output of `env` command after running the `export` script.
+
+```python
+@pytest.mark.usefixtures('test_app_copy')
+def test_cmake(default_idf_env):
+    with tempfile.TemporaryDirectory() as build_dir:
+        # default_idf_env can be passed to subprocess APIs to run other tools
+        subprocess.run(['cmake', '-B', build_dir, '.'], env=default_idf_env, check=True)
+```
+
+Note, `default_idf_env` sets up the environment based on the `IDF_PATH` environment variable set before launching `pytest`.
+
+### `idf_copy` fixture
+
+Copies IDF from `IDF_PATH` into a new temporary directory. `@pytest.mark.idf_copy('name prefix')` can be used to specify the name prefix of the temporary directory.
+
+For the duration of the test, `IDF_PATH` environment variable is set to the newly created copy.
+
+```python
+@pytest.mark.usefixtures('test_app_copy')
+def test_idf_copy(idf_copy):
+    # idf_copy is the temporary IDF copy.
+    # For example, we can check if idf.py build can work without the .git directory:
+    shutil.rmtree(os.path.join(idf_copy, '.git'), ignore_errors=True)
+    # Note that we can't use idf_py fixture, since it uses the default IDF path.
+    # We can use 'get_idf_build_env' with 'run_idf_py', instead:
+    env = get_idf_build_env(idf_copy)
+    run_idf_py('build', env=env)
+```
+
+### Build snapshots
+
+`get_snapshot(list_of_globs)` function takes a list of glob expressions, finds the files matching these expressions, and returns a `Snapshot` instance. `Snapshot` instances record file names and their modification timestamps. Two `Snapshot` instances can be compared using `assert_same` and `assert_different` methods:
+
+```python
+@pytest.mark.usefixtures('test_app_copy')
+def test_build_jsons_updated_by_reconfigure(idf_py):
+    globs = ['build/*.json']
+
+    idf_py('reconfigure')
+    snapshot_1 = get_snapshot(globs)
+    snapshot_2 = get_snapshot(globs)
+    snapshot_2.assert_same(snapshot_1)
+
+    idf_py('reconfigure')
+    snapshot_3 = get_snapshot(globs)
+    snapshot_3.assert_different(snapshot_2)
+```
+
+### Helper functions for file modifications
+
+A few extra functions are provided to make simple file modifications:
+
+* `append_to_file(filename: typing.Union[str, Path], what: str) -> None` — appends the given string to a file.
+* `replace_in_file(filename: typing.Union[str, Path], search: str, replace: str) -> None` — searches the file for occurrences of the string `search` and replaces all of them with `replace`, then writes the result back to the file.
+
+### Application under test
+
+Most build system tests should use the included [`build_test_app`](build_test_app/), if possible.
+
+Using other test apps and examples for the purpose of testing the build system is okay as long as you keep the following in mind:
+
+* Don't use the build system tests to compile examples or test apps under a particular combination of sdkconfig options. Use the `sdkconfig.ci.*` files for that, instead.
+* Examples or test apps may be changed, renamed or removed. If you add a dependency on another example or a test app, your test case might need to be rewritten if someone has to modify or remove the example or a test app your test case depends on.
+
+For convenience, the `build_test_app` app included here provides several placeholders which can be modified using the `replace_in_file` function. You can find the placeholders by running `grep -r placeholder_ build_test_app`.

tools/test_build_system/build_test_app/CMakeLists.txt

@@ -0,0 +1,9 @@
+cmake_minimum_required(VERSION 3.16)
+
+# placeholder_before_include_project_cmake
+
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+
+# placeholder_after_include_project_cmake
+
+project(build_test_app)

tools/test_build_system/build_test_app/README.md

@@ -0,0 +1 @@
+Information about this test app can be found [here](../README.md#application-under-test).

tools/test_build_system/build_test_app/main/CMakeLists.txt

@@ -0,0 +1,5 @@
+# placeholder_before_idf_component_register
+
+idf_component_register(SRCS "build_test_app.c"
+                       # placeholder_inside_idf_component_register
+)

tools/test_build_system/build_test_app/main/build_test_app.c

@@ -0,0 +1,12 @@
+/*
+ * SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
+ *
+ * SPDX-License-Identifier: Apache-2.0
+ */
+#include <stdio.h>
+// placeholder_before_main
+
+void app_main(void)
+{
+    // placeholder_inside_main
+}