Skip to content

Commit

Permalink
Tools: Allow custom Python installation path with IDF_PYTHON_ENV_PATH
Browse files Browse the repository at this point in the history
IDF_PYTHON_ENV_PATH is the path where the Python environment is created
and used. By default it is inside IDF_TOOLS_PATH. IDF_PYTHON_ENV_PATH
was exported by idf_tools.py but was not imported back. This fixes the
issue and ESP-IDF will honor the value of IDF_PYTHON_ENV_PATH.

Closes #10489
  • Loading branch information
dobairoland committed Feb 15, 2023
1 parent 885e501 commit 1feccdc
Show file tree
Hide file tree
Showing 3 changed files with 26 additions and 5 deletions.
4 changes: 2 additions & 2 deletions docs/en/api-guides/tools/idf-tools.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Inside ``IDF_TOOLS_PATH``, the scripts performing tools installation create the
- ``dist`` — where the archives of the tools are downloaded.
- ``tools`` — where the tools are extracted. The tools are extracted into subdirectories: ``tools/TOOL_NAME/VERSION/``. This arrangement allows different versions of tools to be installed side by side.
- ``idf-env.json`` — user install options (targets, features) are stored in this file. Targets are selected chip targets for which tools are installed and kept up-to-date. Features determine the Python package set which should be installed. These options will be discussed later.
- ``python_env`` — not tools related; virtual Python environments are installed in the sub-directories.
- ``python_env`` — not tools related; virtual Python environments are installed in the sub-directories. Note that the Python environment directory can be placed elsewhere by setting the ``IDF_PYTHON_ENV_PATH`` environment variable.
- ``espidf.constraints.*.txt`` — one constraint file for each ESP-IDF release containing Python package version requirements.

GitHub Assets Mirror
Expand Down Expand Up @@ -109,7 +109,7 @@ Any mirror server can be used provided the URL matches the ``github.com`` downlo

* ``check``: For each tool, checks whether the tool is available in the system path and in ``IDF_TOOLS_PATH``.

* ``install-python-env``: Create a Python virtual environment in the ``${IDF_TOOLS_PATH}/python_env`` directory and install there the required Python packages. An optional ``--features`` argument allows one to specify a comma-separated list of features to be added or removed. Feature that begins with ``-`` will be removed and features with ``+`` or without any sign will be added. Example syntax for removing feature ``XY`` is ``--features=-XY`` and for adding ``--features=+XY`` or ``--features=XY``. If both removing and adding options are provided with the same feature, no operation is performed. For each feature a requirements file must exist. For example, feature ``XY`` is a valid feature if ``${IDF_PATH}/tools/requirements/requirements.XY.txt`` is an existing file with a list of Python packages to be installed. There is one mandatory ``core`` feature ensuring core functionality of ESP-IDF (build, flash, monitor, debug in console). There can be an arbitrary number of optional features. The selected list of features is stored in ``idf-env.json``. The requirement files contain a list of the desired Python packages to be installed and ``espidf.constraints.*.txt`` downloaded from https://dl.espressif.com and stored in ``${IDF_TOOLS_PATH}`` the package version requirements for a given ESP-IDF version. Althought it is not recommended, the download and use of constraint files can be disabled with the ``--no-constraints`` argument or setting the ``IDF_PYTHON_CHECK_CONSTRAINTS`` environment variable to ``no``.
* ``install-python-env``: Create a Python virtual environment in the ``${IDF_TOOLS_PATH}/python_env`` directory (or directly in the directory set by the ``IDF_PYTHON_ENV_PATH`` environment variable) and install there the required Python packages. An optional ``--features`` argument allows one to specify a comma-separated list of features to be added or removed. Feature that begins with ``-`` will be removed and features with ``+`` or without any sign will be added. Example syntax for removing feature ``XY`` is ``--features=-XY`` and for adding ``--features=+XY`` or ``--features=XY``. If both removing and adding options are provided with the same feature, no operation is performed. For each feature a requirements file must exist. For example, feature ``XY`` is a valid feature if ``${IDF_PATH}/tools/requirements/requirements.XY.txt`` is an existing file with a list of Python packages to be installed. There is one mandatory ``core`` feature ensuring core functionality of ESP-IDF (build, flash, monitor, debug in console). There can be an arbitrary number of optional features. The selected list of features is stored in ``idf-env.json``. The requirement files contain a list of the desired Python packages to be installed and ``espidf.constraints.*.txt`` downloaded from https://dl.espressif.com and stored in ``${IDF_TOOLS_PATH}`` the package version requirements for a given ESP-IDF version. Althought it is not recommended, the download and use of constraint files can be disabled with the ``--no-constraints`` argument or setting the ``IDF_PYTHON_CHECK_CONSTRAINTS`` environment variable to ``no``.

* ``check-python-dependencies``: Checks if all required Python packages are installed. Packages from ``${IDF_PATH}/tools/requirements/requirements.*.txt`` files selected by the feature list of ``idf-env.json`` are checked with the package versions specified in the ``espidf.constraints.*.txt`` file. The constraint file is downloaded with ``install-python-env`` command. The use of constraints files can be disabled similarly to the ``install-python-env`` command.

Expand Down
4 changes: 2 additions & 2 deletions tools/idf_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -1332,8 +1332,8 @@ def get_python_env_path() -> Tuple[str, str, str, str]:
python_ver_major_minor = '{}.{}'.format(sys.version_info.major, sys.version_info.minor)

idf_version = get_idf_version()
idf_python_env_path = os.path.join(global_idf_tools_path or '', 'python_env',
'idf{}_py{}_env'.format(idf_version, python_ver_major_minor))
idf_python_env_path = os.getenv('IDF_PYTHON_ENV_PATH') or os.path.join(global_idf_tools_path or '', 'python_env',
'idf{}_py{}_env'.format(idf_version, python_ver_major_minor))

python_exe, subdir = get_python_exe_and_subdir()
idf_python_export_path = os.path.join(idf_python_env_path, subdir)
Expand Down
23 changes: 22 additions & 1 deletion tools/test_idf_tools/test_idf_tools_python_env.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# SPDX-FileCopyrightText: 2022 Espressif Systems (Shanghai) CO LTD
# SPDX-FileCopyrightText: 2022-2023 Espressif Systems (Shanghai) CO LTD
# SPDX-License-Identifier: Apache-2.0

import os
import shutil
import subprocess
import sys
import tempfile
import unittest
from typing import List

Expand Down Expand Up @@ -74,5 +75,25 @@ def test_no_constraints(self): # type: () -> None
self.assertIn(REQ_CORE, output)


class TestCustomPythonPathInstall(TestPythonInstall):

def setUp(self): # type: () -> None
self.CUSTOM_PYTHON_DIR = tempfile.mkdtemp()
self.addCleanup(shutil.rmtree, self.CUSTOM_PYTHON_DIR)
os.environ['IDF_PYTHON_ENV_PATH'] = self.CUSTOM_PYTHON_DIR

def test_default_arguments(self): # type: () -> None
output = self.run_idf_tools(['check-python-dependencies'])
self.assertIn(f"{self.CUSTOM_PYTHON_DIR}/bin/python doesn't exist", output)
self.assertNotIn(PYTHON_DIR, output)

output = self.run_idf_tools(['install-python-env'])
self.assertIn(self.CUSTOM_PYTHON_DIR, output)
self.assertNotIn(PYTHON_DIR, output)

output = self.run_idf_tools(['check-python-dependencies'])
self.assertIn(self.CUSTOM_PYTHON_DIR, output)


if __name__ == '__main__':
unittest.main()

0 comments on commit 1feccdc

Please sign in to comment.