Builds no longer respect IDF_PYTHON_ENV_PATH (IDFGH-9085)

[ghost]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.0

Operating System used.

Windows

How did you build your project?

Eclipse IDE

If you are using Windows, please specify command line type.

CMD

What is the expected behavior?

Builds with manual IDF_PYTHON_ENV_PATH succeeding in v5.0 like in v4.4.3

What is the actual behavior?

Build fails with message like "python not found in HOME/python_env".

Steps to reproduce.

if starting with working setup:

• rename python_env to something else
• set environment IDF_PYTHON_ENV_PATH to actual environment inside the renamed folder
• run build

Build or installation Logs.

No response

More Information.

The problem lies in build.cmake:
v5.0 (line 338)
execute_process(COMMAND "${python}" "${idf_path}/tools/idf_tools.py" "check-python-dependencies"
v4.4.3 (line 283)
execute_process(COMMAND "${python}" "${idf_path}/tools/check_python_dependencies.py"

idf_tools.py's check-python-dependencies ignores the IDF_PYTHON_ENV_PATH.

After changing the line to v4.4.3 revision and (just to prove that's the rootcause) circumvention of the new interface_version params by removing
build.cmake
507 idf_build_get_property(component_manager_interface_version __COMPONENT_MANAGER_INTERFACE_VERSION)
512 "--interface_version=${component_manager_interface_version}"

component.cmake
240 "--interface_version=${component_manager_interface_version}"

it works as expected.

[igrr]

@Alim-Oezdemir IDF_PYTHON_ENV_PATH is just an internal variable set by export.sh or export.ps1. Did you find anything in the docs that would suggest that changing this variable manually is supported? We do expect that IDF_PYTHON_ENV_PATH will always match the location of the current Python interpreter (as shown in the output of which python command).

If you need to relocate the python environment, you need to set IDF_TOOLS_PATH variable globally (e.g. in your profile script). Then run install.sh and . ./export.sh again.

[ghost]

@igrr I have custom installs on my machines to prevent the download/install stuff not needed (cmake, ninja, ccache installed globally already; s2/s3/risc/llvm binaries not applicable, PATH, IDF_PATH and IDF_PYTHON_ENV_PATH set manually). Hence IDF_TOOLS_PATH,install, export weren't necessary.
I can't recall where I got the IDF_PYTHON_ENV_PATH from, I thought I followed the "custom toolchain" instructions back
in IDF 2/3 where the MSYS-process was still default. Read through current and old documentation and no reference to it.

[ghost]

Found it, it was in the docs of the eclipse plugin:
https://github.com/espressif/idf-eclipse-plugin#configuring-environment-variables

Required environment variables:
...
IDF_PYTHON_ENV_PATH

[dobairoland]

The Python system was changed in ESP-IDF v5.0. We consider IDF_PYTHON_ENV_PATH to be our tool's internal and we don't support changing the Python path through it.

If you need then you can change IDF_TOOLS_PATH which would change your Python path as well. I hope this is sufficient for you.

[ghost]

That enforces the use of install/export mechanism with it's versioned paths like "tools\cmake\3.24.0" and "python_env\idf5.0_py3.10_env". Which is helpful for easy one-time setup, yet cumbersome on developer machines with custom needs.

[igrr]

@Alim-Oezdemir as @dobairoland said, it is possible to have a different location of the python interpreter. The problem you are running into is happening because the IDF_PYTHON_ENV_PATH doesn't point to the same location as the python interpreter found in PATH. Just make sure you add the custom virtual environment directory at the front of the PATH variable and set IDF_PYTHON_ENV_PATH accordingly. You can debug this by printing the environment variables and checking the location of python using which python. I use a non-standard location of the python environment myself and it works fine, even in master.

[ghost]

@igrr, just switched back to verify your ideas. Doesn't work for me, python_env holds the environment, yet build complains:

ERROR: ...\python_env\idf5.0_py3.10_env\Scripts\python.exe doesn't exist! Please run the install script or "idf_tools.py install-python-env" in order to create it

Because the idf_tools.py in contrast to former check_python_dependencies.py expects subfolder with version (e.g. idf5.0_py3.10_env" inside it.

[igrr]

@Alim-Oezdemir could you please show the values of PATH and IDF_PYTHON_ENV_PATH you have set up before running the build?

[ghost]

IDF_PYTHON_ENV_PATH 
	c:\...\esp32tools\python_env
PATH 
	C:\...\esp32tools\tensa-esp-elf-gdb\bin;
	C:\...\esp32tools\xtensa-esp32-elf\bin;
	C:\...\esp32tools\esp32ulp-elf\bin;
	C:\...\esp32tools\cmake\bin;
	C:\...\esp32tools\openocd-esp32\bin;
	C:\...\esp32tools\ninja;
	C:\...\esp32tools\idf-ex;
	C:\...\esp32tools\ccache;
	C:\...\esp32tools\dfu-util;
	C:\...\esp32tools\python_env\Scripts;
	C:\...\esp-idf\tools;
	C:\...\jdk11\bin\server;
	C:\...\jdk11\bin;
	C:\WINDOWS\system32;
	C:\WINDOWS;
	C:\...\python3\;
	C:\...\python3\Scripts\;
	C:\...\WindowsApps;

[igrr]

Thanks, and could you also check what is the output of where python (in case you are running CMD.exe) or (get-command python.exe).Path (in case you are running Powershell)?

[ghost]

It gives out two lines:

C:\>where python
C:\...\esp32tools\python_env\Scripts\python.exe
C:\...\python3\python.exe

[dobairoland]

If I understand your issue well than you don't like keeping the part idf5.0_py3.10_env of the path or you don't want to relocate all of the .espressif folder. Is this correct?

May I ask why you don't want to keep the idf5.0_py3.10_env part of the path? The IDF and Python versions are encoded into it with a good reason. I hope you don't try to use the same environment with different minor/major versions of ESP-IDF.

[ghost]

@dobairoland, the reason is the same as with the version-ids on the tools folders, stable build environment. For updates it was just necessary to change the contents of the folders. Variables stay same and work across commandline and IDE without duplicate installs.