0.19.1 `kedro new` failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter #3539

BrianCechmanek · 2024-01-21T21:12:32Z

Description

Creating a new kedro project via kedro new --name my_project --verbose fails with the following error:

kedro.framework.cli.utils.KedroCliError: Failed to generate project when running cookiecutter. (full trace below)

Context

I am trying to initiate a new project in a 'clean' conda environment with kedro==0.19.1. I have no other environments (except base which has only defaults installed), and no other kedro installs.
Cookiecutter is 2.5.0 (installs with kedro)
I'm on Windows 10

Issue reproduces on kedro==0.19.0. Issue is absent on `kedro==0.18.14.

Steps to Reproduce

conda create --name my_env python=3.11
conda activate my_env
(my_env) $ kedro new --name my_project--verbose
select any/all/none tools (it fails on all options)
select yes/no for example pipeline (it fails on both options )

Expected Result

A base new project should be created.

Actual Result

Project setup fails in Cookiecutter afaict.

Would you like to include an example pipeline? :
 (no): yes
[01/21/24 20:56:13] ERROR    Stopping generation because post_gen_project hook script didn't exit successfully                                                          hooks.py:159
Traceback (most recent call last):
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\kedro\framework\cli\starters.py", line 864, in _create_project
    result_path = cookiecutter(template=template_path, **cookiecutter_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\main.py", line 182, in cookiecutter
    result = generate_files(
             ^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\generate.py", line 420, in generate_files
    run_hook_from_repo_dir(
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 152, in run_hook_from_repo_dir
    run_hook(hook_name, project_dir, context)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 135, in run_hook
    run_script_with_context(script, project_dir, context)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 118, in run_script_with_context
    run_script(temp.name, cwd)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\cookiecutter\hooks.py", line 89, in run_script
    raise FailedHookException(
cookiecutter.exceptions.FailedHookException: Hook script failed (exit status: 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\kedro\framework\cli\starters.py", line 268, in new
    _create_project(project_template, cookiecutter_args)
  File "C:\Users\bcech\anaconda3\envs\my_env\Lib\site-packages\kedro\framework\cli\starters.py", line 866, in _create_project
    raise KedroCliError(
kedro.framework.cli.utils.KedroCliError: Failed to generate project when running cookiecutter.

Your Environment

Kedro version used (pip show kedro or kedro -V): 0.19.1, 0.19.0
Python version used (python -V): 3.11.7, 3.10.13
Operating system and version: Windows 10, build 19045.3930
conda version: 23.7.4

The text was updated successfully, but these errors were encountered:

SajidAlamQB · 2024-01-22T11:28:27Z

Hi, thanks for reporting the issue should be resolved in kedro 0.19.2 please try that version instead.

astrojuanlu · 2024-01-22T11:48:16Z

This was fixed in 0.19.2 (confirmed by another user), I'm closing @BrianCechmanek but if an upgrade still doesn't do the trick please leave a comment and we'll reconsider.

BrianCechmanek · 2024-01-22T12:28:19Z

Thanks for looking into it!

Unfortunately, the problem persists in 0.19.2 , both after direct upgrade (from 0.19.1) and with a fresh conda env.

Same error as above.

astrojuanlu · 2024-01-22T13:03:13Z

Hi @BrianCechmanek , sorry the problem is still there. Can you confirm the traceback looks the same?

noklam · 2024-01-22T13:08:33Z

@BrianCechmanek Can you try

kedro new --name my_new_project --verbose? I think the problem may be you have a new_project folder already and kedro new would not overwrite existing files. It's showing cookiecutter error because that's what kedro new use under the hood, we can probably surface this error with better error message. Please let me know if it works.

BrianCechmanek · 2024-01-22T13:18:30Z

Thanks @noklam , @astrojuanlu

I can confirm it's the same problem (trace below). I'm trying it in an empty directory. my venv (kedro311 in the traceout), is a fresh conda env, with kedro==0.19.2.

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox> dir     # empty
(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox> kedro new --name my_new_project --verbose

←[1mProject Tools
=============←[0m
These optional tools can help you apply software engineering best practices.
To skip this step in future use --tools
To find out more: https://docs.kedro.org/en/stable/starters/new_project_tools.html

Tools
1) Lint: Basic linting with Ruff
2) Test: Basic testing with pytest
3) Log: Additional, environment-specific logging options
4) Docs: A Sphinx documentation setup
5) Data Folder: A folder structure for data management
6) PySpark: Configuration for working with PySpark
7) Kedro-Viz: Kedro's native visualisation tool

Which tools would you like to include in your project? [1-7/1,3/all/none]:
 (none): none

←[1mExample Pipeline
================←[0m
Select whether you would like an example spaceflights pipeline included in your project.
To skip this step in the future use --example=y/n
To find out more: https://docs.kedro.org/en/stable/starters/new_project_tools.html

Would you like to include an example pipeline? :
 (no): no
[01/22/24 13:15:02] ERROR    Stopping generation because post_gen_project hook script didn't exit successfully                     hooks.py:159
Traceback (most recent call last):
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\kedro\framework\cli\starters.py", line 895, in _create_project
    result_path = cookiecutter(template=template_path, **cookiecutter_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\main.py", line 182, in cookiecutter
    result = generate_files(
             ^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\generate.py", line 420, in generate_files
    run_hook_from_repo_dir(
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\hooks.py", line 152, in run_hook_from_repo_dir
    run_hook(hook_name, project_dir, context)
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\hooks.py", line 135, in run_hook
    run_script_with_context(script, project_dir, context)
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\hooks.py", line 118, in run_script_with_context
    run_script(temp.name, cwd)
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\hooks.py", line 89, in run_script
    raise FailedHookException(
cookiecutter.exceptions.FailedHookException: Hook script failed (exit status: 1)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\kedro\framework\cli\starters.py", line 346, in new
    _create_project(project_template, cookiecutter_args)
  File "C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\kedro\framework\cli\starters.py", line 897, in _create_project
    raise KedroCliError(
kedro.framework.cli.utils.KedroCliError: Failed to generate project when running cookiecutter.

astrojuanlu · 2024-01-22T13:23:59Z

I cannot reproduce on macOS on Python 3.11 (both my old Kedro installation and a newly created environment) but I can reproduce on Python 3.12 (which is nonetheless not supported yet).

@BrianCechmanek could you paste the output of conda info -a, conda env export --from-history, and pip freeze?

astrojuanlu · 2024-01-22T13:30:48Z

Another user reproduced this on Python 3.12 on Windows.

BrianCechmanek · 2024-01-22T13:34:13Z

I'm not using python 3.12 (as it's not supported yet). I can reproduce this issue on python 3.11 and 3.10. On Windows.

conda info -a:


     active environment : kedro311
    active env location : C:\Users\bcech\anaconda3\envs\kedro311
            shell level : 2
       user config file : C:\Users\bcech\.condarc
 populated config files : C:\Users\bcech\.condarc
          conda version : 23.7.4
    conda-build version : 3.26.1
         python version : 3.11.5.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.3=0
                          __win=0=0
       base environment : C:\Users\bcech\anaconda3  (writable)
      conda av data dir : C:\Users\bcech\anaconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\bcech\anaconda3\pkgs
                          C:\Users\bcech\.conda\pkgs
                          C:\Users\bcech\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\bcech\anaconda3\envs
                          C:\Users\bcech\.conda\envs
                          C:\Users\bcech\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.7.4 requests/2.31.0 CPython/3.11.5 Windows/10 Windows/10.0.19041 aau/0.4.2 c/...
          administrator : False
             netrc file : None
           offline mode : False


# conda environments:
#
base                     C:\Users\bcech\anaconda3
kedro311              *  C:\Users\bcech\anaconda3\envs\kedro311
nanogpt                  C:\Users\bcech\anaconda3\envs\nanogpt

sys.version: 3.11.5 | packaged by Anaconda, Inc. | (m...
sys.prefix: C:\Users\bcech\anaconda3
sys.executable: C:\Users\bcech\anaconda3\python.exe
conda location: C:\Users\bcech\anaconda3\Lib\site-packages\conda
conda-build: C:\Users\bcech\anaconda3\Scripts\conda-build.exe
conda-content-trust: C:\Users\bcech\anaconda3\Scripts\conda-content-trust.exe
conda-convert: C:\Users\bcech\anaconda3\Scripts\conda-convert.exe
conda-debug: C:\Users\bcech\anaconda3\Scripts\conda-debug.exe
conda-develop: C:\Users\bcech\anaconda3\Scripts\conda-develop.exe
conda-env: C:\Users\bcech\anaconda3\Scripts\conda-env.exe
conda-index: C:\Users\bcech\anaconda3\Scripts\conda-index.exe
conda-inspect: C:\Users\bcech\anaconda3\Scripts\conda-inspect.exe
conda-metapackage: C:\Users\bcech\anaconda3\Scripts\conda-metapackage.exe
conda-pack: C:\Users\bcech\anaconda3\Scripts\conda-pack.exe
conda-render: C:\Users\bcech\anaconda3\Scripts\conda-render.exe
conda-repo: C:\Users\bcech\anaconda3\Scripts\conda-repo.exe
conda-server: C:\Users\bcech\anaconda3\Scripts\conda-server.exe
conda-skeleton: C:\Users\bcech\anaconda3\Scripts\conda-skeleton.exe
conda-token: C:\Users\bcech\anaconda3\Scripts\conda-token.exe
conda-verify: C:\Users\bcech\anaconda3\Scripts\conda-verify.exe
user site dirs: C:\Users\bcech\AppData\Roaming\Python\Python37
                C:\Users\bcech\AppData\Roaming\Python\Python38
                C:\Users\bcech\AppData\Roaming\Python\share

CIO_TEST: <not set>
CONDA_DEFAULT_ENV: kedro311
CONDA_EXE: C:\Users\bcech\anaconda3\Scripts\conda.exe
CONDA_PREFIX: C:\Users\bcech\anaconda3\envs\kedro311
CONDA_PREFIX_1: C:\Users\bcech\anaconda3
CONDA_PROMPT_MODIFIER: (kedro311)
CONDA_PYTHON_EXE: C:\Users\bcech\anaconda3\python.exe
CONDA_ROOT: C:\Users\bcech\anaconda3
CONDA_SHLVL: 2
CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3
CURL_CA_BUNDLE: <not set>
HOMEPATH: \Users\bcech
LD_PRELOAD: <not set>
NVTOOLSEXT_PATH: C:\Program Files\NVIDIA Corporation\NvToolsExt\
PATH: C:\Users\bcech\anaconda3\envs\kedro311;C:\Users\bcech\anaconda3\envs\kedro311\Library\mingw-w64\bin;C:\Users\bcech\anaconda3\envs\kedro311\Library\usr\bin;C:\Users\bcech\anaconda3\envs\kedro311\Library\bin;C:\Users\bcech\anaconda3\envs\kedro311\Scripts;C:\Users\bcech\anaconda3\envs\kedro311\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.3\libnvvp;C:\Program Files\ffmpeg\bin;C:\Users\bcech\anaconda3\condabin;C:\Program Files (x86)\Razer Chroma SDK\bin;C:\Program Files\Razer Chroma SDK\bin;C:\Program Files (x86)\Razer\ChromaBroadcast\bin;C:\Program Files\Razer\ChromaBroadcast\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\Git\cmd;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\Program Files\CMake\bin;C:\Program Files\PuTTY;C:\Program Files (x86)\Arm GNU Toolchain arm-none-eabi\11.2 2022.02\bin;C:\Users\bcech\AppData\Local\Microsoft\WindowsApps;C:\Users\bcech\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\Plantronics\Spokes3G;C:\Program Files\dotnet;C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.2.0;C:\Users\bcech\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\Users\bcech\AppData\Local\Pub\Cache\bin;C:\Program Files (x86)\Arm GNU Toolchain arm-none-eabi\11.2 2022.02\bin;C:\Users\bcech\AppData\Local\Microsoft\WindowsApps;C:\Users\bcech\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\bcech\AppData\Roaming\nvm;C:\Program Files\nodejs;C:\src\flutter\bin;C:\Users\bcech\AppData\Local\Pub\Cache\bin;.
PSMODULEPATH: C:\Users\bcech\Documents\WindowsPowerShell\Modules;C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: C:\Users\bcech\anaconda3\envs\kedro311\Library\ssl\cacert.pem

conda env export --from-history :

simple history was: $ conda create --name kedro311 python=3.11 and then (kedro311) $ python -m pip install kedro kedro-datasets kedro-viz numpy pandas torch ipykernel

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox> conda env export --from-history
name: kedro311
channels:
  - defaults
dependencies:
  - python=3.11
prefix: C:\Users\bcech\anaconda3\envs\kedro311

and lastly, pip freeze:

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox> python -m pip freeze
aiobotocore==2.11.0
aiofiles==23.2.1
aiohttp==3.9.1
aioitertools==0.11.0
aiosignal==1.3.1
antlr4-python3-runtime==4.9.3
anyio==3.7.1
arrow==1.3.0
asttokens==2.4.1
attrs==23.2.0
binaryornot==0.4.4
botocore==1.34.22
build==1.0.3
cachetools==5.3.2
certifi==2023.11.17
chardet==5.2.0
charset-normalizer==3.3.2
click==8.1.7
colorama==0.4.6
comm==0.2.1
cookiecutter==2.5.0
debugpy==1.8.0
decorator==5.1.1
dynaconf==3.2.4
executing==2.0.1
fastapi==0.109.0
filelock==3.13.1
frozenlist==1.4.1
fsspec==2023.12.2
gitdb==4.0.11
GitPython==3.1.41
graphql-core==3.2.3
greenlet==3.0.3
h11==0.14.0
httptools==0.6.1
idna==3.6
importlib-metadata==7.0.1
importlib-resources==6.1.1
ipykernel==6.29.0
ipython==8.20.0
jedi==0.19.1
Jinja2==3.1.3
jmespath==1.0.1
jupyter_client==8.6.0
jupyter_core==5.7.1
kedro==0.19.2
kedro-datasets==2.0.0
kedro-viz==7.0.0
lazy_loader==0.3
markdown-it-py==3.0.0
MarkupSafe==2.1.4
matplotlib-inline==0.1.6
mdurl==0.1.2
more-itertools==10.2.0
mpmath==1.3.0
multidict==6.0.4
nest-asyncio==1.6.0
networkx==3.2.1
numpy==1.26.3
omegaconf==2.3.0
orjson==3.9.12
packaging==23.2
pandas==2.2.0
parse==1.20.0
parso==0.8.3
platformdirs==4.1.0
plotly==5.18.0
pluggy==1.3.0
pre-commit-hooks==4.5.0
prompt-toolkit==3.0.43
psutil==5.9.8
pure-eval==0.2.2
pydantic==1.10.14
Pygments==2.17.2
pyproject_hooks==1.0.0
python-dateutil==2.8.2
python-dotenv==1.0.0
python-slugify==8.0.1
pytoolconfig==1.3.1
pytz==2023.3.post1
pywin32==306
PyYAML==6.0.1
pyzmq==25.1.2
requests==2.31.0
rich==13.7.0
rope==1.12.0
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
s3fs==2023.12.2
secure==0.3.0
six==1.16.0
smmap==5.0.1
sniffio==1.3.0
SQLAlchemy==2.0.25
stack-data==0.6.3
starlette==0.35.1
strawberry-graphql==0.217.1
sympy==1.12
tenacity==8.2.3
text-unidecode==1.3
toml==0.10.2
toposort==1.10
torch==2.1.2
tornado==6.4
traitlets==5.14.1
types-python-dateutil==2.8.19.20240106
typing_extensions==4.9.0
tzdata==2023.4
urllib3==2.0.7
uvicorn==0.22.0
watchfiles==0.21.0
watchgod==0.8.2
wcwidth==0.2.13
websockets==12.0
wrapt==1.16.0
yarl==1.9.4
zipp==3.17.0

astrojuanlu · 2024-01-22T14:57:53Z

For readers: this consistently fails on Python 3.12 unless the PYTHONWARNINGS environment variable is set. This is because the post-gen hook is run as a subprocess (notice how the warning text is shown twice):

$ export PYTHONWARNINGS='default:Kedro is not yet fully compatible'
$ kedro new --name testkedro_312_8 --tools none --example no --verbose                                                            (kedro312) 
/Users/juan_cano/.micromamba/envs/kedro312/lib/python3.12/site-packages/kedro/__init__.py:25: KedroPythonVersionWarning: Kedro is not yet fully compatible with this Python version.
To proceed at your own risk and ignore this warning,
run Kedro with `python -W "default:Kedro is not yet fully compatible" -m kedro ...`
or set the PYTHONWARNINGS environment variable accordingly.
  warnings.warn(
/Users/juan_cano/.micromamba/envs/kedro312/lib/python3.12/site-packages/kedro/__init__.py:25: KedroPythonVersionWarning: Kedro is not yet fully compatible with this Python version.
To proceed at your own risk and ignore this warning,
run Kedro with `python -W "default:Kedro is not yet fully compatible" -m kedro ...`
or set the PYTHONWARNINGS environment variable accordingly.
  warnings.warn(

Congratulations!
Your project 'testkedro_312_8' has been created in the directory 
/Users/juan_cano/Projects/QuantumBlack Labs/tmp/testkedro-312-8

You have selected no project tools

So in your case @BrianCechmanek it must be something else. We're looking into it but unfortunately we haven't been able to reproduce the issue on our end yet, and the diagnostics information from cookiecutter is not great.

astrojuanlu · 2024-01-22T14:59:36Z

I think the problem may be you have a new_project folder already and kedro new would not overwrite existing files.

That shows a different error, at least on macOS:

Traceback (most recent call last):
  File "/Users/juan_cano/.micromamba/envs/kedro312/lib/python3.12/site-packages/kedro/framework/cli/starters.py", line 895, in _create_project
    result_path = cookiecutter(template=template_path, **cookiecutter_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/juan_cano/.micromamba/envs/kedro312/lib/python3.12/site-packages/cookiecutter/main.py", line 182, in cookiecutter
    result = generate_files(
             ^^^^^^^^^^^^^^^
  File "/Users/juan_cano/.micromamba/envs/kedro312/lib/python3.12/site-packages/cookiecutter/generate.py", line 321, in generate_files
    project_dir, output_directory_created = render_and_create_dir(
                                            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/juan_cano/.micromamba/envs/kedro312/lib/python3.12/site-packages/cookiecutter/generate.py", line 251, in render_and_create_dir
    raise OutputDirExistsException(msg)
cookiecutter.exceptions.OutputDirExistsException: Error: "/Users/juan_cano/Projects/QuantumBlack Labs/tmp/testkedro-312-8" directory already exists

BrianCechmanek · 2024-01-22T17:30:09Z

Thanks @astrojuanlu

I'll have time in a day or two to debug it a bit more. I'll see if it's more one of Conda, Windows, Cookiecutter.

astrojuanlu · 2024-01-22T18:22:57Z

@BrianCechmanek when you get to it, one thing you can try is launching

python -m pdb -m kedro new --name my_new_project --tools none --example no --verbose

and set a breakpoint right before run_script(temp.name, cwd) is run:

(Pdb) b C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\hooks.py:118

when you get to that point (notice that it's called a few times before the problematic one), you can see what's temp.name and cwd, and essentially run the script yourself in another terminal. That way you should see the output.

BrianCechmanek · 2024-01-23T01:57:10Z

Tiny update:

On WSL2 (Ubuntu) kedro new --name my_new_project --tools none --example no --verbose does not replicate the above error, with python 3.10.4 and 3.11.7. that is, kedro in WSL works fine.
Running pdb as above, is unremarkable. If you're curious:

-> run_script(temp.name, cwd)
(Pdb) temp.name
'C:\\Users\\bcech\\AppData\\Local\\Temp\\tmp4gvvu3sc.py'
(Pdb) cwd
'C:\\Users\\bcech\\Documents\\Projects\\sandbox\\my-new-project'
(Pdb)

An old bug report on Cookiecutter seems to match, and be unanswered: Creation failed: cookiecutter.exceptions.FailedHookException: Hook script failed cookiecutter/cookiecutter#1616 . Note this is a very old release of cc 1.7.2. Not sure if this is a red herring or not.

astrojuanlu · 2024-01-23T10:16:26Z

@BrianCechmanek yes, that's expected. C:\\Users\\bcech\\AppData\\Local\\Temp\\tmp4gvvu3sc.py is a temporary copy of the post-build script that's about to run. Inspecting its contents will yield the hook of the template. You can then

> chdir C:\Users\bcech\Documents\Projects\sandbox\my-new-project
> C:\Users\bcech\anaconda3\envs\kedro311\Scripts\python.exe C:\Users\bcech\AppData\Local\Temp\tmp4gvvu3sc.py

and you should observe the error that's crashing kedro new (we're almost there 🙏🏽)

BrianCechmanek · 2024-01-23T12:38:40Z

Thanks for guiding me along this.

Summary: I think the issue is cookiecutter trying to call a python exe that doesn't exist C:\Users\bcech\anaconda3\envs\kedro311\Scripts\python.exe .

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox\my-new-project> C:\Users\bcech\anaconda3\envs\kedro311\Scripts\python.exe C:\Users\bcech\AppData\Local\Temp\tmp4gvvu3sc.p
C:\Users\bcech\anaconda3\envs\kedro311\Scripts\python.exe : The term 'C:\Users\bcech\anaconda3\envs\kedro311\Scripts\python.exe' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if
a path was included, verify that the path is correct and try again.
At line:1 char:1
+ C:\Users\bcech\anaconda3\envs\kedro311\Scripts\python.exe C:\Users\bc ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Users\bcech\...ipts\python.exe:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Looking in my conda env folder...

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox\my-new-project> Get-ChildItem -Path C:\Users\bcech\anaconda3\envs\kedro311\Scripts | Where-Object { $_.Name -like 'p*' }
    Directory: C:\Users\bcech\anaconda3\envs\kedro311\Scripts
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a----        2023-11-22   3:45 PM            216 pip-script.py
-a----        2023-10-02   3:45 PM          41984 pip.exe
-a----        2023-11-22   3:45 PM            216 pip3-script.py
-a----        2023-10-02   3:45 PM          41984 pip3.exe
-a----        2024-01-22  12:15 PM         108410 pre-commit-hooks-removed.exe
-a----        2024-01-22  12:15 PM         108421 pretty-format-json.exe
-a----        2023-12-15   6:09 PM             57 pydoc-script.py
-a----        2023-12-15   6:09 PM          41984 pydoc.exe
-a----        2024-01-22  12:13 PM         108402 pygmentize.exe
-a----        2024-01-22  12:13 PM           6318 pygrun
-a----        2024-01-22  12:15 PM         108412 pyproject-build.exe
-a----        2024-01-22  12:13 PM          27255 pywin32_postinstall.py
-a----        2024-01-22  12:13 PM           3721 pywin32_testall.py

No python.exe indeed (It's actually one level up in C:\Users\bcech\anaconda3\envs\kedro311).

Finally, running the intermediate script via call to python:

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox\my-new-project> python C:\\Users\\bcech\\AppData\\Local\\Temp\\tmpkqt655hr.py

executes without error and my project dir is seemingly correct:

(kedro311) PS C:\Users\bcech\Documents\Projects\sandbox\my-new-project> dir


    Directory: C:\Users\bcech\Documents\Projects\sandbox\my-new-project


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        2024-01-23  11:53 AM                conf
d-----        2024-01-23  11:53 AM                notebooks
d-----        2024-01-23  11:53 AM                src
-a----        2024-01-23  11:53 AM           1901 .gitignore
-a----        2024-01-23  12:09 PM            695 pyproject.toml
-a----        2024-01-23  11:53 AM           2955 README.md
-a----        2024-01-23  12:09 PM             81 requirements.txt

I tried running kedro=0.18.14, with similar breakpoints along run_script, but none triggered (and it creates the project successfully)... I was hoping to see where cookiecutter calls python from ¯\ (°_o)/¯ .

astrojuanlu · 2024-01-23T16:28:41Z

The plot thickens 😬

Disregard the ObjectNotFound errors you got initially - the path I gave you initially was wrong (it should have been C:\Users\bcech\anaconda3\envs\kedro311\python.exe, without Scripts\).

So, the interesting thing then is that running the script manually seems to work, but kedro new still fails.

If you want to keep digging, my recommendation is to try to set up some manual breakpoints in the cookiecutter code. This is how cookiecutter runs the script:

https://github.com/cookiecutter/cookiecutter/blob/5d2b1e37b90ad11ac412c691f131689445840709/cookiecutter/hooks.py#L85-L97

run_thru_shell will be True because your platform is Windows (please double check). So you can try running

subprocess.Popen([sys.executable, script_path], shell=run_thru_shell, cwd=cwd)

manually and see what happens.

Galileo-Galilei · 2024-01-28T16:56:33Z

I can confirm that I am facing a similar issue on windows 11, python=3.9.15, kedro==0.19.2. It looks like there is some tricky conversion on tools which sometimes generate an invalid pyproject.toml by removing quotes around the "tools": "['None']", don't know why.

Edit: Fun fact: it works if I always press enter without writing anything on each entry.

BrianCechmanek · 2024-01-28T19:40:58Z

Next minor update...

run_thru_shell will be True because your platform is Windows (please double check)

> c:\users\bcech\anaconda3\envs\kedro311\lib\site-packages\cookiecutter\hooks.py(77)run_script()
-> run_thru_shell = sys.platform.startswith('win')
(Pdb) sys.platform.startswith('win')
True
(Pdb) s
> c:\users\bcech\anaconda3\envs\kedro311\lib\site-packages\cookiecutter\hooks.py(78)run_script()
-> if script_path.endswith('.py'):
(Pdb) run_thru_shell
True

(I'm not sure if the steps I took were the right thing to do here. but I tried...)

I paused execution at > c:\users\bcech\anaconda3\envs\kedro311\lib\site-packages\cookiecutter\hooks.py(86)run_script() -> proc = subprocess.Popen(script_command, shell=run_thru_shell, cwd=cwd) # nosec
I copy-pasted the script_command and rest into a new python file pdb_me.py :

import subprocess

script_command = ['C:\\Users\\bcech\\anaconda3\\envs\\kedro311\\python.exe', 'C:\\Users\\bcech\\AppData\\Local\\Temp\\tmpw31svgdv.py']
run_thru_shell = True
cwd = 'C:\\Users\\bcech\\Documents\\Projects\\sandbox\\my-new-project'

breakpoint()
proc = subprocess.Popen(script_command, shell=run_thru_shell, cwd=cwd)
breakpoint()
print("great success")

I opened a new terminal, activated my env and chdir'd to \\my-new-project
I ran python -m pdb -m pdb_me.py
I checked proc, the response from subprocess.open(...):

(Pdb) proc
<Popen: returncode: None args: ['C:\\Users\\bcech\\anaconda3\\envs\\kedro311...>

Thanks @Galileo-Galilei -- sorta nice to know it's not just me!

BrianCechmanek · 2024-01-29T00:06:34Z

(sorry for the billion updates. really trying to get this resolved)

Finally ran it by forcing:
C:\Users\bcech\anaconda3\envs\kedro311\Lib\site-packages\cookiecutter\hooks.py:86,
proc = subprocess.Popen(script_command, shell=run_thru_shell, cwd=cwd)
run_thru_shell = False)

and it succeeds (as best I can tell).

ran it with some ad hoc variations on --tools, --example yes , et c. didn't do a full sweep though...

A lot of dead ends looking at other things...

Best I can tell from the Popen docs, is that cookiecutter should probably be using shell=False:

On Windows with shell=True, the COMSPEC environment variable specifies the default shell. The only time you need to specify shell=True on Windows is when the command you wish to execute is built into the shell (e.g. dir or copy). You do not need shell=True to run a batch file or console-based executable.

When running with run_thru_shell=True, the proc.returncode was always None
When running with run_thru_shell=True, exit_status was always 1

Summary: I think this goes to cookiecutter, not Kedro.

Thoughts @astrojuanlu ?

astrojuanlu · 2024-01-29T06:52:30Z

As far as anyone can tell, run_thru_shell=True for Windows has been there "forever" (commit from 11 years ago) cookiecutter/cookiecutter@a1a41c4

While I'm not saying this is the right thing to do (who can say for certain? every time I interact with CPython subprocess I question my life choices) if that was the root cause, someone else would have raised an issue.

...which they did! cookiecutter/cookiecutter#1394 but this seems to apply to MinGW and PowerShell. From looking at your tracebacks I don't think that's the case, but I could be wrong. And regardless, there's not enough info in the upstream issue to be sure. But maybe we're onto something?

Summary: I think this goes to cookiecutter, not Kedro.

Last suggestion: kedro new wraps cookiecutter.

kedro/kedro/framework/cli/starters.py

Line 895 in a14b6fa

result_path = cookiecutter(template=template_path, **cookiecutter_args)

You can try to run the corresponding command from your terminal. For example

> cookeicutter ... --verbose  #whatever

And you'll likely observe an unhelpful traceback. This is something that can be reported upstream - sure, the problem might lie in the template we provide, but if the error message doesn't help users diagnose what's going on, I argue this is indeed cookiecutter's fault.

It looks like there is some tricky conversion on tools which sometimes generate an invalid pyproject.toml by removing quotes around the "tools": "['None']", don't know why.

That's intriguing. If you have more info or a reproducer, please let us know.

BrianCechmanek · 2024-01-29T11:13:35Z

But maybe we're onto something?

Finally, I think so!

I've been running kedro new ... via my Anaconda Powershell
on windows, Popen(..., shell=True) is calling %COMSPEC% -> default C:\Windows\System32\cmd.exe
I'd never normally run cmd.exe (powershell is miserable enough...)
Unrelated - I found out last night - my C:\Windows\System32\cmd.exe is totally broken. No idea why or what happened.

So. When I've been calling script_command = ['C:\\Users\\bcech\\anaconda3\\envs\\kedro311\\python.exe', 'C:\\Users\\bcech\\AppData\\Local\\Temp\\tmpw31svgdv.py'] manually (in my conda powershell), everything is hunky-dorey. But the subprocess.Popen(...) call is trying to go to cmd.exe, which is indeed failing. This makes sense that shell=False works fine -- python is called directly and the script runs as usual.

I'll fix that (my cmd.exe) when I'm able and confirm back here.

But, in the meantime, I'll close this bug now. So sorry to litter up your Issue board with what is ultimately not looking like a Kedro bug at all. Thank you so much for your persistence on this.

What a wild ride.

astrojuanlu · 2024-01-29T11:56:48Z

So happy we got to the bottom of this! And no need to apologize @BrianCechmanek, I know how annoying these rabbit holes can get and glad that I could help you debug this nasty issue.

Maybe as a tribute do you want to send a PR to cookiecutter to amend this clunky setting? 😄

astrojuanlu closed this as completed Jan 22, 2024

astrojuanlu reopened this Jan 22, 2024

astrojuanlu added the Issue: Bug Report 🐞 Bug that needs to be fixed label Jan 22, 2024

github-project-automation bot added this to Kedro Framework Jan 22, 2024

BrianCechmanek closed this as completed Jan 29, 2024

github-project-automation bot moved this to Done in Kedro Framework Jan 29, 2024

github-actions bot mentioned this issue Feb 1, 2024

Monthly issue metrics report #3582

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

0.19.1 `kedro new` failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter #3539

0.19.1 `kedro new` failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter #3539

BrianCechmanek commented Jan 21, 2024

SajidAlamQB commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

noklam commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 23, 2024 •

edited

Loading

astrojuanlu commented Jan 23, 2024

BrianCechmanek commented Jan 23, 2024

astrojuanlu commented Jan 23, 2024

Galileo-Galilei commented Jan 28, 2024 •

edited

Loading

BrianCechmanek commented Jan 28, 2024 •

edited

Loading

BrianCechmanek commented Jan 29, 2024

astrojuanlu commented Jan 29, 2024

BrianCechmanek commented Jan 29, 2024

astrojuanlu commented Jan 29, 2024

0.19.1 kedro new failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter #3539

0.19.1 kedro new failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter #3539

Comments

BrianCechmanek commented Jan 21, 2024

Description

Context

Steps to Reproduce

Expected Result

Actual Result

Your Environment

SajidAlamQB commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

noklam commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 22, 2024

astrojuanlu commented Jan 22, 2024

BrianCechmanek commented Jan 23, 2024 • edited Loading

astrojuanlu commented Jan 23, 2024

BrianCechmanek commented Jan 23, 2024

astrojuanlu commented Jan 23, 2024

Galileo-Galilei commented Jan 28, 2024 • edited Loading

BrianCechmanek commented Jan 28, 2024 • edited Loading

BrianCechmanek commented Jan 29, 2024

astrojuanlu commented Jan 29, 2024

BrianCechmanek commented Jan 29, 2024

astrojuanlu commented Jan 29, 2024

0.19.1 `kedro new` failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter #3539

0.19.1 `kedro new` failes to create new project. Raises KedroCliError: Failed to generate project when running cookiecutter #3539

BrianCechmanek commented Jan 23, 2024 •

edited

Loading

Galileo-Galilei commented Jan 28, 2024 •

edited

Loading

BrianCechmanek commented Jan 28, 2024 •

edited

Loading