[WIP] Version 2

[pjbull]

We've seen a lot of potential features for this where we need to handle forking paths gracefully. By default cookiecutter can't do this (see cookiecutter/cookiecutter#848). It's been years, so we can't reasonably expect this to change upstream...

This implements a monkey-patching workaround to enable this behavior. It introduces a couple of major changes, so here are my recommendations.

Here are the big differences for a user:

• we now need to run ccds <path to repo> instead of cookiecutter <path to repo>
• the options and their defaults are changing
• as a consequence of supporting more environments/dependency managers at setup time, we stop supporting them at execution time. this means that teams will have to pick one of each and stick to it across developers. I think this is pretty widespread already. (i.e. make create_environment will only support one of the options rather than multiple like it does now)

Beta Testing v2

V2 is available to beta testers! Expect some bugs and changes as you go. If you encounter issues, you can file an issue and use the v2-bug label.

To beta test v2, you can install the ccds package with pip:

• pip install git+https://github.com/drivendata/cookiecutter-data-science@v2

And then to make a new project with the v2 template, you can run:

• ccds https://github.com/drivendata/cookiecutter-data-science.git --checkout v2

Note: because this is a separate package, you should be able to run it alongside your normal cookiecutter installation as well and still use v1 when you choose to. Also, always answer "yes" to re-cloning the project so that you get the latest and so v1 continues to work as well.

High-level feedback on choices and design is welcome!

Tracking issues for release

Done:

• we now support multiple storage providers (Google Cloud Storage support #120)
• we allow user to choose package manager and some default packages (Add package list that populates requirements.txt to the workflow #5)
• we allow user to choose a dependency format (Allow user to choose package manager #118)
• src is now {{ cookiecutter.module_name }} (Rename src directory to project_name #140)
• remove test_environment.py
• tag current master as v1 so anyone relying on the current flow/structure can continue to use it easily (V1 tag and deprecation #240)
• implement the rails for the high priority items
• make comprehensive tests run on CI/CD to de-risk
• add support for azure/google cloud as storage backends (Add option to choose different data storage back ends #8)
• add ccds command and make cookiecutter-data-science a proper package
• add support for differenct virtual env providers (Add hooks for virtualenvs #6)
• add lint command to Makefile
• formatting commands / configuration (e.g., black)
• Configuration file format defaults (setup.cfg/pyproject.toml)
• Default to no license (Default license should be "No license" #241)

---

Remaining items

Cookiecutter default structure

• support user supplied config files (Support user-supplied config files #325)
• revise generated python package boilerplate (make optional) (Make Python boilerplate optional in initial setup #49, Improve the quality of the boilerplate python code #326)
• mkdocs in place of sphinx (Switch from sphinx as default project docs to mkdocs #327)

Cookiecutter options

• add deon (Add option to include deon checklist #154)
• add nbautoexport (Add option to include nbautoexport #324)
• test suite configuration (e.g., pytest install and make commands) Consider adding options for test runners and example tests #195

Infrastructure

• tests passing on windows Add Windows to V2 CI #316
• Makefiile working on Windows (and instructions) Solo make command doesn't usually work (generate help text) on Windows #221, Make on Windows link not working #68
• release command and make PyPI release Add CI for releasing package to PyPI #317
• Transfer ownership to drivendataorg organization rather than drivendata user Transfer ownership from drivendata user to drivendataorg organization #315
• Add python 3.12 support (Add Python 3.12 support #328)

Fixes

• Quotes should be escaped on project init #248
• Apostrophes in project description break Sphinx template #142
• Don't show deprecation message when running v2 Running v2 should not generate a v1 deprecation warning #281

Docs

• add v2 beta testing documentation
• Add documentation about major changes between V1 and V2 #318
• update screencast and add screenshots of the new flow (Create a V2 screencast #314)
• Add links to real projects and also a few simple examples Real cases using the template #238, Add examples page and list to examples of analyses that use the project  #38, Add docs page with links to forks for more specific purposes (so people can PR a link to their projects) #48, Request for a tutorial demonstrating simple implementation of cookie-cutter-data-science framework #59
• Make separate docs pages instead of one monolithic page #19
• Add to docs that S3 can be a path (rather than just a bucket name) #109
• [DOCS] Clarify difference in function between data/external and data/raw #136

---

Under consideration

• Add a default (optional) Dockerfile (Create example projects that utilize Docker/Vagrant #13)
• git init as an option Initialize Git repository on creation #56
• Wrap make commands in a check that the right environment is active Make commands operate in current shell environment (not necessarily in the project Python environment) #83
• Add github actions boilerplate (optional)
• pre-commit hooks
• add option to render a completed demo project
• Adding the ability to sort imports in src using isort #276
• adding Citation files to cookiecutter-data-science template #274
• Use pytest-cookies for testing CCDS Use pytest-cookies for testing #166
• Adding in an opinionated pre-commit #188 Version with a make register_hooks is probably reasonable
• Check for existing directory earlier #282 Validation of inputs (at least that dir doesn't already exist)

Scoped out

• Add option to use different DAG manager #9 All of the current options are still worse than make 😿
• Add “apps” as a directory for model serving under main-project-stem #226 Not generic enough of a need

[shortcipher3]

Is this active? It looks like there hasn't been any changes for a while, but I would love to know when it is expected to be released.

[pjbull]

@shortcipher3 Yes, we're hoping to finish the work-in-progress here this summer. Thanks!

[raphaelreimann]

Hey, any updates on when this will be released? :)

[pjbull]

Thanks for the interest @raphaelreimann, work is progressing. Please see the updated beta testing v2 section at the start of this thread. It would be great to get feedback from interested folks that start using the latest version!