Windows OS solution

[avisionh]

"As an Windows OS user, I want to use the data science template to help structure my data science projects."

This is an open-call for help to implement an Windows compatible framework for this. Currently, it has been tested and works on Unix-based systems such as macOS and Linux. What we are looking for is someone to adapt this or add extra features so it works on Windows OS.

[ESKYoung]

To clarify, you can still use govcookiecutter to set up data science projects on Windows. However, you won't have two bits of functionality without further system configuration:

1. Separation of secrets from version-control, whilst still being able to use them locally as environment variables; and
2. Helper commands defined in Makefile.

Both are optional, and do not affect the structure or other configurations.

The first is performed by the Unix shell extension direnv, which loads environment variables from .envrc (version-controlled), and.envrc also sources .secrets (not version-controlled). This allows you to store secrets in .secrets out of version control, but still use them as environment variables.

The second is done using the Unix make tool; if you open the Makefile you'll see what commands are executed for each make command.

If you want to use govcookiecutter as is on Windows, you cannot use the Makefile, and will have to use your own method to store and use secrets.

Advanced - getting Unix functionality on Windows

It is possible to have full govcookiecutter functionality on Windows. One method, however, requires administrative privileges on your Windows machine.

Further details can be found in this Gist, or in the add_windows_support windows-support-for-dotenvrc branch.

We would welcome any contributions that might find a non-administrator way of supporting Windows, and/or remove our Unix dependencies!

[foster999]

On 1., there's a couple of Python tools that provide something similar in an OS-independent way:

• python-dotenv stores variables in a .env file or similar, but requires you to load these in your code: load_dotenv
• python_secrets creates an "environment" outside of the project to store secrets.

Neither is as neat as direnv, as they need some kind of explicit activation (not based on location on the command line). But it might be worth referencing them if someone is looking for similar functionality on a Windows setup, without admin privileges.

On 2., sphinx deals with Windows compatibility by providing an executable make.bat file that is called in the same way (make html). I'd be happy to create something similar here, if that would be useful?

[ESKYoung]

Regarding 1., that sounds like a good thing to edit on the Gist/add_windows_supportwindows-support-for-dotenvrc branch. I'll find some time to add those in the coming days. It maybe that we don't merge that branch in, and instead update the docs at the top-level of govcookiecutter to address possible Windows integration. Also need to make it clear that most of the functionality is language-agnostic (not currently the case, see for example README.md).

RE: creating a .bat file - happy for you to create something if you'd like! We can slowly then integrate partial Windows support (or something to that effect).

[AndreasThinks]

As a starter for ten, I'd say worth clarifying in the docs that Windows support is sub-par, but still pretty good (and better than doing your own cookiecutter variant) - have been spinning up a few projects with this in our windows env and faced no issues, though granted have not got to the makefile stage!

As someone who initially saw the project and thought "oh, not for me, I'm on Windows", would have been nice ;) Happy to pull/draft something if it helps!

[ESKYoung]

@Crimsoneer thanks for the feedback! There's some guidance here already as part of the add_windows_supportwindows-support-for-dotenvrc branch - this is awaiting approval of PR #18. Feel free to add to this branch if you think the instructions are not sufficient. It may (mostly likely will!) have to be rebased once PR #18 is in main though, so fair warning there may be some changes!

I'm also playing around with a possible replacement for direnv for Windows users using python-dotenv - will show more when it's in a better state!

[ESKYoung]

@AndreasThinks - I've had a play around with getting .envrc work on Windows - code is now on the windows-support-for-dotenvrc branch. To create a new project from this, run the following command in your terminal:

cookiecutter https://github.com/ukgovdatascience/govcookiecutter.git --checkout windows-support-for-dotenvrc

It includes a new function src.utils.load_dotenvrc. After setting everything else up, if you add the following lines to the top of each and every script/notebook:

from src.utils import load_dotenvrc
load_dotenvrc()

It should:

1. Parse .envrc, .secrets, and any other file referenced in .envrc with the source_env or source_env_if_exists direnv command
2. Create/update an untracked .env file with these variables (only if there are changes)
3. Load the .env file using python-dotenv so you can access your environment variables using the os package.
   • Note existing environment variables will be overridden with these new values

More details available in the documentation for that branch.

Minimum working example:

from src.utils import load_dotenvrc
import os
load_dotenvrc()

print(os.getenv("DIR_DATA"))  # should print out the folder path to the `data` folder

Let me know if this works! Feel free to push changes to the branch too! If it does the job, I'll raise a PR!

[AndreasThinks]

Oo, very exciting! I'm actually just working on my first project using the template, and am hoping to finish it off tomorrow and tie it up into a nice package/container, so shall give it a shot later this week and let you know how I get on. I was adding "sys.path.insert" and my working directory to each notebook, but this looks a lot cleaner.

[ESKYoung]

Great thanks! It's not a full replacement for direnv - mainly a bunch of regular expressions - but hopefully it will do the job for now!

I supposed on notebooks you may have to call load_dotenvrc to add to the PYTHONPATH before importing other src functions? (If that was what you mean) Would be good to test!

[ESKYoung]

Slightly nightmarish, but notebooks may not be so straightforward. .envrc has the working directory in PYTHONPATH, but you first need to load the environment variables for this to work... which needs you to run load_dotenvrc... Chicken and egg situation.

Without inserting the sys.path and then importing load_dotenvrc, which would get repetitive, a possible solution for Jupyter notebooks on Windows is to have a file in the notebooks folder, say notebook_setup.py consisting of:

from IPython.core.interactiveshell import InteractiveShell
import os
import sys
sys.path.insert(0, os.path.abspath(".."))

from src.utils import load_dotenvrc  # noqa: E402

# Load environment variables, and enable multiple outputs per cell
load_dotenvrc()
InteractiveShell.ast_node_interactivity = "all"  # just to demonstrate you can add any code here you like!

And then import it in the first cell of every notebook, i.e.

import notebook_setup  # must be before any imports from `src`!

Not sure if this is the cleanest way to do it - everything else seems to require setting environment variables, which is kind of the whole point of direnv, before being able to load configuration files!

[ESKYoung]

Thoughts welcome on #28 - revising the way we handle environment variables so that direnv becomes optional may be a welcome change for Windows users! (Doesn't get around the manual loading requirements, e.g. using python-dotenv though!)

[ESKYoung]

As a brief update, I think this issue will be resolved when issues #28 and #34 are resolved.

Issue #28 should make all core govcookiecutter features OS-independent by enabling local usage of secrets without accidentally version-controlling them. direnv will be maintained as an option for Unix-based users.

Issue #34, as an extension of issue #28, should make it easier for Jupyter notebooks to use the src package using the autoreload magic, as the package will be installed locally.

[ESKYoung]

I've also rebased the windows-support-for-dotenvrc branch up to the latest version of govcookiecutter (1.2.2). However, this may become a redundant branch given the upcoming changes!

[harrietrs]

has there been any further update for Windows? I took a look through #28 but can't see anything after Oct last year.
(as a sidebar, the link to the issue is broken in the README.md)

[ESKYoung]

@harrietrs, not sure what progress has been made, although I see PR #48 was raised some time ago. Both @avisionh and I have now left @ukgovdatascience, but will tag @alexander-newton and @Jacobb164 (if they're still at ONS!)

[AndreasThinks]

As a temporary workaround, I'm using python-dotenv for windows and it works quite well for me, eg

dotenv import dotenv_values
config = dotenv_values("../.secrets")

[Jacobb164]

Hi @AndreasThinks I am currently in the process of updating govcookiecutter with this as one of my main priorities. Would love to hear if you have any other suggestions / requests for how govcookiecutter can be improve going forward.

[harrietrs]

What's the update on how the latest release meets this requirement?
EDIT
I can see that this was addressed in https://github.com/best-practice-and-impact/govcookiecutter/releases/tag/1.3.2, could this issue thread be updated accordingly?