Skip to content

Commit

Permalink
Caveats for Windows (#450)
Browse files Browse the repository at this point in the history
* various improvements/updates

* add info about WSL

* formatting fixes

* minor formatting fixes

* tiny wording fixes

* add slash to command
  • Loading branch information
kmoscoe authored Jul 17, 2024
1 parent 38fde7f commit d0d1805
Show file tree
Hide file tree
Showing 6 changed files with 63 additions and 34 deletions.
2 changes: 1 addition & 1 deletion custom_dc/custom_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ docker run -it \
-e DEBUG=true \
--env-file $PWD/custom_dc/sqlite_env.list \
-v $PWD/custom_dc/<var>CUSTOM_DATA_DIRECTORY</var>:/userdata \
[-v $PWD/custom_dc/<var>CUSTOM_DATA_DIRECTORY</var>/datacommons:/sqlite]
[-v $PWD/custom_dc/<var>CUSTOM_DATA_DIRECTORY</var>/datacommons:/sqlite] \
gcr.io/datcom-ci/datacommons-website-compose:stable
</pre>

Expand Down
4 changes: 2 additions & 2 deletions custom_dc/data_cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ If you are prompted to install the Cloud Resource Manager API, press `y` to acce

#### Run with a prebuilt image

If you have not made changes that require a local build, and just want to run the pre-downloaded image, from `website` root of the repository, run:
If you have not made changes that require a local build, and just want to run the pre-downloaded image, from your repository root, run:

```shell
docker run -it \
Expand All @@ -125,7 +125,7 @@ gcr.io/datcom-ci/datacommons-website-compose:stable

#### Run with a locally built repo

If you have made local changes and have a [locally built repo](/custom_dc/build_repo.html), from the `website` root of the repository, run the following:
If you have made local changes and have a [locally built repo](/custom_dc/manage_repo.html#build-repo), from the root of the repository, run the following:

<pre>
docker run -it \
Expand Down
2 changes: 2 additions & 0 deletions custom_dc/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ A custom Data Commons site runs in a Docker container on Google Cloud Platform (
- A [Docker](http://docker.com) account
- If you will be customizing the site's UI, familiarity with the Python [Flask](https://flask.palletsprojects.com/en/3.0.x/#) web framework and [Jinja](https://jinja.palletsprojects.com/en/3.1.x/templates/) HTML templating

**Note:** Data Commons does not support local Windows development natively. If you wish to develop Data Commons on Windows, you will need to use the [Windows Subsystem for Linux](https://learn.microsoft.com/en-us/windows/wsl/about).

If you already have an account with another cloud provider, we can provide a connector; please contact us if you are interested in this option.

In terms of development time and effort, to launch a site with custom data in compatible format and no UI customization, you can expect it to take less than three weeks. If you need substantial UI customization it may take up to four months.
Expand Down
40 changes: 26 additions & 14 deletions custom_dc/manage_repo.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,12 +21,23 @@ Data Commons provides two prebuilt images in the Google Artifact Registry that y
- `gcr.io/datcom-ci/datacommons-website-compose:stable`. This is a tested, stable version but may be several weeks old.
- `gcr.io/datcom-ci/datacommons-website-compose:latest`. This is the latest version built from head.

If you want to pick up the latest prebuilt version, run the following command from the `website` directory:
If you want to pick up the latest prebuilt version, do the following:

```shell
docker pull gcr.io/datcom-ci/datacommons-website-compose:latest
```
Then, [restart Docker](quickstart.md#start-services), specifying that repo as the argument to the `docker run` command.
1. From the root directory (e.g. `website`), run the following command:

```shell
docker pull gcr.io/datcom-ci/datacommons-website-compose:latest
```
1. Rerun the container, specifying that repo as the argument to the `docker run` command:

```shell
docker run -it \
-p 8080:8080 \
-e DEBUG=true \
- -env-file $PWD/custom_dc/sqlite_env.list \
- v $PWD/custom_dc/sample:/userdata \
gcr.io/datcom-ci/datacommons-website-compose:latest
```

## Build a local image {#build-repo}

Expand All @@ -47,17 +58,18 @@ If you are using a version control system other than Github, you can download a

In Github, use the following procedure.

1. If you want to reuse the `website` directory you previously created and cloned, skip to step 3.
1. If you want to reuse the root directory you previously created and cloned, skip to step 3.
If you want to create a new source directory and start from scratch, clone the repo up to the stable release tag:

```
git clone https://github.com/datacommonsorg/website --branch customdc_stable --single-branch
```
1. Change to the `website` directory (or whatever you have named it):
<pre>
git clone https://github.com/datacommonsorg/website --branch customdc_stable --single-branch [<var>DIRECTORY</var>]
</pre>
1. Change to the root directory:

<pre>
cd website | cd <var>DIRECTORY</var>
</pre>

```
cd website
```
1. Create a new branch synced to the stable release:

<pre>
Expand Down Expand Up @@ -136,7 +148,7 @@ docker run -it \
--env-file $PWD/custom_dc/sqlite_env.list \
-p 8080:8080 \
-e DEBUG=true \
[-v $PWD/custom_dc/<var>CUSTOM_DATA_DIRECTORY</var>:/userdata \]
[-v $PWD/custom_dc/<var>CUSTOM_DATA_DIRECTORY</var>:/userdata \]
[-v $PWD/server/templates/custom_dc/custom:/workspace/server/templates/custom_dc/custom \]
[-v $PWD/static/custom_dc/custom:/workspace/static/custom_dc/custom \]
datacommons-website-compose:<var>DOCKER_TAG</var>
Expand Down
37 changes: 20 additions & 17 deletions custom_dc/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,13 @@ This page shows you how to run a local custom Data Commons instance inside a Doc
## Prerequisites

- Obtain a [GCP](https://console.cloud.google.com/welcome) billing account and project.
- Install [Docker Engine](https://docs.docker.com/engine/install/).
- If you are developing on Windows, install [WSL 2](https://learn.microsoft.com/en-us/windows/wsl/install) (any distribution will do, but we recommend the default, Ubuntu), and enable [WSL 2 integration with Docker](https://docs.docker.com/desktop/wsl/).
- Install [Docker Desktop/Engine](https://docs.docker.com/engine/install/).
- Install [Git](https://git-scm.com/).
- Get an API key for Data Commons by submitting the [Data Commons API key request form](https://docs.google.com/forms/d/e/1FAIpQLSePrkVfss9lUIHFClQsVPwPcAVWvX7WaZZyZjJWS99wRQNW4Q/viewform?resourcekey=0-euQU6Kly7YIWVRNS2p4zjw). The key is needed to authorize requests from your site to the base Data Commons site. Typical turnaround times are 24-48 hours.
- Optional: Get a [Github](http://github.com) account, if you would like to browse the Data Commons source repos using your browser.

## One-time setup steps
## One-time setup steps {#setup}

### Enable Google Cloud APIs and get a Maps API key {#maps-key}

Expand All @@ -42,30 +43,31 @@ This page shows you how to run a local custom Data Commons instance inside a Doc

### Clone the Data Commons repository

**Note:** If you are using WSL on Windows, open the Linux distribution app as your command shell. You must use the Linux-style file structure for Data Commons to work correctly.

1. Open a terminal window, and go to a directory to which you would like to download the Data Commons repository.
1. Clone the website Data Commons repository:

```shell
git clone https://github.com/datacommonsorg/website.git
```
<pre>
git clone https://github.com/datacommonsorg/website.git [<var>DIRECTORY</var>]
</pre>
If you don't specify a directory name, this creates a local `website` subdirectory. If you specify a directory name, all files are created under that directory, without a `website` subdirectory.

This creates a local `website` subdirectory.
When the downloads are complete, navigate to the root directory of the repo (e.g. `website`). References to various files and commands in these procedures are relative to this root.

1. When the downloads are complete, navigate to the root directory of the repo, `website`. References to various files and commands in these procedures are relative to this root.
<pre>
cd website | <var>DIRECTORY</var>
</pre>

```shell
cd website
```

### Set API keys as environment variables
### Set API keys as environment variables

1. Using your favorite editor, open `custom_dc/sqlite_env.list`.
1. Enter the relevant values for `DC_API_KEY` and `MAPS_API_KEY`.
1. Leave `ADMIN_SECRET` blank for now.

Warning: Do not use any quotes (single or double) or spaces when specifying the values.

Note: If you are storing your source code in a public/open-source version control system, we recommend that you do not store the environment variables files containing secrets. Instead, store them locally only.
Note: If you are storing your source code in a public/open-source version control system, we recommend that you do not store the environment variables files containing secrets. Instead, store them locally only. If you are using Git/Github to manage your code, you can add the file name to the `.gitignore` file.

## About the downloaded files

Expand Down Expand Up @@ -106,9 +108,8 @@ Note: If you are storing your source code in a public/open-source version contro

## Start the services {#start-services}

From the root directory, `website`, run Docker as follows.

Note: If you are running on Linux, depending on whether you have created a ["sudoless" Docker group](https://docs.docker.com/engine/install/linux-postinstall/), you will need to preface every `docker` invocation with `sudo`.
1. If you are running on Windows or Mac, start Docker Desktop and ensure that the Docker Engine is running.
1. Open a terminal window, and from the root directory, run the following command:

```shell
docker run -it \
Expand All @@ -119,6 +120,8 @@ docker run -it \
gcr.io/datcom-ci/datacommons-website-compose:stable
```

Note: If you are running on Linux, depending on whether you have created a ["sudoless" Docker group](https://docs.docker.com/engine/install/linux-postinstall/), you will need to preface every `docker` invocation with `sudo`.

This command does the following:

- The first time you run it, downloads the latest stable Data Commons image, `gcr.io/datcom-ci/datacommons-website-compose:stable`, from the Google Cloud Artifact Registry, which may take a few minutes. Subsequent runs use the locally stored image.
Expand All @@ -135,7 +138,7 @@ If you need to restart the services for any reason, do the following:

Tip: If you close the terminal window in which you started the Docker container, you can kill it as follows:

1. Open another terminal window, and from the `website` directory, get the Docker container ID.
1. Open another terminal window, and from the root directory, get the Docker container ID.

```shell
docker ps
Expand Down
12 changes: 12 additions & 0 deletions custom_dc/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,18 @@ docker: Error response from daemon: pull access denied for datacommons-website-c
1. Use `sudo` with your `docker` invocations or set up a "sudoless" docker group, as described in [Linux post-installation steps for Docker Engine](https://docs.docker.com/engine/install/linux-postinstall/).
1. If you've just installed Docker, try rebooting the machine.

## Startup errors

### "Failed to create metadata: failed to create secret manager client: google: could not find default credentials."

If you try to run the services and fail with this error:

```
Failed to create metadata: failed to create secret manager client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
```

This indicates that you have not specified API keys in the environment file. Follow procedures in [One-time setup steps](/custom_dc/quickstart.html#setup) to obtain and configure API keys.

## Local build errors

### "file not found in build context"
Expand Down

0 comments on commit d0d1805

Please sign in to comment.