Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename services container throughout #474

Merged
merged 70 commits into from
Aug 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
6af1ba7
integrate custom docs with new UI
kmoscoe Jul 11, 2024
cd0fe76
more edits
kmoscoe Jul 11, 2024
c45d61b
use website wording for intro
kmoscoe Jul 15, 2024
f5cbbca
fix numbering in table
kmoscoe Jul 15, 2024
90c57e7
Merge branch 'master' into custom_dc
kmoscoe Jul 15, 2024
57ca62f
rename and some edits
kmoscoe Jul 15, 2024
2277cb1
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Jul 15, 2024
fb33722
rename manage_repo file, per Bo
kmoscoe Jul 16, 2024
0ebdd67
Merge.
kmoscoe Jul 17, 2024
e3148c4
merge
kmoscoe Jul 17, 2024
ee5f580
Merge branch 'datacommonsorg:master' into master
kmoscoe Jul 22, 2024
5993fb7
Merge branch 'master' of https://github.com/datacommonsorg/docsite
kmoscoe Jul 22, 2024
61ce06d
Merge branch 'custom_dc'
kmoscoe Jul 22, 2024
2b37137
Merge branch 'datacommonsorg:master' into master
kmoscoe Jul 22, 2024
37f3e87
formatting edits
kmoscoe Jul 23, 2024
d83db88
updates per Keyur's feedback
kmoscoe Jul 24, 2024
03b906f
Fix typos
kmoscoe Jul 24, 2024
03538af
fix nav order
kmoscoe Jul 24, 2024
d7fb58f
fix link to API key request form
kmoscoe Jul 24, 2024
c746a7e
update form link
kmoscoe Jul 24, 2024
d5e04af
update key request form and output dir env var
kmoscoe Jul 24, 2024
0183130
Merge branch 'master' into custom_dc
kmoscoe Jul 25, 2024
b6f53cf
Revert to gerund
kmoscoe Jul 26, 2024
e538860
Merge branch 'master' into master
kmoscoe Jul 26, 2024
d368ba4
new troubleshooting entry
kmoscoe Jul 31, 2024
db99da0
Merge branch 'datacommonsorg:master' into master
kmoscoe Jul 31, 2024
535962c
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Jul 31, 2024
fd071e9
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Jul 31, 2024
771ccca
fix typo
kmoscoe Jul 31, 2024
7f2bccf
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 1, 2024
7286ff2
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 5, 2024
6e61dac
new data container procedures
kmoscoe Aug 5, 2024
24ea3c4
more work
kmoscoe Aug 5, 2024
03b9786
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 6, 2024
534461b
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 6, 2024
d4d81c8
more work
kmoscoe Aug 6, 2024
5d52e4b
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 6, 2024
8054ac7
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 6, 2024
166e7e1
complete data draft
kmoscoe Aug 6, 2024
079fe62
more changes
kmoscoe Aug 7, 2024
5ccaba5
more changes
kmoscoe Aug 8, 2024
6d80bcc
more revisions
kmoscoe Aug 8, 2024
21917fa
update troubleshooting doc etc.
kmoscoe Aug 9, 2024
586973a
new version of diagrams
kmoscoe Aug 12, 2024
c207f6b
remove data loading problems troubleshooting entry; can't reproduce
kmoscoe Aug 12, 2024
6fa91d3
revert title change
kmoscoe Aug 12, 2024
9cde771
add example for not mixing entity types
kmoscoe Aug 12, 2024
83bbaf2
changes from Keyur
kmoscoe Aug 12, 2024
86800ba
add screenshots for GCP, and related changes
kmoscoe Aug 13, 2024
1a23559
fixed one image
kmoscoe Aug 13, 2024
b4e6bd4
added screenshots for Cloud Run service
kmoscoe Aug 13, 2024
a68f8b4
resize images
kmoscoe Aug 13, 2024
05b1894
more changes from Keyur
kmoscoe Aug 13, 2024
c606037
fix a tiny error
kmoscoe Aug 13, 2024
9bb0355
delete unused images
kmoscoe Aug 13, 2024
c497e03
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 13, 2024
914bc76
Merge branch 'datacommonsorg:master' into custom_dc
kmoscoe Aug 13, 2024
75e9b41
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 13, 2024
edcb5bc
fix missing dash
kmoscoe Aug 13, 2024
e8ca73e
update services container name
kmoscoe Aug 14, 2024
cab4484
Merge branch 'datacommonsorg:master' into master
kmoscoe Aug 14, 2024
529b734
Merge branch 'master' into custom_dc
kmoscoe Aug 14, 2024
740a1e6
Merge branch 'custom_dc' of https://github.com/kmoscoe/docsite into c…
kmoscoe Aug 14, 2024
9bf96d7
Merge branch 'master' of https://github.com/kmoscoe/docsite into cust…
kmoscoe Aug 14, 2024
ae46ea2
update build file
kmoscoe Aug 14, 2024
4ce0fb9
adjust build command
kmoscoe Aug 14, 2024
68a3f51
Revert "adjust build command"
kmoscoe Aug 14, 2024
d8f1ab7
update docker file
kmoscoe Aug 14, 2024
d67d280
more fixes
kmoscoe Aug 14, 2024
15a0a18
one last fix
kmoscoe Aug 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 20 additions & 20 deletions custom_dc/build_image.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,26 +18,26 @@ While you are just testing out data changes, you don't need to build the website

Data Commons provides two prebuilt images in the Google Artifact Registry that you can download to run in a Docker container:

- `gcr.io/datcom-ci/datacommons-website-compose:stable`. This is a tested, stable version but may be several weeks old.
- `gcr.io/datcom-ci/datacommons-website-compose:latest`. This is the latest version built from head.
- `gcr.io/datcom-ci/datacommons-services:stable`. This is a tested, stable version but may be several weeks old.
- `gcr.io/datcom-ci/datacommons-services:latest`. This is the latest version built from head.

If you want to pick up the latest prebuilt version, do the following:

1. From the root directory (e.g. `website`), run the following command:

```shell
docker pull gcr.io/datcom-ci/datacommons-website-compose:latest
docker pull gcr.io/datcom-ci/datacommons-services:latest
```
1. Rerun the container, specifying that repo as the argument to the `docker run` command:

```shell
docker run -it \
-p 8080:8080 \
-e DEBUG=true \
--env-file $PWD/custom_dc/env.list \
-v $PWD/custom_dc/sample:$PWD/custom_dc/sample \
gcr.io/datcom-ci/datacommons-website-compose:latest
```
```shell
docker run -it \
-p 8080:8080 \
-e DEBUG=true \
--env-file $PWD/custom_dc/env.list \
-v $PWD/custom_dc/sample:$PWD/custom_dc/sample \
gcr.io/datcom-ci/datacommons-services:latest
```

## Build a local image {#build-repo}

Expand All @@ -61,9 +61,9 @@ In Github, use the following procedure.
1. If you want to reuse the root directory you previously created and cloned, skip to step 3.
If you want to create a new source directory and start from scratch, clone the repo up to the stable release tag:

<pre>
git clone https://github.com/datacommonsorg/website --branch customdc_stable --single-branch [<var>DIRECTORY</var>]
</pre>
<pre>
git clone https://github.com/datacommonsorg/website --branch customdc_stable --single-branch [<var>DIRECTORY</var>]
</pre>
1. Change to the root directory:

<pre>
Expand All @@ -88,7 +88,7 @@ If you want to create a new source directory and start from scratch, clone the r
...
```

Verify that the last commit in the output matches that listed in [https://github.com/datacommonsorg/website/releases/tag/customdc_stable](https://github.com/datacommonsorg/website/releases/tag/customdc_stable).
Verify that the last commit in the output matches that listed in https://github.com/datacommonsorg/website/releases/tag/customdc_stable.

1. Press `q` to exit the output log.

Expand Down Expand Up @@ -124,12 +124,12 @@ If you want to create a new source directory and start from scratch, clone the r
Run the following command to build the repo:

<pre>
docker build --tag datacommons-website-compose:<var>DOCKER_TAG</var> \
-f build/web_compose/Dockerfile \
-t website-compose .
docker build --tag <var>IMAGE_NAME</var>:<var>IMAGE_TAG</var> \
-f build/cdc_services/Dockerfile .
</pre>

The _DOCKER_TAG_ is a meaningful description of the version you are building.
- The image name is a meaningful name, such as `datacommons-services`.
- The image tag is a meaningful description of the version you are building, such as `latest`.

It will take several minutes to build.

Expand All @@ -149,5 +149,5 @@ docker run -it \
-v <var>OUTPUT_DIRECTORY</var>:<var>OUTPUT_DIRECTORY</var> \
[-v $PWD/server/templates/custom_dc/custom:/workspace/server/templates/custom_dc/custom \]
[-v $PWD/static/custom_dc/custom:/workspace/static/custom_dc/custom \]
datacommons-website-compose:<var>DOCKER_TAG</var>
<var>IMAGE_NAME</var>:<var>IMAGE_TAG</var>
</pre>
4 changes: 2 additions & 2 deletions custom_dc/custom_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -276,9 +276,9 @@ In another terminal window, from the root directory, run the following command t
docker run -it \
-p 8080:8080 \
-e DEBUG=true \
--env-file $PWD/custom_dc/env.list \ \
--env-file $PWD/custom_dc/env.list \
-v <var>OUTPUT_DIRECTORY</var>:<var>OUTPUT_DIRECTORY</var> \
gcr.io/datcom-ci/datacommons-website-compose:stable
gcr.io/datcom-ci/datacommons-services:stable
</pre>

Any time you make changes to the CSV or JSON files and want to reload the data, you will need to rerun the data management container, and then restart the services container.
Expand Down
2 changes: 1 addition & 1 deletion custom_dc/data_cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ docker run -it \
-v <var>OUTPUT_DIRECTORY</var>:<var>OUTPUT_DIRECTORY</var> \
[-v $PWD/server/templates/custom_dc/custom:/workspace/server/templates/custom_dc/custom \]
[-v $PWD/static/custom_dc/custom:/workspace/static/custom_dc/custom \]
datacommons-website-compose:<var>DOCKER_TAG</var>
<var>IMAGE_NAME</var>:<var>IMAGE_TAG</var>
</pre>


Expand Down
15 changes: 7 additions & 8 deletions custom_dc/deploy_cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,26 +47,25 @@ This procedure creates a "dev" Docker package that you upload to the Google Clou
1. Generate credentials for the Docker package you will build in the next step. Docker package names must be in the format <code><var>LOCATION</var>-docker-pkg.dev</code>, where the _LOCATION_ is the region you have selected in the repository creation step previously; for example, `us-central1`.

<pre>
gcloud auth configure-docker <var>LOCATION</var>-docker.pkg.dev
gcloud auth configure-docker <var>LOCATION</var>-docker.pkg.dev
</pre>

When prompted to confirm creating the credentials file, click `Y` to accept.

1. Create a package from the source image created in step 1:

<pre>
docker tag datacommons-website-compose:<var>DOCKER_TAG</var> \
<var>LOCATION</var>-docker.pkg.dev/<var>PROJECT_ID</var>/<var>ARTIFACT_REPO</var>/<var>IMAGE_NAME</var>:<var>TARGET_IMAGE_TAG</var>
docker tag <var>SOURCE_IMAGE_NAME</var>:<var>SOURCE_IMAGE_TAG</var> \
<var>LOCATION</var>-docker.pkg.dev/<var>PROJECT_ID</var>/<var>ARTIFACT_REPO</var>/<var>TARGET_IMAGE_NAME</var>:<var>TARGET_IMAGE_TAG</var>
</pre>

- The `_ARTIFACT_REPO`_ must be an Artifact Registry repository you have created previously.
- The `_IMAGE_NAME`_ may be the same as the source (`datacommons-website-compose`) or any other string.
- The _`TARGET_IMAGE_TAG`_ can be the same as the source, or any other string.

- The artifact repo must be an Artifact Registry repository you have created previously.
- The target image name and tag can be the same as the source or different.

1. Push the image to the registry:

<pre>
docker push <var>LOCATION</var>-docker.pkg.dev/<var>PROJECT_ID</var>/<var>ARTIFACT_REPO</var>/<var>IMAGE_NAME</var>:<var>TARGET_IMAGE_TAG</var>
docker push <var>LOCATION</var>-docker.pkg.dev/<var>PROJECT_ID</var>/<var>ARTIFACT_REPO</var>/<var>TARGET_IMAGE_NAME</var>:<var>TARGET_IMAGE_TAG</var>
</pre>

This will take several minutes to upload.
Expand Down
16 changes: 8 additions & 8 deletions custom_dc/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,12 +124,12 @@ To load the sample data:
1. If you are running on Windows or Mac, start Docker Desktop and ensure that the Docker Engine is running.
1. Open a terminal window, and from the root directory, run the following command to run the data management Docker container:

```shell
docker run \
--env-file $PWD/custom_dc/env.list \
-v $PWD/custom_dc/sample:$PWD/custom_dc/sample \
gcr.io/datcom-ci/datacommons-data:stable
```
```shell
docker run \
--env-file $PWD/custom_dc/env.list \
-v $PWD/custom_dc/sample:$PWD/custom_dc/sample \
gcr.io/datcom-ci/datacommons-data:stable
```
This does the following:

- The first time you run it, downloads the latest stable Data Commons data image, `gcr.io/datcom-ci/datacommons-data:stable`, from the Google Cloud Artifact Registry, which may take a few minutes. Subsequent runs use the locally stored image.
Expand All @@ -151,14 +151,14 @@ docker run -it \
-e DEBUG=true \
--env-file $PWD/custom_dc/env.list \
-v $PWD/custom_dc/sample:$PWD/custom_dc/sample \
gcr.io/datcom-ci/datacommons-website-compose:stable
gcr.io/datcom-ci/datacommons-services:stable
```

Note: If you are running on Linux, depending on whether you have created a ["sudoless" Docker group](https://docs.docker.com/engine/install/linux-postinstall/), you may need to preface every `docker` invocation with `sudo`.

This command does the following:

- The first time you run it, downloads the latest stable Data Commons image, `gcr.io/datcom-ci/datacommons-website-compose:stable`, from the Google Cloud Artifact Registry, which may take a few minutes. Subsequent runs use the locally stored image.
- The first time you run it, downloads the latest stable Data Commons image, `gcr.io/datcom-ci/datacommons-services:stable`, from the Google Cloud Artifact Registry, which may take a few minutes. Subsequent runs use the locally stored image.
- Starts a services Docker container.
- Starts development/debug versions of the Web Server, NL Server, and Mixer, as well as the Nginx proxy, inside the container.
- Maps the output sample data to a Docker path.
Expand Down
2 changes: 1 addition & 1 deletion custom_dc/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ dial unix /var/run/docker.sock: connect: permission denied.
or this:

```
docker: Error response from daemon: pull access denied for datacommons-website-compose, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
docker: Error response from daemon: pull access denied for datacommons-services, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
```

1. Use `sudo` with your `docker` invocations or set up a "sudoless" docker group, as described in [Linux post-installation steps for Docker Engine](https://docs.docker.com/engine/install/linux-postinstall/).
Expand Down