From d6319825516993c197505c5a6bf79badcd145a4d Mon Sep 17 00:00:00 2001 From: eecavanna Date: Thu, 14 Nov 2024 14:52:49 -0800 Subject: [PATCH 1/4] Containerize the build process and fill in developer documentation --- README.md | 83 +++++++++++++++++++++++++++++++++++++++++++--- builder.Dockerfile | 31 +++++++++++++++++ 2 files changed, 109 insertions(+), 5 deletions(-) create mode 100644 builder.Dockerfile diff --git a/README.md b/README.md index b17459d9..d2b87c62 100644 --- a/README.md +++ b/README.md @@ -20,13 +20,86 @@ Note that the while this repo is named `submission-schema`, the generated artifa ## Developer Documentation -
-Use the `make` command to generate project artefacts: +### Updating the submission schema -- `make all`: make everything -- ~~`make deploy`: deploys site~~ +Here's how you can update the submission schema to use a new version of `nmdc-schema`: -
+1. Update the one occurrence of the `nmdc-schema` version number in `project.Makefile`: + + ```diff + local/nmdc.yaml: + - wget -O $@ https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.0.1/nmdc_schema/nmdc_materialized_patterns.yaml + + wget -O $@ https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.1.0/nmdc_schema/nmdc_materialized_patterns.yaml + ``` + +2. Update all the `nmdc-schema` version numbers in `sheets_and_friends/tsv_in/import_slots_regardless.tsv` + + ```diff + - schema_fp: https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.0.1/nmdc_schema/nmdc_materialized_patterns.yaml; source_class: Biosample; destination_class: BuiltEnvInterface + + schema_fp: https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.1.0/nmdc_schema/nmdc_materialized_patterns.yaml; source_class: Biosample; destination_class: BuiltEnvInterface + ``` + > That is one of the _many_ occurrences in that file. + > + > **Note:** The above search-and-replacement (of all occurrences in that file) can be done inside Vim via: + > ```vim + > :%s/nmdc-schema\/v11.0.1/nmdc-schema\/v11.1.0/g + > ``` + +### Building the submission schema + +Here's how you can generate the submission schema release artifacts: + +#### Container-based process + +##### Prerequisites + +- Docker is installed on your computer +- You are in the root directory of the repository + +##### Procedure + +1. Build the container image you will later use to build the submission schema: + ```shell + docker build -t submission-schema-builder -f builder.Dockerfile . + ``` +2. Use that container image to build the submission schema: + ```shell + docker run --rm -it -v ${PWD}:/submission-schema submission-schema-builder + ``` +3. (Optional) Delete the container image: + ```shell + docker image rm submission-schema-builder + ``` + +#### Direct process + +##### Prerequisites + +- `yq` is installed on your computer, such that the following command shows a version number instead of an error message. + ```shell + bash -c 'yq --version' + ``` + If `yq` is not installed, you can install it by running this, assuming you're using macOS: + ```shell + brew install yq + ``` + +##### Procedure + +1. Install Python dependencies: + ```shell + poetry shell + poetry install + ``` +2. Generate the release artefacts: + ```shell + make all + ``` +3. Commit the changes, using the new `nmdc-schema` version number as the commit message; like this: + ```shell + git add . + git commit -m "11.1.0" + ``` ## Credits diff --git a/builder.Dockerfile b/builder.Dockerfile new file mode 100644 index 00000000..0f19fc2a --- /dev/null +++ b/builder.Dockerfile @@ -0,0 +1,31 @@ +# Introduction: This Dockerfile can be used to build a container image that can, in turn, be used +# to "build" (hence, its name, "builder") the submission schema release artifacts. + +# Use Python 3.9 because that's the Python version listed in `pyproject.toml`. +FROM python:3.9 + +WORKDIR /submission-schema + +# Download and install yq. +# Reference: https://github.com/mikefarah/yq#install +RUN wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq && \ + chmod +x /usr/bin/yq + +# Install Poetry, a package manager for Python (an alternative to pip). +RUN pip install poetry + +# Copy the entire repository into the container image. +# +# Note: Copying _only_ the dependency lists (i.e. `pyproject.toml` and `poetry.lock`) and then +# mounting the entire repository as a volume in the container image at `docker run` time +# results in `make all` failing due to the absence of an `nmdc_submission_schema` package. +# I haven't yet figured out why that happens, but it doesn't happen if I have copied the +# entire repository into the container image here at build time—so, I do the latter. +# +ADD . /submission-schema + +# Install the project's Python dependencies. +RUN poetry install --no-interaction + +# Run the command that builds the submission schema release artifacts. +CMD ["make", "all"] \ No newline at end of file From e17c45743708628358490007fcb2f42760397f86 Mon Sep 17 00:00:00 2001 From: eecavanna Date: Thu, 14 Nov 2024 15:41:59 -0800 Subject: [PATCH 2/4] Add note about `wget` being a dependency of the build process --- README.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/README.md b/README.md index d2b87c62..ff91ef6a 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,14 @@ Here's how you can generate the submission schema release artifacts: ```shell brew install yq ``` +- `wget` is installed on your computer, such that the following command shows a version number instead of an error message. + ```shell + bash -c 'wget --version' + ``` + If `wget` is not installed, you can install it by running this, assuming you're using macOS: + ```shell + brew install wget + ``` ##### Procedure From 13ab9429331a98728324b749fb36ee688e890366 Mon Sep 17 00:00:00 2001 From: eecavanna Date: Thu, 14 Nov 2024 15:44:22 -0800 Subject: [PATCH 3/4] Make indentation amount consistent within a given bullet point --- README.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index ff91ef6a..2e664634 100644 --- a/README.md +++ b/README.md @@ -76,21 +76,21 @@ Here's how you can generate the submission schema release artifacts: ##### Prerequisites - `yq` is installed on your computer, such that the following command shows a version number instead of an error message. - ```shell - bash -c 'yq --version' - ``` - If `yq` is not installed, you can install it by running this, assuming you're using macOS: - ```shell - brew install yq - ``` + ```shell + bash -c 'yq --version' + ``` + If `yq` is not installed, you can install it by running this, assuming you're using macOS: + ```shell + brew install yq + ``` - `wget` is installed on your computer, such that the following command shows a version number instead of an error message. - ```shell - bash -c 'wget --version' - ``` - If `wget` is not installed, you can install it by running this, assuming you're using macOS: - ```shell - brew install wget - ``` + ```shell + bash -c 'wget --version' + ``` + If `wget` is not installed, you can install it by running this, assuming you're using macOS: + ```shell + brew install wget + ``` ##### Procedure From f601b3bea27479ed84f8b9a14d26835f5cc56553 Mon Sep 17 00:00:00 2001 From: eecavanna <134325062+eecavanna@users.noreply.github.com> Date: Fri, 6 Dec 2024 11:08:33 -0800 Subject: [PATCH 4/4] Fix incorrect diff so it shows accurate TSV snippet Co-authored-by: Patrick Kalita --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 2e664634..8adf944e 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,8 @@ Here's how you can update the submission schema to use a new version of `nmdc-sc 2. Update all the `nmdc-schema` version numbers in `sheets_and_friends/tsv_in/import_slots_regardless.tsv` ```diff - - schema_fp: https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.0.1/nmdc_schema/nmdc_materialized_patterns.yaml; source_class: Biosample; destination_class: BuiltEnvInterface - + schema_fp: https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.1.0/nmdc_schema/nmdc_materialized_patterns.yaml; source_class: Biosample; destination_class: BuiltEnvInterface + - Biosample https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.0.1/nmdc_schema/nmdc_materialized_patterns.yaml abs_air_humidity ... + + Biosample https://raw.githubusercontent.com/microbiomedata/nmdc-schema/v11.1.0/nmdc_schema/nmdc_materialized_patterns.yaml abs_air_humidity ... ``` > That is one of the _many_ occurrences in that file. >