Updating Docker build configuration to fix missing calabash. (#1005)

Also made docs Docker container dependent on the build container, which should ease future maintenance.
Updated readmes to address docker 20.10 compose syntax, and to add new dependencies.
Changed location of profile-resolution.html.
Resolves #1004.

build/Dockerfile

@@ -1,12 +1,17 @@
-FROM ubuntu:20.04
+FROM ubuntu:20.04 as base-environment
 
 ARG saxonversion
 ARG hugoversion
+ARG calabashversion
 
 ENV TZ=US/Eastern
 RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
 
-RUN apt-get update && apt-get install -y wget apt-utils libxml2-utils jq maven nodejs npm build-essential python3-pip git && apt-get clean
+RUN apt-get update && apt-get dist-upgrade -y
+
+FROM base-environment as oscal-dependencies
+
+RUN apt-get install -y apt-utils build-essential git jq libxml2-utils maven nodejs npm python3-pip unzip wget && apt-get clean
 RUN npm install -g npm n
 RUN n latest
 RUN npm install --loglevel verbose -g ajv-cli@"^4.0.x" ajv-formats@"^1.5.x" json-diff markdown-link-check yaml-convert@"^1.0.x" yargs
@@ -15,15 +20,28 @@ RUN pip3 install lxml
 #RUN useradd --create-home --home-dir /home/user user
 #USER user
 
-RUN mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=${saxonversion}
+ENV SAXON_VERSION ${saxonversion}
+
+RUN mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=${SAXON_VERSION}
 
-RUN wget https://github.com/gohugoio/hugo/releases/download/v${hugoversion}/hugo_extended_${hugoversion}_Linux-64bit.deb
-RUN dpkg -i hugo_extended_${hugoversion}_Linux-64bit.deb
+ENV HUGO_VERSION ${hugoversion}
+RUN wget https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_Linux-64bit.deb
+RUN dpkg -i hugo_extended_${HUGO_VERSION}_Linux-64bit.deb
+
+# calabash
+ENV CALABASH_VERSION ${calabashversion}
+ENV CALABASH_HOME /dependencies/calabash
+RUN wget https://github.com/ndw/xmlcalabash1/releases/download/${CALABASH_VERSION}/xmlcalabash-${CALABASH_VERSION}.zip
+RUN mkdir -p "${CALABASH_HOME}"
+RUN unzip -d "${CALABASH_HOME}" "xmlcalabash-${CALABASH_VERSION}.zip"
+RUN f=`ls -d "${CALABASH_HOME}"/*| xargs` && mv "${CALABASH_HOME}"/*/* "${CALABASH_HOME}" && rmdir "${f}"
 
 #RUN chown -R user:user /home/user
 
-ENV SAXON_VERSION ${saxonversion} 
+FROM oscal-dependencies as oscal-base
 VOLUME ["/oscal"]
 WORKDIR /oscal
 
+FROM oscal-base as cli
+
 ENTRYPOINT ["/bin/bash"]

build/README.md

@@ -6,7 +6,7 @@ This subdirectory contains a set of build scripts used to create OSCAL-related a
 
 If using Docker:
 
-- [Docker 19.03+](https://docs.docker.com/install/)
+- [Docker 20.10+](https://docs.docker.com/install/)
 
 If not using Docker:
 
@@ -26,7 +26,7 @@ A Docker container configuration is provided that establishes the runtime enviro
     You can build the Docker container for the build environment using Docker Compose as follows from the OSCAL `/build` directory:
 
     ```
-    docker-compose build
+    docker compose build
     ```
 
 3. Run the Docker container
@@ -36,14 +36,14 @@ A Docker container configuration is provided that establishes the runtime enviro
     You can run the Docker container for the build environment using Docker Compose as follows:
 
     ```
-    docker-compose run cli
+    docker compose run cli
     ```
 
     On Windows environments, you may need to execute in a pty that allows for using an interactive shell. In such a case you can run the Docker container as follows:
 
 
     ```
-    winpty docker-compose run cli
+    winpty docker compose run cli
     ```
 
     This should launch an interactive shell.
@@ -58,12 +58,16 @@ The following steps are known to work on [Ubuntu](https://ubuntu.com/) (tested i
 
     - SAXON_VERSION - Defines which version of Saxon-HE to use
     - HUGO_VERSION - Defines which version of Hugo to use
+    - CALABASH_VERSION - Defines which version of XML Calabash to use
+    - CALABASH_HOME - Defines where calabash will be installed
 
     The following is an example of how to configure the environment.
 
     ```bash
     export SAXON_VERSION="9.9.1-3"
-    export HUGO_VERSION="0.74.3"
+    export HUGO_VERSION="0.83.1"
+    export CALABASH_VERSION="1.2.5-100"
+    export CALABASH_HOME="$HOME/calabash"
     ```
 
     You may want to add this export to your `~/.bashrc` to persist the configuration.
@@ -117,6 +121,16 @@ The following steps are known to work on [Ubuntu](https://ubuntu.com/) (tested i
     mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=${SAXON_VERSION}
     ```
 
+1. Install Calabash
+
+    To install Calabash, run the following:
+
+    ```bash
+    wget https://github.com/ndw/xmlcalabash1/releases/download/${CALABASH_VERSION}/xmlcalabash-${CALABASH_VERSION}.zip
+    mkdir -p "${CALABASH_HOME}"
+    unzip -d "${CALABASH_HOME}" "xmlcalabash-${CALABASH_VERSION}.zip"
+    mv "${CALABASH_HOME}"/*/* "${CALABASH_HOME}"
+
 Your environment should be setup.
 
 ## Running the Build Scripts

build/ci-cd/README.md

@@ -150,18 +150,15 @@ export SAXON_VERSION=9.9.1-3
 mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=$SAXON_VERSION
 ```
 
-mkdir $HOME/oscal-oss
+You will also need a copy of the ISO Schematron skeleton.
 
+```
+mkdir $HOME/oscal-oss
 export SCHEMATRON_HOME=$HOME/oscal-oss/git-schematron
 git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME"
 cd "$SCHEMATRON_HOME"
 git checkout master -- trunk/schematron/code
-
-
-export OSCAL_TOOLS_DIR=$HOME/oscal-oss/oscal_tools
-git clone --depth 1 https://github.com/usnistgov/oscal-tools.git "${OSCAL_TOOLS_DIR}"
-cd $OSCAL_TOOLS_DIR/json-cli
-mvn clean install
+```
 
 Finally, export instructions in the preceding must also be copied into .bashrc so they persist in your environment.
 
@@ -171,7 +168,6 @@ prettyson
 sudo npm install -g prettyjson
 ```
 
-
 jq
 
 ```

build/ci-cd/generate-specification-documentation.sh

@@ -57,7 +57,7 @@ if [ "$VERBOSE" = "true" ]; then
 fi
 
 SPEC_SOURCE="${OSCALDIR}/src/specifications/profile-resolution/profile-resolution-specml.xml"
-SPEC_OUTPUT="$WORKING_DIR/docs/content/documentation/processing/profile-resolution.html"
+SPEC_OUTPUT="$WORKING_DIR/docs/content/concepts/processing/profile-resolution.html"
 
 result=$(xsl_transform "$OSCALDIR/src/specifications/profile-resolution/specml-html-hugo-uswds.xsl" "${SPEC_SOURCE}" "${SPEC_OUTPUT}" 2>&1)
 cmd_exitcode=$?

build/docker-compose.yml

@@ -5,11 +5,14 @@ services:
     tty: true
     build:
       context: .
+      target: cli
       args:
         saxonversion: 9.9.1-3
         hugoversion: 0.83.1
+        calabashversion: 1.2.5-100
     volumes:
       - "../:/oscal"
-    environment:
-      - SAXON_VERSION=9.9.1-3
-      - JSON_CLI_VERSION=0.0.1-SNAPSHOT
+#    environment:
+#      - SAXON_VERSION=9.9.1-3
+#      - JSON_CLI_VERSION=0.0.1-SNAPSHOT
+#      - CALABASH_VERSION=1.2.5-100

docs/README.md

@@ -9,7 +9,7 @@ The website is built using the [Hugo](https://gohugo.io/) static site generator
 If using Docker:
 
 - [Saxon-HE for Java](http://saxon.sourceforge.net/#F9.9HE)
-- [Docker 19.03+](https://docs.docker.com/install/)
+- [Docker 20.10+](https://docs.docker.com/install/)
 
 If not using Docker:
 
@@ -35,7 +35,7 @@ Instructions for installing the Hugo CLI on your OS can be found [here](https://
 
 The website's visual styling is also backed by the U.S. Web Design System (USWDS) via an open source Hugo theme at https://github.com/usnistgov/hugo-uswds.
 
-The USWDS framework, a Jekyll customization we are using, is documented here: https://designsystem.digital.gov/.
+The USWDS framework is documented here: https://designsystem.digital.gov/.
 
 ### Building the site with LiveReload
 
@@ -86,8 +86,8 @@ The website can also be developed and built using the included Docker resources.
 Assuming you've [installed Docker](https://docs.docker.com/install/) and [Docker Compose](https://docs.docker.com/compose/install/) for your system, you can build and serve the site using Docker Compose as follows:
 
 ```
-docker-compose build
-docker-compose up
+docker compose build
+docker compose up
 ```
 
 Once the site is running, it can be accessed at http://localhost:1313/OSCAL. Whenever you make any changes to the content with the Hugo server running, you'll notice that the site automatically updates itself to reflect those changes.