Skip to content

Commit

Permalink
Fixing broken links in docs.
Browse files Browse the repository at this point in the history
  • Loading branch information
david-waltermire committed Aug 27, 2021
1 parent 355c529 commit 1a78944
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 48 deletions.
65 changes: 32 additions & 33 deletions build/ci-cd/README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,33 @@
# Continuous Integration and Continuous Deployment (CI/CD)

This GitHub repository uses a CI/CD process that provides automation for:

- Validation of contributed content and models.
- Generation of documentation, content converters, and content in alternate formats.
- Methods to update the [project website](https://pages.nist.gov/OSCAL).

Using a CI/CD process ensures that all artifacts in [pull requests](https://github.com/usnistgov/OSCAL/blob/master/CONTRIBUTING.md) and in the [master branch](https://github.com/usnistgov/OSCAL) are valid and usable.

# Provided Scripts
## Provided Scripts

The primary CircleCI supported CI/CD workflow, ```build```, is uses a series of bash scripts. The following scripts are used in this workflow for all PRs that are submitted:

1. [run-all.sh](run-all.sh): Runs all of the scripts in workflow order.
1. [validate-metaschema.sh](validate-metaschema.sh): Ensures that all [metaschema](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema) are valid according to the [metaschema XML schema](https://github.com/usnistgov/metaschema/blob/master/toolchains/oscal-m2/lib/metaschema.xsd).
1. [validate-metaschema.sh](validate-metaschema.sh): Ensures that all [metaschema](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema) are valid according to the [metaschema XML schema](https://github.com/usnistgov/metaschema/blob/master/toolchains/xslt-M4/validate/metaschema.xsdmetaschema.xsd).
1. [generate-schema.sh](generate-schema.sh): Generates [XML schema](https://github.com/usnistgov/OSCAL/tree/master/xml/schema) and [JSON schema](https://github.com/usnistgov/OSCAL/tree/master/json/schema) files for each OSCAL model based on their [respective metaschemas](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema).
1. [validate-content.sh](validate-content.sh): Validates OSCAL content in the repository's [src](https://github.com/usnistgov/OSCAL/tree/master/src) against the content's respective OSCAL model and format.
1. [generate-content-converters.sh](generate-content-converters.sh): Generates [JSON to XML](https://github.com/usnistgov/OSCAL/tree/master/xml/convert) and [XML to JSON](https://github.com/usnistgov/OSCAL/tree/master/json/convert) content conversion scripts for each OSCAL model based on their [respective metaschemas](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema).
1. [copy-and-convert-content.sh](copy-and-convert-content.sh): Copies selective content from the [src/content](https://github.com/usnistgov/OSCAL/tree/master/src/content) directory, and generates instance of this content in alternate OSCAL formats using the appropriate content converters. Note __This script requires that the process have write access to the Git repository branch that the CI/CD is operating on to work completely.__
1. [validate-content-conversion-round-trips.sh](validate-content-conversion-round-trips.sh): Performs round trip validations of the content from XML->JSON->XML to confirm that the conversions are lossless and that resulting files comply with their schemas.

Additional scripts are provided that generate content for the OSCAL project website, supported by the CircleCI ```build``` workflow on the ```master``` branch.
Additional scripts are provided that generate content for the OSCAL project website, supported by the CircleCI ```build``` workflow on the ```master``` branch.

1. [generate-model-documentation.sh](generate-model-documentation.sh): This is used as part of the site generation pipeline to build HTML documentation for the OSCAL models.

# Running the Scripts Locally
## Running the Scripts Locally

The bash scripts used in the CI/CD workflow can be run locally in the correct environment. In addition to a bash shell, to run these scripts the following must be installed:

- [xmlint](http://xmlsoft.org/xmllint.html)
- [Java](https://www.java.com/en/) 8.0 or greater
- [Apache Maven](https://maven.apache.org/) 3.6.1 or greater
Expand All @@ -36,123 +39,119 @@ Additionally, the following packages need to be installed.

NodeJS Packages:

```
```bash
npm install -g ajv-cli
```

Python Packages:

```
```bash
pip install lxml
```

The Saxon jar file needs to be retrieved. This can be two ways:

1) You can set the environment variable ```SAXON_VERSION``` to the Saxon version to use. This must correspond to a valid Saxon-HE version in [Maven Central](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE). Maven will be used to automatically download the Saxon-HE jar dependency.
1. You can set the environment variable ```SAXON_VERSION``` to the Saxon version to use. This must correspond to a valid Saxon-HE version in [Maven Central](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE). Maven will be used to automatically download the Saxon-HE jar dependency.

For example:
For example:

```
export SAXON_VERSION=9.9.1-3
```
```bash
export SAXON_VERSION=9.9.1-3
```

2) You can manually download any variant of Saxon (e.g., HE, ) from [Saxonica](https://www.saxonica.com/download/java.xml) and set the ```SAXON_HOME``` environment variable to the location of the Saxon JAR.
2. You can manually download any variant of Saxon (e.g., HE, ) from [Saxonica](https://www.saxonica.com/download/java.xml) and set the ```SAXON_HOME``` environment variable to the location of the Saxon JAR.

```
```bash
export SAXON_HOME=~/saxon-jar-directory
```

Finally, the Schematron Skeleton needs to be retrieved. The following commands can be used to download it.

```
```bash
export SCHEMATRON_HOME=~/schematron-skeleton
git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME"
cd "$SCHEMATRON_HOME"
git checkout master -- trunk/schematron/code
```

## Further installation notes
### Further installation notes

Setting up to run in WSL (Windows subsystem for Linux) avoids many Windows-related configuration issues, but requires the entire environment be available from within the subsystem. The easiest way to accomplish this is to install the components into the subsystem directly.

To check whether xmllint is installed:

```
```bash
xmllint -version
```

To install it

```
```bash
sudo apt install libxml2-utils
```

To install Java:

```
```bash
sudo apt install openjdk-11-jre-headless
```

To install NodeJS:

```
```bash
sudo apt install nodejs
```

And npm (Node package manager, separate from NodeJS):

```
```bash
sudo apt install npm
```


To install AJV (nodeJS JSON validator)

```
```bash
npm install -g ajv-cli
```


or if you want to install globally (for all users):

```
```bash
sudo npm install -g ajv-cli
```

Ubuntu comes with Python. To check the version:

```
```bash
python3 -V
```

Scripts were tested with version 3.7.3 but 3.6.8 (the default in Ubuntu under WSL) should work.

Along with Python you also need a library, for which you need pip (a Python installer):

```
```bash
sudo apt install python-pip
pip install lxml
```


To install Maven:

```
```bash
sudo apt install maven
```

Then run Maven to acquire Saxon:


```
```bash
export SAXON_VERSION=9.9.1-3
mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=$SAXON_VERSION
```

You will also need a copy of the ISO Schematron skeleton.

```
```bash
mkdir $HOME/oscal-oss
export SCHEMATRON_HOME=$HOME/oscal-oss/git-schematron
git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME"
Expand All @@ -164,12 +163,12 @@ Finally, export instructions in the preceding must also be copied into .bashrc s

prettyson

```
```bash
sudo npm install -g prettyjson
```

jq

```
```bash
sudo apt-get install jq
```
28 changes: 13 additions & 15 deletions src/metaschema/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@ OSCAL Metaschemas are used to generate other OSCAL artifacts based on the metasc

Use of Metaschemas in OSCAL allow us to maintain seamless and consistent support for multiple OSCAL model formats, including XML and JSON. Content can also be kept up-to-date in multiple formats using generated content converters, and can be validated using generated schema. Adding support for new formats (e.g., YAML) can accomplished by extending the Metaschema tooling to produce schema and converters for other formats.

The Metaschema syntax (an XML application) is also described and constrained with its [own schema](https://github.com/usnistgov/metaschema/blob/master/toolchains/xslt-M4/validate/metaschema.xsd), and with a [Schematron constraints set](https://github.com/usnistgov/metaschema/blob/master/toolchains/xslt-M4/validate/metaschema-check.sch). The latter is able to enforce some of the rules described below.
The Metaschema syntax (an XML application) is also described and constrained with its [own schema](https://github.com/usnistgov/metaschema/blob/master/toolchains/xslt-M4/validate/metaschema.xsd), and with a [Schematron constraints set](https://github.com/usnistgov/metaschema/blob/master/toolchains/xslt-M4/validate/metaschema-composition-check.sch). The latter is able to enforce some of the rules described below.

#### Enumerated values
## Enumerated values

Additionally, flags may be constrained to a set of known values listed in advance.

Expand All @@ -34,6 +34,7 @@ Within it `valid-values`, a `value` element's `@name` attribute assigns the perm
<value name="RIPEMD-160"/>
</valid-values> ...
```

## Metaschema modeling

In the case of field and flag objects, the modeling constraints to be imposed by the result schemas (either XSD or JSON Schema) over the data set, can be determined on the basis of how they are described. Assembly definitions, however, permit not only flags to be assigned to assemblies of the defined type; additionally, they contain a `model` element for a *mode declaration*. This declaration names the subcomponents to be permitted (in documents valid to the target schemas) within any assembly of the type being defined.
Expand All @@ -44,13 +45,12 @@ Among these elements, no single `@named` attribute value (which refers a model c

With these limitations, a model may be defined to contain any mix of fields and assemblies.

* `field` refers to a field definition and permits a single occurrence of the indicated field
* `field/@required='yes'` a field component is to be required in a model by any schema based on the metaschema
* `assembly` refers to an assembly definition and permits a single occurrence of the indicated assembly.
* `fields` - same as `field`, but permits the field to be repeated. In the JSON representation the multiple values are represented as any array unless `@address` is given
* `assemblies` - same as `fields`, but for assemblies. In JSON, this construct is also presented as an array unless there is an `@address`
* `prose` refers to a "region of prose", that is, a section of prose text of unspecified length. In XML, prose is represented in conventional fashion as (a sequence of) `<p>` and list elements (`<ul>` or `<ol>`) perhaps with inline markup (indicating further formatting). For consistency across metaschema applications, the permitted tagging will always be conformant to the same model for prose, managed to reflect (echo) a clean HTML subset. This specification also permits the markup vocabulary to be mapped to a text-based markdown syntax, suitable for use within JSON expressions of the same or similar data.

- `field` refers to a field definition and permits a single occurrence of the indicated field
- `field/@required='yes'` a field component is to be required in a model by any schema based on the metaschema
- `assembly` refers to an assembly definition and permits a single occurrence of the indicated assembly.
- `fields` - same as `field`, but permits the field to be repeated. In the JSON representation the multiple values are represented as any array unless `@address` is given
- `assemblies` - same as `fields`, but for assemblies. In JSON, this construct is also presented as an array unless there is an `@address`
- `prose` refers to a "region of prose", that is, a section of prose text of unspecified length. In XML, prose is represented in conventional fashion as (a sequence of) `<p>` and list elements (`<ul>` or `<ol>`) perhaps with inline markup (indicating further formatting). For consistency across metaschema applications, the permitted tagging will always be conformant to the same model for prose, managed to reflect (echo) a clean HTML subset. This specification also permits the markup vocabulary to be mapped to a text-based markdown syntax, suitable for use within JSON expressions of the same or similar data.

## JSON Enhancement features

Expand All @@ -60,20 +60,18 @@ One problem with zero-or-more cardinality as supported by `fields` and `assembli

This works, but there are also occasions when a much more concise mapping may also be supported -- if the data can be ensured to follow another rule, namely that data elements (string data) can be known to be uniquely-valued. In these cases there is a different option, namely to promote a flag of a particular known (and controlled) type, to a role as "address" -- which can (incidentally) serve as a label on a JSON property, thus improving both presentation, and addressability.

Accordingly, `@address` on `field` or `assemblies` indicates that their contents (components, that is each field or assembly in the series) may be addressed using the flag (attribute) of the given name. So if `address='id'`, for example, and an `id` flag is included on the field or assembly, such flag is assumed to be unique and validable as such (at least within the scope of its parent or containing structure), thus making it suitable for use as a label; consequently, in JSON, the field or assembly can be represented as a labeled property (of an object) rather than an unlabeled member of an array (of similar objects). This both reduces the data footprint and renders the data more addressable via key constructs such as identifiers.
Accordingly, `@address` on `field` or `assemblies` indicates that their contents (components, that is each field or assembly in the series) may be addressed using the flag (attribute) of the given name. So if `address='id'`, for example, and an `id` flag is included on the field or assembly, such flag is assumed to be unique and valid as such (at least within the scope of its parent or containing structure), thus making it suitable for use as a label; consequently, in JSON, the field or assembly can be represented as a labeled property (of an object) rather than an unlabeled member of an array (of similar objects). This both reduces the data footprint and renders the data more addressable via key constructs such as identifiers.

To support this, flags used as addresses should be declared as type `ID`, providing "an extra layer of protection". On the JSON side, validating the uniqueness of these values (on same-named properties across document scope) remains TBD.



```
```xml
<define-field name="title" as="mixed"/>
```

```
```xml
<title>Water (H<sub>2</sub>0)</title>
```

```
```xml
"title" : "Water (H~~2~~0)"
```

0 comments on commit 1a78944

Please sign in to comment.