Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes to Metapath syntax in constraints #1012

Merged
65 changes: 32 additions & 33 deletions build/ci-cd/README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,33 @@
# Continuous Integration and Continuous Deployment (CI/CD)

This GitHub repository uses a CI/CD process that provides automation for:

- Validation of contributed content and models.
- Generation of documentation, content converters, and content in alternate formats.
- Methods to update the [project website](https://pages.nist.gov/OSCAL).

Using a CI/CD process ensures that all artifacts in [pull requests](https://github.com/usnistgov/OSCAL/blob/master/CONTRIBUTING.md) and in the [master branch](https://github.com/usnistgov/OSCAL) are valid and usable.

# Provided Scripts
## Provided Scripts

The primary CircleCI supported CI/CD workflow, ```build```, is uses a series of bash scripts. The following scripts are used in this workflow for all PRs that are submitted:

1. [run-all.sh](run-all.sh): Runs all of the scripts in workflow order.
1. [validate-metaschema.sh](validate-metaschema.sh): Ensures that all [metaschema](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema) are valid according to the [metaschema XML schema](https://github.com/usnistgov/metaschema/blob/master/toolchains/oscal-m2/lib/metaschema.xsd).
1. [validate-metaschema.sh](validate-metaschema.sh): Ensures that all [metaschema](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema) are valid according to the [metaschema XML schema](https://github.com/usnistgov/metaschema/blob/master/toolchains/xslt-M4/validate/metaschema.xsd).
1. [generate-schema.sh](generate-schema.sh): Generates [XML schema](https://github.com/usnistgov/OSCAL/tree/master/xml/schema) and [JSON schema](https://github.com/usnistgov/OSCAL/tree/master/json/schema) files for each OSCAL model based on their [respective metaschemas](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema).
1. [validate-content.sh](validate-content.sh): Validates OSCAL content in the repository's [src](https://github.com/usnistgov/OSCAL/tree/master/src) against the content's respective OSCAL model and format.
1. [generate-content-converters.sh](generate-content-converters.sh): Generates [JSON to XML](https://github.com/usnistgov/OSCAL/tree/master/xml/convert) and [XML to JSON](https://github.com/usnistgov/OSCAL/tree/master/json/convert) content conversion scripts for each OSCAL model based on their [respective metaschemas](https://github.com/usnistgov/OSCAL/tree/master/src/metaschema).
1. [copy-and-convert-content.sh](copy-and-convert-content.sh): Copies selective content from the [src/content](https://github.com/usnistgov/OSCAL/tree/master/src/content) directory, and generates instance of this content in alternate OSCAL formats using the appropriate content converters. Note __This script requires that the process have write access to the Git repository branch that the CI/CD is operating on to work completely.__
1. [validate-content-conversion-round-trips.sh](validate-content-conversion-round-trips.sh): Performs round trip validations of the content from XML->JSON->XML to confirm that the conversions are lossless and that resulting files comply with their schemas.

Additional scripts are provided that generate content for the OSCAL project website, supported by the CircleCI ```build``` workflow on the ```master``` branch.
Additional scripts are provided that generate content for the OSCAL project website, supported by the CircleCI ```build``` workflow on the ```master``` branch.

1. [generate-model-documentation.sh](generate-model-documentation.sh): This is used as part of the site generation pipeline to build HTML documentation for the OSCAL models.

# Running the Scripts Locally
## Running the Scripts Locally

The bash scripts used in the CI/CD workflow can be run locally in the correct environment. In addition to a bash shell, to run these scripts the following must be installed:

- [xmlint](http://xmlsoft.org/xmllint.html)
- [Java](https://www.java.com/en/) 8.0 or greater
- [Apache Maven](https://maven.apache.org/) 3.6.1 or greater
Expand All @@ -36,123 +39,119 @@ Additionally, the following packages need to be installed.

NodeJS Packages:

```
```bash
npm install -g ajv-cli
```

Python Packages:

```
```bash
pip install lxml
```

The Saxon jar file needs to be retrieved. This can be two ways:

1) You can set the environment variable ```SAXON_VERSION``` to the Saxon version to use. This must correspond to a valid Saxon-HE version in [Maven Central](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE). Maven will be used to automatically download the Saxon-HE jar dependency.
1. You can set the environment variable ```SAXON_VERSION``` to the Saxon version to use. This must correspond to a valid Saxon-HE version in [Maven Central](https://search.maven.org/artifact/net.sf.saxon/Saxon-HE). Maven will be used to automatically download the Saxon-HE jar dependency.

For example:
For example:

```
export SAXON_VERSION=9.9.1-3
```
```bash
export SAXON_VERSION=9.9.1-3
```

2) You can manually download any variant of Saxon (e.g., HE, ) from [Saxonica](https://www.saxonica.com/download/java.xml) and set the ```SAXON_HOME``` environment variable to the location of the Saxon JAR.
2. You can manually download any variant of Saxon (e.g., HE, ) from [Saxonica](https://www.saxonica.com/download/java.xml) and set the ```SAXON_HOME``` environment variable to the location of the Saxon JAR.

```
```bash
export SAXON_HOME=~/saxon-jar-directory
```

Finally, the Schematron Skeleton needs to be retrieved. The following commands can be used to download it.

```
```bash
export SCHEMATRON_HOME=~/schematron-skeleton
git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME"
cd "$SCHEMATRON_HOME"
git checkout master -- trunk/schematron/code
```

## Further installation notes
### Further installation notes

Setting up to run in WSL (Windows subsystem for Linux) avoids many Windows-related configuration issues, but requires the entire environment be available from within the subsystem. The easiest way to accomplish this is to install the components into the subsystem directly.

To check whether xmllint is installed:

```
```bash
xmllint -version
```

To install it

```
```bash
sudo apt install libxml2-utils
```

To install Java:

```
```bash
sudo apt install openjdk-11-jre-headless
```

To install NodeJS:

```
```bash
sudo apt install nodejs
```

And npm (Node package manager, separate from NodeJS):

```
```bash
sudo apt install npm
```


To install AJV (nodeJS JSON validator)

```
```bash
npm install -g ajv-cli
```


or if you want to install globally (for all users):

```
```bash
sudo npm install -g ajv-cli
```

Ubuntu comes with Python. To check the version:

```
```bash
python3 -V
```

Scripts were tested with version 3.7.3 but 3.6.8 (the default in Ubuntu under WSL) should work.

Along with Python you also need a library, for which you need pip (a Python installer):

```
```bash
sudo apt install python-pip
pip install lxml

```


To install Maven:

```
```bash
sudo apt install maven
```

Then run Maven to acquire Saxon:


```
```bash
export SAXON_VERSION=9.9.1-3
mvn org.apache.maven.plugins:maven-dependency-plugin:2.10:get -DartifactId=Saxon-HE -DgroupId=net.sf.saxon -Dversion=$SAXON_VERSION
```

You will also need a copy of the ISO Schematron skeleton.

```
```bash
mkdir $HOME/oscal-oss
export SCHEMATRON_HOME=$HOME/oscal-oss/git-schematron
git clone --depth 1 --no-checkout https://github.com/Schematron/schematron.git "$SCHEMATRON_HOME"
Expand All @@ -164,12 +163,12 @@ Finally, export instructions in the preceding must also be copied into .bashrc s

prettyson

```
```bash
sudo npm install -g prettyjson
```

jq

```
```bash
sudo apt-get install jq
```
2 changes: 1 addition & 1 deletion build/metaschema
Submodule metaschema updated 251 files
36 changes: 15 additions & 21 deletions src/metaschema/oscal_assessment-common_metaschema.xml
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,10 @@
<enum value="objective"></enum>
<enum value="assessment"></enum>
</allowed-values>
<has-cardinality target="part[@name='objective']" min-occurs="0" max-occurs="1" />
<has-cardinality target="part[@name='objective']" max-occurs="1" />
<has-cardinality target="part[@name='assessment']/prop[@name='method']" min-occurs="1" max-occurs="1" />
<has-cardinality target="part[@name='assessment']/part[@name='objects']" min-occurs="1" max-occurs="1" />
<has-cardinality target="part[@name='objective']/prop[@name='method-id']" min-occurs="1" max-occurs="unbounded" />
<has-cardinality target="part[@name='objective']/prop[@name='method-id']" min-occurs="1" />
</constraint>
</define-assembly>
<define-assembly name="assessment-method">
Expand Down Expand Up @@ -157,7 +157,7 @@
<field ref="remarks" in-xml="WITH_WRAPPER" min-occurs="0" max-occurs="1"/>
</model>
<constraint>
<is-unique name="unique-step-responsible-role" target="responsible-role">
<is-unique id="unique-step-responsible-role" target="responsible-role">
<key-field target="@role-id"/>
<remarks>
<p>Since multiple <code>party-uuid</code> entries can be provided, each role-id must be referenced only once.</p>
Expand Down Expand Up @@ -190,7 +190,7 @@
<enum value="EXAMINE">The process of reviewing, inspecting, observing, studying, or analyzing one or more assessment objects (i.e., specifications, mechanisms, or activities).</enum>
<enum value="TEST">The process of exercising one or more assessment objects (i.e., activities or mechanisms) under specified conditions to compare actual with expected behavior.</enum>
</allowed-values>
<is-unique name="unique-activity-responsible-role" target="responsible-role">
<is-unique id="unique-activity-responsible-role" target="responsible-role">
<key-field target="@role-id"/>
<remarks>
<p>Since <code>responsible-role</code> associates multiple <code>party-uuid</code> entries with a single <code>role-id</code>, each role-id must be referenced only once.</p>
Expand Down Expand Up @@ -331,7 +331,7 @@
--> <field ref="remarks" in-xml="WITH_WRAPPER" min-occurs="0" max-occurs="1"/>
</model>
<constraint>
<is-unique name="unique-associated-activity-responsible-role" target="responsible-role">
<is-unique id="unique-associated-activity-responsible-role" target="responsible-role">
<key-field target="@role-id"/>
<remarks>
<p>Since <code>responsible-role</code> associates multiple <code>party-uuid</code> entries with a single <code>role-id</code>, each role-id must be referenced only once.</p>
Expand Down Expand Up @@ -687,7 +687,7 @@
<field ref="remarks" in-xml="WITH_WRAPPER"/>
</model>
<constraint>
<is-unique name="unique-ssp-uses-component-responsible-party" target="responsible-party">
<is-unique id="unique-ssp-uses-component-responsible-party" target="responsible-party">
<key-field target="@role-id"/>
<remarks>
<p>Since <code>responsible-party</code> associates multiple <code>party-uuid</code> entries with a single <code>role-id</code>, each role-id must be referenced only once.</p>
Expand All @@ -700,7 +700,7 @@
</define-assembly>
</model>
<constraint>
<is-unique name="unique-ssp-assessment-assets-component" target="component">
<is-unique id="unique-ssp-assessment-assets-component" target="component">
<key-field target="@uuid"/>
<remarks>
<p>Since multiple assessment <code>component</code> entries can be provided, each component must have a unique <code>uuid</code>.</p>
Expand Down Expand Up @@ -887,18 +887,18 @@
</define-field>
<field ref="remarks" in-xml="WITH_WRAPPER" min-occurs="0" max-occurs="1"/>
</model>
<constraint>
<!-- TODO: review these and figure out where these go -->
<!-- <allowed-values target="origin/@type" allow-other="no">
<!-\- CHANGED: "tool" to "******" -\->
<!-- <constraint>
<!-\- TODO: review these and figure out where these go -\->
<allowed-values target="origin/@type" allow-other="no">
<!-\\- CHANGED: "tool" to "******" -\\->
<enum value="tool">An assessment tool, defined in the assets section of the assessment plan or results.</enum>
<enum value="test-method">A test method defined in the assessment-activities section of the assessment plan or results.</enum>
<enum value="task">A task defined in the schedule of the assessment plan or results.</enum>
<enum value="included-activity">An assessment activity defined in the assessment-activities section of the assessment plan or results.</enum>
<enum value="other">The UUID points elsewhere in the this file or an imported file.</enum>
</allowed-values>
--> </constraint>
</define-assembly>
</constraint>
--> </define-assembly>

<define-assembly name="origin">
<formal-name>Origin</formal-name>
Expand Down Expand Up @@ -994,7 +994,7 @@
<field ref="remarks" in-xml="WITH_WRAPPER" min-occurs="0" max-occurs="1"/>
</model>
<constraint>
<is-unique name="unique-ssp-related-task-responsible-party" target="responsible-party">
<is-unique id="unique-ssp-related-task-responsible-party" target="responsible-party">
<key-field target="@role-id"/>
<remarks>
<p>Since <code>responsible-party</code> associates multiple <code>party-uuid</code> entries with a single <code>role-id</code>, each role-id must be referenced only once.</p>
Expand Down Expand Up @@ -1194,7 +1194,7 @@
<allowed-values target="prop/@name" allow-other="yes">
<enum value="type">The type of remediation tracking entry. Can be multi-valued.</enum>
</allowed-values>
<allowed-values target="prop/[@name='type']/@value" allow-other="yes">
<allowed-values target="prop[@name='type']/@value" allow-other="yes">
<enum value="vendor-check-in">Contacted vendor to determine the status of a pending fix to a known vulnerability.</enum>
<enum value="status-update">Information related to the current state of response to this risk.</enum>
<enum value="milestone-complete">A significant step in the response plan has been achieved.</enum>
Expand Down Expand Up @@ -1587,12 +1587,6 @@
</assembly>
<field ref="remarks" in-xml="WITH_WRAPPER" min-occurs="0" max-occurs="1"/>
</model>
<constraint>
<allowed-values target="origin/@type" allow-other="no">
<enum value="party">The UUID of the person or organization who made the recommendation</enum>
<enum value="tool">The UUID of the tool that made the recommendation</enum>
</allowed-values>
</constraint>
</define-assembly>
<assembly ref="task" min-occurs="0" max-occurs="unbounded">
<group-as name="tasks" in-json="ARRAY"/>
Expand Down
4 changes: 2 additions & 2 deletions src/metaschema/oscal_assessment-plan_metaschema.xml
Original file line number Diff line number Diff line change
Expand Up @@ -67,13 +67,13 @@
<field ref="remarks" in-xml="WITH_WRAPPER" min-occurs="0" max-occurs="1"/>
</model>
<constraint>
<is-unique name="unique-ap-local-definitions-component" target="component">
<is-unique id="unique-ap-local-definitions-component" target="component">
<key-field target="@uuid"/>
<remarks>
<p>Since multiple <code>component</code> entries can be provided, each component must have a unique <code>uuid</code>.</p>
</remarks>
</is-unique>
<is-unique name="unique-ap-local-definitions-user" target="user">
<is-unique id="unique-ap-local-definitions-user" target="user">
<key-field target="@uuid"/>
<remarks>
<p>A given <code>uuid</code> must be assigned only once to a user.</p>
Expand Down
6 changes: 3 additions & 3 deletions src/metaschema/oscal_assessment-results_metaschema.xml
Original file line number Diff line number Diff line change
Expand Up @@ -141,13 +141,13 @@
</assembly>
</model>
<constraint>
<is-unique name="unique-ar-local-definitions-component" target="component">
<is-unique id="unique-ar-local-definitions-component" target="component">
<key-field target="@uuid"/>
<remarks>
<p>Since multiple <code>component</code> entries can be provided, each component must have a unique <code>uuid</code>.</p>
</remarks>
</is-unique>
<is-unique name="unique-ar-local-definitions-user" target="user">
<is-unique id="unique-ar-local-definitions-user" target="user">
<key-field target="@uuid"/>
<remarks>
<p>A given <code>uuid</code> must be assigned only once to a user.</p>
Expand Down Expand Up @@ -179,7 +179,7 @@
</assembly>
</model>
<constraint>
<is-unique name="unique-ar-attestation-responsible-party" target="responsible-party">
<is-unique id="unique-ar-attestation-responsible-party" target="responsible-party">
<key-field target="@role-id"/>
<remarks>
<p>Since <code>responsible-party</code> associates multiple <code>party-uuid</code> entries with a single <code>role-id</code>, each role-id must be referenced only once.</p>
Expand Down
3 changes: 0 additions & 3 deletions src/metaschema/oscal_catalog_metaschema.xml
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,6 @@
</remarks>
</assembly>
</model>
<constraint>

</constraint>
<remarks>
<p>Catalogs may use one or more <code>group</code> objects to subdivide the control contents of a catalog.</p>
<p>An OSCAL catalog model provides a structured representation of control information.</p>
Expand Down
2 changes: 1 addition & 1 deletion src/metaschema/oscal_complete_metaschema.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<!-- OSCAL GRAND UNIFIED MEGALOMETASCHEMA -->
<!-- validate with XSD and Schematron (linked) -->
<!DOCTYPE METASCHEMA [
<!ENTITY allowed-values-control-group-property-name SYSTEM "shared-constraints/allowed-values-control-group-property-name.ent">
<!ENTITY allowed-values-control-group-property-name SYSTEM "./shared-constraints/allowed-values-control-group-property-name.ent">
]>
<METASCHEMA xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0">
<schema-name>OSCAL Unified Model of Models</schema-name>
Expand Down
Loading