Skip to content

Commit

Permalink
Issue 468 364 478 integration (usnistgov#492)
Browse files Browse the repository at this point in the history 
* Removed unnecessary metaschema check on json-value-key

* Now implementing in-xml="with-wrapper | unwrapped" on markup multiline fields, with default `with-wrapper` and corresponding changes to catalog and profile metaschemas.

* Updating catalog and profile schemas for (prose) unwrapping behavior

* Renamed 'json-behavior' in Metaschema to 'in-json' for consistency with 'in-xml'.

* Updating syntax in metaschemas 'in-json' for 'json-behavior'

* Updating json converter generator to provide in-xml behavior 'GROUPED'

* Refining json converter production from Metaschema

* Repairing egregious syntax error

* Rectify bug introduced with new feature

* Adding support for 'date' and 'dateTime' datatypes, with no time zones (usnistgov#480), including unit tests

* Improvements and corrections to datatypes docs

* More small repairs on datatypes descriptions (table markup)

* Built out validations of allowed (enumerated) values, including Metaschema validation checks on whether given allowed values conform to their nominal type.

* Straightening kinks in datatype metaschema meta-validation

* More special handling of recalcitrant datatypes, this time 'NMTOKENS' and 'IDREFS'

* Addressing usnistgov#463, adding `id`, `class` and `title` to `group` in profiles.

* Schematron improvements; documentation revisions (removing references to subcontrols)

* New Schematron for SP800-53-style catalogs

* Removed using in implementation to new metaschema for storage. Started on example, which is still invalid

* More work on an example, which is still not valid

* refactored schematron processing

* more work on SSP example

* Refactored generate-schema.sh to allow a metaschema to be provided on the CLI

* refactored SSP model to be more flat, while allowing component references

* slight cleanup

* old example

* Fixed content generation

* updated metaschema to new specs

* fixed content errors

* added metaschema examples

* Updated metaschema example.

* Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

* Added 'param' to profile 'group' model

* JSON converter now properly handling flags assigned to assemblies identified by json key

* Added 'sort-id' property to controls in SP800-53 catalog

* Added support for direct generation of schema and converters

* Minor updates to SSP example and metaschema

* Adding computer build metaschema examples

* Extending allowed-values support to fields; unit tests: usnistgov#437

* More work on allowed-values, now in XSD with unit tests for usnistgov#437

* Repairing small datatyping bug thanks to regression testing

* Further adjustment for allowed-values in docs

* Renamed unit tests, now working

* Adding allowed-values unit tests

* Addressing usnistgov#452: cleaned up character handling in XML-to-JSON conversion. Fixed the bug this exposed in markdown->HTML conversion. Added some light unit testing.

* More adjustments in view of comments to usnistgov#452

* Repairing more bugs in character handling in Markdown and plain text, with unit testing: usnistgov#452

* Improvements to Schematron for SP800-53 usnistgov#400

* For usnistgov#452, modified copy-and-convert script to suppress character-level post-processing in generated JSON; added unit tests for URIs to detect issues with solidus characters '/'

* Edited docs, including datatypes page per usnistgov#452 (regarding character escapes).

* Cleanup metaschema valdiation errors

* changed use of the ID and IDREF datatype to NCName

* Removed ID and IDREF datatypes

* refactored components out of ssp metaschema

* Updated SP800-53 catalog and NIST profiles to current model

* Added name to information type.

* Continued refinement of the SSP model

* Updated metadata syntax

* changing CM-6 constraint to guidance

* continued work on SSP

* refactored responsible party to metadata

* fixed validation issues

* Fixed overriding enumerated values

* Fixed content validation errors based on latest metaschema changes

* Completed full pass through ssp model

* added prop to resource

* Fixed broken links in Markdown files. Small adjustments to SSP example.

* FDixed relapath error caused when the generated schema directory doesn't exist. Also attempting a fix for CircleCI colorized escape codes causing extra characters to appear.

* Fixed validation issue with component metaschema

* attempting to fix extra characters in CircleCI colorized messages

* Updated enumerated values to remove values that should appear in a given context

* quick reordering of elements

* hardcode ascii color codes

* Fixing broken links in readmes; file name update for consistency

* added unit tests for bug

* fixed color escaping in scripts

* Removed bogus unit test and fixed a small cardinality issue in the SSP model

* Improvements to docs generation

* Added JSON value keys to fields with flags (that did not have them)

* Bug reduction in XSD production mainly related to datatypes and allowed values

* Fixing glitch in JSON converter generator

* Adjustments repairing Metaschema issues

* CSS tweak in docs (model maps to use USWDS font)

* Adding to Milestone 2 release notes (in progress)

* Change to parameter setting in profile model (usnistgov#494, usnistgov#288); release notes towards M2

* Small improvements to copy in release notes (draft)

* Updating metaschemas to latest syntax, with improved Metaschema Schematron and unit test tweakage

* Adjustments to flag unit test and release notes

* SSP Metaschema review and update

* cleanup of unused role-id field

* reflactored role-id

* updated documentation for release.

* cleanup of IDs

* Minor schema and metaschema improvements

* M1 to M2 content converter

* Merging local and upstream changes to json example

* Relaxing rule over 'ID' flag to accept any string not only XML NCName

* reorganizing content for release

* more file moves

* Fixed changed path

* Updated FedRAMP profiles and catalog to current models

* Bug fix to XML-to-JSON converter generator

* Fixed copy script

* Updated FedRAMP baselines and catalog to match new metaschema syntax

* fixed merge error

* fixed another merge error

* fixed spelling issues

* Fix from Wendell
  • Loading branch information
@david-waltermire @bradh
david-waltermire authored and bradh committed Dec 4, 2019
1 parent 854ebef commit 4e7b229
Show file tree
Hide file tree
Showing 142 changed files with 23,658 additions and 3,666 deletions.
1 change: 1 addition & 0 deletions .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@ jobs: # a collection of
- install-maven-dependencies
- install-jq
- install-prettyjson
- install-xmllint
- run:
name: Generate OSCAL converters
command: |
Expand Down
1 change: 1 addition & 0 deletions build/ci-cd/config/content
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# path to source|format of source|model of source|format(s) to convert to
src/content/ssp-example/ssp-example.json|json|ssp|xml
src/content/fedramp.gov/xml/*catalog.xml|xml|catalog|json
src/content/fedramp.gov/xml/*profile.xml|xml|profile|json
src/content/nist.gov/SP800-53/rev4/xml/*catalog.xml|xml|catalog|json
Expand Down
2 changes: 0 additions & 2 deletions build/ci-cd/config/metaschema
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,3 @@ src/metaschema/oscal_catalog_metaschema.xml|xml,json|xml,json|xml,json
src/metaschema/oscal_profile_metaschema.xml|xml,json|xml,json|xml,json
src/metaschema/oscal_component_metaschema.xml|xml,json|xml,json|xml,json
src/metaschema/oscal_ssp_metaschema.xml|xml,json|xml,json|xml,json


1 change: 1 addition & 0 deletions build/ci-cd/config/release-content
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ content/fedramp.gov/**/*.yaml
content/nist.gov/**/*.xml
content/nist.gov/**/*.json
content/nist.gov/**/*.yaml
content/ssp-example/ssp-example.json
xml/convert/*.xsl
xml/schema/*.xsd
json/convert/*.xsl
Expand Down
79 changes: 47 additions & 32 deletions build/ci-cd/copy-and-convert-content.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,18 +57,18 @@ done

OTHER_ARGS=$@ # save the remaining args

echo ""
echo "${P_INFO}Copying and Converting Content${P_END}"
echo "${P_INFO}==============================${P_END}"
echo -e ""
echo -e "${P_INFO}Copying and Converting Content${P_END}"
echo -e "${P_INFO}==============================${P_END}"

if [ "$VERBOSE" = "true" ]; then
echo "${P_INFO}Using working directory:${P_END} ${WORKING_DIR}"
echo -e "${P_INFO}Using working directory:${P_END} ${WORKING_DIR}"
fi

# check for perl
result=$(which perl 2>&1)
if [ $? -ne 0 ]; then
echo "${P_ERROR}Perl is not installed. Perl is needed by this script.${P_END}"
echo -e "${P_ERROR}Perl is not installed. Perl is needed by this script.${P_END}"
exit 1
fi

Expand All @@ -92,16 +92,15 @@ while IFS="|" read path format model converttoformats || [[ -n "$path" ]]; do
file_relative=$(realpath --relative-to="${OSCALDIR}" "$file")
dest="$WORKING_DIR/${file/$OSCALDIR\/src\//}"
dest_dir=${dest%/*} # remove filename
dest_relative=$(realpath --relative-to="${WORKING_DIR}" "$dest")

mkdir -p "$dest_dir"
dest_relative=$(realpath --relative-to="${WORKING_DIR}" "$dest")
result=$(cp "$file" "$dest" 2>&1)
cmd_exitcode=$?
if [ $cmd_exitcode -ne 0 ]; then
echo "${P_ERROR}Unable to copy '${P_END}${file_relative}${P_ERROR}' to '${P_END}${dest_relative}${P_ERROR}'.${P_END}"
echo "${P_ERROR}${result}${P_END}"
echo -e "${P_ERROR}Unable to copy '${P_END}${file_relative}${P_ERROR}' to '${P_END}${dest_relative}${P_ERROR}'.${P_END}"
echo -e "${P_ERROR}${result}${P_END}"
else
echo "${P_OK}Copied '${P_END}${file_relative}${P_OK}' to '${P_END}${dest_relative}${P_OK}'.${P_END}"
echo -e "${P_OK}Copied '${P_END}${file_relative}${P_OK}' to '${P_END}${dest_relative}${P_OK}'.${P_END}"
fi

IFS=","
Expand All @@ -110,32 +109,48 @@ while IFS="|" read path format model converttoformats || [[ -n "$path" ]]; do
newpath="${newpath/\/$format\///$altformat/}" # change path from old to new format dir
newpath="${newpath%.*}" # strip extension

dest="$WORKING_DIR/${newpath}-min.${altformat}"
dest_relative=$(realpath --relative-to="${WORKING_DIR}" "$dest")
converter="$WORKING_DIR/$altformat/convert/oscal_${model}_${format}-to-${altformat}-converter.xsl"
converter_relative=$(realpath --relative-to="${WORKING_DIR}" "$converter")

if [ "$VERBOSE" = "true" ]; then
echo "${P_INFO}Generating ${altformat^^} file '${P_END}${dest_relative}${P_INFO}' from '${P_END}${file_relative}${P_INFO}' using converter '${P_END}${converter_relative}${P_INFO}'.${P_END}"
fi
result=$(xsl_transform "$converter" "$file" "$dest" 2>&1)

case $format in
xml)
dest="$WORKING_DIR/${newpath}-min.${altformat}"
dest_relative=$(realpath --relative-to="${WORKING_DIR}" "$dest")
if [ "$VERBOSE" = "true" ]; then
echo -e "${P_INFO}Generating ${altformat^^} file '${P_END}${dest_relative}${P_INFO}' from '${P_END}${file_relative}${P_INFO}' using converter '${P_END}${converter_relative}${P_INFO}'.${P_END}"
fi
result=$(xsl_transform "$converter" "$file" "$dest" 2>&1)
;;
json)
dest="$WORKING_DIR/${newpath}.${altformat}"
dest_relative=$(realpath --relative-to="${WORKING_DIR}" "$dest")
if [ "$VERBOSE" = "true" ]; then
echo -e "${P_INFO}Generating ${altformat^^} file '${P_END}${dest_relative}${P_INFO}' from '${P_END}${file_relative}${P_INFO}' using converter '${P_END}${converter_relative}${P_INFO}'.${P_END}"
fi
result=$(xsl_transform "$converter" "" "$dest" "-it" "json-file=${file}" 2>&1)
;;
*)
echo -e "${P_WARN}Conversion from '${format} to '${altformat^^}' is unsupported for '${P_END}${file_relative}${P_OK}'.${P_END}"
continue;
esac
cmd_exitcode=$?
if [ $cmd_exitcode -ne 0 ]; then
echo "${P_ERROR}Content conversion to ${altformat^^} failed for '${P_END}${file_relative}${P_ERROR}' using converter '${P_END}${converter_relative}${P_ERROR}'.${P_END}"
echo "${P_ERROR}${result}${P_END}"
echo -e "${P_ERROR}Content conversion to ${altformat^^} failed for '${P_END}${file_relative}${P_ERROR}' using converter '${P_END}${converter_relative}${P_ERROR}'.${P_END}"
echo -e "${P_ERROR}${result}${P_END}"
exitcode=1
continue
else
if [ "$VERBOSE" = "true" ]; then
echo "${P_OK}Content conversion to ${altformat^^} succeeded for '${P_END}${file_relative}${P_OK}'.${P_END}"
echo -e "${P_OK}Content conversion to ${altformat^^} succeeded for '${P_END}${file_relative}${P_OK}'.${P_END}"
fi
fi

# Format specific post-processing
case $altformat in
json)
# Remove extra slashes
perl -pi -e 's,\\/,/,g' "${dest}"
# perl -pi -e 's,\\/,/,g' "${dest}"
# translate OSCAL mime types
perl -pi -e 's,(application/oscal\.[a-z]+\+)xml\",\1json\",g' "${dest}"
# relative content paths
Expand All @@ -152,21 +167,21 @@ while IFS="|" read path format model converttoformats || [[ -n "$path" ]]; do
result=$(validate_json "$schema" "$dest")
cmd_exitcode=$?
if [ $cmd_exitcode -ne 0 ]; then
echo "${P_ERROR}JSON Schema validation failed for '${P_END}${dest_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo "${P_ERROR}${result}${P_END}"
echo -e "${P_ERROR}JSON Schema validation failed for '${P_END}${dest_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo -e "${P_ERROR}${result}${P_END}"
exitcode=1
continue
else
echo "${P_OK}JSON Schema validation passed for '${P_END}${dest_relative}${P_OK}' using schema '${P_END}${schema_relative}${P_OK}'.${P_END}"
echo -e "${P_OK}JSON Schema validation passed for '${P_END}${dest_relative}${P_OK}' using schema '${P_END}${schema_relative}${P_OK}'.${P_END}"
fi

# produce pretty JSON
dest_pretty="$WORKING_DIR/${newpath}.${altformat}"
dest_pretty_relative=$(realpath --relative-to="${WORKING_DIR}" "$dest_pretty")
result=$(jq . "$dest" > "$dest_pretty" 2>&1)
if [ $? -ne 0 ]; then
echo "${P_ERROR}Unable to execute jq on '${P_END}${dest_pretty_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo "${P_ERROR}${result}${P_END}"
echo -e "${P_ERROR}Unable to execute jq on '${P_END}${dest_pretty_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo -e "${P_ERROR}${result}${P_END}"
exitcode=1
continue
fi
Expand All @@ -176,12 +191,12 @@ while IFS="|" read path format model converttoformats || [[ -n "$path" ]]; do
result=$(validate_json "$schema" "$dest_pretty" 2>&1)
cmd_exitcode=$?
if [ $cmd_exitcode -ne 0 ]; then
echo "${P_ERROR}JSON Schema validation failed for '${P_END}${dest_pretty_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo "${P_ERROR}${result}${P_END}"
echo -e "${P_ERROR}JSON Schema validation failed for '${P_END}${dest_pretty_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo -e "${P_ERROR}${result}${P_END}"
exitcode=1
continue
else
echo "${P_OK}JSON Schema validation passed for '${P_END}${dest_pretty_relative}${P_OK}' using schema '${P_END}${schema_relative}${P_OK}'.${P_END}"
echo -e "${P_OK}JSON Schema validation passed for '${P_END}${dest_pretty_relative}${P_OK}' using schema '${P_END}${schema_relative}${P_OK}'.${P_END}"
fi

# produce yaml
Expand All @@ -198,15 +213,15 @@ while IFS="|" read path format model converttoformats || [[ -n "$path" ]]; do
result=$(xmllint --noout --schema "$schema" "$dest" 2>&1)
cmd_exitcode=$?
if [ $cmd_exitcode -ne 0 ]; then
echo "${P_ERROR}XML Schema validation failed for '${P_END}${dest_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo "${P_ERROR}${result}${P_END}"
echo -e "${P_ERROR}XML Schema validation failed for '${P_END}${dest_relative}${P_ERROR}' using schema '${P_END}${schema_relative}${P_ERROR}'.${P_END}"
echo -e "${P_ERROR}${result}${P_END}"
exitcode=1
else
echo "${P_OK}XML Schema validation passed for '${P_END}${dest_relative}${P_OK}' using schema '${P_END}${schema_relative}${P_OK}'.${P_END}"
echo -e "${P_OK}XML Schema validation passed for '${P_END}${dest_relative}${P_OK}' using schema '${P_END}${schema_relative}${P_OK}'.${P_END}"
fi
;;
*)
echo "${P_WARN}Post processing of '${altformat^^}' is unsupported for '${P_END}${dest_relative}${P_OK}'.${P_END}"
echo -e "${P_WARN}Post processing of '${altformat^^}' is unsupported for '${P_END}${dest_relative}${P_OK}'.${P_END}"
continue;
esac
done
Expand Down
155 changes: 84 additions & 71 deletions build/ci-cd/generate-content-converters.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ HELP=false

usage() { # Function: Print a help message.
cat << EOF
Usage: $0 [options]
Usage: $0 [options] [metaschema paths]
-h, --help Display help
-w DIR, --working-dir DIR Generate artifacts in DIR
Expand Down Expand Up @@ -54,81 +54,94 @@ done

OTHER_ARGS=$@ # save the remaining args

echo ""
echo "${P_INFO}Generating XML <-> JSON Content Converters${P_END}"
echo "${P_INFO}==========================================${P_END}"
echo -e ""
echo -e "${P_INFO}Generating XML <-> JSON Content Converters${P_END}"
echo -e "${P_INFO}==========================================${P_END}"

if [ "$VERBOSE" = "true" ]; then
echo "${P_INFO}Using working directory:${P_END} ${WORKING_DIR}"
echo -e "${P_INFO}Using working directory:${P_END} ${WORKING_DIR}"
fi

exitcode=0
shopt -s nullglob
shopt -s globstar
while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; do
shopt -s extglob
[[ "$path" =~ ^[[:space:]]*# ]] && continue
# remove leading space
path="${path##+([[:space:]])}"
# remove trailing space
gen_docs="${gen_docs%%+([[:space:]])}"
shopt -u extglob

([ -z "$path" ] || [ -z "$gen_converter" ]) && continue;

files_to_process="$OSCALDIR"/"$path"

IFS= # disable word splitting
for metaschema in $files_to_process
do
filename=$(basename -- "$metaschema")
extension="${filename##*.}"
filename="${filename%.*}"
model="${filename/_metaschema/}"
metaschema_relative=$(realpath --relative-to="${OSCALDIR}" "$metaschema")

#split on commas
IFS=, read -a formats <<< "$gen_converter"
for target_format in "${formats[@]}"; do
if [ -z "$target_format" ]; then
# skip blanks
continue;
fi

# Run the XSL template for the format
case $target_format in
xml)
source_format="json"
;;
json)
source_format="xml"
;;
*)
echo "${P_WARN}Generating converter from '${source_format^^}' to '${target_format^^}' is unsupported for '${P_END}${metaschema_relative}${P_WARN}'.${P_END}"
continue;
;;
esac

converter="$WORKING_DIR/${target_format}/convert/${model}_${source_format}-to-${target_format}-converter.xsl"
converter_relative=$(realpath --relative-to="${WORKING_DIR}" "$converter")

if [ "$VERBOSE" = "true" ]; then
echo "${P_INFO}Generating ${source_format^^} to ${target_format^^} converter for '${P_END}${metaschema_relative}${P_INFO}' as '${P_END}${converter_relative}${P_INFO}'.${P_END}"
fi

result=$(xsl_transform "$OSCALDIR/build/metaschema/$source_format/produce-${source_format}-converter.xsl" "$metaschema" "$converter" 2>&1)
cmd_exitcode=$?
if [ $cmd_exitcode -ne 0 ]; then
echo "${P_ERROR}Generating ${source_format^^} to ${target_format^^} converter failed for '${P_END}${metaschema_relative}${P_ERROR}'.${P_END}"
echo "${P_ERROR}${result}${P_END}"
exitcode=1
else
echo "${P_OK}Generating ${source_format^^} to ${target_format^^} converter passed for '${P_END}${metaschema_relative}${P_OK}'.${P_END}"
fi
declare -a paths
declare -a formats
if [ "$#" -ne 0 ]; then
paths=("$@")
for i in "${!paths[@]}"; do
formats[$i]="xml,json"
done
else
while IFS="|" read path gen_schema gen_converter gen_docs || [[ -n "$path" ]]; do
[[ "$path" =~ ^[[:space:]]*# ]] && continue
# remove leading space
path="${path##+([[:space:]])}"

([ -z "$path" ] || [ -z "$gen_converter" ]) && continue;

path_absolute="$OSCALDIR"/"$path"

IFS_OLD=$IFS
IFS= # disable word splitting
for metaschema in $path_absolute
do
paths+=("$metaschema")
formats+=("$gen_converter")
done
IFS=$IFS_OLD
done < "$OSCALDIR/build/ci-cd/config/metaschema"
fi

exitcode=0
for i in "${!paths[@]}"; do
metaschema="${paths[$i]}"
gen_converter="${formats[$i]}"

filename=$(basename -- "$metaschema")
extension="${filename##*.}"
filename="${filename%.*}"
model="${filename/_metaschema/}"
metaschema_relative=$(realpath --relative-to="${OSCALDIR}" "$metaschema")

#split on commas
IFS_OLD=$IFS
IFS=, #read -a formats <<< "$gen_schema"
for target_format in ${gen_converter}; do
if [ -z "$target_format" ]; then
# skip blanks
continue;
fi

# Run the XSL template for the format
case $target_format in
xml)
source_format="json"
;;
json)
source_format="xml"
;;
*)
echo -e "${P_WARN}Generating converter from '${source_format^^}' to '${target_format^^}' is unsupported for '${P_END}${metaschema_relative}${P_WARN}'.${P_END}"
continue;
;;
esac

converter="$WORKING_DIR/${target_format}/convert/${model}_${source_format}-to-${target_format}-converter.xsl"
converter_relative=$(realpath --relative-to="${WORKING_DIR}" "$converter")

if [ "$VERBOSE" = "true" ]; then
echo -e "${P_INFO}Generating ${source_format^^} to ${target_format^^} converter for '${P_END}${metaschema_relative}${P_INFO}' as '${P_END}${converter_relative}${P_INFO}'.${P_END}"
fi

result=$(xsl_transform "$OSCALDIR/build/metaschema/$source_format/produce-${source_format}-converter.xsl" "$metaschema" "$converter" 2>&1)
cmd_exitcode=$?
if [ $cmd_exitcode -ne 0 ]; then
echo -e "${P_ERROR}Generating ${source_format^^} to ${target_format^^} converter failed for '${P_END}${metaschema_relative}${P_ERROR}'.${P_END}"
echo -e "${P_ERROR}${result}${P_END}"
exitcode=1
else
echo -e "${P_OK}Generating ${source_format^^} to ${target_format^^} converter passed for '${P_END}${metaschema_relative}${P_OK}'.${P_END}"
fi
done
done < $OSCALDIR/build/ci-cd/config/metaschema
shopt -u nullglob
shopt -u globstar
IFS=$IFS_OLD
done

exit $exitcode
Loading

0 comments on commit 4e7b229

Please sign in to comment.