Develop a Testing Strategy and Requirements Document

[howieavp76]

User Story:

As an OSCAL content owner, I need to document requirements for testing to allow future stage prototyping and development

Goals:

OSCAL testing requirements document

Dependencies:

N/A

Acceptance Criteria

NIST approval of the requirements document

[brian-ruf]

3/7/2019 - @howieavp76 has just started. Has become top priority now that other issues are resolved. Expect significant progress by next status meeting.

[howieavp76]

First draft of the requirements document is attached. It lays out the overall requirements, phases, and strategy for development. I was unclear on the Product Owner v/s Product Sponsor role but we can address those changes during the review/comment process for this document.
OSCAL Testing Requirements Document rev0.docx

[howieavp76]

I have also created the App Shell and began to frame out the testing harness. Assuming the requirements document is generally accepted, I can begin build out throughout the next sprint.

Recommend closing this issue as the requirements document is complete (or closing after review/comment) and opening a new issue for Testing MVP 1.

[howieavp76]

Early preview of the testing app splash page:

It now implements the USWDS theme so that it will functionally match the main website.

[david-waltermire]

3/14/2019

This document needs review. Early review of the document has identified a need to better document the initial set of tests to be implemented. Some discussion/brainstorming is needed to flesh out the initial test scope. Testing can then be iteratively enhanced on an ongoing basis. @wendellpiez and @david-waltermire-nist will meet early the week of 3/18 to identify initial test scope. @wendellpiez and @howieavp76 will update the test documentation to include this initial scope in the next sprint.

[iMichaela]

03/21/2019

@howieavp76 and @wendellpiez met today and got the necessary information to develop a harness. Understood how to do the testing. An API will be provided to testers. The issue identifies not the testing but the document that defines the testing methodology. The documentation will be provided leveraging the existing draft. MVP 1 - details,2 & 3 - higher level

[iMichaela]

03/28/2019

Not much progress was done due to @howieavp76 personal issues

[howieavp76]

Successfully resolved CORS issues and completed end to end API call for validation. Working through .NET configurations to allow the schema validation to pass:

[howieavp76]

Successfully took a known good file, validated it against the OSCAL catalog schema, and output the results. Moved all files to local inside the assets directory to avoid opening security issue with remote XML calls. While I made it work, this isn't something you would want to do in a PROD server so I am pulling everything down locally.

[howieavp76]

Updated the splash page to use the OSCAL logo and to match the hero graphic from the OSCAL site (NOTE: Both are subject to change based on NIST review. Just aligning the template for now.)

[howieavp76]

Now pulling dynamic local files and passing to the backend for validation. Also added improved styling to success and error messages.

[redhatrises]

One thing that I would add to be helpful is to be able to send a request for validation remotely say from GitHub and get a response back as to being valid OSCAL or not.

[iMichaela]

@howieavp76 - please remove asap the logo from the OSCAL page. We do not have any logo approved for use to start with. Also, NIST did not receive any logo files, so, as of today, this is not our logo because we do not have it. We can use a logo when we will have the approval from DoC.

[howieavp76]

@iMichaela - no problem, I just had it on the test kit on my local box (it wasn't on the website). I have removed the logo as you see in the screenshot below.

[howieavp76]

@redhatrises - completely agree. The front end sits on top of a set of validation APIs. The goal is to be able to run the code in a container locally to validate or to do remote calls via API as part of the CI/CD to validate.

[howieavp76]

@iMichaela - I am rebasing to Wendell's latest version then I will push up the logo files so you have them.

[howieavp76]

Test case showing a fail with detailed error log:

[anweiss]

@howieavp76 is this testing interface something that you plan to merge into master? It's probably best to take whatever backend validations you've incorporated into this UI and incorporate them into the CircleCI-based CI/CD pipeline. Otherwise, this is another UI that has to be developed and maintained.

[howieavp76]

@anweiss - right now I have it as a separate project. I could easily merge it in but worry the Git management will get complex. Goal is to merge into the CI/CD pipeline. This could be done with a script to call the back-end API or curl commands to pass local files back for validation. I built a front-end so it is more approachable for the non-technical. This does not prohibit backend calls to script test processes in CI/CD. Hopefully, it will allow a best of both worlds approach.

[howieavp76]

Added a a toggle for upload new XML files to validate, see below:

[howieavp76]

@anweiss - my other goal is to create a front end tool for creating OSCAL XML so that policy owners can convert their documents to OSCAL without having to be XML gurus. That would let people create new OSCAL files and then test them in the GUI to see how it works (why I called it the OSCAL UX). They could then test their new OSCAL XML directly in the tool to make sure it is compliant before using it in a downstream tool. Plan to make this super simple as a PoC of what should be possible to tool vendors.

[brian-ruf]

@howieavp76 We need to get on the same page. I'm already working on a tool to manage catalog and profile content. You've been in meetings where we discussed this, such as the working session where we discussed the milestone release target.
That tool has also had the ability to generate syntax errors based on schema validation for over two months now. We have too much to accomplish to duplicate effort. Let's clarify swim lanes in today's status meeting.

[howieavp76]

@brianrufgsa - happy to chat more, this is not intended to compete or replace the tool you were working on. It is just a simple UI to show the basic functionality for schema validation. My scope is now to develop test documents that push all the edge cases and then run them in batch to make sure the schema holds up and the files created are valid. There is only overlap because I am on the very basic MVP side, this tool will diverge to be heavy focused on batching testing to integrate with the CI/CD going forward. Now that I have the basics in place for XML, will put the basics in place for JSON. From there, I can build the test sets, batch processes, and integration with CI/CD. The UI will never advance beyond basic PoC, the value is in the test suite and its comprehensive coverage.

[anweiss]

You could also take a look at GitHub actions for running these tests as part of CI/CD, rather than relying on an externally-managed testing API, server, etc.

[brian-ruf]

4/4/2019 @david-waltermire-nist proposing near-term testing should focus on steps in the CI/CD pipeline.

For all PR submitted against NIST OSCAL Master Branch, we do the following:

1. All metaschema instances are validated against the metaschema XSD.
2. All schemas generated from the metaschemas (validated in the previous step) are correctly generated in the proper location with the proper names. (Schema creation date should be later than the metaschema from which it is generated.)
3. All published OSCAL files in the PRs /content directory are validated against their respective schemas (eg XML or JSON schema) (which were generated in the previous step).
4. All conversion scripts (e.g., XML-to-JSON, JSON-to-XML) are generated from their metaschemas in the proper location with the proper names. (Schema creation date should be later than the metaschema from which it is generated.)
5. All content in the PRs/content directory will successfully undergo a round-trip conversion (ie JSON -> XML -> JSON, or XML -> JSON -> XML) with no difference between the initial and final documents (A -> B -> C, where A = C).
6. All generated documentation files are created to reflect changes to the metaschema.

Successful or failed completion of these actions are indicated as a status check in the PR.

[howieavp76]

@brianrufgsa - thanks for the clarity on requirements. Met with Wendall today and talked through strategy. He will provide me with a key value pair of XML docs to validate against XSD schemas. I have a Python script built now that can be called via command line for this function. I tested it with 4 test examples and the results were as expected, see below:

jareds-mbp:oscalTest jaredhowerton$ sh schemaTests.sh
File: catalogs/oscal_testing_bad_catalog_element.xml, Schema: schemas/oscal-catalog-schema.xsd
SUCCESS: XML well formed, syntax ok.
File: catalogs/oscal_testing_bad_catalog_element.xml, Schema: schemas/oscal-catalog-schema.xsdschemas/oscal-catalog-schema.xsd
ERROR: Schema validation error, see error_schema.log
:5:0:ERROR:SCHEMASV:SCHEMAV_CVC_ELT_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}catalogue': No matching global declaration available for the validation root.
File: catalogs/oscal_testing_bad_catalog_no_title.xml, Schema: schemas/oscal-catalog-schema.xsd
SUCCESS: XML well formed, syntax ok.
File: catalogs/oscal_testing_bad_catalog_no_title.xml, Schema: schemas/oscal-catalog-schema.xsdschemas/oscal-catalog-schema.xsd
ERROR: Schema validation error, see error_schema.log
:6:0:ERROR:SCHEMASV:SCHEMAV_ELEMENT_CONTENT: Element '{http://csrc.nist.gov/ns/oscal/1.0}section': This element is not expected. Expected is ( {http://csrc.nist.gov/ns/oscal/1.0}title ).
File: catalogs/oscal_testing_dinosaur_catalog.xml, Schema: schemas/oscal-catalog-schema.xsd
SUCCESS: XML well formed, syntax ok.
File: catalogs/oscal_testing_dinosaur_catalog.xml, Schema: schemas/oscal-catalog-schema.xsd
SUCCESS: XML valid, schema validation ok.
File: catalogs/oscal_testing_mini-testing_catalog.xml, Schema: schemas/oscal-catalog-schema.xsd
SUCCESS: XML well formed, syntax ok.
File: catalogs/oscal_testing_mini-testing_catalog.xml, Schema: schemas/oscal-catalog-schema.xsdschemas/oscal-catalog-schema.xsd
ERROR: Schema validation error, see error_schema.log
:4:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_4: Element '{http://csrc.nist.gov/ns/oscal/1.0}catalog': The attribute 'id' is required but missing.
:4:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_4: Element '{http://csrc.nist.gov/ns/oscal/1.0}catalog': The attribute 'model-version' is required but missing.
:24:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:29:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:35:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:40:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:70:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:74:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:78:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
💯0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:125:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:151:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:178:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:192:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:213:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:233:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:255:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:261:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:283:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:300:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:316:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:338:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:384:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:388:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:394:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:397:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:402:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.
:440:0:ERROR:SCHEMASV:SCHEMAV_CVC_COMPLEX_TYPE_3_2_1: Element '{http://csrc.nist.gov/ns/oscal/1.0}p', attribute 'class': The attribute 'class' is not allowed.

Once I get the right priority docs/schemas to test, I will update the file to run the full suite of validations and we can plug that into CI/CD. Still looking into the best way to do the transformation testing. I have an approach on the timestamp checking but haven't coded that yet.

[howieavp76]

Another example with real data, attempting to validate the FedRAMP profiles against the OSCAL profile schema:

jareds-mbp:oscalTest jaredhowerton$ sh schemaTests.sh
File: catalogs/FedRAMP_HIGH-baseline_profile.xml, Schema: schemas/oscal-profile-schema.xsd
SUCCESS: XML well formed, syntax ok.
File: catalogs/FedRAMP_HIGH-baseline_profile.xml, Schema: schemas/oscal-profile-schema.xsdschemas/oscal-profile-schema.xsd
ERROR: Schema validation error, see error_schema.log
<string>:4:0:ERROR:SCHEMASV:SCHEMAV_ELEMENT_CONTENT: Element '{http://csrc.nist.gov/ns/oscal/1.0}publication_information': This element is not expected. Expected is one of ( {http://csrc.nist.gov/ns/oscal/1.0}metadata, {http://csrc.nist.gov/ns/oscal/1.0}import, {http://csrc.nist.gov/ns/oscal/1.0}merge, {http://csrc.nist.gov/ns/oscal/1.0}modify ).
File: catalogs/FedRAMP_LOW-baseline_profile.xml, Schema: schemas/oscal-profile-schema.xsd
SUCCESS: XML well formed, syntax ok.
File: catalogs/FedRAMP_LOW-baseline_profile.xml, Schema: schemas/oscal-profile-schema.xsdschemas/oscal-profile-schema.xsd
ERROR: Schema validation error, see error_schema.log
<string>:4:0:ERROR:SCHEMASV:SCHEMAV_ELEMENT_CONTENT: Element '{http://csrc.nist.gov/ns/oscal/1.0}publication_information': This element is not expected. Expected is one of ( {http://csrc.nist.gov/ns/oscal/1.0}metadata, {http://csrc.nist.gov/ns/oscal/1.0}import, {http://csrc.nist.gov/ns/oscal/1.0}merge, {http://csrc.nist.gov/ns/oscal/1.0}modify ).
File: catalogs/FedRAMP_MODERATE-baseline_profile.xml, Schema: schemas/oscal-profile-schema.xsd
SUCCESS: XML well formed, syntax ok.
File: catalogs/FedRAMP_MODERATE-baseline_profile.xml, Schema: schemas/oscal-profile-schema.xsdschemas/oscal-profile-schema.xsd
ERROR: Schema validation error, see error_schema.log
<string>:4:0:ERROR:SCHEMASV:SCHEMAV_ELEMENT_CONTENT: Element '{http://csrc.nist.gov/ns/oscal/1.0}publication_information': This element is not expected. Expected is one of ( {http://csrc.nist.gov/ns/oscal/1.0}metadata, {http://csrc.nist.gov/ns/oscal/1.0}import, {http://csrc.nist.gov/ns/oscal/1.0}merge, {http://csrc.nist.gov/ns/oscal/1.0}modify ).

All of the FedRAMP profiles failed the initial validation but it looks like an easy fix to correct.

[brian-ruf]

@howieavp76 - Yes. The original FedRAMP profiles are out of alignment with the current syntax. They were built with an early tool that was more of a proof-of-concept. They have a few other short-falls as well. For example, they don't reflect subtle differences in FedRAMP's control objectives, nor have they been expanded to tag FedRAMP's critical controls.

I've avoided updating them because I hope to have the new open-source tool ready in the next few weeks, and plan to use it to re-generate the FedRAMP profiles more completely and using the latest syntax.

[howieavp76]

For the XML/XSD validation, I am using xmlint. It is open source with some a simple Python script that can be called from the command line.

[howieavp76]

@brianrufgsa - sounds awesome :) . Hope to have the testing tools ready so that you can output your new OSCAL xml files and have them checked in the CI/CD. Will be really cool to see it all working end to end.

[redhatrises]

@howieavp76 have you checked out https://github.com/usnistgov/OSCAL/blob/master/util/oscal-content-validator.py ?

[howieavp76]

@redhatrises - had not seen that but our approaches are similar. I added some more formatted output with some enhanced telemetry that could be parsed in CI/CD for pass/fail logic. I also am working on a batch file to comprehensively cover all of the schemas to supported files that are being published. Haven't started on JSON yet but have the first 7 tests working for XML/XSD validation for catalogs and profiles.

[wendellpiez]

@howieavp76 @brianrufgsa yep, the FedRAMP samples are "known invalid" until further notice. That's actually a comfort, it slipped my mind that would be the case. (And the tool appears to be wired correctly.)

[howieavp76]

OSCAL Testing Requirements Document rev2.docx
Round 2 of the testing requirements is attached.

[iMichaela]

04/11/2019

The document listed above covers all the requirements from @david-waltermire-nist, @wendellpiez and @brianrufgsa . The document should be a live document to accomodate teh agile development.

[howieavp76]

OSCAL Testing Requirements Document rev3.docx
Attaching version 3 without a signature page to allow agile development and progressive elaboration.

[howieavp76]

PR submitted with test scripts to allow integration with CI/CD. See:

Testing Framework #340 .

In addition - attached is the latest requirements document that shows which tests have been implemented. NOTE: Many currently fail which was the expected outcome.
OSCAL Testing Requirements Document rev3.docx

[david-waltermire]

04-19-19

@david-waltermire-nist will review the testing requirements document and then comment or close this issue.

[iMichaela]

04/25/2019

Waiting on @david-waltermire-nist to provide the comments.

[iMichaela]

5/2/2019

pending on @david-waltermire-nist to provide feedback

[brian-ruf]

5/9/2019

@david-waltermire-nist to email @howieavp76 to setup meeting to discuss CI/CD integration.

@david-waltermire-nist investigating NIST-approved build services as part of this effort.

[david-waltermire]

We have unit and round trip testing. This was implemented a long time ago. Closing.