Unit and regression testing for schemas and tools

[wendellpiez]

User Story:

As an OSCAL developer, I have access to unit testing and regression testing to ensure and show my code is correct. As a stakeholder, I can examine these unit tests to see what is going on.

Goals:

1. Provide developers with a stable foundation for further development of OSCAL
2. Expose contracts (expected outputs for inputs, both correct and incorrect) to all stakeholders
3. Support design as complexity increases
4. Support documentation and alignment of tools with specs

Dependencies:

1. More or less stable schemas and specs, with some samples to start with
2. Some further definition of scope and priorities (across layers and functionalities) for this initial cut
3. Agreement on tooling (at least provisional)

Acceptance Criteria

1. We can demonstrate unit testing over a significant set of features
2. We can demonstrate regression testing for schemas, Schematrons and/or XSLTs and other pipelines (as prioritized)
3. Our means and methods are replicable and documented appropriately

Hints

For schemas, a test suite of canonical samples can ensure compatibility going forward. Backward-incompatible changes can be planned and samples updated accordingly.

For Schematron and XSLT, the state of the art for unit and regression testing is XSpec: https://github.com/xspec/xspec

[anweiss]

@wendellpiez is the plan to use Saxon-HE to execute all of the transformations from the CLI? Otherwise, I don't know of any other CLI tools that fully conform to XSLT 3.0 and XPath 3.1

[wendellpiez]

@anweiss , yes, Saxon-HE is an informal reference implementation for the W3C Recs and developed by the editor of the spec. There are other XSLT 3.0 processors but as you say they are not complete or as accessible. Saxon is available in both Java and C.

BTW this Issue should have been referenced by #186 --

[anweiss]

Ok cool. It’s what I’ve been using on my side for running the transformations and XSpec tests so was just confirming.

[anweiss]

Hey @wendellpiez, is there a way to execute a transformation using the CLI to achieve the same mocking context behavior that one gets with XSpec <x:context>? So for example, I'd to be able to do the following:

$ echo '<title>Three Controls from SP800-53</title>' | java -jar ~/saxon/saxon9he.jar -s:- -xsl:oscal-catalog-xml-converter.xsl

which returns the following error:

Error at char 12 in xsl:value-of/@select on line 37 column 74 of oscal-catalog-xml-converter.xsl:
  FOJS0006: A valid document must contain no text outside the outermost element
A valid document must contain no text outside the outermost element

Instead, I have to create a more complete XML artifact as follows in order to get it to work:

$ echo '<?xml version="1.0" encoding="UTF-8"?><catalog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://csrc.nist.gov/ns/oscal/1.0 ../../../schema/xml/oscal-catalog-schema.xsd" xmlns="http://csrc.nist.gov/ns/oscal/1.0"><title>Three Controls from SP800-53</title></catalog>' | java -jar ~/saxon/saxon9he.jar -s:- -xsl:oscal-catalog-xml-converter.xsl

which returns the following, as expected:

{"title":"Three Controls from SP800-53"}

I ask because I want to make sure the feature tests on the JavaScript side via the Saxon CLI are executing using the same context as in XSpec.

[anweiss]

@wendellpiez you can take a look at how this is coming together at https://github.com/anweiss/OSCAL/blob/json-tests/lib/XML-JSON/test/features/produce_json_from_xml.feature. The actual implementation of the test in JavaScript is at https://github.com/anweiss/OSCAL/blob/json-tests/lib/XML-JSON/test/features/step_definitions/produce_json_from_xml_stepdefs.js. It's actually pretty straight forward: run the transformation through Saxon, run a JSON.parse() against the raw JSON string, and conduct a deep object comparison using a JavaScript assertion library.

I'm simply working off of your feature-metaschema branch.

[wendellpiez]

@anweiss there is also a "mode" often given on the XSpec context, which indicates a mode of operation to the stylesheet. The mode can also be set on the Saxon CL as IIRC 'initial-mode' or some such. For example, in this case the mode might be "xml2json" (effectively the default mode of this XSLT, but made explicit to facilitate calling this logic from other stylesheets without clashing).

I am not sure that will solve the problem but it might help.

It is awesome that you are able to replicate the functionality on the JSON side, that is killer. (And that testing spec is sooo pretty.)

[anweiss]

June 21 Status: Per previous comments, JSON unit tests are being developed in a feature branch, anweiss/json-tests. Once #202 is complete, will push a PR with tests.

[david-waltermire]

We have been working on JSON schema generation unit tests in PR #464. This is a start of a long-running epic.