SPIKE: Profile Resolution Testing Integration

[aj-stein-nist]

User Story

As an OSCAL developer, in order to sufficiently understand and rely on the profile resolution specification, I want access to a test suite I can use with a profile resolve implementation to show conformance with the specification.

Goals

This is a spike where we will determine the path forward for #1087.

• Review the draft PR with tests for the specification linked to 1087 with @wendellpiez.
  • Configure the unit tests on your workstation, use them, understand their mechanics.
• Work with assigned teammates to design an approach to link the specification tests (as written and the results of their specifcation) to the itemized specification requirements
• Design a method for integrating the execution of the tests in our CI/CD pipeline

Dependencies

No response

Acceptance Criteria

• An ADR that documents profile resolution testing method and documents future integration into our CI/CD pipeline is created, reviewed, and merged here by the end of the sprint
• A written or verbal presentation to the NIST OSCAL Team with findings from the analysis to be presented before or during sprint review
• One or more issues added to the backlog for any actionable findings from the report the team can integrate into website enhancements, either before or during sprint review

[aj-stein-nist]

I worked with Chris and Nikita today, and spent the afternoon troubleshooting why the current tests would pass inside of Oxygen with XSpec but not on the command line (in docker or otherwise using XSpec directly) to advance an integration planning. After much troubleshooting, it was not the tests like example-set.xspec that were failing, but sadly an older version of xspec was not working well. That has been partially fixed with usnistgov/metaschema#347 but needs to be updated and integrated into the collaboration branch I set up with Chris and Nikita so they could share updates instead of working on Wendell's fork. More to follow ...

Note to self: figuring this blocker out took faaaaar longer than it should have and there is a good lessons learned for "insist team members reach for easiest troubleshooting step or fix first" strategies when pairing or getting unstuck, more on that later (and that is on me, so I hope other teammates call me out on that in the future).

[Compton-US]

Just sharing the xspec output errors (only showing one xspec file):

=== Testing Suite /oscal/src/utils/util/resolver-pipeline/testing/5_finished/finish.xspec ===
/oscal/src/utils/util/resolver-pipeline/testing/4_modified/modify.xspec,compile_failed
Creating Test Stylesheet...
Type error at char 5 in expression in xsl:sequence/@select on line 33 column 45 of uri-utils.xsl:
  XPTY0004  An empty sequence is not allowed as the first argument of x:resolve-xml-uri-with-catalog()
at function x:document-actual-uri on line 26 column 60 of uri-utils.xsl:
     invoked by function call at file:/xspec/src/compiler/base/resolve-import/gather/gather-specs.xsl#36
  In template rule with match="element(Q{http://www.jenitennison.com/xslt/xspec}description)" on line 32 of gather-specs.xsl
     invoked by xsl:apply-templates at file:/xspec/src/compiler/base/resolve-import/resolve-import.xsl#21
at function x:resolve-import on line 13 column 55 of resolve-import.xsl:
     invoked by unknown caller (class net.sf.saxon.value.MemoClosure)
at function x:resolve-import on line 13 column 55 of resolve-import.xsl:
     invoked by function call at file:/xspec/src/compiler/base/main.xsl#91
  In template rule with match="/" on line 87 of main.xsl
An empty sequence is not allowed as the first argument of x:resolve-xml-uri-with-catalog()

*** Error compiling the test suite
/oscal/src/utils/util/resolver-pipeline/testing/5_finished/finish.xspec,compile_failed

[aj-stein-nist]

@Compton-NIST, can you DM on Element/Gitter please? I have some questions about the container work referenced in this issue so I can test in Nikita's absence and bring myself up to speed before sprint planning on Tuesday.

[aj-stein-nist]

I talked with @nikitawootten-nist about this earlier today and there is some wrap-up work we need to so we can finalize hopefully in the beginning of the sprint.

[nikitawootten-nist]

@Compton-NIST and I had some follow up work (see #1771). Otherwise, the original spirit of the issue has been completed!