XSpec test suites script & CI #1754

nikitawootten-nist · 2023-04-13T19:10:27Z

Committer Notes

This PR provides a simple helper script to run XSpec tests & an acompanying GitHub action.

All Submissions:

Have you selected the correct base branch per Contributing guidance?
Have you set "Allow edits and access to secrets by maintainers
"?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Have you squashed any non-relevant commits and commit messages? [instructions]
Do all automated CI/CD checks pass?

By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
~~Have you included examples of how to use your new feature(s)?~~
~~Have you updated all OSCAL website and readme documentation affected by the changes you made? Changes to the OSCAL website can be made in the docs/content directory of your branch.~~

Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of #1754.

aj-stein-nist · 2023-04-19T16:57:48Z

@nikitawootten-nist, things are looking up now that we tweaked the subcommit: it passed the step where it last failed:

https://github.com/usnistgov/OSCAL/actions/runs/4745778013/jobs/8428564087

nikitawootten-nist · 2023-04-20T16:37:22Z

@aj-stein-nist this PR should be ready, can you work with me to try to get the XSpec Test Suite job to run?

aj-stein-nist · 2023-04-20T17:31:27Z

@aj-stein-nist this PR should be ready, can you work with me to try to get the XSpec Test Suite job to run?

Do you know how to simulate a complete merge on your fork? (I know we love it, but this is the one case where we can use the fork to our advantage to check "what happens on main?" before merging there.)
Also what is the plan with the target branch after this merge is complete? (That branch gets merged into develop, for example?)

nikitawootten-nist · 2023-04-20T18:39:48Z

Also what is the plan with the target branch after this merge is complete? (That branch gets merged into develop, for example?)

Yeah ideally the target branch is merged into develop after the fact. The tests do currently fail, so the goal would be for us to work with Wendell to get the tests aligned. For now the workflow will only fail if the XSpec tests fail to compile.

build/pull-oscal-env-dockerfile.sh

.github/workflows/xspec-tests.yml

src/utils/util/resolver-pipeline/testing/test.sh

Co-authored-by: A.J. Stein <[email protected]>

nikitawootten-nist · 2023-04-21T19:43:16Z

Ok so I've confirmed the actions work on my fork: https://github.com/nikitawootten-nist/OSCAL/actions/runs/4768051498/jobs/8477350797

* XSpec test suites script & CI * Helper script to build and tag the OSCAL env * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of #1754. * Save XSpec test summary + pull appropriate tag * Escape TEST_SUITE in XSpec runner Co-authored-by: A.J. Stein <[email protected]> * Update test script to address shellcheck guidance * Grab branch from GHA * Fix for builds on the M1 --------- Co-authored-by: A.J. Stein <[email protected]>

@david-waltermire-nist

* Profile resolution unit tests and specification updates (#1219) * Updates to tests including whitespace scrub test * Adding oXygen project * Integration work on specs and unit tests / XSLT and Schematron * Profile resolution testing infrastructure, plus some tests * Some test files with expected results * Updated and improved XSpec generation and tests - now driven directly from SpecML source * Adjusted gitignore; extended spec with more bindings; illustrator XSLT * Added XSLT producing Markdown summary of requirements w/ examples; updates * Added a small utility for converting JSON "prop" fields to "props" fields (emended 1.0.2 syntax) * Updated to Profile Resolution spec (with example files now tagged) and readme.md * Updated profile resolution (spec) readme with clarification regarding unit testing * Adding XSLT provisionally to produce a unit testing harness after @david-waltermire-nist model, with example and schema; also cleanup * Removed working file included by accident * More unit tests with edits to spec for tracking. * Catching up various outstanding profile resolution work * Small update to oXygen setup * Update metaschema submodule to existing commit. * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of #1754. * XSpec test suites script & CI (#1754) * XSpec test suites script & CI * Helper script to build and tag the OSCAL env * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of #1754. * Save XSpec test summary + pull appropriate tag * Escape TEST_SUITE in XSpec runner Co-authored-by: A.J. Stein <[email protected]> * Update test script to address shellcheck guidance * Grab branch from GHA * Fix for builds on the M1 --------- Co-authored-by: A.J. Stein <[email protected]> * Fixed logic for when a tag does not exist --------- Co-authored-by: Wendell Piez <[email protected]> Co-authored-by: A.J. Stein <[email protected]>

@david-waltermire-nist

* Profile resolution unit tests and specification updates (#1219) * Updates to tests including whitespace scrub test * Adding oXygen project * Integration work on specs and unit tests / XSLT and Schematron * Profile resolution testing infrastructure, plus some tests * Some test files with expected results * Updated and improved XSpec generation and tests - now driven directly from SpecML source * Adjusted gitignore; extended spec with more bindings; illustrator XSLT * Added XSLT producing Markdown summary of requirements w/ examples; updates * Added a small utility for converting JSON "prop" fields to "props" fields (emended 1.0.2 syntax) * Updated to Profile Resolution spec (with example files now tagged) and readme.md * Updated profile resolution (spec) readme with clarification regarding unit testing * Adding XSLT provisionally to produce a unit testing harness after @david-waltermire-nist model, with example and schema; also cleanup * Removed working file included by accident * More unit tests with edits to spec for tracking. * Catching up various outstanding profile resolution work * Small update to oXygen setup * Update metaschema submodule to existing commit. * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of usnistgov/OSCAL#1754. * XSpec test suites script & CI (#1754) * XSpec test suites script & CI * Helper script to build and tag the OSCAL env * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of usnistgov/OSCAL#1754. * Save XSpec test summary + pull appropriate tag * Escape TEST_SUITE in XSpec runner Co-authored-by: A.J. Stein <[email protected]> * Update test script to address shellcheck guidance * Grab branch from GHA * Fix for builds on the M1 --------- Co-authored-by: A.J. Stein <[email protected]> * Fixed logic for when a tag does not exist --------- Co-authored-by: Wendell Piez <[email protected]> Co-authored-by: A.J. Stein <[email protected]>

@david-waltermire-nist

* Profile resolution unit tests and specification updates (#1219) * Updates to tests including whitespace scrub test * Adding oXygen project * Integration work on specs and unit tests / XSLT and Schematron * Profile resolution testing infrastructure, plus some tests * Some test files with expected results * Updated and improved XSpec generation and tests - now driven directly from SpecML source * Adjusted gitignore; extended spec with more bindings; illustrator XSLT * Added XSLT producing Markdown summary of requirements w/ examples; updates * Added a small utility for converting JSON "prop" fields to "props" fields (emended 1.0.2 syntax) * Updated to Profile Resolution spec (with example files now tagged) and readme.md * Updated profile resolution (spec) readme with clarification regarding unit testing * Adding XSLT provisionally to produce a unit testing harness after @david-waltermire-nist model, with example and schema; also cleanup * Removed working file included by accident * More unit tests with edits to spec for tracking. * Catching up various outstanding profile resolution work * Small update to oXygen setup * Update metaschema submodule to existing commit. * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of usnistgov/OSCAL#1754. * XSpec test suites script & CI (#1754) * XSpec test suites script & CI * Helper script to build and tag the OSCAL env * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of usnistgov/OSCAL#1754. * Save XSpec test summary + pull appropriate tag * Escape TEST_SUITE in XSpec runner Co-authored-by: A.J. Stein <[email protected]> * Update test script to address shellcheck guidance * Grab branch from GHA * Fix for builds on the M1 --------- Co-authored-by: A.J. Stein <[email protected]> * Fixed logic for when a tag does not exist --------- Co-authored-by: Wendell Piez <[email protected]> Co-authored-by: A.J. Stein <[email protected]>

@david-waltermire-nist

* Profile resolution unit tests and specification updates (usnistgov#1219) * Updates to tests including whitespace scrub test * Adding oXygen project * Integration work on specs and unit tests / XSLT and Schematron * Profile resolution testing infrastructure, plus some tests * Some test files with expected results * Updated and improved XSpec generation and tests - now driven directly from SpecML source * Adjusted gitignore; extended spec with more bindings; illustrator XSLT * Added XSLT producing Markdown summary of requirements w/ examples; updates * Added a small utility for converting JSON "prop" fields to "props" fields (emended 1.0.2 syntax) * Updated to Profile Resolution spec (with example files now tagged) and readme.md * Updated profile resolution (spec) readme with clarification regarding unit testing * Adding XSLT provisionally to produce a unit testing harness after @david-waltermire-nist model, with example and schema; also cleanup * Removed working file included by accident * More unit tests with edits to spec for tracking. * Catching up various outstanding profile resolution work * Small update to oXygen setup * Update metaschema submodule to existing commit. * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of usnistgov#1754. * XSpec test suites script & CI (usnistgov#1754) * XSpec test suites script & CI * Helper script to build and tag the OSCAL env * Move up Metaschema submodule Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of usnistgov#1754. * Save XSpec test summary + pull appropriate tag * Escape TEST_SUITE in XSpec runner Co-authored-by: A.J. Stein <[email protected]> * Update test script to address shellcheck guidance * Grab branch from GHA * Fix for builds on the M1 --------- Co-authored-by: A.J. Stein <[email protected]> * Fixed logic for when a tag does not exist --------- Co-authored-by: Wendell Piez <[email protected]> Co-authored-by: A.J. Stein <[email protected]>

nikitawootten-nist linked an issue Apr 13, 2023 that may be closed by this pull request

SPIKE: Profile Resolution Testing Integration #1735

Closed

7 tasks

nikitawootten-nist force-pushed the nikitawootten-nist/issue1735 branch 2 times, most recently from 5e4246c to c71c91d Compare April 13, 2023 19:29

XSpec test suites script & CI

b8647d0

nikitawootten-nist force-pushed the nikitawootten-nist/issue1735 branch from c71c91d to b8647d0 Compare April 13, 2023 21:40

Helper script to build and tag the OSCAL env

521c21d

aj-stein-nist marked this pull request as ready for review April 19, 2023 16:01

aj-stein-nist requested a review from wendellpiez as a code owner April 19, 2023 16:01

Move up Metaschema submodule

2ffb5d4

Prior work on pushing up the XSpec dependency and pulling in other file relocation improvements were merged and it appears I pushed the wrong commit into this branch and inadvertently led to more confusion where builds failed in current version of #1754.

Save XSpec test summary + pull appropriate tag

8c50f93

aj-stein-nist self-requested a review April 20, 2023 17:32

aj-stein-nist approved these changes Apr 20, 2023

View reviewed changes

build/pull-oscal-env-dockerfile.sh Show resolved Hide resolved

.github/workflows/xspec-tests.yml Show resolved Hide resolved

src/utils/util/resolver-pipeline/testing/test.sh Outdated Show resolved Hide resolved

nikitawootten-nist and others added 4 commits April 20, 2023 17:05

Escape TEST_SUITE in XSpec runner

0c5a9c3

Co-authored-by: A.J. Stein <[email protected]>

Update test script to address shellcheck guidance

e9ad5cd

Grab branch from GHA

38a6686

Fix for builds on the M1

0966775

nikitawootten-nist requested a review from aj-stein-nist April 21, 2023 19:49

aj-stein-nist removed a link to an issue Apr 21, 2023

SPIKE: Profile Resolution Testing Integration #1735

Closed

7 tasks

aj-stein-nist approved these changes Apr 21, 2023

View reviewed changes

aj-stein-nist merged commit 893b8f5 into feature-profile-resolution-unittestingA Apr 21, 2023

nikitawootten-nist deleted the nikitawootten-nist/issue1735 branch June 12, 2023 19:09

nikitawootten-nist mentioned this pull request Jul 10, 2023

Move web content from OSCAL, and other streamlining. #1824

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

XSpec test suites script & CI #1754

XSpec test suites script & CI #1754

nikitawootten-nist commented Apr 13, 2023 •

edited by aj-stein-nist

Loading

aj-stein-nist commented Apr 19, 2023

nikitawootten-nist commented Apr 20, 2023

aj-stein-nist commented Apr 20, 2023 •

edited

Loading

nikitawootten-nist commented Apr 20, 2023 •

edited

Loading

nikitawootten-nist commented Apr 21, 2023

XSpec test suites script & CI #1754

XSpec test suites script & CI #1754

Conversation

nikitawootten-nist commented Apr 13, 2023 • edited by aj-stein-nist Loading

Committer Notes

All Submissions:

Changes to Core Features:

aj-stein-nist commented Apr 19, 2023

nikitawootten-nist commented Apr 20, 2023

aj-stein-nist commented Apr 20, 2023 • edited Loading

nikitawootten-nist commented Apr 20, 2023 • edited Loading

nikitawootten-nist commented Apr 21, 2023

nikitawootten-nist commented Apr 13, 2023 •

edited by aj-stein-nist

Loading

aj-stein-nist commented Apr 20, 2023 •

edited

Loading

nikitawootten-nist commented Apr 20, 2023 •

edited

Loading