Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance Metaschema Unit Testing Framework #186

Open
10 tasks
aj-stein-nist opened this issue Feb 23, 2022 · 3 comments
Open
10 tasks

Enhance Metaschema Unit Testing Framework #186

aj-stein-nist opened this issue Feb 23, 2022 · 3 comments
Labels
enhancement New feature or request EPIC A collection of issues to be worked on over a series of sprints

Comments

@aj-stein-nist
Copy link
Collaborator

aj-stein-nist commented Feb 23, 2022

User Story:

As a Metaschema-tooling developer, to have more confidence that I am properly implementing my own tools that parse Metaschema, I would like to have a more organized collection of unit and integration tests, where I can use this repository's test runner or reuse the test data inputs and outputs with my own alternative test runner for conformance testing.

Goals:

Unit Tests should be reorganized with an organization scheme and implementation that identifies how the test implements one or more of the following goals:

  • The Metaschema itself is syntactically valid (#47)
  • A generated XML Schema for a valid Metaschema is well-formed
  • A generated JSON Schema for a valid Metaschema is well-formed
  • The Metaschema emits an expected XML Schema structure for given Metaschema
  • The Metaschema emits an expected JSON Schema structure for given Metaschema

Tests should be reorganized with an organization scheme and implementation that identifies how the test implements one or more of the following goals:

  • How the JSON Schema validation behave in common language runtimes (aka conformance test)
  • How the XML Schema validation behave in common language runtimes (aka conformance test)

Dependencies:

Iterating and prototyping a first attempt or an interrim solution as part of ongoing work in #183 and #184.

Acceptance Criteria

  • All website and readme documentation affected by the changes in this issue have been updated. Changes to the website can be made in the docs/content directory of your branch.
  • A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
  • The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

{The items above are general acceptance criteria for all User Stories. Please describe anything else that must be completed for this issue to be considered resolved.}

@aj-stein-nist aj-stein-nist added enhancement New feature or request EPIC A collection of issues to be worked on over a series of sprints labels Feb 23, 2022
@aj-stein-nist
Copy link
Collaborator Author

@david-waltermire-nist, before I forget: an epic tracker given our discussion today on enhancing the unit tests, and maybe later integration tests. Look forward to syncing up again on Friday to discuss this further.

@aj-stein-nist
Copy link
Collaborator Author

Preliminary work has begun on some work in this epic and new issues will be drawn out. Will move this to assigned to developer.

david-waltermire pushed a commit to wendellpiez/metaschema that referenced this issue Apr 28, 2022
… #201 - now everything is green or grey (pending) no pink
david-waltermire pushed a commit to wendellpiez/metaschema that referenced this issue Apr 28, 2022
… #201 - now everything is green or grey (pending) no pink
david-waltermire added a commit that referenced this issue Apr 28, 2022
* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for #198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - #81 #186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
david-waltermire added a commit that referenced this issue Jun 21, 2022
* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for #198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - #81 #186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
@wendellpiez
Copy link
Collaborator

I am working on this Issue here, with provisional XSpec underway: https://github.com/wendellpiez/metaschema/tree/feature-metaschema-relocation-plus-enhancementsA

david-waltermire added a commit that referenced this issue Dec 7, 2022
* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for #198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - #81 #186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Jan 9, 2023
…tgov#199)

* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for usnistgov#198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - usnistgov#81 usnistgov#186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Jan 10, 2023
…tgov#199)

* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for usnistgov#198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - usnistgov#81 usnistgov#186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Jan 10, 2023
…tgov#199)

* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for usnistgov#198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - usnistgov#81 usnistgov#186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Jan 10, 2023
…tgov#199)

* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for usnistgov#198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - usnistgov#81 usnistgov#186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Jan 10, 2023
…tgov#199)

* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for usnistgov#198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - usnistgov#81 usnistgov#186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Jan 10, 2023
…tgov#199)

* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for usnistgov#198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - usnistgov#81 usnistgov#186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
david-waltermire added a commit that referenced this issue Mar 9, 2023
* Metaschema composition XSpecs
* Adding 'a8' prune-unused-definitions filter for comparison (most advanced yet).
* Added new prune filter with slight adjustments to interface; new test for running it standalone
* Removing noisy XSpec result
* More tuning up v9 pruning step for #198
* Cleanup; restoring top-level compose XSLT
* Updating files before merge
* First draft of testing approach with current WIP prune phase transform.
* Fix tests to align with Wendell exception msg check.
* Move pruning tests back to their own folder.
* Refactor test input XML docs into separate files
* Make the tests more reusable across implementations.
* Relocate input files to subdir per convo with @wendellpiez.
* Slight improvement to prototype
* Much cleanup and rearrangement; readme docs
* Updating readme plus entry (shell) XSLT and XProc to current
* Removing obsolete XSpec reports
* Adding .gitignore to exclude XSpec results
* More rearrangement and cleanup; adding step 4 XSpec
* New filter detecting and removing unused definitions passes unit tests
* Removing early copy of remedied step 4 XSLT
* Updating Metaschema Schematron XSpecs to functional state: they are now stable, addressing #47
* More adjustments prepping to M4 unit tests - #81 #186 #201 - now everything is green or grey (pending) no pink
* Added more clarification to pruning XSpec
* Edits in response to feedback from @aj-stein-nist thanks AJ!
* adding blank lines at end of files

Co-authored-by: Alexander Stein <[email protected]>
Co-authored-by: David Waltermire <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request EPIC A collection of issues to be worked on over a series of sprints
Projects
None yet
Development

No branches or pull requests

2 participants