Profile Group Element Should Match Catalog #463

brian-ruf · 2019-07-22T16:11:10Z

User Story:

As an OSCAL Profile Creator, I should be able to carry groups forward from catalogs with the same information. Currently in a profile, the group element has no attributes, and only "call" and "match" elements, leaving no way to carry forward the group class, id or title; nor to assign new ones. A profile grouping has far more value if it can be named.

Goals:

Expand the profile syntax to allow the group element to have class and id attributes as well as a title element.

Dependencies:

None

Acceptance Criteria

The metaschema for OSCAL profiles defines @Class and @id flags for group, as well as a title field for group.
All OSCAL website and readme documentation affected by the changes in this issue have been updated. Changes to the OSCAL website can be made in the docs/content directory of your branch.
A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

wendellpiez · 2019-08-14T18:27:55Z

In the profile model, merge/custom/group is for making a new grouping of controls selected from catalog(s). Presumably a title, class and id for such a group would also be new. This is different from copying a grouping from a source catalog, which is done by using the profile mechanism merge/as-is , which designates that the selected controls should be grouped as they are in the source catalogs.

The solution described here is reasonable, but not a solution for the problem as stated, but rather for a different problem -- that currently, new groupings of controls in a resolved catalog cannot have titles, class assignments or IDs.

We should also document the use of merge/custom and merge/as-is better to help profile producers.

wendellpiez · 2019-08-28T14:53:28Z

In a catalog (valid to the Catalog Schema) a group can have any of the following:

flag id
flag class
field title
assemblies param
fields prop
fields link
assemblies part
assemblies group or control

To support the requirement for profiles described in this Issue, we could permit group inside a profile to contain, minimally

flag id
flag class
field title
assemblies group or match or group (for calling controls or creating new groups)

Alternatively, permitting more elaboration of new groups in profiles:

flag id
flag class
field title
fields prop
fields link
assemblies part
assemblies group or match or group (instead of control)

Questions:

Are we agreed that profile/group needs id, class and title?
Does profile/group need prop, link or part? (It would be nice if not.)

Additionally, new questions:

Currently in the profile model , a group contents is described as a choice between element types, either all calls, all matches, or all groups. This is probably not right. Probably, we need to permit any or all three to occur together. (Unfortunately in Metaschema this means constraining their order.)
In both the catalog and profile models, a flag class appears on group with no ns. Shouldn't we confirm that ns is permitted everwhere class is permitted?

Suggest that we discuss (at least @david-waltermire-nist and @brianrufgsa) to produce decisions for me to code to.

brian-ruf · 2019-08-28T15:09:46Z

@wendellpiez on your two questions, I firmly agree with the first. We minimally need @ id, @ class and /title in a group element under profile.

On the second, I can live with out those within my foreseeable use cases; however, we may want to discuss them to make sure we don't short change other use cases.

wendellpiez · 2019-08-29T15:34:33Z

I will move forward with this with help from @david-waltermire-nist on final decisions.

iMichaela · 2019-09-05T14:21:51Z

9/5/2019

Issue was implemented and is part of the PR #484, waiting to be reviewed. @wendellpiez will do a sanity check first.

wendellpiez · 2019-09-11T14:00:16Z

In my branch behind PR #484 I am providing for flags id and class and field title to appear on group in the profile model.

We can call this done if:

This doesn't break the build
@david-waltermire-nist agrees we need nothing more on group in profile, for now, such as properties or parts. (My own feeling is if you need these, you might be writing a catalog not making a profile of someone else's. But I could be shown to be wrong.)

… in profiles.

david-waltermire · 2019-09-12T14:09:28Z

@wendellpiez As discussed this AM, we need to support the full range of data items for groups in Profiles. These should be the same as what is available in Catalogs.

wendellpiez · 2019-09-13T13:45:56Z

The content model for profile groups now has (title?, param*, prop*, part*, (group* | call* | match*)). This is the same as groups in catalog except for call or match instead of control.

I am not sure of the use case for param inside profiles, but the patching mechanism permits adding new parameters, so why not here as well, for consistency.

@david-waltermire-nist please review. This is in the issue-468-364-478-integration working branch.

brian-ruf · 2019-09-19T14:15:58Z

@wendellpiez reports the work is complete, and in a branch ready for merge. @brianrufgsa to perform peer review and provide feedback. @david-waltermire-nist still needs to review as well.

wendellpiez · 2019-09-19T14:16:01Z

Here is the branch in my repo: https://github.com/wendellpiez/OSCAL/tree/issue-468-364-478-integration

… in profiles.

" This reverts commit ff20820.

… `group` in profiles." This reverts commit b32faa9.

" This reverts commit ff20820.

… `group` in profiles." This reverts commit b32faa9.

… in profiles.

* Removed unnecessary metaschema check on json-value-key * Now implementing in-xml="with-wrapper | unwrapped" on markup multiline fields, with default `with-wrapper` and corresponding changes to catalog and profile metaschemas. * Updating catalog and profile schemas for (prose) unwrapping behavior * Renamed 'json-behavior' in Metaschema to 'in-json' for consistency with 'in-xml'. * Updating syntax in metaschemas 'in-json' for 'json-behavior' * Updating json converter generator to provide in-xml behavior 'GROUPED' * Refining json converter production from Metaschema * Repairing egregious syntax error * Rectify bug introduced with new feature * Adding support for 'date' and 'dateTime' datatypes, with no time zones (#480), including unit tests * Improvements and corrections to datatypes docs * More small repairs on datatypes descriptions (table markup) * Built out validations of allowed (enumerated) values, including Metaschema validation checks on whether given allowed values conform to their nominal type. * Straightening kinks in datatype metaschema meta-validation * More special handling of recalcitrant datatypes, this time 'NMTOKENS' and 'IDREFS' * Addressing #463, adding `id`, `class` and `title` to `group` in profiles. * Schematron improvements; documentation revisions (removing references to subcontrols) * New Schematron for SP800-53-style catalogs * Removed using in implementation to new metaschema for storage. Started on example, which is still invalid * More work on an example, which is still not valid * refactored schematron processing * more work on SSP example * Refactored generate-schema.sh to allow a metaschema to be provided on the CLI * refactored SSP model to be more flat, while allowing component references * slight cleanup * old example * Fixed content generation * updated metaschema to new specs * fixed content errors * added metaschema examples * Updated metaschema example. * Adding 'prop' and 'part' to Profile Metaschema per #463 * Added 'param' to profile 'group' model * JSON converter now properly handling flags assigned to assemblies identified by json key * Added 'sort-id' property to controls in SP800-53 catalog * Added support for direct generation of schema and converters * Minor updates to SSP example and metaschema * Adding computer build metaschema examples * Extending allowed-values support to fields; unit tests: #437 * More work on allowed-values, now in XSD with unit tests for #437 * Repairing small datatyping bug thanks to regression testing * Further adjustment for allowed-values in docs * Renamed unit tests, now working * Adding allowed-values unit tests * Addressing #452: cleaned up character handling in XML-to-JSON conversion. Fixed the bug this exposed in markdown->HTML conversion. Added some light unit testing. * More adjustments in view of comments to #452 * Repairing more bugs in character handling in Markdown and plain text, with unit testing: #452 * Improvements to Schematron for SP800-53 #400 * For #452, modified copy-and-convert script to suppress character-level post-processing in generated JSON; added unit tests for URIs to detect issues with solidus characters '/' * Edited docs, including datatypes page per #452 (regarding character escapes). * Cleanup metaschema valdiation errors * changed use of the ID and IDREF datatype to NCName * Removed ID and IDREF datatypes * refactored components out of ssp metaschema * Updated SP800-53 catalog and NIST profiles to current model * Added name to information type. * Continued refinement of the SSP model * Updated metadata syntax * changing CM-6 constraint to guidance * continued work on SSP * refactored responsible party to metadata * fixed validation issues * Fixed overriding enumerated values * Fixed content validation errors based on latest metaschema changes * Completed full pass through ssp model * added prop to resource * Fixed broken links in Markdown files. Small adjustments to SSP example. * FDixed relapath error caused when the generated schema directory doesn't exist. Also attempting a fix for CircleCI colorized escape codes causing extra characters to appear. * Fixed validation issue with component metaschema * attempting to fix extra characters in CircleCI colorized messages * Updated enumerated values to remove values that should appear in a given context * quick reordering of elements * hardcode ascii color codes * Fixing broken links in readmes; file name update for consistency * added unit tests for bug * fixed color escaping in scripts * Removed bogus unit test and fixed a small cardinality issue in the SSP model * Improvements to docs generation * Added JSON value keys to fields with flags (that did not have them) * Bug reduction in XSD production mainly related to datatypes and allowed values * Fixing glitch in JSON converter generator * Adjustments repairing Metaschema issues * CSS tweak in docs (model maps to use USWDS font) * Adding to Milestone 2 release notes (in progress) * Change to parameter setting in profile model (#494, #288); release notes towards M2 * Small improvements to copy in release notes (draft) * Updating metaschemas to latest syntax, with improved Metaschema Schematron and unit test tweakage * Adjustments to flag unit test and release notes * SSP Metaschema review and update * cleanup of unused role-id field * reflactored role-id * updated documentation for release. * cleanup of IDs * Minor schema and metaschema improvements * M1 to M2 content converter * Merging local and upstream changes to json example * Relaxing rule over 'ID' flag to accept any string not only XML NCName * reorganizing content for release * more file moves * Fixed changed path * Updated FedRAMP profiles and catalog to current models * Bug fix to XML-to-JSON converter generator * Fixed copy script * Updated FedRAMP baselines and catalog to match new metaschema syntax * fixed merge error * fixed another merge error * fixed spelling issues * Fix from Wendell

david-waltermire · 2019-11-03T21:15:02Z

This was completed in PR #492.

* Removed unnecessary metaschema check on json-value-key * Now implementing in-xml="with-wrapper | unwrapped" on markup multiline fields, with default `with-wrapper` and corresponding changes to catalog and profile metaschemas. * Updating catalog and profile schemas for (prose) unwrapping behavior * Renamed 'json-behavior' in Metaschema to 'in-json' for consistency with 'in-xml'. * Updating syntax in metaschemas 'in-json' for 'json-behavior' * Updating json converter generator to provide in-xml behavior 'GROUPED' * Refining json converter production from Metaschema * Repairing egregious syntax error * Rectify bug introduced with new feature * Adding support for 'date' and 'dateTime' datatypes, with no time zones (usnistgov#480), including unit tests * Improvements and corrections to datatypes docs * More small repairs on datatypes descriptions (table markup) * Built out validations of allowed (enumerated) values, including Metaschema validation checks on whether given allowed values conform to their nominal type. * Straightening kinks in datatype metaschema meta-validation * More special handling of recalcitrant datatypes, this time 'NMTOKENS' and 'IDREFS' * Addressing usnistgov#463, adding `id`, `class` and `title` to `group` in profiles. * Schematron improvements; documentation revisions (removing references to subcontrols) * New Schematron for SP800-53-style catalogs * Removed using in implementation to new metaschema for storage. Started on example, which is still invalid * More work on an example, which is still not valid * refactored schematron processing * more work on SSP example * Refactored generate-schema.sh to allow a metaschema to be provided on the CLI * refactored SSP model to be more flat, while allowing component references * slight cleanup * old example * Fixed content generation * updated metaschema to new specs * fixed content errors * added metaschema examples * Updated metaschema example. * Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463 * Added 'param' to profile 'group' model * JSON converter now properly handling flags assigned to assemblies identified by json key * Added 'sort-id' property to controls in SP800-53 catalog * Added support for direct generation of schema and converters * Minor updates to SSP example and metaschema * Adding computer build metaschema examples * Extending allowed-values support to fields; unit tests: usnistgov#437 * More work on allowed-values, now in XSD with unit tests for usnistgov#437 * Repairing small datatyping bug thanks to regression testing * Further adjustment for allowed-values in docs * Renamed unit tests, now working * Adding allowed-values unit tests * Addressing usnistgov#452: cleaned up character handling in XML-to-JSON conversion. Fixed the bug this exposed in markdown->HTML conversion. Added some light unit testing. * More adjustments in view of comments to usnistgov#452 * Repairing more bugs in character handling in Markdown and plain text, with unit testing: usnistgov#452 * Improvements to Schematron for SP800-53 usnistgov#400 * For usnistgov#452, modified copy-and-convert script to suppress character-level post-processing in generated JSON; added unit tests for URIs to detect issues with solidus characters '/' * Edited docs, including datatypes page per usnistgov#452 (regarding character escapes). * Cleanup metaschema valdiation errors * changed use of the ID and IDREF datatype to NCName * Removed ID and IDREF datatypes * refactored components out of ssp metaschema * Updated SP800-53 catalog and NIST profiles to current model * Added name to information type. * Continued refinement of the SSP model * Updated metadata syntax * changing CM-6 constraint to guidance * continued work on SSP * refactored responsible party to metadata * fixed validation issues * Fixed overriding enumerated values * Fixed content validation errors based on latest metaschema changes * Completed full pass through ssp model * added prop to resource * Fixed broken links in Markdown files. Small adjustments to SSP example. * FDixed relapath error caused when the generated schema directory doesn't exist. Also attempting a fix for CircleCI colorized escape codes causing extra characters to appear. * Fixed validation issue with component metaschema * attempting to fix extra characters in CircleCI colorized messages * Updated enumerated values to remove values that should appear in a given context * quick reordering of elements * hardcode ascii color codes * Fixing broken links in readmes; file name update for consistency * added unit tests for bug * fixed color escaping in scripts * Removed bogus unit test and fixed a small cardinality issue in the SSP model * Improvements to docs generation * Added JSON value keys to fields with flags (that did not have them) * Bug reduction in XSD production mainly related to datatypes and allowed values * Fixing glitch in JSON converter generator * Adjustments repairing Metaschema issues * CSS tweak in docs (model maps to use USWDS font) * Adding to Milestone 2 release notes (in progress) * Change to parameter setting in profile model (usnistgov#494, usnistgov#288); release notes towards M2 * Small improvements to copy in release notes (draft) * Updating metaschemas to latest syntax, with improved Metaschema Schematron and unit test tweakage * Adjustments to flag unit test and release notes * SSP Metaschema review and update * cleanup of unused role-id field * reflactored role-id * updated documentation for release. * cleanup of IDs * Minor schema and metaschema improvements * M1 to M2 content converter * Merging local and upstream changes to json example * Relaxing rule over 'ID' flag to accept any string not only XML NCName * reorganizing content for release * more file moves * Fixed changed path * Updated FedRAMP profiles and catalog to current models * Bug fix to XML-to-JSON converter generator * Fixed copy script * Updated FedRAMP baselines and catalog to match new metaschema syntax * fixed merge error * fixed another merge error * fixed spelling issues * Fix from Wendell

* Removed unnecessary metaschema check on json-value-key * Now implementing in-xml="with-wrapper | unwrapped" on markup multiline fields, with default `with-wrapper` and corresponding changes to catalog and profile metaschemas. * Updating catalog and profile schemas for (prose) unwrapping behavior * Renamed 'json-behavior' in Metaschema to 'in-json' for consistency with 'in-xml'. * Updating syntax in metaschemas 'in-json' for 'json-behavior' * Updating json converter generator to provide in-xml behavior 'GROUPED' * Refining json converter production from Metaschema * Repairing egregious syntax error * Rectify bug introduced with new feature * Adding support for 'date' and 'dateTime' datatypes, with no time zones (usnistgov#480), including unit tests * Improvements and corrections to datatypes docs * More small repairs on datatypes descriptions (table markup) * Built out validations of allowed (enumerated) values, including Metaschema validation checks on whether given allowed values conform to their nominal type. * Straightening kinks in datatype metaschema meta-validation * More special handling of recalcitrant datatypes, this time 'NMTOKENS' and 'IDREFS' * Addressing usnistgov#463, adding `id`, `class` and `title` to `group` in profiles. * Schematron improvements; documentation revisions (removing references to subcontrols) * New Schematron for SP800-53-style catalogs * Removed using in implementation to new metaschema for storage. Started on example, which is still invalid * More work on an example, which is still not valid * refactored schematron processing * more work on SSP example * Refactored generate-schema.sh to allow a metaschema to be provided on the CLI * refactored SSP model to be more flat, while allowing component references * slight cleanup * old example * Fixed content generation * updated metaschema to new specs * fixed content errors * added metaschema examples * Updated metaschema example. * Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463 * Added 'param' to profile 'group' model * JSON converter now properly handling flags assigned to assemblies identified by json key * Added 'sort-id' property to controls in SP800-53 catalog * Added support for direct generation of schema and converters * Minor updates to SSP example and metaschema * Adding computer build metaschema examples * Extending allowed-values support to fields; unit tests: usnistgov#437 * More work on allowed-values, now in XSD with unit tests for usnistgov#437 * Repairing small datatyping bug thanks to regression testing * Further adjustment for allowed-values in docs * Renamed unit tests, now working * Adding allowed-values unit tests * Addressing usnistgov#452: cleaned up character handling in XML-to-JSON conversion. Fixed the bug this exposed in markdown->HTML conversion. Added some light unit testing. * More adjustments in view of comments to usnistgov#452 * Repairing more bugs in character handling in Markdown and plain text, with unit testing: usnistgov#452 * Improvements to Schematron for SP800-53 #400 * For usnistgov#452, modified copy-and-convert script to suppress character-level post-processing in generated JSON; added unit tests for URIs to detect issues with solidus characters '/' * Edited docs, including datatypes page per usnistgov#452 (regarding character escapes). * Cleanup metaschema valdiation errors * changed use of the ID and IDREF datatype to NCName * Removed ID and IDREF datatypes * refactored components out of ssp metaschema * Updated SP800-53 catalog and NIST profiles to current model * Added name to information type. * Continued refinement of the SSP model * Updated metadata syntax * changing CM-6 constraint to guidance * continued work on SSP * refactored responsible party to metadata * fixed validation issues * Fixed overriding enumerated values * Fixed content validation errors based on latest metaschema changes * Completed full pass through ssp model * added prop to resource * Fixed broken links in Markdown files. Small adjustments to SSP example. * FDixed relapath error caused when the generated schema directory doesn't exist. Also attempting a fix for CircleCI colorized escape codes causing extra characters to appear. * Fixed validation issue with component metaschema * attempting to fix extra characters in CircleCI colorized messages * Updated enumerated values to remove values that should appear in a given context * quick reordering of elements * hardcode ascii color codes * Fixing broken links in readmes; file name update for consistency * added unit tests for bug * fixed color escaping in scripts * Removed bogus unit test and fixed a small cardinality issue in the SSP model * Improvements to docs generation * Added JSON value keys to fields with flags (that did not have them) * Bug reduction in XSD production mainly related to datatypes and allowed values * Fixing glitch in JSON converter generator * Adjustments repairing Metaschema issues * CSS tweak in docs (model maps to use USWDS font) * Adding to Milestone 2 release notes (in progress) * Change to parameter setting in profile model (usnistgov#494, #288); release notes towards M2 * Small improvements to copy in release notes (draft) * Updating metaschemas to latest syntax, with improved Metaschema Schematron and unit test tweakage * Adjustments to flag unit test and release notes * SSP Metaschema review and update * cleanup of unused role-id field * reflactored role-id * updated documentation for release. * cleanup of IDs * Minor schema and metaschema improvements * M1 to M2 content converter * Merging local and upstream changes to json example * Relaxing rule over 'ID' flag to accept any string not only XML NCName * reorganizing content for release * more file moves * Fixed changed path * Updated FedRAMP profiles and catalog to current models * Bug fix to XML-to-JSON converter generator * Fixed copy script * Updated FedRAMP baselines and catalog to match new metaschema syntax * fixed merge error * fixed another merge error * fixed spelling issues * Fix from Wendell

brian-ruf added enhancement User Story labels Jul 22, 2019

david-waltermire added this to the OSCAL 1.0 Milestone 2 milestone Aug 14, 2019

david-waltermire assigned brian-ruf Aug 22, 2019

david-waltermire added Scope: Content Development of OSCAL content and examples. Scope: Modeling Issues targeted at development of OSCAL formats labels Sep 4, 2019

wendellpiez added a commit to wendellpiez/OSCAL that referenced this issue Sep 11, 2019

Addressing usnistgov#463, adding id, class and title to group…

a54cc4f

… in profiles.

wendellpiez added a commit to wendellpiez/OSCAL that referenced this issue Sep 13, 2019

Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

f7e3b03

brian-ruf mentioned this issue Sep 19, 2019

Patching control info at arbitrary granularity / addressability of control contents #70

Closed

wendellpiez added a commit to wendellpiez/OSCAL that referenced this issue Sep 23, 2019

Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

b7c3c4e

david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue Sep 23, 2019

Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

e299c2b

david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue Sep 26, 2019

Addressing usnistgov#463, adding id, class and title to group…

b32faa9

… in profiles.

david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue Sep 26, 2019

Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

ff20820

david-waltermire added a commit to david-waltermire/OSCAL that referenced this issue Sep 26, 2019

Revert "Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

137e675

" This reverts commit ff20820.

david-waltermire added a commit to david-waltermire/OSCAL that referenced this issue Sep 26, 2019

Revert "Addressing usnistgov#463, adding id, class and title to…

9065f6c

… `group` in profiles." This reverts commit b32faa9.

david-waltermire added a commit to david-waltermire/OSCAL that referenced this issue Sep 26, 2019

Revert "Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

36cd674

" This reverts commit ff20820.

david-waltermire added a commit to david-waltermire/OSCAL that referenced this issue Sep 26, 2019

Revert "Addressing usnistgov#463, adding id, class and title to…

295ac12

… `group` in profiles." This reverts commit b32faa9.

david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue Oct 1, 2019

Addressing usnistgov#463, adding id, class and title to group…

b084591

… in profiles.

david-waltermire pushed a commit to david-waltermire/OSCAL that referenced this issue Oct 1, 2019

Adding 'prop' and 'part' to Profile Metaschema per usnistgov#463

05fac54

david-waltermire closed this as completed Nov 3, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Profile Group Element Should Match Catalog #463

Profile Group Element Should Match Catalog #463

brian-ruf commented Jul 22, 2019

wendellpiez commented Aug 14, 2019

wendellpiez commented Aug 28, 2019

brian-ruf commented Aug 28, 2019 •

edited

Loading

wendellpiez commented Aug 29, 2019

iMichaela commented Sep 5, 2019

wendellpiez commented Sep 11, 2019

david-waltermire commented Sep 12, 2019

wendellpiez commented Sep 13, 2019

brian-ruf commented Sep 19, 2019

wendellpiez commented Sep 19, 2019

david-waltermire commented Nov 3, 2019

Profile Group Element Should Match Catalog #463

Profile Group Element Should Match Catalog #463

Comments

brian-ruf commented Jul 22, 2019

User Story:

Goals:

Dependencies:

Acceptance Criteria

wendellpiez commented Aug 14, 2019

wendellpiez commented Aug 28, 2019

brian-ruf commented Aug 28, 2019 • edited Loading

wendellpiez commented Aug 29, 2019

iMichaela commented Sep 5, 2019

9/5/2019

wendellpiez commented Sep 11, 2019

david-waltermire commented Sep 12, 2019

wendellpiez commented Sep 13, 2019

brian-ruf commented Sep 19, 2019

wendellpiez commented Sep 19, 2019

david-waltermire commented Nov 3, 2019

brian-ruf commented Aug 28, 2019 •

edited

Loading