Feature/all valid test (#896)

* all valid test * make ssp all valid Co-Authored-By: Gabeblis <[email protected]> * update oscal version Co-Authored-By: Gabeblis <[email protected]> * Update fedramp_extensions.feature Co-Authored-By: Gabeblis <[email protected]> * Update fedramp_extensions.feature Co-authored-by: A.J. Stein <[email protected]> * Update module.mk Co-authored-by: A.J. Stein <[email protected]> * add integration npm commands * Update fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * get latest ssp-all valid * all valid test * make ssp all valid Co-Authored-By: Gabeblis <[email protected]> * update oscal version Co-Authored-By: Gabeblis <[email protected]> * Update fedramp_extensions.feature Co-Authored-By: Gabeblis <[email protected]> * Update fedramp_extensions.feature Co-authored-by: A.J. Stein <[email protected]> * Update module.mk Co-authored-by: A.J. Stein <[email protected]> * add integration npm commands * Update fedramp-external-constraints.xml Co-authored-by: Gabeblis <[email protected]> * get latest ssp-all valid * Update fedramp_extensions.feature Co-Authored-By: Rene Tshiteya <[email protected]> * use env variable to control quiet --------- Co-authored-by: Gabeblis <[email protected]> Co-authored-by: A.J. Stein <[email protected]> Co-authored-by: Rene Tshiteya <[email protected]>
GSA · Nov 20, 2024 · 595e071 · 595e071
1 parent a79a5fb
commit 595e071
Show file tree

Hide file tree

Showing 5 changed files with 181 additions and 157 deletions.
diff --git a/features/fedramp_extensions.feature b/features/fedramp_extensions.feature
@@ -3,7 +3,134 @@ Feature: OSCAL Document Constraints
 @style-guide
 Scenario Outline: Validating OSCAL constraints with metaschema constraints
   Then I should verify that all constraints follow the style guide constraint
-
+
+@integration
+Scenario Outline: Documents that should be valid are pass
+  Then I should have valid results "<valid_file>"
+Examples:
+| valid_file     |
+| ssp-all-VALID.xml |
+# | ../../../content/awesome-cloud/xml/AwesomeCloudSSP1.xml |
+# | ../../../content/awesome-cloud/xml/AwesomeCloudSSP2.xml |
+
+@full-coverage
+Scenario: Preparing constraint coverage analysis
+Given I have loaded all Metaschema extensions documents
+And I have collected all YAML test files in the test directory
+When I extract all constraint IDs from the Metaschema extensions
+And I analyze the YAML test files for each constraint ID
+
+@full-coverage
+Scenario Outline: Ensuring full test coverage for "<constraint_id>"
+Then I should have both FAIL and PASS tests for constraint ID "<constraint_id>"
+Examples:
+| constraint_id |
+#BEGIN_DYNAMIC_CONSTRAINT_IDS
+  | address-type |
+  | attachment-type |
+  | authorization-type |
+  | categorization-has-correct-system-attribute |
+  | categorization-has-information-type-id |
+  | cia-impact-has-adjustment-justification |
+  | cia-impact-has-selected |
+  | cloud-service-model |
+  | component-type |
+  | control-implementation-status |
+  | data-center-alternate |
+  | data-center-count |
+  | data-center-country-code |
+  | data-center-primary |
+  | data-center-us |
+  | deployment-model |
+  | fedramp-version |
+  | fully-operational-date-is-valid |
+  | fully-operational-date-type |
+  | has-authenticator-assurance-level |
+  | has-authorization-boundary-diagram |
+  | has-authorization-boundary-diagram-caption |
+  | has-authorization-boundary-diagram-description |
+  | has-authorization-boundary-diagram-link |
+  | has-authorization-boundary-diagram-link-href-target |
+  | has-authorization-boundary-diagram-link-rel |
+  | has-authorization-boundary-diagram-link-rel-allowed-value |
+  | has-cloud-deployment-model |
+  | has-cloud-deployment-model-remarks |
+  | has-cloud-service-model |
+  | has-cloud-service-model-remarks |
+  | has-configuration-management-plan |
+  | has-data-flow |
+  | has-data-flow-description |
+  | has-data-flow-diagram |
+  | has-data-flow-diagram-caption |
+  | has-data-flow-diagram-description |
+  | has-data-flow-diagram-link |
+  | has-data-flow-diagram-link-href-target |
+  | has-data-flow-diagram-link-rel |
+  | has-data-flow-diagram-link-rel-allowed-value |
+  | has-data-flow-diagram-uuid |
+  | has-federation-assurance-level |
+  | has-fully-operational-date |
+  | has-identity-assurance-level |
+  | has-incident-response-plan |
+  | has-information-system-contingency-plan |
+  | has-network-architecture |
+  | has-network-architecture-diagram |
+  | has-network-architecture-diagram-caption |
+  | has-network-architecture-diagram-description |
+  | has-network-architecture-diagram-link |
+  | has-network-architecture-diagram-link-href-target |
+  | has-network-architecture-diagram-link-rel |
+  | has-network-architecture-diagram-link-rel-allowed-value |
+  | has-published-date |
+  | has-rules-of-behavior |
+  | has-security-impact-level |
+  | has-security-sensitivity-level |
+  | has-separation-of-duties-matrix |
+  | has-system-id |
+  | has-system-name-short |
+  | has-user-guide |
+  | import-profile-has-available-document |
+  | import-profile-resolves-to-fedramp-content |
+  | information-type-800-60-v2r1 |
+  | information-type-has-availability-impact |
+  | information-type-has-confidentiality-impact |
+  | information-type-has-integrity-impact |
+  | information-type-system |
+  | interconnection-direction |
+  | interconnection-security |
+  | inventory-item-allows-authenticated-scan |
+  | inventory-item-public |
+  | inventory-item-virtual |
+  | marking |
+  | missing-response-components |
+  | party-has-name |
+  | privilege-level |
+  | prop-response-point-has-cardinality-one |
+  | resource-has-base64-or-rlink |
+  | resource-has-title |
+  | responsible-party-is-person |
+  | responsible-party-prepared-by |
+  | responsible-party-prepared-by-location-valid |
+  | responsible-party-prepared-for |
+  | responsible-party-prepared-for-location-valid |
+  | role-defined-authorizing-official-poc |
+  | role-defined-information-system-security-officer |
+  | role-defined-prepared-by |
+  | role-defined-prepared-for |
+  | role-defined-system-owner |
+  | scan-type |
+  | security-level |
+  | security-sensitivity-level-matches-security-impact-level |
+  | user-has-authorized-privilege |
+  | user-has-privilege-level |
+  | user-has-role-id |
+  | user-has-sensitivity-level |
+  | user-has-user-type |
+  | user-privilege-level |
+  | user-sensitivity-level |
+  | user-type |
+#END_DYNAMIC_CONSTRAINT_IDS
+
 @constraints
 Scenario Outline: Validating OSCAL documents with metaschema constraints
   Given I have Metaschema extensions documents
@@ -230,123 +357,3 @@ Examples:
   | user-type-FAIL.yaml |
   | user-type-PASS.yaml |
 #END_DYNAMIC_TEST_CASES
-
-@full-coverage
-Scenario: Preparing constraint coverage analysis
-Given I have loaded all Metaschema extensions documents
-And I have collected all YAML test files in the test directory
-When I extract all constraint IDs from the Metaschema extensions
-And I analyze the YAML test files for each constraint ID
-
-@full-coverage
-Scenario Outline: Ensuring full test coverage for "<constraint_id>"
-Then I should have both FAIL and PASS tests for constraint ID "<constraint_id>"
-Examples:
-| constraint_id |
-#BEGIN_DYNAMIC_CONSTRAINT_IDS
-  | address-type |
-  | attachment-type |
-  | authorization-type |
-  | categorization-has-correct-system-attribute |
-  | categorization-has-information-type-id |
-  | cia-impact-has-adjustment-justification |
-  | cia-impact-has-selected |
-  | cloud-service-model |
-  | component-type |
-  | control-implementation-status |
-  | data-center-alternate |
-  | data-center-count |
-  | data-center-country-code |
-  | data-center-primary |
-  | data-center-us |
-  | deployment-model |
-  | fedramp-version |
-  | fully-operational-date-is-valid |
-  | fully-operational-date-type |
-  | has-authenticator-assurance-level |
-  | has-authorization-boundary-diagram |
-  | has-authorization-boundary-diagram-caption |
-  | has-authorization-boundary-diagram-description |
-  | has-authorization-boundary-diagram-link |
-  | has-authorization-boundary-diagram-link-href-target |
-  | has-authorization-boundary-diagram-link-rel |
-  | has-authorization-boundary-diagram-link-rel-allowed-value |
-  | has-cloud-deployment-model |
-  | has-cloud-deployment-model-remarks |
-  | has-cloud-service-model |
-  | has-cloud-service-model-remarks |
-  | has-configuration-management-plan |
-  | has-data-flow |
-  | has-data-flow-description |
-  | has-data-flow-diagram |
-  | has-data-flow-diagram-caption |
-  | has-data-flow-diagram-description |
-  | has-data-flow-diagram-link |
-  | has-data-flow-diagram-link-href-target |
-  | has-data-flow-diagram-link-rel |
-  | has-data-flow-diagram-link-rel-allowed-value |
-  | has-data-flow-diagram-uuid |
-  | has-federation-assurance-level |
-  | has-fully-operational-date |
-  | has-identity-assurance-level |
-  | has-incident-response-plan |
-  | has-information-system-contingency-plan |
-  | has-inventory-items |
-  | has-network-architecture |
-  | has-network-architecture-diagram |
-  | has-network-architecture-diagram-caption |
-  | has-network-architecture-diagram-description |
-  | has-network-architecture-diagram-link |
-  | has-network-architecture-diagram-link-href-target |
-  | has-network-architecture-diagram-link-rel |
-  | has-network-architecture-diagram-link-rel-allowed-value |
-  | has-published-date |
-  | has-rules-of-behavior |
-  | has-security-impact-level |
-  | has-security-sensitivity-level |
-  | has-separation-of-duties-matrix |
-  | has-system-id |
-  | has-system-name-short |
-  | has-user-guide |
-  | import-profile-has-available-document |
-  | import-profile-resolves-to-fedramp-content |
-  | information-type-800-60-v2r1 |
-  | information-type-has-availability-impact |
-  | information-type-has-confidentiality-impact |
-  | information-type-has-integrity-impact |
-  | information-type-system |
-  | interconnection-direction |
-  | interconnection-security |
-  | inventory-item-allows-authenticated-scan |
-  | inventory-item-public |
-  | inventory-item-virtual |
-  | marking |
-  | missing-response-components |
-  | party-has-name |
-  | privilege-level |
-  | prop-response-point-has-cardinality-one |
-  | resource-has-base64-or-rlink |
-  | resource-has-title |
-  | responsible-party-is-person |
-  | responsible-party-prepared-by |
-  | responsible-party-prepared-by-location-valid |
-  | responsible-party-prepared-for |
-  | responsible-party-prepared-for-location-valid |
-  | role-defined-authorizing-official-poc |
-  | role-defined-information-system-security-officer |
-  | role-defined-prepared-by |
-  | role-defined-prepared-for |
-  | role-defined-system-owner |
-  | scan-type |
-  | security-level |
-  | security-sensitivity-level-matches-security-impact-level |
-  | unique-inventory-item-asset-id |
-  | user-has-authorized-privilege |
-  | user-has-privilege-level |
-  | user-has-role-id |
-  | user-has-sensitivity-level |
-  | user-has-user-type |
-  | user-privilege-level |
-  | user-sensitivity-level |
-  | user-type |
-#END_DYNAMIC_CONSTRAINT_IDS