diff --git a/content/06.quality_criteria.md b/content/06.quality_criteria.md index 0da733c..398a80e 100644 --- a/content/06.quality_criteria.md +++ b/content/06.quality_criteria.md @@ -105,7 +105,53 @@ early detection of failures in the development cycle. * **[QC.Uni04.1]** When working on automated testing, the use of testing doubles is RECOMMENDED to mimic a simplistic behavior of objects and procedures. -## 4.7. Functional Testing [QC.Fun] +## **PROPOSAL** + +## 4.6. Test Harness [QC.Har] + +In software development, a test harness +[@url:https://searchsoftwarequality.techtarget.com/definition/test-harness], +is a collection of software and test data used by developers to unit test software models during +development. A test harness will specifically refer to test doubles, which are programs +that interact with the software being tested. Once a test harness is used to execute a test, they +can also utilize a test library to generate reports. + +It is also a simple form of Integration Testing, where interaction and integration with external +components are substituted by a Double. + +Test Double is a generic term for any case where you replace a production object for testing +purposes. There are various kinds of double [@isbn:9780131495050]: + +* Dummy objects are passed around but never actually used. Usually they are just used to fill + parameter lists. + +* Fake objects actually have working implementations, but usually take some shortcut which makes + them not suitable for production (an InMemoryTestDatabase is a good example). + +* Stubs provide canned answers to calls made during the test, usually not responding at all to + anything outside what's programmed in for the test. + +* Spies are stubs that also record some information based on how they were called. One form of this + might be an email service that records how many messages it was sent. + +* Mocks are pre-programmed with expectations which form a specification of the calls they are + expected to receive. They can throw an exception if they receive a call they don't expect and + are checked during verification to ensure they got all the calls they were expecting. + +As such the following criteria is defined for Test Harness: + +* **[QC.Har01]** When working on automated testing, the use of Test Doubles + is RECOMMENDED to mimic a simplistic behavior of objects and procedures. + +* **[QC.Har02]** Test Doubles SHOULD reside in the software component repository code + base but separated from the main code. + +* **[QC.Har03]** Regression testing, that checks the conformance with previous tests, is + covered at this stage by executing the complete set of Test Doubles available. + +* **[QC.Har04]** Test Doubles and regression, MUST be checked on change basis. + +## 4.XX. Functional Testing [QC.Fun] - **TO BE REMOVED** Functional testing is a type of black-box testing. It involves the verification of the software component’s @@ -115,24 +161,25 @@ functionality that the software component exposes, leaving apart any internal design analysis or side-effects to external systems. * **[QC.Fun01]** Functional testing SHOULD tend to cover the full set of functionality that - the software component claims to provide. + the software component claims to provide. **[SvcQC.Fun01]** * **[QC.Fun02]** Functional tests SHOULD be checked automatically with the exception of those functionality that require human interaction, such as Graphical User - Interfaces (GUI). + Interfaces (GUI). **[SvcQC.Fun02]** * **[QC.Fun03]** When working on automated testing, the use of testing doubles is RECOMMENDED to mimic a simplistic behavior of objects and procedures. + **TEST HARNESS** * **[QC.Fun04]** Functional tests SHOULD reside in the software component repository code - base but separated from the main code. + base but separated from the main code. **[SvcQC.Fun03]** **TEST HARNESS** * **[QC.Fun05]** Regression testing, that checks the conformance with previous tests, is - covered at this stage by executing the complete set of functional tests available. + covered at this stage by executing the complete set of functional tests available. **TEST HARNESS** -* **[QC.Fun06]** Functional and regression testing MUST be checked on change basis. +* **[QC.Fun06]** Functional and regression testing MUST be checked on change basis. **TEST HARNESS** -* **[QC.Fun07]** Functional and regression testing coverage MAY NOT be suitable for automated testing. +* **[QC.Fun07]** Functional and regression testing coverage MAY NOT be suitable for automated testing. **REMOVE** ## 4.8. Test-Driven Development [QC.Tdd]