[ENHANCEMENT] Inclusion of test harness for pilot testbeds

[orviz]

Please elaborate here on the enhancement request.
Inclusion of test harness for pilot testbeds

Describe the solution you'd like
Quoting from @samuelbernardo:

I think that the test pilot should have defined multiple stages where the development will be processed. This would define the steps for using test harness methodology. In the end we can get the test environment necessary to integrate with automated testing tools, but as a MAY requirement.

Additional context
[1] https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e5bbef5c65&appId=PPGMS
[2] https://intercor-project.eu/wp-content/uploads/sites/15/2019/07/InterCor_M10-Pilot-RollOut-Operations_v1.0_INEA.pdf
[3] https://www.guru99.com/test-environment-software-testing.html
[4] https://www.guru99.com/what-is-test-harness-comparison.html
[5] https://en.wikipedia.org/wiki/Test_harness
[6] https://github.com/indigo-dc/sqa-baseline/blob/87bc18c1eee457d01484e3f061724896695ff03c/content/06.quality_criteria.md

[samuelbernardolip]

As an example for application of test harness we could start reviewing the implementation for Jenkins CI:
https://github.com/jenkinsci/jenkins-test-harness

[mariojmdavid]

this issue is invalid after discussion
it is implemented in the service QA baseline in the integration test criteria

[mariojmdavid]

comment from @samuelbernardolip , seems it did not get recorded here

Hi Mario,

I don't understand your comment when you mention "this issue is invalid" and then that is implemented "in the integration test criteria".

Test harness have multiple definitions, but I suggest to adopt the following:

A test harness or automated test framework is a collection of software and test data configured to test a program unit by running it under varying conditions and monitoring its behavior and outputs. It has two main parts: the test execution engine and the test script repository.

In this context there is a set of patterns that allow to specific tests to run, orchestrate a runtime environment, and provide a capability to analyse results. The typical objectives of a test harness are to:

• automate the testing process
• execute test suites of test cases
• generate associated test reports

On the baseline context, test harnesses are external to the application being tested and simulate services or functionality not available in a test environment. On the other side you have integration testing (not the same), where test stubs or mocks, as components or building blocks of the application, are replaced by working components.

For example, imagine that you need to run an application using a real HPC system interface, but you haven't access to that system in development stage. So to prepare the virtual environment that reproduces the HPC system, a test harness may be built to be used as a substitute.

So this is not directly related to the system testing, but in a kind of previous stage where the real infrastructure is not yet available. Test harness can be scoped as a previous testing phase before integration testing. In the end is a kind of unit testing applied to system testing.

So in my opinion, this issue is valid for both SQA and Service baseline. Besides this kind of testing could be very useful to provide an automated platform for user testing, it is also very expensive to develop, and so the only consequence is that we can not implement it for the time being of this project.

Cheers,

Samuel

[mariojmdavid]

indeed this is a good question, the points of discussion where that the SQA baseline we would keep all criteria that could be not only automated but on the source code, and white testing, and this covers unit testing
and eventually some mix that can already be considered functional testing.

As such, that plan is to remove API, functional, integration from the SQA base, the test harness as you describe is also blackbox but against mocks and stubs

anyway I have not only reopen the issue, but will use it for further discussion because I also have kind of mixed feelings in this regard

[mariojmdavid]

added a section/criteria for test harness
commit cdaec1c