-
Notifications
You must be signed in to change notification settings - Fork 0
/
analysis.qmd
128 lines (64 loc) · 10.7 KB
/
analysis.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
::: {.callout-important}
This version of the AQuA book is a preliminary ALPHA draft. It is still in development, and we are still working to ensure that it meets user needs.
The draft currently has no official status. It is a work in progress and is subject to further revision and reconfiguration (possibly substantial change) before it is finalised.
:::
# Analysis
## Introduction and overview
The analysis stage is where planned analysis is undertaken and assured, and progress and relevance are monitored. During this stage, the design may be amended to account for changing circumstances, emerging information or unexpected difficulties or limitations encountered. This stage also includes maintaining appropriate and traceable records of the analysis and assurance activities conducted, changes, decisions and assumptions made. In some cases, changes or limitations encountered may necessitate a return to either the design or scoping stage.
## Roles and responsibilities in the analysis stage
### The Commissioner's responsibilities during the analysis stage
* The Commissioner should be available to provide input and clarifications to the Analyst.
* The Commissioner’s should review any changes in design or methodology that the Analyst brings to their attention.
### The Analyst's responsibilities during the analysis stage
* The Analyst shall follow the conduct the verification and validation activities that were designed as part of the analytical plan in [the design stage](design.html#the-design-stage). They shall provide traceable documentation of the assurance they have undertaken. They shall respond to recommendations from the Assurer and act on them as appropriate.
* When the analysis includes coding, the Analyst shall proportionately follow [best practice for code development](#assurance-of-code).
* The Analyst shall produce documentation of the data (see [The Government Data Quality Framework](https://www.gov.uk/government/publications/the-government-data-quality-framework/the-government-data-quality-framework)) and methods used. The Analyst shall ensure these are sufficient for the Assurer to understand the approach.
* The Analyst shall document any changes to the analytical plan in a proportionate manner.
* The Analyst shall maintain appropriate contact with Commissioner and Assurer. This provides and opportunity for them to advise on whether the analysis is still meeting the Commissioner's needs or whether there are any new requirements.
### The Assurer's responsibilities during the analysis stage
* The Assurer shall review the assurance completed by the Analyst, carry out any further validation and verification they may see as appropriate, and report errors and areas for improvement to the Analyst. The Assurer may then need to re-review the analytical work completed, as required.
* When the analysis includes coding, the Assurer shall review that the work proportionately adheres to [best practice for code development](#assurance-of-code).
* The Assurer may be required to provide feedback on changes to the analytical plan, and consider whether they are qualified to provide rigorous assurance on the revised methodology.
### The Approver's responsibilities during the analysis stage
The Approver should be aware of the progress of the analysis and ensure that they are available for approving the work at the delivery stage.
## Assurance activities in the analysis stage
### Verification and validation
Verification that the implemented methodology meets the design requirements should be incorporated as part the analysis. [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) reviewed verification techniques in relation to simulation modelling. These techniques extend to analysis more broadly. These include:
- Informal analysis: techniques that rely on human reasoning and subjectivity.
- Static analysis: tests that the implementation of the analysis before it is run. For example, checking that code adheres to code conventions, structural analysis of the code by examining graphs of control and data flows, .
- Dynamic analysis: tests the behaviour of the system, model or code to find errors that arise during execution. This includes [unit testing](https://en.wikipedia.org/wiki/Unit_testing), [integration testing](https://en.wikipedia.org/wiki/Integration_testing) and [stress testing](https://en.wikipedia.org/wiki/Stress_testing_(computing))
- Symbolic analysis: particularly relevant to modelling and tests the transformation of symbolic proxies of model inputs into outputs during the execution of a model. Includes path tracing and cause-effect testing (see [Whitener and Balci (1989)](https://typeset.io/pdf/guidelines-for-selecting-and-using-simulation-model-143kzp6h5s.pdf) )
- Constraint analysis: particularly relevant to modelling and tests the implementation of constraints during model execution. This includes checking the assertions of the model and boundary analysis.
- Formal analysis: tests logical correctness through [formal verification](https://en.wikipedia.org/wiki/Formal_verification#Formal_verification_for_software) such as logic or mathematical proofs.
Validation refers to testing whether the product meets the requirements of users. Hence, it is important to involve the users in the process. [Methods for validation](https://en.wikipedia.org/wiki/Verification_and_validation#Aspects_of_analytical_methods_validation_) include quantification and judgment of acceptable sensitivity, specificity, accuracy, precision and reproducibility.
Validation of models includes testing the validity of the conceptual model, and testing the operational validity of any computerized model. Techniques that may be useful in validation of models are reviewed by [Sargent (2011)](https://www.informs-sim.org/wsc11papers/016.pdf).
The Analyst has primary responsibility for conducting verification and validation. The Assurer is responsible for reviewing the verification and validation that is carried out by the Analyst, and for conducting or recommending additional verification and validation as required. The Assurer may refer to the [specification document](definitions_and_key_concepts.html#specification-documentation) to assure that the analysis meets the specification.
### Data validity and data considerations
Testing data validity (i.e. that data meet the specification for which they are used) is a vital part of analysis. Procedures for assuring data validity include testing for internal consistency, screening for data characteristics (outliers, trends, expected distributions etc), and assuring robust data management practices (e.g. automating data creation and data sourcing).
It is rare to have the perfect dataset for an analytical commission. Reasons for this include:
* The data is not available in the time frame required for the ideal analysis;
* The data definition does not perfectly align with the commission;
* There are data or coverage gaps;
* The data may be experimental or there are other reasons why it is not ‘mature’.
Often, no data is available that are directly and precisely relevant to the parameter and conditions of interest. In such cases, it is often possible to use surrogate data. This is measurements of another parameter, or of the parameter of interest under different conditions, that is related to the parameter and conditions of interest. This implies an extrapolation between parameters, or between conditions for the same parameter. The use of surrogate data introduces further uncertainty, additional to that associated with the data itself. It may be possible to quantify this additional uncertainty using expert knowledge of the relationship between the surrogate and the parameter of interest.
The effect of using a proxy dataset should be explored and, if the uncertainty associated with the dataset has a large bearing on the analysis, its appropriateness should be revisited. This exploration, and the decision to use a particular dataset or input, should be recorded for the benefit of the Assurer.
### Assurance of code
The [Duck Book](https://best-practice-and-impact.github.io/qa-of-code-guidance/intro.html) provides detailed guidance on developing and assurance for delivering quality code. This includes guidance on structuring code, producing documentation, using version control, data management, testing, peer review, and automation. The Analyst shall follow the guidance for good quality code development in a proportionate manner, and the Assurer shall review this accordingly.
## Documentation in the analysis stage
The Analyst should:
* Maintain appropriate records of the work;
* Fully document any code following agreed standards;
* Log the data, assumptions and inputs used in the analysis, and decisions made (see [documentation](definitions_and_key_concepts.html/#documentation));
* Record the verification and validation that has been undertaken, documenting any activities that are outstanding and noting what remedial action has been taken and its effect on the analysis;
* Produce [user and technical documentation](definitions_and_key_concepts.html#user-technical-documentation). For modelling, the Analyst may include a model map that describes data flows and transformations.
## Treatment of uncertainty in the analysis stage
While the Scoping and Design stages identified and described risks and uncertainties, the Analysis stage aims to assess and quantify how uncertainty may influence the analytical outcome and their contribution to the range and likelihoods of possible outcomes. [The Uncertainty Toolkit for Analysts](hhttps://analystsuncertaintytoolkit.github.io/UncertaintyWeb/chapter_5.html) reviews methods of quantifying uncertainty. The verification and validation by the Analyst and Assurer should assure the appropriate treatment of uncertainty.
## Black box models and the analysis stage
[Black box models](definitions_and_key_concepts.html/#black-box-models) such as AI and ML models are not as transparent as traditionally coded models. This adds challenge to the assurance of these models as compared to other forms of analysis.
Assurance activities during the Analysis stage:
* may include performance testing in a live environment and
* should include the verification steps set out in the Design Phase
* should include validation and verification of automatic tests to ensure the model behave as expected
See the [Introduction to AI Assurance](https://www.gov.uk/government/publications/introduction-to-ai-assurance) for further details.
## Multi-use models and the analysis stage
In multi-use models, analysis and edits may be carried out on individual elements of the model at differing times. This calls for mechanisms for assuring that the changes integrate into the larger model as expected, for example, through the use of test-suites.