From bffc5d57b90adb5a0b5a93b887e46bbc9ae0c5ba Mon Sep 17 00:00:00 2001 From: amyheather Date: Tue, 11 Jun 2024 16:58:12 +0100 Subject: [PATCH] Minor updates --- evaluation/badges.qmd | 1 - evaluation/reporting.qmd | 8 ++++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/evaluation/badges.qmd b/evaluation/badges.qmd index 40409be..8268ee6 100644 --- a/evaluation/badges.qmd +++ b/evaluation/badges.qmd @@ -3,7 +3,6 @@ title: "Badges" format: html: code-fold: true -jupyter: python3 bibliography: ../quarto_site/references.bib --- diff --git a/evaluation/reporting.qmd b/evaluation/reporting.qmd index eed9500..7600cce 100644 --- a/evaluation/reporting.qmd +++ b/evaluation/reporting.qmd @@ -51,7 +51,7 @@ Of the **24** items in the checklist: | 5.3 Model execution | State the event processing mechanism used e.g. three phase, event, activity, process interaction.
*Note that in some commercial software the event processing mechanism may not be published. In these cases authors should adhere to item 5.1 software recommendations.*
State all priority rules included if entities/activities compete for resources.
If the model is parallel, distributed and/or use grid or cloud computing, etc., state and preferably reference the technology used. For parallel and distributed simulations the time management algorithms used. If the HLA is used then state the version of the standard, which run-time infrastructure (and version), and any supporting documents (FOMs, etc.) | ❌ Not met | Not described, but understand from Tom that as it used SimPy, it was a process-based simulation worldview. | | 5.4 System specification | State the model run time and specification of hardware used. This is particularly important for large scale models that require substantial computing power. For parallel, distributed and/or use grid or cloud computing, etc. state the details of all systems used in the implementation (processors, network, etc.) | 🟡 Partially | Do not state model run time, but do provide hardware specification, and use parallelism

`Materials and methods - analysis environment`: "The dialysis model results were run on an Intel i9-7980XE CPU with 64GB RAM running Ubuntu 19.10 Linux."

`Conclusion`: "We have designed the tools to scale to large international dialysis planning situations and exploit parallelism and fast well known heuristics." | | **Code access** | -| 6.1 Computer model sharing statement | Describe how someone could obtain the model described in the paper, the simulation software and any other associated software (or hardware) needed to reproduce the results. Provide, where possible, the link and DOIs to these. | ✅ Fully | `Data availability statement`: "The python code and anonymous aggregate level data used this study are available from Zenodo . Production code is hosted on University of Exeter’s GitLab ." +| 6.1 Computer model sharing statement | Describe how someone could obtain the model described in the paper, the simulation software and any other associated software (or hardware) needed to reproduce the results. Provide, where possible, the link and DOIs to these. | ✅ Fully | `Data availability statement`: "The python code and anonymous aggregate level data used this study are available from Zenodo . Production code is hosted on University of Exeter’s GitLab ." | : {tbl-colwidths="[12, 36, 12, 40]"} ## DES checklist derived from ISPOR-SDM @@ -74,7 +74,7 @@ Of the **18** items in the checklist: | - | - | - | - | | **Model conceptualisation** | | 1 Is the focused health-related decision problem clarified? | ...the decision problem under investigation was defined. DES studies included different types of decision problems, eg, those listed in previously developed taxonomies. | ✅ Fully | Planning service delivery when need to keep COVID-positive and COVID-negative patients seperate, testing the plan under worst-case scenario three-month spread.

`Introduction`: "Rapid guidelines for dialysis service delivery have been published. These include separation of COVID-positive and COVID-negative patients; dialysis units working with transport providers to minimise the risk of cross-infection; and continuing to treat patients as close to home as possible. Planning service delivery that separates COVID-positive patients is complicated, due to the uncertainty of the spread of SARS-CoV-2, the variability seen in symptom onset, length of infectivity, and regional delivery of dialysis. We therefore sought to support decision making in the period prior to peak infection by developing mathematical models of dialysis service delivery"

`Materials and methods - study setting`: "During the epidemic, COVID-positive patients will be treated separately from negative and recovered. The Queen Alexandra will be used as the primary site for positive outpatients and inpatients with spillover to a second site (Basingstoke) when capacity is insufficient. Patient transport services will provide COVID only ambulances with a policy of single patient transport" | -| 2 Is the modeled healthcare setting/health condition clarified? | ...the physical context/scope (eg, a certain healthcare unit or a broader system) or disease spectrum simulated was described. | ✅ Fully | Patients with Chronic Kidney Disease visiting dialysis units during COVID-19

`Introduction`: "Severe Acute Respiratory Syndrome-Corona Virus-2 (SARS-CoV-2) and the disease it causes COVID-19 (henceforth known as COVID) is causing widespread disruption to normal healthcare services, as the number COVID-positive cases increases... Although social distancing measures are in place both in the UK and internationally, patients with Chronic Kidney Disease who must visit dialysis units are limited in their ability to be fully isolated." +| 2 Is the modeled healthcare setting/health condition clarified? | ...the physical context/scope (eg, a certain healthcare unit or a broader system) or disease spectrum simulated was described. | ✅ Fully | Patients with Chronic Kidney Disease visiting dialysis units during COVID-19

`Introduction`: "Severe Acute Respiratory Syndrome-Corona Virus-2 (SARS-CoV-2) and the disease it causes COVID-19 (henceforth known as COVID) is causing widespread disruption to normal healthcare services, as the number COVID-positive cases increases... Although social distancing measures are in place both in the UK and internationally, patients with Chronic Kidney Disease who must visit dialysis units are limited in their ability to be fully isolated." | | 3 Is the model structure described? | ...the model’s conceptual structure was described in the form of either graphical or text presentation. | ✅ Fully | `Figure 1`: "Schematic representation of patient pathway"
![Figure 1: Schematic representation of patient pathway](../original_study/article_fig1.png){.lightbox}

`Materials and methods - Dialysis model` section provides a detailed description of how the model works, and how patients flow through it | | 4 Is the time horizon given? | ...the time period covered by the simulation was reported. | ✅ Fully | Three months - `Materials and methods - patient progression model`: "The baseline model takes a worst case progression of COVID, infecting 80% of the dialysis population over 3 months." | | 5 Are all simulated strategies/scenarios specified? | ...the comparators under test were described in terms of their components, corresponding variations, etc | ✅ Fully | There is a single scenario: worst-case three month spread. The parameters for this scenario are described and justified.

`Materials and methods - patient progression model`: "The baseline model takes a worst case progression of COVID, infecting 80% of the dialysis population over 3 months." | @@ -85,13 +85,13 @@ Of the **18** items in the checklist: | 9 Are model uncertainties discussed? | ...the uncertainty surrounding parameter estimations and adopted statistical methods (eg, 95% confidence intervals or possibility distributions) were reported. | ✅ Fully | All figures include the minimum and maximum from 30 trials (shown with fainter lines), alongside the median results (shown with bolder lines). Example from `Figure 2`:
![Figure 2: Patient state over time](../original_study/article_fig2.png){.lightbox} | | 10 Are sensitivity analyses performed and reported? | ...the robustness of model outputs to input uncertainties was examined, for example via deterministic (based on parameters’ plausible ranges) or probabilistic (based on a priori-defined probability distributions) sensitivity analyses, or both. | ❌ Not met | Not reported. | | **Validation** | -| 11 Is face validity evaluated and reported? | ...it was reported that the model was subjected to the examination on how well model designs correspond to the reality and intuitions. It was assumed that this type of validation should be conducted by external evaluators with no stake in the study. | ✅ Fully | `Materials and methods - verification and validation`: They "worked closely with clinicians, managers and informatics specialists within the local health system to review iterative versions of the model. We also opted to model a range of likely scenarios including what is widely believed to be the worst case." +| 11 Is face validity evaluated and reported? | ...it was reported that the model was subjected to the examination on how well model designs correspond to the reality and intuitions. It was assumed that this type of validation should be conducted by external evaluators with no stake in the study. | ✅ Fully | `Materials and methods - verification and validation`: They "worked closely with clinicians, managers and informatics specialists within the local health system to review iterative versions of the model. We also opted to model a range of likely scenarios including what is widely believed to be the worst case." | | 12 Is cross validation performed and reported | ...comparison across similar modeling studies which deal with the same decision problem was undertaken. | N/A | This was not possible as, at the point the study was conducted, similar studies on the same decision problem were not available.

`Materials and methods - verification and validation`: "Quantitative validation of models (checking models are appropriate detailed and sufficiently accurate) is challenging in the COVID epidemic as the forecast is of unprecedented conditions" | | 13 Is external validation performed and reported? | ...the modeler(s) examined how well the model’s results match the empirical data of an actual event modeled. | N/A | This was not possible as, at the point the study was conducted, no empirical data was available.

`Materials and methods - verification and validation`: "Quantitative validation of models (checking models are appropriate detailed and sufficiently accurate) is challenging in the COVID epidemic as the forecast is of unprecedented conditions" | | 14 Is predictive validation performed or attempted? | ...the modeler(s) examined the consistency of a model’s predictions of a future event and the actual outcomes in the future. If this was not undertaken, it was assessed whether the reasons were discussed. | ✅ Fully | Not performed but justified. `Discussion - strengths of the study`: "Our toolkit complements the existing epidemiological and ICU literature and aligns itself with the recent call to build new tools that can support operational decision making in health services during the pandemic. Indeed a strength of our toolkit is its practical utility in supporting the continuous safe treatment of a vulnerable group during a national or regional outbreak of COVID infection. We argue that more work like ours is needed to support health services and that the data science community must now focus on research translation. While the world’s population awaits the results of vaccine trials, the risk of a secondary wave of infections remains a reality. To protect the public and support health services under these circumstances we urgently need more research and development of novel practical decision support tools. These tools must be findable, accessible and verified. Our approach has been to make our modelling tools freely available, verifiable and open online." | | **Generalisability and stakeholder involvement** | | 15 Is the model generalizability issue discussed? | ...the modeler(s) discussed the potential of the resulting model for being applicable to other settings/populations (single/multiple application). | 🟡 Partially | The authors state that the tools from this work can be used in other populations - `Conclusion`: "The study we present applies these tools in the South of England where they were used to stress test capacity planning plans. As the pandemic progresses internationally there are risks of regional outbreaks or secondary national spikes in cases. At the start of the pandemic, there were limited operational modelling tools available to support decision makers. Our simulation toolkit is now available for use immediately by health planners. We have designed the tools to scale to large international dialysis planning situations and exploit parallelism and fast well known heuristics."

The authors provide a description of the sample in the paper, with further information on the study setting and dialysis units in the `Supplementary Materials 1`: Study setting geography. This supports readers understanding of potential generalisaibility.

The study encourages reuse of the tools in other settings, but doesn't explicitly describe whether the results of this study are generalisable or not, and to whom. | | 16 Are decision makers or other stakeholders involved in modeling? | ...the modeler(s) reported in which part throughout the modeling process decision makers and other stakeholders (eg, subject experts) were engaged. | ✅ Fully | Review of model - `Materials and methods - verification and validation`: they "worked closely with clinicians, managers and informatics specialists within the local health system to review iterative versions of the model." | -| 17 Is the source of funding stated? | ...the sponsorship of the study was indicated. | ✅ Fully | `Funding`: "This article presents independent research funded by the National Institute for Health Research (NIHR) Applied Research Collaboration (ARC) South West Peninsula (MA, SL). The views expressed in this publication are those of the author(s) and not necessarily those of the National Health Service, the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." +| 17 Is the source of funding stated? | ...the sponsorship of the study was indicated. | ✅ Fully | `Funding`: "This article presents independent research funded by the National Institute for Health Research (NIHR) Applied Research Collaboration (ARC) South West Peninsula (MA, SL). The views expressed in this publication are those of the author(s) and not necessarily those of the National Health Service, the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." | | 18 Are model limitations discussed? | ...limitations of the assessed model, especially limitations of interest to decision makers, were discussed. | ✅ Fully | Limitations are detailed in `Discussion - limitations of the study - dialysis network model`. | : {tbl-colwidths="[20, 25, 15, 40]"} \ No newline at end of file