Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: Cite renv #15

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: Cite renv #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/MainText.Rmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Negative controls will be picked to represent exposure-outcome pairs where no ca

## General

Various standardized analytics available in the OHDSI community will be applied. The Strategus pipeline will be used to call various packages in the HADES library for A) data characterization (A1‑cohort diagnostics, A2‑cohort features, A3‑incidence rates, A4-time-to-event), B) population-level effect estimation (C1--comparative cohort study, C2‑‑self-controlled case-series).[@noauthor_ohdsistrategus_2024; @schuemie_health-analytics_2024] The R package versioning history will be recorded using the renv file and stored in the study Github repository.
Various standardized analytics available in the OHDSI community will be applied. The Strategus pipeline will be used to call various packages in the HADES library for A) data characterization (A1‑cohort diagnostics, A2‑cohort features, A3‑incidence rates, A4-time-to-event), B) population-level effect estimation (C1--comparative cohort study, C2‑‑self-controlled case-series).[@noauthor_ohdsistrategus_2024; @schuemie_health-analytics_2024] The R package versioning history will be recorded using the renv file and stored in the study Github repository.[@ushey_renv_2024]

## Data Characterization

Expand Down
8 changes: 8 additions & 0 deletions docs/Protocol.bib
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -451,3 +451,11 @@ @article{swerdel_phevaluator_2022
keywords = {Algorithms, Databases, Factual, Machine Learning, Phenotype, Phenotype algorithms, Positive predictive value, Reproducibility of Results, Sensitivity, Specificity},
pages = {104177},
}

@Manual{ushey_renv_2024,
title = {renv: Project Environments},
author = {Kevin Ushey and Hadley Wickham},
year = {2024},
note = {R package version 1.0.7, https://github.com/rstudio/renv},
url = {https://rstudio.github.io/renv/},
}
27 changes: 15 additions & 12 deletions docs/protocol.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3321,7 +3321,7 @@ <h2><span class="header-section-number">8.6</span> Negative Controls</h2>
<h1><span class="header-section-number">9</span> Data Analysis Plan</h1>
<div id="general" class="section level2" number="9.1">
<h2><span class="header-section-number">9.1</span> General</h2>
<p>Various standardized analytics available in the OHDSI community will be applied. The Strategus pipeline will be used to call various packages in the HADES library for A) data characterization (A1‑cohort diagnostics, A2‑cohort features, A3‑incidence rates, A4-time-to-event), B) population-level effect estimation (C1–comparative cohort study, C2‑‑self-controlled case-series).<span class="citation">[<a href="#ref-noauthor_ohdsistrategus_2024">15</a>,<a href="#ref-schuemie_health-analytics_2024">16</a>]</span> The R package versioning history will be recorded using the renv file and stored in the study Github repository.</p>
<p>Various standardized analytics available in the OHDSI community will be applied. The Strategus pipeline will be used to call various packages in the HADES library for A) data characterization (A1‑cohort diagnostics, A2‑cohort features, A3‑incidence rates, A4-time-to-event), B) population-level effect estimation (C1–comparative cohort study, C2‑‑self-controlled case-series).<span class="citation">[<a href="#ref-noauthor_ohdsistrategus_2024">15</a>,<a href="#ref-schuemie_health-analytics_2024">16</a>]</span> The R package versioning history will be recorded using the renv file and stored in the study Github repository.<span class="citation">[<a href="#ref-ushey_renv_2024">17</a>]</span></p>
</div>
<div id="data-characterization" class="section level2" number="9.2">
<h2><span class="header-section-number">9.2</span> Data Characterization</h2>
Expand All @@ -3343,14 +3343,14 @@ <h4><span class="header-section-number">9.2.3.1</span> Calculation of time-at-ri
</div>
<div id="time-to-event" class="section level3" number="9.2.4">
<h3><span class="header-section-number">9.2.4</span> Time-to-Event</h3>
<p>Time to the outcomes of NAION or DR worsening (as defined above) will be calculated for each exposure cohort.<span class="citation">[<a href="#ref-reps_design_2018">17</a>]</span></p>
<p>Time to the outcomes of NAION or DR worsening (as defined above) will be calculated for each exposure cohort.<span class="citation">[<a href="#ref-reps_design_2018">18</a>]</span></p>
</div>
</div>
<div id="population-level-effect-estimation" class="section level2" number="9.3">
<h2><span class="header-section-number">9.3</span> Population-Level Effect Estimation</h2>
<div id="comparative-cohort-study" class="section level3" number="9.3.1">
<h3><span class="header-section-number">9.3.1</span> Comparative Cohort Study</h3>
<p>The CohortMethod and Cyclops packages in Hades will be used.14 Large-scale propensity score methods will be used to match the targe exposure cohort with the comparator export cohort (e.g., semaglutide vs empagliflozin) using 1:1 propensity score matching. Given the concern for increasing usage of semaglutide in recent years, we will also perform a sensitivity analysis with calendar year restriction: Dec2017-Jan2020, Feb2020-June2021, July2021-Dec2023. Cox proportional hazards models will be used to estimate the risk of NAION and separately DR progression while on treatment using the intent-to-treat design.<span class="citation">[<a href="#ref-cox_regression_1972">18</a>]</span> Negative controls will be used to assess residual bias.</p>
<p>The CohortMethod and Cyclops packages in Hades will be used.14 Large-scale propensity score methods will be used to match the targe exposure cohort with the comparator export cohort (e.g., semaglutide vs empagliflozin) using 1:1 propensity score matching. Given the concern for increasing usage of semaglutide in recent years, we will also perform a sensitivity analysis with calendar year restriction: Dec2017-Jan2020, Feb2020-June2021, July2021-Dec2023. Cox proportional hazards models will be used to estimate the risk of NAION and separately DR progression while on treatment using the intent-to-treat design.<span class="citation">[<a href="#ref-cox_regression_1972">19</a>]</span> Negative controls will be used to assess residual bias.</p>
</div>
<div id="self-controlled-case-series" class="section level3" number="9.3.2">
<h3><span class="header-section-number">9.3.2</span> Self-Controlled Case Series</h3>
Expand All @@ -3360,8 +3360,8 @@ <h3><span class="header-section-number">9.3.2</span> Self-Controlled Case Series
</div>
<div id="study-diagnostics" class="section level1" number="10">
<h1><span class="header-section-number">10</span> Study Diagnostics</h1>
<p>Residual bias can still remain in observational studies even after PS adjustment is applied to control for measured confounding.<span class="citation">[<a href="#ref-schuemie_empirical_2018">19</a>,<a href="#ref-schuemie_robust_2016">20</a>]</span> To address this, we conduct negative control (falsification) outcome experiments for each comparison and outcome, assuming the null hypothesis of no differential effect (i.e., risk ratio (RR) = 1) is true for each negative control outcome. We identified 96 negative controls using a data-driven algorithm that selects OMOP condition concept occurrences with similar prevalence to the outcomes of interest but lacking evidence of association with exposures in published literature, drug product labels, and spontaneous reports.<span class="citation">[<a href="#ref-voss_accuracy_2017">21</a>]</span> These were then verified through expert review. The list of negative controls is provided below. From these experiments, we derive an empirical null distribution to account for residual study bias. Using this empirical null, we calibrate each RR estimate, its 95% confidence interval (CI), and the p-value to test for the null hypothesis. We consider an RR significantly different from the null if the calibrated p-value is below 0.05, without adjusting for multiple testing.</p>
<p>To ensure the reliability and generalizability of all comparisons, we evaluate study diagnostics while blinded to the results, and only present estimates that successfully pass these diagnostics.<span class="citation">[<a href="#ref-schuemie_how_2020">22</a>,<a href="#ref-schuemie_improving_2018">23</a>]</span> For the primary analysis using the active comparator new-user cohort design, our diagnostics include: (1) preference score distributions between the target and comparator cohorts to evaluate empirical equipoise and population generalizability; (2) cohort balance before and after PS adjustment, defined by the absolute standardized mean differences (SMDs) on extensive patient characteristics for each target-comparator-analysis; (3) negative control calibration plots to assess residual bias, quantified by the Expected Absolute Systematic Error (EASE) derived from the empirical null distribution; and for the primary analysis (Cox PH model), (4) Kaplan-Meier plots to visually examine hazard ratio proportionality assumptions. A study passes diagnostics and contributes to the meta analysis if more than 10% of patients have preference scores between 0.3 and 0.7 on both arms, maximum SMD &lt; 0.1 after PS adjustment, and EASE &lt; 0.25. For the self-controlled case-series analysis, study diagnostics include (1) time trend check that tests for stable background hazards over time periods after PS adjustment, and (2) pre-exposure check to ensure comparative effects in 30 days pre-exposure do not significantly differ from the null to rule out reverse causality; a study passes diagnostics if the p-values for both checks are &gt; 0.05. For both analyses we further evaluate the meta-analytic minimally detectable risk ratio (MDRR) as a proxy of statistical power and only admit a meta-analytic RR estimate if meta-analytic MDRR &lt; 10.</p>
<p>Residual bias can still remain in observational studies even after PS adjustment is applied to control for measured confounding.<span class="citation">[<a href="#ref-schuemie_empirical_2018">20</a>,<a href="#ref-schuemie_robust_2016">21</a>]</span> To address this, we conduct negative control (falsification) outcome experiments for each comparison and outcome, assuming the null hypothesis of no differential effect (i.e., risk ratio (RR) = 1) is true for each negative control outcome. We identified 96 negative controls using a data-driven algorithm that selects OMOP condition concept occurrences with similar prevalence to the outcomes of interest but lacking evidence of association with exposures in published literature, drug product labels, and spontaneous reports.<span class="citation">[<a href="#ref-voss_accuracy_2017">22</a>]</span> These were then verified through expert review. The list of negative controls is provided below. From these experiments, we derive an empirical null distribution to account for residual study bias. Using this empirical null, we calibrate each RR estimate, its 95% confidence interval (CI), and the p-value to test for the null hypothesis. We consider an RR significantly different from the null if the calibrated p-value is below 0.05, without adjusting for multiple testing.</p>
<p>To ensure the reliability and generalizability of all comparisons, we evaluate study diagnostics while blinded to the results, and only present estimates that successfully pass these diagnostics.<span class="citation">[<a href="#ref-schuemie_how_2020">23</a>,<a href="#ref-schuemie_improving_2018">24</a>]</span> For the primary analysis using the active comparator new-user cohort design, our diagnostics include: (1) preference score distributions between the target and comparator cohorts to evaluate empirical equipoise and population generalizability; (2) cohort balance before and after PS adjustment, defined by the absolute standardized mean differences (SMDs) on extensive patient characteristics for each target-comparator-analysis; (3) negative control calibration plots to assess residual bias, quantified by the Expected Absolute Systematic Error (EASE) derived from the empirical null distribution; and for the primary analysis (Cox PH model), (4) Kaplan-Meier plots to visually examine hazard ratio proportionality assumptions. A study passes diagnostics and contributes to the meta analysis if more than 10% of patients have preference scores between 0.3 and 0.7 on both arms, maximum SMD &lt; 0.1 after PS adjustment, and EASE &lt; 0.25. For the self-controlled case-series analysis, study diagnostics include (1) time trend check that tests for stable background hazards over time periods after PS adjustment, and (2) pre-exposure check to ensure comparative effects in 30 days pre-exposure do not significantly differ from the null to rule out reverse causality; a study passes diagnostics if the p-values for both checks are &gt; 0.05. For both analyses we further evaluate the meta-analytic minimally detectable risk ratio (MDRR) as a proxy of statistical power and only admit a meta-analytic RR estimate if meta-analytic MDRR &lt; 10.</p>
<div id="diagnostic-thresholds-for-cohort-method" class="section level2" number="10.1">
<h2><span class="header-section-number">10.1</span> Diagnostic Thresholds for Cohort Method</h2>
<table class="table table-striped" style="margin-left: auto; margin-right: auto;">
Expand Down Expand Up @@ -3529,26 +3529,29 @@ <h1>References</h1>
<div id="ref-schuemie_health-analytics_2024" class="csl-entry">
<div class="csl-left-margin">16 </div><div class="csl-right-inline">Schuemie M, Reps J, Black A, <em>et al.</em> Health-<span>Analytics</span> <span>Data</span> to <span>Evidence</span> <span>Suite</span> (<span>HADES</span>): <span>Open</span>-<span>Source</span> <span>Software</span> for <span>Observational</span> <span>Research</span>. IOS Press 2024. 966–70. doi:<a href="https://doi.org/10.3233/SHTI231108">10.3233/SHTI231108</a></div>
</div>
<div id="ref-ushey_renv_2024" class="csl-entry">
<div class="csl-left-margin">17 </div><div class="csl-right-inline">Ushey K, Wickham H. <em>Renv: Project environments</em>. 2024. <a href="https://rstudio.github.io/renv/">https://rstudio.github.io/renv/</a></div>
</div>
<div id="ref-reps_design_2018" class="csl-entry">
<div class="csl-left-margin">17 </div><div class="csl-right-inline">Reps JM, Schuemie MJ, Suchard MA, <em>et al.</em> Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. <em>Journal of the American Medical Informatics Association: JAMIA</em> 2018;<strong>25</strong>:969–75. doi:<a href="https://doi.org/10.1093/jamia/ocy032">10.1093/jamia/ocy032</a></div>
<div class="csl-left-margin">18 </div><div class="csl-right-inline">Reps JM, Schuemie MJ, Suchard MA, <em>et al.</em> Design and implementation of a standardized framework to generate and evaluate patient-level prediction models using observational healthcare data. <em>Journal of the American Medical Informatics Association: JAMIA</em> 2018;<strong>25</strong>:969–75. doi:<a href="https://doi.org/10.1093/jamia/ocy032">10.1093/jamia/ocy032</a></div>
</div>
<div id="ref-cox_regression_1972" class="csl-entry">
<div class="csl-left-margin">18 </div><div class="csl-right-inline">Cox DR. Regression <span>Models</span> and <span>Life</span>-<span>Tables</span>. <em>Journal of the Royal Statistical Society: Series B (Methodological)</em> 1972;<strong>34</strong>:187–202. doi:<a href="https://doi.org/10.1111/j.2517-6161.1972.tb00899.x">10.1111/j.2517-6161.1972.tb00899.x</a></div>
<div class="csl-left-margin">19 </div><div class="csl-right-inline">Cox DR. Regression <span>Models</span> and <span>Life</span>-<span>Tables</span>. <em>Journal of the Royal Statistical Society: Series B (Methodological)</em> 1972;<strong>34</strong>:187–202. doi:<a href="https://doi.org/10.1111/j.2517-6161.1972.tb00899.x">10.1111/j.2517-6161.1972.tb00899.x</a></div>
</div>
<div id="ref-schuemie_empirical_2018" class="csl-entry">
<div class="csl-left-margin">19 </div><div class="csl-right-inline">Schuemie MJ, Hripcsak G, Ryan PB, <em>et al.</em> Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data. <em>Proceedings of the National Academy of Sciences of the United States of America</em> 2018;<strong>115</strong>:2571–7. doi:<a href="https://doi.org/10.1073/pnas.1708282114">10.1073/pnas.1708282114</a></div>
<div class="csl-left-margin">20 </div><div class="csl-right-inline">Schuemie MJ, Hripcsak G, Ryan PB, <em>et al.</em> Empirical confidence interval calibration for population-level effect estimation studies in observational healthcare data. <em>Proceedings of the National Academy of Sciences of the United States of America</em> 2018;<strong>115</strong>:2571–7. doi:<a href="https://doi.org/10.1073/pnas.1708282114">10.1073/pnas.1708282114</a></div>
</div>
<div id="ref-schuemie_robust_2016" class="csl-entry">
<div class="csl-left-margin">20 </div><div class="csl-right-inline">Schuemie MJ, Hripcsak G, Ryan PB, <em>et al.</em> Robust empirical calibration of p‐values using observational data. <em>Statistics in Medicine</em> 2016;<strong>35</strong>:3883–8. doi:<a href="https://doi.org/10.1002/sim.6977">10.1002/sim.6977</a></div>
<div class="csl-left-margin">21 </div><div class="csl-right-inline">Schuemie MJ, Hripcsak G, Ryan PB, <em>et al.</em> Robust empirical calibration of p‐values using observational data. <em>Statistics in Medicine</em> 2016;<strong>35</strong>:3883–8. doi:<a href="https://doi.org/10.1002/sim.6977">10.1002/sim.6977</a></div>
</div>
<div id="ref-voss_accuracy_2017" class="csl-entry">
<div class="csl-left-margin">21 </div><div class="csl-right-inline">Voss EA, Boyce RD, Ryan PB, <em>et al.</em> Accuracy of an automated knowledge base for identifying drug adverse reactions. <em>Journal of Biomedical Informatics</em> 2017;<strong>66</strong>:72–81. doi:<a href="https://doi.org/10.1016/j.jbi.2016.12.005">10.1016/j.jbi.2016.12.005</a></div>
<div class="csl-left-margin">22 </div><div class="csl-right-inline">Voss EA, Boyce RD, Ryan PB, <em>et al.</em> Accuracy of an automated knowledge base for identifying drug adverse reactions. <em>Journal of Biomedical Informatics</em> 2017;<strong>66</strong>:72–81. doi:<a href="https://doi.org/10.1016/j.jbi.2016.12.005">10.1016/j.jbi.2016.12.005</a></div>
</div>
<div id="ref-schuemie_how_2020" class="csl-entry">
<div class="csl-left-margin">22 </div><div class="csl-right-inline">Schuemie MJ, Cepeda MS, Suchard MA, <em>et al.</em> How <span>Confident</span> <span>Are</span> <span>We</span> about <span>Observational</span> <span>Findings</span> in <span>Healthcare</span>: <span>A</span> <span>Benchmark</span> <span>Study</span>. <em>Harvard Data Science Review</em> 2020;<strong>2</strong>. doi:<a href="https://doi.org/10.1162/99608f92.147cc28e">10.1162/99608f92.147cc28e</a></div>
<div class="csl-left-margin">23 </div><div class="csl-right-inline">Schuemie MJ, Cepeda MS, Suchard MA, <em>et al.</em> How <span>Confident</span> <span>Are</span> <span>We</span> about <span>Observational</span> <span>Findings</span> in <span>Healthcare</span>: <span>A</span> <span>Benchmark</span> <span>Study</span>. <em>Harvard Data Science Review</em> 2020;<strong>2</strong>. doi:<a href="https://doi.org/10.1162/99608f92.147cc28e">10.1162/99608f92.147cc28e</a></div>
</div>
<div id="ref-schuemie_improving_2018" class="csl-entry">
<div class="csl-left-margin">23 </div><div class="csl-right-inline">Schuemie MJ, Ryan PB, Hripcsak G, <em>et al.</em> Improving reproducibility by using high-throughput observational studies with empirical calibration. <em>Philosophical Transactions Series A, Mathematical, Physical, and Engineering Sciences</em> 2018;<strong>376</strong>:20170356. doi:<a href="https://doi.org/10.1098/rsta.2017.0356">10.1098/rsta.2017.0356</a></div>
<div class="csl-left-margin">24 </div><div class="csl-right-inline">Schuemie MJ, Ryan PB, Hripcsak G, <em>et al.</em> Improving reproducibility by using high-throughput observational studies with empirical calibration. <em>Philosophical Transactions Series A, Mathematical, Physical, and Engineering Sciences</em> 2018;<strong>376</strong>:20170356. doi:<a href="https://doi.org/10.1098/rsta.2017.0356">10.1098/rsta.2017.0356</a></div>
</div>
</div>
</div>
Expand Down