The R package ‘DQAstats’ provides core functionalities to perform data quality assessment (DQA) of electronic health record data (EHR).
Currently implemented features are:
- descriptive (univariate) analysis of categorical and continuous variables of a source database and a target database
- checks of the extract-transform-load (ETL) pipeline:
- comparing of distinct values and valid values between the source database and the target database
- identification of missing/duplicate Data based on a comparison of timestamps
- value conformance checks by comparing the resulting statistics to value constraints (given in a meta data repository (MDR))
- ‘atemporal plausibility’ checks (multivariate)
- ‘uniqueness plausibility’ checks (multivariate)
The tool provides one main function, dqa()
, to create a comprehensive
PDF document, which presents all statistics and results of the data
quality assessment.
Currently supported input data formats / databases:
- CSV files (via R package
data.table
) - PostgreSQL (via R package
RPostgres
) - ORACLE (via R package
RJDBC
)
DQAstats
can be installed directly from CRAN with:
install.packages("DQAstats")
You can install the latest development version of DQAstats
with:
install.packages("remotes")
remotes::install_github("miracum/dqa-dqastats")
Note: A working LaTeX installation is a prerequisite for using this
software (e.g. using the R package
tinytex
)!
💡 If you want to run this in a dockerized environment you can use
the rocker/verse
image which
has TeX already installed.
The configuration of databases, be it CSV files or SQL-based databases,
is done with environment variables, which can be set using the base R
command Sys.setenv()
.
A detailed description, which environment variables need to be set for the specific databases can be found here.
The following code example is intended to provide a minimal working
example on how to apply the DQA tool to data. Example data and a
corresponding MDR are provided with the R package DQAstats (a working
LaTeX installation is a prerequisite for using this software, e.g. by
using the R package tinytex
; please
refer to the DQAstats
wiki for
further installation instructions).
- Example data: https://github.com/miracum/dqa-dqastats/tree/master/inst/demo_data
- Example MDR: https://github.com/miracum/dqa-dqastats/blob/master/inst/demo_data/utilities/MDR/mdr_example_data.csv
# Load library DQAstats:
library(DQAstats)
# Set environment vars to demo files paths:
Sys.setenv("EXAMPLECSV_SOURCE_PATH" = system.file("demo_data",
package = "DQAstats"))
Sys.setenv("EXAMPLECSV_TARGET_PATH" = system.file("demo_data",
package = "DQAstats"))
# Set path to utilities folder where to find the mdr and template files:
utils_path <- system.file("demo_data/utilities",
package = "DQAstats")
# Execute the DQA and generate a PDF report:
results <- DQAstats::dqa(
source_system_name = "exampleCSV_source",
target_system_name = "exampleCSV_target",
utils_path = utils_path,
mdr_filename = "mdr_example_data.csv",
output_dir = "output/",
parallel = FALSE
)
# The PDF report is stored at "./output/"
You can test the package without needing to install anything except docker. 💡 For further details, see the Wiki: https://github.com/miracum/dqa-dqastats/wiki/Deployment.
L.A. Kapsner, J.M. Mang, S. Mate, S.A. Seuchter, A. Vengadeswaran, F. Bathelt, N. Deppenwiese, D. Kadioglu, D. Kraska, and H.-U. Prokosch, Linking a Consortium-Wide Data Quality Assessment Tool with the MIRACUM Metadata Repository, Appl Clin Inform. 12 (2021) 826–835. doi:10.1055/s-0041-1733847.
@article{kapsner2021,
title = {Linking a {{Consortium}}-{{Wide Data Quality Assessment Tool}} with the {{MIRACUM Metadata Repository}}},
author = {Kapsner, Lorenz A. and Mang, Jonathan M. and Mate, Sebastian and Seuchter, Susanne A. and Vengadeswaran, Abishaa and Bathelt, Franziska and Deppenwiese, Noemi and Kadioglu, Dennis and Kraska, Detlef and Prokosch, Hans-Ulrich},
year = {2021},
month = aug,
journal = {Applied Clinical Informatics},
volume = {12},
number = {04},
pages = {826--835},
issn = {1869-0327},
doi = {10.1055/s-0041-1733847},
language = {en}
}
- about MIRACUM: https://www.miracum.org/
- about the Medical Informatics Initiative: https://www.medizininformatik-initiative.de/index.php/de
- the precursor’s version of the MIRACUM DQA tool (publication): https://gitlab.miracum.org/miracum/dqa/dqa-p21-i2b2