Skip to content

Commit

Permalink
added table of performance metrics
Browse files Browse the repository at this point in the history
  • Loading branch information
ntatonetti committed Nov 3, 2022
1 parent bc05ff2 commit 19f8346
Show file tree
Hide file tree
Showing 6 changed files with 88 additions and 96 deletions.
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,14 @@ A resource of adverse drug effects extracted from FDA structured product labels.

Second release of the OnSIDES database of adverse reactions and boxed warnings extracted from the FDA structured product labels (SPLs). This version contains significant model improvements as well as updated labels. All labels available to download from DailyMed (https://dailymed.nlm.nih.gov/dailymed/spl-resources-all-drug-labels.cfm) as of November 2, 2022 were processed in this analysis. In total XXX million adverse reactions were extracted from XX,000 labels for just under X,000 drug ingredients or combination of ingredients.

OnSIDES was created using the PubMedBERT language model and 200 manually curated labels available from [Denmer-Fushman et al.](https://pubmed.ncbi.nlm.nih.gov/29381145/). The model achieves an F1 score of 0.90, AUROC of 0.92, and AUPR of 0.94 at extracting effects from the ADVERSE REACTIONS section of the label. This constitutes an absolute increase of 4% in each of the performance metrics over V01. For the BOXED WARNINGS section, the model achieves a F1 score of 0.76, AUROC of 0.83, and AUPR of 0.77. This constitutes an absolute increase of 10-17% in the performance metrics over V01. Compared against the TAC reference standard using the official evaluation script the model achieves a F1 score of 0.85.
OnSIDES was created using the [PubMedBERT language model](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract) and 200 manually curated labels available from [Denmer-Fushman et al.](https://pubmed.ncbi.nlm.nih.gov/29381145/). The model achieves an F1 score of 0.90, AUROC of 0.92, and AUPR of 0.94 at extracting effects from the ADVERSE REACTIONS section of the label. This constitutes an absolute increase of 4% in each of the performance metrics over V01. For the BOXED WARNINGS section, the model achieves a F1 score of 0.76, AUROC of 0.83, and AUPR of 0.77. This constitutes an absolute increase of 10-17% in the performance metrics over V01. Compared against the TAC reference standard using the official evaluation script the model achieves a F1 score of 0.85.

| Metric | TAC (Best Model) | SIDER | OnSIDES V01 | OnSIDES V02 |
| ----------- | ---------------- | ----- | ----------- | ----------- |
| F1 Score | 82.19 | 74.36 | 82.01 | 87.67 |
| Precision | 80.69 | 43.49 | 88.76 | 93.65 |
| Recall | 85.05 | 52.89 | 77.12 | 82.40 |
*Performance metrics in table are evaluated on the TAC gold standard test set.*

### Download

Expand Down
Binary file modified figures/Experiment10E-bestepoch-summary-stats.pdf
Binary file not shown.
Binary file modified figures/Experiment10E-bestepoch.pdf
Binary file not shown.
Binary file modified figures/Experiment10E-final-summary-stats.pdf
Binary file not shown.
Binary file modified figures/Experiment10E-final.pdf
Binary file not shown.

Large diffs are not rendered by default.

0 comments on commit 19f8346

Please sign in to comment.