Skip to content

Files

Latest commit

baf8d9a · Oct 15, 2024

History

History
30 lines (21 loc) · 2.75 KB

README.md

File metadata and controls

30 lines (21 loc) · 2.75 KB

Translator Answer Appraiser

The Answer Appraiser is a micro-service that attaches a Confidence, Clinical Information, and Novelty score to each result of a given TRAPI message.

Scoring Metrics

  • The Confidence Score is currently a product of ARA scores that gives a strong weight to any ARA that scores a result very high.

  • The Clinical Information Score is returned for queries where the results are drug-chemical pairs. That factor takes into account the weighted average odds ratio of an observed association between a drug (query result) and a disease outcome (query) calculated across 3 independent knowledge providers.

  • The Novelty Score captures many dimensions and holds the strong potential to differentiate results for researchers, beyond the use of a single and more conventional summarization based on published literature. These dimensions are contributed by a microservice termed the Annotator and include:

    • Publication recency. There is a tradeoff in weighting highly cited earlier publications, versus newer publications that represent the latest novel findings, even if they have not had a chance to be cited widely yet. The recency factor evaluates the amount and year publications into a combined sigmoid function. A recency factor near 1 denotes a query result that is supported by a very recent publication.
    • FDA Approval. Drugs that are in a late phase (IV) of a US clinical trial are down weighted while those in earlier phases (I, II, III) get a greater novelty score. Predicted drugs that are not in an FDA approval status are not scored and the approval status is specific to the disease being queried.
    • MolDist (Molecular Distinctiveness). MolDist is calculated using the Jaccard coefficient between compounds shown in results and more established compounds. Structurally identical compounds are scored as 1.
    • TDL (Gene Target Development Levels). Over 80% of the genes named in the results have a TDL: TClin, TChem, TBio or TDark, in decreasing levels of study knowledge of that gene (and thereby increasing novelty).
    • Gene distinctiveness. This metric denotes the number of pathways that a gene belongs to.

    The Novelty score calculation starts with recency. Then if the result is a chemical, MolSim and FDA approval may up or down-regulate the score. Conversely, if the result is a gene, then the TDL and gene distinctiveness may up or down-regulate the score.

Development Setup

Answer Appraiser can be run locally through either python or Docker.

python

Note: redis is also required and is easily stood up in docker.

  1. Create and activate a virtual python environment (python3.12 -m venv <path_to_venv>, source <path_to_venv>)
  2. pip install -r requirements-lock.txt
  3. python run.py

docker

  1. docker compose up