Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate example results for "creative/inferred" mode for relay #453

Closed
andrewsu opened this issue May 31, 2022 · 15 comments
Closed

generate example results for "creative/inferred" mode for relay #453

andrewsu opened this issue May 31, 2022 · 15 comments
Assignees

Comments

@andrewsu
Copy link
Member

There is an issue to track the implementation of a "creative/inferred" mode. At the upcoming relay, there is a session on "creative mode" on Monday and Tuesday in which we would like to explore some preliminary results. Organizers would like us to come with some example output for two queries from the list of example queries. Specifically, organizers would like to see results for "Alzheimer disease MONDO:0004975" and "neurofibromatosis, familial spinal MONDO:0008078". In advance of our full technical implementation, we can show up with simply a list of ARAX links (each corresponding to one of the query templates in https://github.com/Carolina1396/drug_mechanisms_rare_diseases_BioThingsExplorer/tree/main/src/query_templates).

This is meant to be a very rough approximation of what will come out later, just so we have something to discuss at the relay. So if the plan described above needs to be modified for the sake of easy feasibility, no problem... Initially assigning to @colleenXu.

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 1, 2022

Posting from Slack, David K. from Translator's ask:

in preparation for the Relay next week, please have creative answers to “what drugs may treat disease X?” available to share for the following diseases: MONDO:0004975 (Alzheimers), MONDO:0008078 (neurofibromatosis, familial spinal).
At this point, some teams’ answers require manual intervention, but if able, please attempt as many as you can in this list of diseases provided by the UI team.
Optimally, the answers will be in TRAPI form, but links to ARAX-displayed results (eg. if you ran an SME developed workflow), graphical representations of results, etc. are also acceptable.
Please recall that the goal is to have all ARAs responsive to the creative query “what drugs may treat disease X?” in a fully automated fashion.

EDIT: there are 9 diseases/phenotypes posted below as of 6/6/22 8:49PM PDT.
6 are common, 3 are rare:

Common:

Rare:

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 6, 2022

Alzheimers (MONDO:0004975)

Initial query

Assuming the query given will be...

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

Other methods that didn't return results:

  • Method 2: Took ~ 2 min 35 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 5030 entities).
  • Method 3: Took ~ 8 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eB (dev: 2477 entities).
  • Method 4: Took ~ 1 min 9-42 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eB (dev: 4963 entities).
  • Method 6: Took ~ 2 min 50 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 5133 entities).

Note! If biolink version is updated, we may need to change the predicates used in the methods below.


"Straightforward method"

  • Change "biolink:Drug" -> "biolink:ChemicalEntity", and remove the "knowledge_type: inferred" (so the "normal" behavior will happen)
  • Like a modified version of this template
"straightforward" query
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                },
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               }
            },
            "edges": {
                "eA": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"]
                }
            }
        }
    }
}

Method 1: Disease X -(has_phenotype)-> PhenotypicFeature -(treated_by)-> ChemicalEntity

"method 1" query
{
    "message": {
        "query_graph": {
            "nodes": {
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               },
                "nA": {
                    "categories":["biolink:PhenotypicFeature"]
                },
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "eA": {
                    "subject": "disease",
                    "object": "nA",
                    "predicates": ["biolink:has_phenotype"]
                },
                "eB": {
                    "subject": "nA",
                    "object": "drug",
                    "predicates": ["biolink:treated_by"]
                }
            }
        }
    }
}

Method 2: Disease X -(superclass_of)-> Disease --> Gene -(affected_by)-> ChemicalEntity

  • Modified from this template. Note the reversed predicate
"method 2" query
{
    "message": {
        "query_graph": {
            "nodes": {
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               },
                "nA": {
                    "categories":["biolink:Disease"]
                },
                "nB": {
                    "categories":["biolink:Gene"]
                },
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "eA": {
                    "subject": "disease",
                    "object": "nA",
                    "predicates": ["biolink:superclass_of"]
                },
                "eB": {
                    "subject": "nA",
                    "object": "nB"
                },
                "eC": {
                    "subject": "nB",
                    "object": "drug",
                    "predicates": ["biolink:affected_by"]
                }
            }
        }
    }
}

Method 3: Disease X --> SequenceVariant --> Gene -(affected_by)-> ChemicalEntity

"method 3" query
{
    "message": {
        "query_graph": {
            "nodes": {
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               },
                "nA": {
                    "categories":["biolink:SequenceVariant"]
                },
                "nB": {
                    "categories":["biolink:Gene"]
                },
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "eA": {
                    "subject": "disease",
                    "object": "nA"
                },
                "eB": {
                    "subject": "nA",
                    "object": "nB"
                },
                "eC": {
                    "subject": "nB",
                    "object": "drug",
                    "predicates": ["biolink:affected_by"]
                }
            }
        }
    }
}

Method 4: Disease X --> Gene -(affected_by)-> ChemicalEntity

"method 4" query
{
    "message": {
        "query_graph": {
            "nodes": {
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               },
                "nA": {
                    "categories":["biolink:Gene"]
                },
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "eA": {
                    "subject": "disease",
                    "object": "nA"
                },
                "eB": {
                    "subject": "nA",
                    "object": "drug",
                    "predicates": ["biolink:affected_by"]
                }
            }
        }
    }
}

Method 5: Disease X -(has_phenotype)-> PhenotypicFeature -(affected_by)-> Gene -(affected_by)-> ChemicalEntity

"method 5" query
{
    "message": {
        "query_graph": {
            "nodes": {
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               },
                "nA": {
                    "categories":["biolink:PhenotypicFeature"]
                },
                "nB": {
                    "categories":["biolink:Gene"]
                },
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "eA": {
                    "subject": "disease",
                    "object": "nA",
                    "predicates": ["biolink:has_phenotype"]
                },
                "eB": {
                    "subject": "nA",
                    "object": "nB",
                    "predicates": ["biolink:affected_by"]
                },
                "eC": {
                    "subject": "nB",
                    "object": "drug",
                    "predicates": ["biolink:affected_by"]
                }
            }
        }
    }
}

Method 6: Disease X -(caused_by)-> Gene -(affected_by)-> Gene -(affected_by)-> ChemicalEntity

  • Modified from this template. Note that another template has the same structure, but removes the predicate caused_by from e1...
  • Note that affected_by includes entity_regulated_by_entity in the biolink version 2.2.13 we're on
"method 6" query
{
    "message": {
        "query_graph": {
            "nodes": {
                "disease": {
                    "ids":["MONDO:0004975"],
                    "categories":["biolink:Disease"],
                    "name": "Alzheimers"
               },
                "nA": {
                    "categories":["biolink:Gene"]
                },
                "nB": {
                    "categories":["biolink:Gene"]
                },
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "eA": {
                    "subject": "disease",
                    "object": "nA",
                    "predicates": ["biolink:caused_by"]
                },
                "eB": {
                    "subject": "nA",
                    "object": "nB",
                    "predicates": ["biolink:affected_by"]
                },
                "eC": {
                    "subject": "nB",
                    "object": "drug",
                    "predicates": ["biolink:affected_by"]
                }
            }
        }
    }
}

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 6, 2022

neurofibromatosis, familial spinal (MONDO:0008078)

Initial query

Assuming the query given will be...

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["MONDO:0008078"],
                    "categories":["biolink:Disease"],
                    "name": "neurofibromatosis, familial spinal"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

Other methods that didn't return results:

  • Straightforward method: took < 3 sec, BTE didn't find any records for the QEdge, returns 0 results (TRAPI Response JSON)
  • Method 5: had a series of errors happen and didn't return a response, ran for ~71 min. See below.

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

                    "ids":["MONDO:0008078"],
                    "categories":["biolink:Disease"],
                    "name": "neurofibromatosis, familial spinal"

Method 5 issues: walking through the logs

Used the older format for the query (n0-n3, e1-e3)

old method 5 query
{
    "message": {
        "query_graph": {
            "nodes": {
                "n0": {
                    "ids":["MONDO:0008078"],
                    "categories":["biolink:Disease"],
                    "name": "neurofibromatosis, familial spinal"
               },
                "n1": {
                    "categories":["biolink:PhenotypicFeature"]
                },
                "n2": {
                    "categories":["biolink:Gene"]
                },
                "n3": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "e1": {
                    "subject": "n0",
                    "object": "n1",
                    "predicates": ["biolink:has_phenotype"]
                },
                "e2": {
                    "subject": "n1",
                    "object": "n2",
                    "predicates": ["biolink:affected_by"]
                },
                "e3": {
                    "subject": "n2",
                    "object": "n3",
                    "predicates": ["biolink:affected_by"]
                }
            }
        }
    }
}

First two hops only take ~ 1 min

logs before the last hop
bte:biothings-explorer-trapi:NewQNode Node "n1" kept (63) curies... +10ms 
bte:biothings-explorer-trapi:main (10) Edge successfully queried. +19s 
bte:biothings-explorer-trapi:edge-manager (4) Edges not yet executed = 1 +1s 
bte:biothings-explorer-trapi:edge-manager (5) Sending next edge 'e3' WITH entity count...(979) +1ms 
bte:biothings-explorer-trapi:edge-manager Checking entity max : (0)--(979) +0ms 
bte:biothings-explorer-trapi:edge-manager 'e1' : (1) --> (63) +0ms 
bte:biothings-explorer-trapi:edge-manager 'e2' : (63) --> (979) +0ms 
bte:biothings-explorer-trapi:edge-manager 'e3' : (979) --> (0) +0ms 
bte:biothings-explorer-trapi:main (5) Executing current edge >> "e3" +1ms 
bte:biothings-explorer-trapi:batch_edge_query Node Update Start +3s 
  • Final hop took ~ 20 min to execute sub-querying, then ~2-3 min to run/parse Node-Normalizer (ID resolution of output IDs)
  • final edge-managing step takes~43 min to run, partially because the final QNode n3 "intersection" steps of edge-managing take ~5-7 min each time...and it seems to run 6 times.
  • in the logs below, the job is checked every minute which is what the worker threads are doing
related logs
  bte:biothings-explorer-trapi:worker41 Worker thread 41 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 41 completed task, terminated successfully. +1m
  bte:call-apis:query id annotation completes +3m
  bte:call-apis:query qEdge queries complete in 1167s +0ms
  bte:biothings-explorer-trapi:batch_edge_query APIEdges are successfully queried.... +22m
  bte:biothings-explorer-trapi:batch_edge_query Filtering out any "undefined" items in (370970) records +0ms
  bte:biothings-explorer-trapi:batch_edge_query Total number of records is (370970) +12ms
  bte:biothings-explorer-trapi:batch_edge_query Start to update nodes... +2ms
  bte:biothings-explorer-trapi:batch_edge_query Update nodes completed! +78ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +22m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 370970 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture","ChemicalMixture"] +5s
  bte:biothings-explorer-trapi:NewQNode Node "n3" saving (18550) curies... +23m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +10ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" intersecting (979)/(967) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" kept (946) curies... +1s
  bte:biothings-explorer-trapi:edge-manager 'e3' Reversed[false] (946)--(18550) entities / (370970) records. +23m
  bte:biothings-explorer-trapi:worker42 Worker thread 42 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 42 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker43 Worker thread 43 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 43 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker44 Worker thread 44 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 44 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker45 Worker thread 45 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 45 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker46 Worker thread 46 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 46 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker47 Worker thread 47 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 47 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker48 Worker thread 48 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 48 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker49 Worker thread 49 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 49 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:edge-manager 'e3' dropped (2020) records. +8m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +8m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 368950 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture","ChemicalMixture"] +5s
  bte:biothings-explorer-trapi:NewQNode Node "n3" intersecting (18550)/(18445) curies... +8m
  bte:biothings-explorer-trapi:worker50 Worker thread 50 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 50 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker51 Worker thread 51 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 51 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker52 Worker thread 52 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 52 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker53 Worker thread 53 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 53 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker54 Worker thread 54 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 54 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker55 Worker thread 55 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 55 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker56 Worker thread 56 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 56 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:NewQNode Node "n3" kept (18445) curies... +7m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +7m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" intersecting (946)/(967) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" kept (946) curies... +1s
  bte:biothings-explorer-trapi:edge-manager Updating all other edges... +7m
  bte:biothings-explorer-trapi:edge-manager Updating "e1"... +0ms
  bte:biothings-explorer-trapi:edge-manager 'e1' Reversed[false] (1)--(63) entities / (64) records. +0ms
  bte:biothings-explorer-trapi:edge-manager 'e1' dropped (0) records. +2ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +1s
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 64 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e1" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["BehavioralFeature","ClinicalFinding","PhenotypicFeature"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["PhenotypicFeature"] +1ms
  bte:biothings-explorer-trapi:NewQNode Node "n1" intersecting (63)/(63) curies... +3ms
  bte:biothings-explorer-trapi:NewQNode Node "n1" kept (63) curies... +10ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e1" +10ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Disease"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Disease"] +0ms
  bte:biothings-explorer-trapi:NewQNode Node "n0" intersecting (1)/(49) curies... +0ms
  bte:biothings-explorer-trapi:NewQNode Node "n0" kept (1) curies... +0ms
  bte:biothings-explorer-trapi:edge-manager 'e3' Reversed[false] (946)--(18445) entities / (368950) records. +11ms
  bte:biothings-explorer-trapi:worker57 Worker thread 57 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 57 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker58 Worker thread 58 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 58 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker59 Worker thread 59 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 59 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker60 Worker thread 60 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 60 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker61 Worker thread 61 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 61 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker62 Worker thread 62 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 62 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker63 Worker thread 63 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 63 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:edge-manager 'e3' dropped (0) records. +8m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +8m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 368950 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +1ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture","ChemicalMixture"] +5s
  bte:biothings-explorer-trapi:NewQNode Node "n3" intersecting (18445)/(18445) curies... +8m
  bte:biothings-explorer-trapi:worker64 Worker thread 64 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 64 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker65 Worker thread 65 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 65 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker66 Worker thread 66 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 66 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker67 Worker thread 67 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 67 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker68 Worker thread 68 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 68 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker69 Worker thread 69 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 69 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker70 Worker thread 70 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 70 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:NewQNode Node "n3" kept (18445) curies... +7m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +7m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" intersecting (946)/(967) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" kept (946) curies... +993ms
  bte:biothings-explorer-trapi:edge-manager Updating "e2"... +7m
  bte:biothings-explorer-trapi:edge-manager 'e2' Reversed[false] (63)--(946) entities / (2908) records. +0ms
  bte:biothings-explorer-trapi:edge-manager 'e2' dropped (35) records. +218ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +1s
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 2873 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e2" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +20ms
  bte:biothings-explorer-trapi:NewQNode Node "n2" intersecting (946)/(946) curies... +238ms
  bte:biothings-explorer-trapi:NewQNode Node "n2" kept (946) curies... +998ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e2" +998ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["BehavioralFeature","ClinicalFinding","PhenotypicFeature"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["PhenotypicFeature"] +27ms
  bte:biothings-explorer-trapi:NewQNode Node "n1" intersecting (63)/(61) curies... +28ms
  bte:biothings-explorer-trapi:NewQNode Node "n1" kept (61) curies... +9ms
  bte:biothings-explorer-trapi:edge-manager 'e3' Reversed[false] (946)--(18445) entities / (368950) records. +1s
  bte:biothings-explorer-trapi:worker71 Worker thread 71 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 71 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker72 Worker thread 72 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 72 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker73 Worker thread 73 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 73 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker74 Worker thread 74 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 74 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker75 Worker thread 75 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 75 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker76 Worker thread 76 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 76 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker77 Worker thread 77 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 77 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker78 Worker thread 78 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 78 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:edge-manager 'e3' dropped (0) records. +8m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +8m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 368950 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +1ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture","ChemicalMixture"] +5s
  bte:biothings-explorer-trapi:NewQNode Node "n3" intersecting (18445)/(18445) curies... +8m
  bte:biothings-explorer-trapi:worker79 Worker thread 79 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 79 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker80 Worker thread 80 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 80 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker81 Worker thread 81 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 81 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker82 Worker thread 82 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 82 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker83 Worker thread 83 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 83 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker84 Worker thread 84 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 84 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:NewQNode Node "n3" kept (18445) curies... +6m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "e3" +6m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" intersecting (946)/(967) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "n2" kept (946) curies... +986ms
  bte:biothings-explorer-trapi:main (10) Edge successfully queried. +1h
logs showing how large the size is now
  bte:biothings-explorer-trapi:edge-manager (11) Collecting records... +6m
  bte:biothings-explorer-trapi:edge-manager (11) 'e1' keeps (64) records! +0ms
  bte:biothings-explorer-trapi:edge-manager ---------- +0ms
  bte:biothings-explorer-trapi:edge-manager (11) 'e2' keeps (2873) records! +0ms
  bte:biothings-explorer-trapi:edge-manager ---------- +0ms
  bte:biothings-explorer-trapi:edge-manager (11) 'e3' keeps (368950) records! +12ms
  bte:biothings-explorer-trapi:edge-manager ---------- +0ms
  bte:biothings-explorer-trapi:edge-manager (12) Collected records for: ["e1","e2","e3"]! +0ms
  bte:biothings-explorer-trapi:edge-manager (12) Collected (371887) records! +0ms
  bte:biothings-explorer-trapi:edge-manager (13) Edge Manager reporting combined records... +0ms
  bte:biothings-explorer-trapi:Graph Updating BTE Graph now. +1h
  bte:biothings-explorer-trapi:edge-manager (13) Edge Manager reporting organized records... +4s
  bte:biothings-explorer-trapi:QueryResult Updating query results now! +1h
ERROR 1: SCORING FAILS
  bte:biothings-explorer-trapi:Score Querying 774910 combos. +1h
  bte:biothings-explorer-trapi:Score Failed to query for scores:  Error: Request failed with status code 500
    at createError (/Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/axios/lib/adapters/http.js:269:11)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  config: {
    url: 'http://biothings.ncats.io/semmeddb/query/ngd',
    method: 'post',
    data: '{"umls":[["C3887682","C0030233"],["C3887682","C0006931"],

and the query is super long and then these logs look relevant

  response: {
    status: 500,
    statusText: 'Internal Server Error',
    headers: {
      server: 'nginx/1.14.0 (Ubuntu)',
      date: 'Mon, 06 Jun 2022 10:37:31 GMT',
      'content-type': 'text/html',
      'content-length': '202',
      connection: 'close'
    },

    data: '<html>\r\n' +
      '<head><title>500 Internal Server Error</title></head>\r\n' +
      '<body bgcolor="white">\r\n' +
      '<center><h1>500 Internal Server Error</h1></center>\r\n' +
      '<hr><center>nginx/1.14.0 (Ubuntu)</center>\r\n' +
      '</body>\r\n' +
      '</html>\r\n'
ERROR 2: can't return the response due to string issue? looks like a TON of results too (326849) and took ~ 2 min to generate the logs during results-assembly
  bte:biothings-explorer-trapi:QueryResult Successfully scored 0 results, couldn't score 326849 results. +787ms
  bte:biothings-explorer-trapi:Graph pruning BTEGraph nodes/edges... +3m
  bte:biothings-explorer-trapi:Graph pruned 2 nodes and 2 edges from BTEGraph. +908ms
  bte:biothings-explorer-trapi:main (14) TRAPI query finished. +3m
  bte:biothings-explorer-trapi:worker88 Worker thread 88 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job RBjIxHQsHz +0ms
  bte:biothings-explorer-trapi:threading Worker thread 88 completed task, terminated successfully. +1m
RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at /Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/asyncquery.js:61:46
    at Array.map (<anonymous>)
    at storeQueryResponse (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/asyncquery.js:57:52)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async exports.asyncqueryResponse (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/asyncquery.js:132:13)
    at async jobToBeDone (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/processors/async_v1.js:17:20)
    at async module.exports (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/processors/async_v1.js:22:10)
    at async /Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/bull/lib/process/master.js:92:27

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 6, 2022

noonan syndrome (MONDO:0018997)

Initial query

Assuming the query given will be...

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["MONDO:0018997"],
                    "categories":["biolink:Disease"],
                    "name": "Noonan syndrome"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

Other methods that didn't return results:

  • Method 6: had a series of errors happen and didn't return a response, ran for ~47 min. See below.

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

                    "ids":["MONDO:0018997"],
                    "categories":["biolink:Disease"],
                    "name": "Noonan syndrome"

Method 6 issues: walking through the logs

Very similar to what happened above with neurofibromatosis in the above post

query
method_6 = {
    "message": {
        "query_graph": {
            "nodes": {
                "disease": {
                    "ids":["MONDO:0018997"],
                    "categories":["biolink:Disease"],
                    "name": "Noonan syndrome"
               },
                "nA": {
                    "categories":["biolink:Gene"]
                },
                "nB": {
                    "categories":["biolink:Gene"]
                },
                "drug": {
                    "categories":["biolink:ChemicalEntity"]
                }
            },
            "edges": {
                "eA": {
                    "subject": "disease",
                    "object": "nA",
                    "predicates": ["biolink:caused_by"]
                },
                "eB": {
                    "subject": "nA",
                    "object": "nB",
                    "predicates": ["biolink:affected_by"]
                },
                "eC": {
                    "subject": "nB",
                    "object": "drug",
                    "predicates": ["biolink:affected_by"]
                }
            }
        }
    }
}

First two hops only take ~ 1 min

logs before the last hop
  bte:biothings-explorer-trapi:main (10) Edge successfully queried. +16s
  bte:biothings-explorer-trapi:edge-manager (4) Edges not yet executed = 1 +775ms
  bte:biothings-explorer-trapi:edge-manager (5) Sending next edge 'eC' WITH entity count...(728) +0ms
  bte:biothings-explorer-trapi:edge-manager Checking entity max : (0)--(728) +0ms
  bte:biothings-explorer-trapi:edge-manager 'eA' : (1) --> (15) +0ms
  bte:biothings-explorer-trapi:edge-manager 'eB' : (15) --> (728) +0ms
  bte:biothings-explorer-trapi:edge-manager 'eC' : (728) --> (0) +0ms
  bte:biothings-explorer-trapi:main (5) Executing current edge >> "eC" +0ms
  bte:biothings-explorer-trapi:batch_edge_query Node Update Start +2s
  • Final hop took ~ 15 min to execute sub-querying, then ~1-2 min to run/parse Node-Normalizer (ID resolution of output IDs)
  • final edge-managing step takes ~26 min to run, partially because the final QNode n3 "intersection" steps of edge-managing take ~4 min each time...and it seems to run 6 times.
  • in the logs below, the job is checked every minute which is what the worker threads are doing
related logs
  bte:biothings-explorer-trapi:threading Worker thread 18 completed task, terminated successfully. +1m
  bte:call-apis:query id annotation completes +2m
  bte:call-apis:query qEdge queries complete in 905s +1ms
  bte:biothings-explorer-trapi:batch_edge_query APIEdges are successfully queried.... +17m
  bte:biothings-explorer-trapi:batch_edge_query Filtering out any "undefined" items in (278096) records +0ms
  bte:biothings-explorer-trapi:batch_edge_query Total number of records is (278096) +14ms
  bte:biothings-explorer-trapi:batch_edge_query Start to update nodes... +1ms
  bte:biothings-explorer-trapi:batch_edge_query Update nodes completed! +56ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +17m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 278096 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +1ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture"] +4s
  bte:biothings-explorer-trapi:NewQNode Node "drug" saving (14222) curies... +17m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +10ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" intersecting (728)/(726) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" kept (712) curies... +737ms
  bte:biothings-explorer-trapi:edge-manager 'eC' Reversed[false] (712)--(14222) entities / (278096) records. +17m
  bte:biothings-explorer-trapi:worker19 Worker thread 19 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 19 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker20 Worker thread 20 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 20 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker21 Worker thread 21 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 21 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker22 Worker thread 22 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 22 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:edge-manager 'eC' dropped (2929) records. +5m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +5m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 275167 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +1ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture"] +4s
  bte:biothings-explorer-trapi:NewQNode Node "drug" intersecting (14222)/(14098) curies... +5m
  bte:biothings-explorer-trapi:worker23 Worker thread 23 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 23 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker24 Worker thread 24 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 24 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker25 Worker thread 25 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 25 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker26 Worker thread 26 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 26 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:NewQNode Node "drug" kept (14098) curies... +4m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +4m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +1ms
  bte:biothings-explorer-trapi:worker27 Worker thread 27 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 27 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" intersecting (712)/(725) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" kept (711) curies... +658ms
  bte:biothings-explorer-trapi:edge-manager Updating all other edges... +4m
  bte:biothings-explorer-trapi:edge-manager Updating "eA"... +0ms
  bte:biothings-explorer-trapi:edge-manager 'eA' Reversed[false] (1)--(15) entities / (25) records. +0ms
  bte:biothings-explorer-trapi:edge-manager 'eA' dropped (0) records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +658ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 25 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eA" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +1ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +0ms
  bte:biothings-explorer-trapi:NewQNode Node "nA" intersecting (15)/(15) curies... +1ms
  bte:biothings-explorer-trapi:NewQNode Node "nA" kept (15) curies... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eA" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Disease"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Disease"] +0ms
  bte:biothings-explorer-trapi:NewQNode Node "disease" intersecting (1)/(1) curies... +0ms
  bte:biothings-explorer-trapi:NewQNode Node "disease" kept (1) curies... +0ms
  bte:biothings-explorer-trapi:edge-manager 'eC' Reversed[false] (711)--(14098) entities / (275167) records. +1ms
  bte:biothings-explorer-trapi:worker28 Worker thread 28 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 28 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker29 Worker thread 29 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 29 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker30 Worker thread 30 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 30 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker31 Worker thread 31 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 31 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:edge-manager 'eC' dropped (0) records. +5m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +5m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 275167 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture"] +4s
  bte:biothings-explorer-trapi:NewQNode Node "drug" intersecting (14098)/(14098) curies... +5m
  bte:biothings-explorer-trapi:worker32 Worker thread 32 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 32 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker33 Worker thread 33 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 33 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker34 Worker thread 34 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 34 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker35 Worker thread 35 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 35 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:NewQNode Node "drug" kept (14098) curies... +4m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +4m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" intersecting (711)/(725) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" kept (711) curies... +691ms
  bte:biothings-explorer-trapi:edge-manager Updating "eB"... +4m
  bte:biothings-explorer-trapi:edge-manager 'eB' Reversed[false] (15)--(711) entities / (1222) records. +0ms
  bte:biothings-explorer-trapi:edge-manager 'eB' dropped (21) records. +65ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +756ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 1201 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +1ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eB" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +18ms
  bte:biothings-explorer-trapi:NewQNode Node "nB" intersecting (711)/(711) curies... +84ms
  bte:biothings-explorer-trapi:NewQNode Node "nB" kept (711) curies... +640ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eB" +640ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +1ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +16ms
  bte:biothings-explorer-trapi:NewQNode Node "nA" intersecting (15)/(711) curies... +17ms
  bte:biothings-explorer-trapi:NewQNode Node "nA" kept (15) curies... +11ms
  bte:biothings-explorer-trapi:edge-manager 'eC' Reversed[false] (711)--(14098) entities / (275167) records. +687ms
  bte:biothings-explorer-trapi:worker36 Worker thread 36 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 36 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker37 Worker thread 37 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 37 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker38 Worker thread 38 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 38 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker39 Worker thread 39 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 39 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker40 Worker thread 40 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 40 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:edge-manager 'eC' dropped (0) records. +5m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Storing records... +5m
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) Applying Node Constraints to 275167 records. +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (6) No constraints. Skipping... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating nodes based on edge records... +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["MolecularEntity","SmallMolecule","NucleicAcidEntity","Exon","Transcript","RnaProduct","RnaProductIsoform","NoncodingRnaProduct","MicroRna","SiRna","CodingSequence","ChemicalMixture","MolecularMixture","Drug","ComplexMolecularMixture","ProcessedMaterial","Food","EnvironmentalFoodContaminant","FoodAdditive","Nutrient","Macronutrient","Micronutrient","Vitamin","ChemicalEntity"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["SmallMolecule","ChemicalEntity","MolecularMixture","ComplexMolecularMixture"] +4s
  bte:biothings-explorer-trapi:NewQNode Node "drug" intersecting (14098)/(14098) curies... +5m
  bte:biothings-explorer-trapi:worker41 Worker thread 41 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 41 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker42 Worker thread 42 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 42 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker43 Worker thread 43 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 43 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:worker44 Worker thread 44 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 44 completed task, terminated successfully. +1m
  bte:biothings-explorer-trapi:NewQNode Node "drug" kept (14098) curies... +4m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Updating Entities in "eC" +4m
  bte:biothings-explorer-trapi:QueryExecutionEdge (7) Collecting Types: "["Gene"]" +0ms
  bte:biothings-explorer-trapi:QueryExecutionEdge Collected entity ids in records: ["Gene"] +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" intersecting (711)/(725) curies... +2s
  bte:biothings-explorer-trapi:NewQNode Node "nB" kept (711) curies... +642ms
  bte:biothings-explorer-trapi:main (10) Edge successfully queried. +43m
  bte:biothings-explorer-trapi:main 32 additional queries to https://automat.renci.org/ctd/1.2 were skipped as the API was unavailable. +0ms
logs showing how large the size is now
  bte:biothings-explorer-trapi:edge-manager (11) Collecting records... +4m
  bte:biothings-explorer-trapi:edge-manager (11) 'eA' keeps (25) records! +1ms
  bte:biothings-explorer-trapi:edge-manager ---------- +0ms
  bte:biothings-explorer-trapi:edge-manager (11) 'eB' keeps (1201) records! +0ms
  bte:biothings-explorer-trapi:edge-manager ---------- +0ms
  bte:biothings-explorer-trapi:edge-manager (11) 'eC' keeps (275167) records! +8ms
  bte:biothings-explorer-trapi:edge-manager ---------- +0ms
  bte:biothings-explorer-trapi:edge-manager (12) Collected records for: ["eA","eB","eC"]! +0ms
  bte:biothings-explorer-trapi:edge-manager (12) Collected (276393) records! +0ms
  bte:biothings-explorer-trapi:edge-manager (13) Edge Manager reporting combined records... +0ms
  bte:biothings-explorer-trapi:Graph Updating BTE Graph now. +0ms
  bte:biothings-explorer-trapi:edge-manager (13) Edge Manager reporting organized records... +3s
  bte:biothings-explorer-trapi:QueryResult Updating query results now! +0ms
ERROR 1: SCORING FAILS
  bte:biothings-explorer-trapi:Score Querying 711324 combos. +0ms
  bte:biothings-explorer-trapi:Score Failed to query for scores:  Error: Request failed with status code 500
    at createError (/Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/axios/lib/adapters/http.js:269:11)
    at IncomingMessage.emit (node:events:539:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  config: {
    url: 'http://biothings.ncats.io/semmeddb/query/ngd',
    method: 'post',
    data: '{"umls":[["C1413287","C4319543"],["C1413287","C0068253"],

and the query is super long and then these logs look relevant

  response: {
    status: 500,
    statusText: 'Internal Server Error',
    headers: {
      server: 'nginx/1.14.0 (Ubuntu)',
      date: 'Mon, 06 Jun 2022 20:55:55 GMT',
      'content-type': 'text/html',
      'content-length': '202',
      connection: 'close'
    },

    data: '<html>\r\n' +
      '<head><title>500 Internal Server Error</title></head>\r\n' +
      '<body bgcolor="white">\r\n' +
      '<center><h1>500 Internal Server Error</h1></center>\r\n' +
      '<hr><center>nginx/1.14.0 (Ubuntu)</center>\r\n' +
      '</body>\r\n' +
      '</html>\r\n'
ERROR 2: can't return the response due to string issue? looks like a TON of results too (289878) and took ~ 1 min to generate the logs during results-assembly
  bte:biothings-explorer-trapi:QueryResult Successfully scored 0 results, couldn't score 289878 results. +683ms
  bte:biothings-explorer-trapi:Graph pruning BTEGraph nodes/edges... +2m
  bte:biothings-explorer-trapi:Graph pruned 3 nodes and 55 edges from BTEGraph. +871ms
  bte:biothings-explorer-trapi:main (14) TRAPI query finished. +2m
  bte:biothings-explorer-trapi:worker46 Worker thread 46 beginning task. +0ms
  bte:biothings-explorer-trapi:EdgeReverse BioLink-model class is initiated. +0ms
  bte:biothings-explorer-trapi:async checking query status of job YARh5urbqG +0ms
  bte:biothings-explorer-trapi:threading Worker thread 46 completed task, terminated successfully. +1m
RangeError: Invalid string length
    at JSON.stringify (<anonymous>)
    at /Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/asyncquery.js:61:46
    at Array.map (<anonymous>)
    at storeQueryResponse (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/asyncquery.js:57:52)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async exports.asyncqueryResponse (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/asyncquery.js:132:13)
    at async jobToBeDone (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/processors/async_v1.js:17:20)
    at async module.exports (/Users/colleenxu/Desktop/bte-trapi-workspace/packages/@biothings-explorer/bte-trapi/src/controllers/async/processors/async_v1.js:22:10)
    at async /Users/colleenxu/Desktop/bte-trapi-workspace/node_modules/bull/lib/process/master.js:92:27

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 6, 2022

acanthosis nigricans (MONDO:0007035)

Initial query

Assuming the query given will be...

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["MONDO:0007035"],
                    "categories":["biolink:Disease"],
                    "name": "acanthosis nigricans"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

  • straightforward method: TRAPI Response JSON. Took ~ 5 sec to run, <80kB, 32 results
  • Method 2: TRAPI Response JSON. Took ~ 10 min 33 sec to run, very large ~151 MB, 25543 results (a TON)
  • Method 3: TRAPI Response JSON. Took ~ 32 sec to run, ~6 MB, 2520 results (a lot (1000<x<10000))
  • Method 4: TRAPI Response JSON. Took ~ 8 min 31 sec to run, very large ~144 MB, 25543 results (a TON)
  • Method 6: TRAPI Response JSON. Took ~37 min to run, very large ~415 MB, 88648 results (a TON)

Other methods that didn't return results:

  • Method 1: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 5: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

                    "ids":["MONDO:0007035"],
                    "categories":["biolink:Disease"],
                    "name": "acanthosis nigricans"

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 6, 2022

Abnormal blood glucose concentration (HP:0011015 / Phenotype)

Initial query

Assuming the query given will be...(assigning this ID as a Disease even though it's not...BTE will add PhenotypicFeature to its categories)

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["HP:0011015"],
                    "categories":["biolink:Disease"],
                    "name": "Abnormal blood glucose concentration"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

Other methods that didn't return results:

  • straightforward: took < 2 sec, BTE didn't find any records for the QEdge, returns 0 results (TRAPI Response JSON)
  • Method 1: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 2: Took ~ 42 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 2600 entities).
  • Method 3: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 5: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 6: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

(assigning this ID as a Disease even though it's not...BTE will add PhenotypicFeature to its categories)

                    "ids":["HP:0011015"],
                    "categories":["biolink:Disease"],
                    "name": "Abnormal blood glucose concentration"

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 6, 2022

chronic obstructive pulmonary disease (COPD, MONDO:0005002)

Initial query

Assuming the query given will be...

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["MONDO:0005002"],
                    "categories":["biolink:Disease"],
                    "name": "COPD"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

  • straightforward method: TRAPI Response JSON. Took ~ 14 sec to run, ~3.7 MB, 888 results
  • Method 1: TRAPI Response JSON. Took ~ 37 sec to run, ~4.8 MB, 2784 results (a lot (1000<x<10000))
  • Method 5: TRAPI Response JSON. Took ~ 34 min to run, very large ~393 MB, 105924 results (a TON). Used the older format for the query (n0-n3, e1-e3)

Other methods that didn't return results:

  • Method 2: Took ~ 58 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 3026 entities).
  • Method 3: Took ~ 5 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eB (dev: 1287 entities).
  • Method 4: Took ~ 29 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eB (dev: 2538 entities).
  • Method 6: Took ~ 33 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 2063 entities).

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

                    "ids":["MONDO:0005002"],
                    "categories":["biolink:Disease"],
                    "name": "COPD"

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 7, 2022

cirrhosis of liver (MONDO:0005155)

Initial query

Assuming the query given will be...

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["MONDO:0005155"],
                    "categories":["biolink:Disease"],
                    "name": "cirrhosis of liver"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

  • straightforward method: TRAPI Response JSON. Took ~ 18 sec to run, ~6.2 MB, 1252 results (a lot (1000<x<10000))
  • Method 3: TRAPI Response JSON. Took ~ 15 min to run, very large ~200 MB, 49600 results (a TON)

Other methods that didn't return results:

  • Method 1: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 2: Took ~ 38 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 1856 entities).
  • Method 4: Took ~ 22 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eB (dev: 1651 entities).
  • Method 5: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 6: Took ~ 22 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 1122 entities).

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

                    "ids":["MONDO:0005155"],
                    "categories":["biolink:Disease"],
                    "name": "cirrhosis of liver"

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 7, 2022

Colon cancer (HP term that likely maps to disease in Node Normalizer) (HP:0003003)

Initial query

Assuming the query given will be...

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["HP:0003003"],
                    "categories":["biolink:Disease"],
                    "name": "Colon cancer (phenotype)"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

  • straightforward method: TRAPI Response JSON. Took ~ 15 sec to run, ~6.5 MB, 1262 results (a lot (1000<x<10000))
  • Method 1: TRAPI Response JSON. Took ~ 49 sec to run, ~6.1 MB, 3544 results (a lot (1000<x<10000))
  • Method 3: TRAPI Response JSON. Took ~ 28 min to run, very large ~354 MB, 120522 results (a TON)
  • Method 5: TRAPI Response JSON. Took ~ 35 min to run, very large ~400 MB, 108854 results (a TON). Used the older format for the query (n0-n3, e1-e3)

Other methods that didn't return results:

  • Method 2: Took ~ 2 min 24 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 4413 entities).
  • Method 4: Took ~ 43 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eB (dev: 3847 entities).
  • Method 6: Took ~ 1 min 4 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 3437 entities).

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

                    "ids":["HP:0003003"],
                    "categories":["biolink:Disease"],
                    "name": "Colon cancer (phenotype)"

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 7, 2022

Diarrhea (HP:0002014 / Phenotype)

Initial query

Assuming the query given will be...(assigning this ID as a Disease even though it's not...BTE will add PhenotypicFeature to its categories)

Query starting point
{
    "message": {
        "query_graph": {
            "nodes": {
                "drug": {
                    "categories":["biolink:Drug"]
                },
                "disease": {
                    "ids":["HP:0002014"],
                    "categories":["biolink:Disease"],
                    "name": "Diarrhea"
               }
            },
            "edges": {
                "e1": {
                    "subject": "drug",
                    "object": "disease",
                    "predicates": ["biolink:treats"],
                    "knowledge_type": "inferred"
                }
            }
        }
    }
}

Future Responses:

  • straightforward: TRAPI Response JSON. Took ~ 19 sec to run, ~ 6.2 MB, 1375 results (a lot (1000<x<10000))
  • Method 3: TRAPI Response JSON. Took ~ 3 min 4 sec to run, ~ 38 MB, 14512 results (a TON)

Other methods that didn't return results:

  • Method 1: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 2: Took ~ 51 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 1769 entities).
  • Method 4: Took ~ 21 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eB (dev: 1518 entities).
  • Method 5: took < 2 sec, BTE didn't find any records for the first QEdge, returns 0 results (TRAPI Response JSON)
  • Method 6: Took ~ 52 sec, returns 501 Not Implemented (TRAPI response JSON), BTE stops execution early because > 1000 entities set as input for eC (dev: 2710 entities).

Dev notes:

replace Alzheimer's node in the above query-templates for the methods, with this info:

(assigning this ID as a Disease even though it's not...BTE will add PhenotypicFeature to its categories)

                    "ids":["HP:0002014"],
                    "categories":["biolink:Disease"],
                    "name": "Diarrhea"

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 22, 2022

Modified Method 1: add is_set to Pheno node

  • Alzheimers: less results (527) vs w/o is_set (1244)
    • comparable performance 1.2 MB, 18 sec
  • neurofibromatosis, familial spinal: less results (3911) vs w/o is_set (8286)
    • comparable performance 11.2MB, 2 min 45 sec
  • noonan syndrome: less results (1058) vs w/o is_set (1474)
    • comparable performance 2.4 MB, 32 sec
  • acanthosis nigricans: still 0 results. This is because monarch-biolink's disease-to-pheno x-bte-operation is set to direct relationships only. If we allowed non-direct relationships, monarch has 10 phenotypes. However, other diseases like noonan syndrome would go from 55 direct phenos to 511 direct+non-direct phenos.....so explosions....
  • Abnormal blood glucose concentration: still 0 results. This is because there isn't much data/operations for PhenotypicFeature having other PhenotypicFeatures....
  • COPD: less results (1841) vs w/o is_set (2784)
    • comparable performance 4.1 MB, 40 sec
  • cirrhosis of liver: still 0 results. Has the same issue as "acanthosis nigricans"; monarch has 47 non-direct relationships.
  • colon cancer: less results (2180) vs w/o is_set (3544)
    • comparable performance 5.3 MB, 54 sec
  • Diarrhea: still 0 results. Has the same issue as "Abnormal blood glucose concentration".

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 22, 2022

Modified Method 2 (set predicates between disease-gene, use more-specific predicate between gene-chem, add is_set on intermediate nodes):

Of note:

  • neurofibromatosis: 134 results, 14 sec, 885kB (previously 135 results)
  • noonan syndrome: 3525 results, 3 min 47 sec, ~60MB (previously > 10k results, took > 10 min, > 150 MB).
  • acanthosis nigricans: 6298 results, 7 min 39 sec, ~ 118MB (previously > 10k results, took > 10 min, > 150 MB).....hmmm still large.

Basically the same behavior:

  • alzheimers: stopped after 1 min 58 sec (return 501) because > 1000 entities set as input for eC (dev: 4175)
  • Abnormal blood glucose concentration: stopped after 24 sec (return 501) because > 1000 entities set as input for eC (dev: 2060)
  • COPD: stopped after 55 sec (return 501) because > 1000 entities set as input for eC (dev: 2126)
  • cirrhosis of liver: stopped after 40 sec (return 501) because > 1000 entities set as input for eC (dev: 1374)
  • Colon cancer: stopped after 2 min 20 sec (return 501) because > 1000 entities set as input for eC (dev: 3270)
  • Diarrhea: stopped after 1 min 17 sec (return 501) because > 1000 entities set as input for eC (dev: 1348)

@colleenXu
Copy link
Collaborator

colleenXu commented Jun 22, 2022

Modified Method 3 (use more-specific predicate between gene-chem, add is_set on intermediate nodes):

May not want to use this because we can't score any results (SequenceVariants don't have UMLS IDs)...

Of note:

  • neurofibromatosis: 134 results, 3 min 45 sec, 5.4 MB (previously took less time but had way too many results >25k)
  • noonan syndrome: 1047 results, 9 min 19 sec, 23.4 MB (previously took less time but had way too many results >70k)
  • acanthosis nigricans: 734 results, 31 sec, 4.9 MB (previously 2520 results)
  • Diarrhea: 2565 results, 3 min 4 sec, 30.5 MB (previously had way too many results >14k)

Still problematic:

  • cirrhosis of liver: still takes > 10 min (took 14 min 39 sec). 7156 results (previously way too many results >49k), 174 MB...
  • colon cancer: still takes > 10 min, was 1->692-> (wasn't done doing SeqVar -> Gene qEdge). likely large # of results and response size...

Basically the same behavior:

  • alzheimers: stopped after 12 sec (return 501) because > 1000 entities set as input for eC (dev: 2477)
  • Abnormal blood glucose concentration: stopped after 4 sec (return "empty" KG/results) because no records found for first QEdge
  • COPD: stopped after 7 sec (return 501) because > 1000 entities set as input for eB (same as previous behavior)

@colleenXu
Copy link
Collaborator

Modified Method 4 (add predicates between disease-gene, use more-specific predicate between gene-chem, add is_set on intermediate nodes):

Of note:

  • Abnormal blood glucose concentration: 0 results because no records found for first qEdge (previously took ~ 20 sec to run, ~380kB, 59 results)
  • noonan syndrome: 1802 results, 1 min 10 sec, 19.8 MB (previously had way too many results >19k)

Still problematic:

  • acanthosis nigricans: 6298 results, 6 min 8 sec, 115.5 MB (previously took ~ 8 min 31 sec to run, very large ~144 MB, 25543 results (a TON))

Basically the same behavior:

  • alzheimers: stopped after 42 sec (return 501) because > 1000 entities set as input for eB (dev: 4160)
  • neurofibromatosis: 134 results, 16 sec, < 1 MB (previously took ~ 31 sec to run, < 1 MB, 135 results)
  • COPD: stopped after 16 sec (return 501) because > 1000 entities set as input for eB (dev: 1770)
  • cirrhosis of liver: stopped after 14 sec (return 501) because > 1000 entities set as input for eB (dev: 1282)
  • colon cancer: stopped after 26 sec (return 501) because > 1000 entities set as input for eB (dev: 2994)
  • Diarrhea: stopped after 13 sec (return 501) because > 1000 entities set as input for eB (dev: 1238)

@colleenXu
Copy link
Collaborator

Relay is over; closing in favor of #461

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants