semmeddb: generating x-bte annotations using metatriple file

[colleenXu]

@erikyao has code to create a meta-triple table file (original ask) during the semmeddb api deployment process.

Yao got me the meta-triple file for the data used in semmeddb/semmeddb2 during Jan/Feb 2023.

My task is to refactor the x-bte-annotation generating notebook to use this file.

Benefits:

• will add operations for the ncbigene namespace FINALLY
• get code that can be reused for auto-generating x-bte annotations! (but note that one needs to tweak / test query structure for other APIs)

Notes:

• will be setting an arbitrary count cutoff for creating operations from meta-triples. This cutoff will need adjusting because semmddb2 will have a diff number of records for sure

[colleenXu]

Done and deployed (registration refreshed): NCATS-Tangerine/translator-api-registry@b08b9b3

It uses a new metatriple + counts file generated by Yao Monday evening. However, it still uses predication counts (not the record / triple counts of the current API). I still use a cutoff of predication counts > 100

I still kept the original notebook (renamed to AutoGen_SEMMEDDB_Old.ipynb).

The custom processing we still do (outside of Translator-curated exclusions) is written up in the collapsed section of:
#669 (comment)

---

EDIT: added comment

I think this effort added 739 operations (from 3936 to 4675), which would all be Gene-related (have NCBIGene ID-namespace as either input, output, or both).

It's a little hard to tell because the ordering of operations also changed when using the metatriples file.

[colleenXu]

Going to leave this open, since I'm waiting for a response from @erikyao on generating a metatriple file with:

• number of API records (basically unique triple count), rather than what seems like predication count
• number of API records where pmid_count > 3 (helpful given the current constraint)

If this isn't possible, then this issue can be closed as completed.
If this can be done, then maybe this issue can be kept open until that file is made, I adjust the notebook to use it, and generate / deploy x-bte annotations based on this data...

[colleenXu]

Related to #645

[colleenXu]

Yao generated the file with the counts, and I saved a copy of this file to the same repo as the notebook. NCATS-Tangerine/translator-api-registry@205e32d

I also adjusted the notebook to use this file, and with discussion from Andrew (recorded in the notebook), we set the cutoff to metatriples w/ >20 documents with pmid_count > 3. See the commit link for details.

I've deployed the updated SmartAPI yaml (refreshed registration).

So we can close this issue as complete. We can open new issues when we want to update the metatriples+counts file, notebook, and x-bte annotations / SmartAPI yaml...