Testing

[cbizon]

This is a bit of an experiment to see if we can increase the speed of conversations around some key topics, so that we're not limited to just the time on our architecture calls.

Specifically, I wanted to turn to a discussion of testing. There are two resources to point out here. One is this repo that Mark has started: https://github.com/NCATSTranslator/testing. The second are these quick slides that we discussed a bit at the relay meeting: https://docs.google.com/presentation/d/1cllV-KtX6mYoEPiOTCBRMwzInkUgA_kJvKHQnv-G6fc/edit#slide=id.p

So I'd like to open up a couple of questions for discussion:

1. We can go ahead and start working on the plan schematically laid out on slide 6, which would first build a corpus of triples from each KP, and then systematically test the system to see under what kinds of transformations we can find those triples. Are there any alternate approaches or modifications that we should think about before getting started?

2. What other aspects of the Translator system are amenable to testing? For instance, part of the test harness should, I think, check that results are TRAPI compliant, and that they are biolink compliant. I can imagine testing correct smartAPI tags or elements of TRAPI endpoints. I'd love to hear the thoughts of others on what we can have automated testing of that will improve and ensure compliance with standards.

[sstemann]

@NCATSTranslator/architecture-core please share your thoughts in preparation for the architecture call this today

[andrewsu]

Regarding the question "What other aspects of the Translator system are amenable to testing?" I think Noel's conversation starter at NCATSTranslator/Relay#123 fits into that discussion for ARA-level testing. But to reiterate what I said on last week's call, I think the vision on slide 6 for KP testing would be an outstanding starting point. If we standardize and operationalize some basic assumptions about how KPs should work, I think that will make everyone's life easier.

[patrickkwang]

We can go ahead and start working on the plan schematically laid out on slide 6, which would first build a corpus of triples from each KP, and then systematically test the system to see under what kinds of transformations we can find those triples. Are there any alternate approaches or modifications that we should think about before getting started?

This makes sense. We'll want to lay out in more detail the various transformations that we expect should still retrieve the data. These come to mind:

• invert the predicate, e.g. -biolink:treats-> for <-biolink:treated_by-
• use a synonymous id, e.g. MONDO:0005737 for DOID:4325
• use a more general predicate, e.g. biolink:affects for biolink:treats
• use a more general category, e.g. biolink:BiologicalEntity for biolink:Disease
• remove a predicate/category/id
• combinations of the above

[marcdubybroad]

#1 can be split into 2:

• id on object, so predicate stays the same
• reverse predicate and expect similar edges between reversed nodes

[cbizon]

OK, so here is what I'm imagining:

1. We create a corpus of triples that live in https://github.com/NCATSTranslator/testing somewhere.
2. We write a test that compares that corpus of triples against the metaKG and / or /predicates endpoints of KPs in smartAPI registry, depending on whatever the current best source of truth is at the moment. This helps us keep the list of triples up to date and makes sure we have full coverage.
3. We have a suite of tests that implements the basic lookup, plus the transformations above. These tests run on the ARS, and individually on the KPs
4. Each test makes sure we can return the correct edge, from the correct KP, and also checks for cross-cutting concerns like valid TRAPI.
5. These tests run daily on a Jenkins stood up by the deployment WG.

Is that right?

[stevencox]

+1 the above thoughts.

Sifting through Translator's architecture commitments for other possible testable things:

• KPs expose data in ways described in (S3).
• Appropriate use of equivalent node identifiers (S5).
• Provenance (S16)

...look like candidates for automated testing.

I'm curious how we identify and test things that make Translator different from other approaches to the biomedical KG problem, especially: What kinds of inference are components of each kind supposed to be able to do?

[cbizon]

I think these are great ideas. Should we try to incorporate them into the one-hop test suite above or should we think about suites that go after these in different ways?

I also think that there may be other things we can test around endpoint metadata and tags perhaps? But maybe those are more in scope for smartAPI validation...