pyvo.registry search etc: option to show auxiliary capabilities

[theresadower]

Currently, the pyvo.registry search and ivoid2service functions match on servicetypes in a way that excludes auxiliary capabilities, like TAP endpoints associated with a simpler ConeSearch record. While this is fine in many cases and may be an explicit design decision, we've found in working with NAVO tutorial notebooks that it could be very handy and in some cases an easier search pattern if the user could see these.

Requesting feature flag to include auxiliary capability information or add it to the default behavior. I'm inclined to add a flag and happy to write it myself either way, but wanted input on preference from more seasoned pyvo users/maintainers, or to be pointed at documentation if I've missed a way for this to be enumerated already.

Cheers!

[msdemlei]

On Thu, Sep 17, 2020 at 01:38:28PM -0700, theresa dower wrote: Requesting feature flag to include auxiliary capability information or add it to the default behavior. I'm inclined to add a flag and happy to write it myself either way, but wanted input on preference from more seasoned pyvo users/maintainers, or to be pointed at documentation if I've missed a way for this to be enumerated already.

I believe for pyVO is most common discovery case is "all-VO query with constraints", which is service enumeration in the language of http://ivoa.net/documents/discovercollections/: Essentially, you want to hit every service of a given type exactly once. Hence, I'd rather confidently say the default should remain the exclusion of auxiliary capabilities. Supporting data discovery (and thus aux capabilities), of course, is still a good idea, in particular when we expect usage through notebook-like interfaces. Out of curiosity: Do you have a particular scenario in mind?

[theresadower]

Agreed that the default behavior should skip them, especially if we're in most use cases looping over results at the resource level. I'll work on a feature with a flag for including aux and not touch the defaults.

The scenario in which this came up was an evolution of a use case notebook in https://github.com/NASA-NAVO/navo-workshop: we are looking for catalog data from a specific mission, finding the cone search, and then doing a deeper dive into more complex TAP queries. The ability to follow logical links from the HEASARC's ConeSearch service for the data as found in the registry to the main HEASARC xamin TAP service hit a snag when the aux capability doesn't come back.

[msdemlei]

Uh, apologies I let this drop in September.

PR #258 was a good wakeup call; but I'm afraid I'm not too happy with it as it stands. That's because I believe the use cases "(constrained) service enumeration" (what we're doing so far) and "data discovery" (again in the language of https://ivoa.net/documents/discovercollections/) are different enough that we perhaps ought to at least briefly think about whether data discovery shouldn't have a different API entirely.

Me, I'd say we should cover the second use case with function, say, discoverdata that accepts keywords, UCDs, STC location and perhaps some additional constraints. That would return registry records exposing the capabilities found (e.g., SCS and TAP). These objects could then be used to produce service objects of certain types to do the work.

For instance, a client could define do_work_via_tap and fall_back_to_scs functions to do whatever they want to do using the different protocols and then write:

for matched_resource in registry.discoverdata(
    ucd="phot.mag;em.opt.K", location=SkyCoord(23, -40.3)):
  try:
    do_work_via_tap(matched_resource.as_tap())
  except registry.NoSuchCapability:
    fall_back_to_scs(matched_resource.as_scs())

If someone else would be in for helping out with producing tests and test data, I'd be happy to write such a thing.

[theresadower]

Having spent some time tweaking the code in #258, I think some additional discoverdata search pattern could be useful, but the small addition there fits an existing workflow and functions folks are familiar with. I'd rather do this now and then iterate over this new proposed workflow with input from the next round of workshops and user experience. I can pull some test data/resources from this June's.

[msdemlei]

Closing now that #289 is merged.