Implement VRO API Gateway to serve multiple APIs

[yoomlam]

VRO currently serves up its API using Spring MVC. This constrains implementations of new APIs to be written using Spring MVC and Java. To support serving up APIs implemented in other languages (i.e., Python used by the CC Team), implement a VRO Gateway that queries for available APIs and merges them to present a single VRO API.

AC:

• VRO API requests to Team CC's API works
• VRO API requests to Xample API works

Reference

• https://marutitech.com/api-gateway-in-microservices-architecture/
• https://www.moesif.com/blog/technical/api-gateways/How-to-Choose-The-Right-API-Gateway-For-Your-Platform-Comparison-Of-Kong-Tyk-Apigee-And-Alternatives/
• Using Spring: https://piotrminkowski.com/2020/02/20/microservices-api-documentation-with-springdoc-openapi/, https://blog.wick.technology/distributed-api-docs/
  • different implementations and library versions for Spring Boot 2 vs 3. Solution: run using Spring Boot 3 as separate Docker container.
• Using Node.js: https://blog.risingstack.com/building-an-api-gateway-using-nodejs/ or https://www.express-gateway.io/about/
  • Doesn't have OpenAPI (v3) support yet (https://www.express-gateway.io/docs/roadmap/), so can't import APIs from domain-rrd or domain-cc
• Using Apache ApiSix: https://apisix.apache.org/, https://apisix.apache.org/blog/2022/10/27/ten-use-cases-api-gateway/
  • versatile, included in the CNCF (standard); requires etcd container, need script to import OpenAPIs (https://apisix.apache.org/docs/dashboard/modules/data_loader/openapi3/, https://apisix.apache.org/docs/dashboard/IMPORT_OPENAPI_USER_GUIDE/) and seems rather new (cannot export APIs?).
• Using KrackenD: https://www.krakend.io/
  • Need enterprise license for advanced (useful) features

Benefits of having a gateway:

• allows us scale implementations of APIs behind the scenes
• allows us to add logging and manual retries independent of the domain API implementation
• allows authentication and security to be done by VRO, reducing the load on the domain developers
• "internal" domain APIs can be tested independently and without the burden of passing through security boundaries

[lukey-luke]

I think as an engineer, it's important to understand tradeoffs and risks. I agree that VRO api serves as a security blanket, this is not without a cost. It does become a single point of failure for any dependent API's, so glad to see we have some e2e tests setup already.

Additional benefits of having a gateway: We can consolidate how we monitor API performance and get that info in a single place.

[yoomlam]

Thoughts about how to incorporate an API gateway:

• create a new top-level folder api-gateway
  • implement the API gateway and create an api-gateway Docker container, listening on ports 80 (http) and 443 (https)
  • it queries for the OpenAPI specs from app and domain-cc/python-src to present a single merged API spec
  • it routes requests to the corresponding API
  • update Istio config to expose port 80 (or 443?) instead of 8080
• app provides the API for Java controllers (i.e., domain-*/*-api-controller) on ports 8080 and 8081
  • security and domain-independent authentication code should be moved to api-gateway
  • This can be deployed as part of the api-gateway Docker container if they're both Java-based
• domain-cc/python-src provides the API for domain-cc's Python-based API

[dianagriffin]

Reference thread for prioritization -- this is becoming a blocker for Contention Classification team's ability to publish their API spec.

[va-albers]

LHDI has an existing Kong API Gateway in place, my understanding is that they use it for both internal and external-facing partner APIs today. Please check with them to see if their gateway satisfies our requirements.

[yoomlam]

@va-albers Who's a good Kong contact?

[va-albers]

@yoomlam we could check with the LHDI during their Wednesday 11 AM ET office hours. Also Steven Bair mentioned a few names:
"The LH project uses Kong today (hosted in VSP AWS) - we use the same instance of Kong for a team that required public availability on LHDI. There is a team (I believe it was Quokka but moved to another team) looking into what the future for API GW would look like for Lighthouse. Pam Lougheed should be able to point you in the right direction (in terms of what team is doing that research today and a good POC if desired)"
"It's also used as a VA-internal GW. Beau Grantham would be a good POC if interested in using it as an LHDI tenant"

Thanks @yoomlam !

[yoomlam]

Posted question.

[dianagriffin]

As of 5/10, while the CC API is our only use case, we decided not to solve for serving multiple APIs yet. Instead, we will temporarily point the single VRO API hostname to the CC API as the single API within VRO (decoupling it from the existing, RRD-focused VRO API). Then when we surface our next use case for having an external system call an additional API within VRO, we'll come back around to solving for serving up multiple APIs. (Further context on this decision in Slack here).

[dianagriffin]

@yoomlam I'd like to either revamp this ticket to be a tech spec/tech design ticket for solving for serving up multiple APIs (either via gateway or K8s solution), or, close this ticket and reference it in a new tech spec/tech design ticket. Any preference or opinion on that?

[yoomlam]

Let's keep this ticket as is in case we do want an API Gateway as the solution, so "close this ticket and reference it in a new tech spec/tech design ticket".

[dianagriffin]

Update as of 5/15, with a reduced LOE on this API gateway effort we decided to go ahead with it instead of the workaround of pointing the VRO API to the CC API mentioned above. (Slack post here)

[yoomlam]

PR #1572 unblocks Team CC -- Slack.

[yoomlam]

Finally got it working in LHDI. Need to clean up the code and document on the wiki.

[dianagriffin]

@yoomlam I'd like to succinctly capture the decision history on this one, namely:

• why we went with this solution for right now, instead of either Update VRO API to point to Contention Classification (CC) API #1556 or using LHDI's Kong
• the assumption that we can/will revisit and potentially replace this solution once we have additional API use cases to consider

Could you incorporate that into the wiki documentation you mentioned?

[yoomlam]

Created new follow-on ticket #1591 with the same ticket labels as this one.

[yoomlam]

@dianagriffin Added decision history: https://github.com/department-of-veterans-affairs/abd-vro/wiki/Software-Conventions#decision-history

[yoomlam]

Deployed and tested in all LHDI envs.