[RFC] REST API Versioning

[nknize]

Is your feature request related to a problem? Please describe.
Originally posted in opensearch-project/opensearch-clients#17, this RFC aims to solve the broader problem but using a REST API Version mechanism:

"Coming from opensearch-project/opensearch-clients#12, clients are being upgraded to support OpenSearch 2.0 while breaking compatibility with 1.0. For example, opensearch-java 1.0 works against OpenSearch 1.x, opensearch-java 2.0 works against OpenSearch 2.x, but there's no version of client that works against both OpenSearch 1.x and 2.x. Thus, in order to upgrade from OpenSearch 1.x to 2.x one needs to either run a cluster in mixed mode for a long time, or take downtime."

Describe the solution you'd like

Think Big: An API version mechanism to enable users to specify their desired API version to ensure client compatibility is independent of server upgrades while providing full compatibility across clients without requiring to carry deprecated features or tech debt across more than one major server version.

User obsession: User's have control over what API their client is requesting for compatibility.. no down time.

Bias For Action: Start with a simple rest-high-level-compatibility module that provides Version.onOrAfter checks to ensure endpoint compatibility. Let's not "boil the ocean" and handle every corner case. We can release the first version in a 2.x minor.

High Standards: Do not require tech debt or deprecated feature maintenance across more than one version.

Frugality: One module to solve the client compatibility issues. Can even be a plugin and require users to install if they don't need the compatibility

Describe alternatives you've considered
Limit to two major version compatibilty

Additional context
This is also described in #2447 but there was confusion on whether that API versioning was specific to Plugin SDKs.

[dlvenable]

Is this proposal to have OpenSearch provide backward compatible APIs after breaking changes? Would the clients supply a field indicating the version it expects to be compatible with and the server allows that?

So for example, a client can make a "v1" request to an OpenSearch 2.0 cluster to get a document and the type parameter which was removed with mapping types is thus populated?

[dblock]

The client would supply an Accept header, e.g. Accept: application/vnd.opensearch.api+json;version=1.1 and the server would be able to handle that by either implementing a working compatibility layer, but also by rejecting the request in case server-side behavior has changed and would be unexpected by an older client. In your example it would probably make sense to return type. Note that none of this prevents implementing similar behavior client-side, where clients can send different requests to different versions of the server, or handle different responses.

[dblock]

I think this proposal would be fairly easy to implement if we can mechanize the generation of two separate RESTful API versions (#3090). This way we'd author separate specs, generate the interfaces, then provide implementations underneath for compatibility.

[wbeckler]

The 2.x java client now has backward compatibility with the 1.x server: opensearch-project/opensearch-java#152

This is a critical feature of any client, given that double incompatibility (v1 can only talk to v1 and v2 can only talk to v2) prevents a zero-downtime upgrade of a server version.

The clients are moving toward a spec-generated API layer, which allows for several solutions to achieve this. But the solution can't be, generate a new client for the new API, as that reinforces double incompatibility.

If an API on the server changes, maybe it makes sense to rename the route and carefully deprecate the old route.

[dblock]

Thanks @wbeckler. This proposal above is that the server supports multiple API versions (e.g. a backwards compatible 1.x and 2.x) at the same time. Assuming sever-side code works we can use Accept headers to negotiate which version to talk, reducing the client burden to deal with multiple versions of the server.