Generate RESTful APIs from spec

[dblock]

Is your feature request related to a problem? Please describe.

Coming from opensearch-project/opensearch-api-specification#30, RESTful APIs are hand-rolled, making it difficult to inspect the available routes, required or optional parameters.

Describe the solution you'd like
Use https://github.com/opensearch-project/opensearch-api-specification to generate code in OpenSearch.

History
In this issue we discussed an approach that either starts with code that can be inspected/reflected, or starts with a spec, but not both. We proposed using https://github.com/opensearch-project/opensearch-api-specification to generate code in OpenSearch, vs. usng OpenSearch to export the spec in https://github.com/opensearch-project/opensearch-api-specification.

[dblock]

@sachetalva I believe you've looked at what effort it would take to enable "use OpenSearch to export the spec"?

[nknize]

💯 We really need help in this area. We can version the REST APIs all we want to help with client compatibility but keeping the clients NSync w/ the API is another matter. Would love to see someone from the community step up and help here as well!

[sachetalva]

Hi @dblock and @nknize, yes ideally it would be good if we can generate the OpenSearch rest api handlers auto generated from the opensearch-api-specification. However currently there is no direct solution to achieve this.
One of the possible approaches currently is to generate Java POJO classes from the Smithy/OpenAPI models and use these to update the request / response handling in OpenSearch. This would ensure that the api-specification is always in sync with the OpenSearch code. However, we have not explored this approach yet and would need some further research.
To be able to achieve this we would need the smithy models for all the APIs as a pre-req. Currently we are working on setting up api models for the basic CRUD OpenSearch APIs. We are also setting up a CI framework which will help to automatically validate all model files, which will help in faster development and validation of the new smithy model files.

[dblock]

@sachetalva You're proposing a spec -> code path. I am saying there's probably also a code -> spec path.

[sachetalva]

Code -> spec should also be possible but this has not been explored yet. From initial investigation there is no direct approach available to generate Smithy models from code. Custom annotations could be created around the request and response handling code in OpenSearch to declare the request and response body parameters. However we would have an additional overhead to maintain code which translates the structures defined in annotations / code to corresponding Smithy structures and data types.

[reta]

@dblock a few thoughts on that, as far as I can tell, the Opensearch core does NOT have a formal, structured models for APIs, in many cases it is buried behind ToXContent implementations, fe:

    @Override
    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
        builder.startObject(Fields.JVM);
        builder.field(Fields.TIMESTAMP, timestamp);
        builder.humanReadableField(Fields.UPTIME_IN_MILLIS, Fields.UPTIME, new TimeValue(uptime));

        ...
    }

Having formal models (POJOs) would have made specs generation easier but it is huge amount of work. However, there are different approaches out there which we may loot at. For example, Spring REST Docs [1] relies on auto-generated snippets produced with test scaffolding (in this case, Spring MVC Test [2]). The main idea here is to capture API calls + requests + responses in tests and publish them as documentation / specification (we do not need to use these great projects as-is, but borrow the idea). Something to explore, wdyt?

The reverse spec -> code path would be the best option (in my opinion) but still, it very likely would require to replace free style ToXContent transformations with formal (generated) models.

[1] https://spring.io/projects/spring-restdocs#overview
[2] https://docs.spring.io/spring/docs/current/spring-framework-reference/htmlsingle#spring-mvc-test-framework

[rursprung]

i would strongly suggest the spec -> code path: this allows designing the API first, leading to cleaner APIs for consumers rather than starting it with a technology-specific API which just happens to carry some annotations turning it into something which you can call over HTTP (and might not follow any proper RESTful guidelines).

yes, the initial move will be quite a bit of effort, but it'll ultimately pay off in the long run. and probably it doesn't have to be done as a big-bang approach but can instead be done endpoint by endpoint.

as far as i can see smithy specs can be converted to OpenAPI specs and these in turn can be used to generate both client and server stubs e.g. using the org.openapi.generator gradle plugin.

what will definitely not work mid- to long-term is just trying to maintain the specs manually based on changes to the code, this is guaranteed to diverge and is just a waste of time for everyone (maintainers trying to constantly keep it aligned and consumers trying to use it just to learn that there was yet another difference between spec and real API).

[dblock]

It's pretty clear that most people recommend generating code from spec. I agree. I edited this issue to reflect that proposal, leaving history to how we came here.

If anyone wants to take a stab at replacing any part of the code in OpenSearch by code generated from https://github.com/opensearch-project/opensearch-api-specification, you'll be my hero. Re: OpenAPI we need a pipeline that automatically publishes the Smithy IDL in that format, I opened opensearch-project/opensearch-api-specification#34.