[Doc-Meta] Redesign of Documentation Website

[hdhalter]

The OpenSearch.org documentation website does not raise the bar when compared to other open-source documentation websites. As we continue to grow our documentation library, we need to make sure it is built to scale, and also provides a delightful experience.

Improvement ideas:

• Update the summary information
• Update the rest of the content on the doc home page to quickly direct users to the information they need most, for example: Log Analytics, Site Search, Observability, Getting Started, What's New, Latest Release Notes, etc.
• [Meta] Restructure the TOC to make it easier for users to find content #1036
• Redesign the home page into a more modern look and feel (e.g., use different Jekyll template or React) #1035
• Include a feedback mechanism (sentiment measurement, aka, thumbs up/down, comments)

Feel free to comment. More details to follow.

[mattweber]

Honestly we just need documentation. The current state is horrible, I can’t count the number of times I have redirected people to Elastic’s 7.10 docs. I have tried to contribute some new docs in #416 but had it closed because I copied some of the Elastic’s open source docs from 7.10.2. (Same as original code fork). I would personally start by forking some of those original docs and/or allowing users to contribute it.

[navneet1v]

Apart from the above items mentioned, there are a lot of features that we support but are not present in the documentation. Example: We support different types of aggregation of GeoPoints like GeoBounds, GeoCentroid, GeoTile, GeoHashGrid etc but some of them is only mentioned(Geo Bounds, Geo Hash Gird are present in terms of documentation) this lead of create confusion among the users whether a particular feature is supported or not. These are provided from the start when OpenSearch was launched

Also, please add the steps/SOP for updating the documentation for a new user.

[reta]

AFAIK, the efforts in this direction are ongoing, please check [1]

[1] #419

[asfoorial]

I think our efforts should focus at this point on having a comprehensive documentation. For instance, a lot of details are missing related to DSL query, such as composite queries, analyzers and scripting.

As for the structure, the way elasticsearch is doing it seems to work just fine. We can kind of memic their docs structure since customers are already familiar with it.

[hdhalter]

Hi all, thanks so much for your feedback! We recently conducted an audit of the documentation, and have a good idea of the areas that are lacking (but are always happy for additional input). Our primary goal is to provide comprehensive documentation for OpenSearch. But we also know that the current structure, built for Open Distro, is not adequate to support all the new content and features we plan to offer in our documentation. So while we continue to expand our content, we'll also work on improving the experience. Please continue to provide feedback. Your suggestions are very helpful!

[lukas-vlcek]

I think the reference HTTP REST API should be one of the main focus areas. First it is essential for anyone who is trying to consume HTTP REST API, second it can create false impression that undocumented features are not supported. Please notice that the legacy OD4ES documentation can not be always used because it is dated and in some cases inaccurate. Instead of taking the content from OD4ES I would recommend creating a whole new writings. I am more than happy to contribute here.

From my experience the reference HTTP REST API needs to be documented by someone who is both familiar with the API itself (ie. is a frequent user/consumer of the API, ideally someone who implemented some HTTP client for OpenSearch) and can double-check the details in the source code itself (ideally an OpenSearch developer or someone with similar skills). This could be a challenge for doc writers if they do not have prior experience of working with OpenSearch. The best solution would be if OpenSearch developers themselves would spend time writing the reference docs along the way as they contribute to OpenSearch. This is not always happening now.

@navneet1v I believe the steps are documented here https://github.com/opensearch-project/documentation-website#contribute-content At least I was able to follow it and contribute to the documentation in the past.

[asfoorial]

I think the reference HTTP REST API should be one of the main focus areas. First it is essential for anyone who is trying to consume HTTP REST API, second it can create false impression that undocumented features are not supported. Please notice that the legacy OD4ES documentation can not be always used because it is dated and in some cases inaccurate. Instead of taking the content from OD4ES I would recommend creating a whole new writings. I am more than happy to contribute here.

From my experience the reference HTTP REST API needs to be documented by someone who is both familiar with the API itself (ie. is a frequent user/consumer of the API, ideally someone who implemented some HTTP client for OpenSearch) and can double-check the details in the source code itself (ideally an OpenSearch developer or someone with similar skills). This could be a challenge for doc writers if they do not have prior experience of working with OpenSearch. The best solution would be if OpenSearch developers themselves would spend time writing the reference docs along the way as they contribute to OpenSearch. This is not always happening now.

@navneet1v I believe the steps are documented here https://github.com/opensearch-project/documentation-website#contribute-content At least I was able to follow it and contribute to the documentation in the past.

Expanding on your point, I suggest to provide an OpenAPI specification (Swagger) for OpenSearch and make it part of the product.

[lukas-vlcek]

@asfoorial FYI I think there is some plan to use Smithy.

Direct quote from Smithy site:

What is the main difference between Smithy and OpenAPI?
The primary difference between Smithy and OpenAPI is that Smithy is protocol-agnostic, allowing Smithy to describe a broader range of services, metadata, and capabilities. Smithy can be used alongside OpenAPI by converting Smithy models to OpenAPI.

BTW: There is also rest-api-spec folder in OpenSearch repo but unfortunately I am not sure how much complete and updated it is.

[Naarcha-AWS]

Hey @lukas-vlcek and @asfoorial:

We do have an effort to document the HTTP Rest APIs manually for the time being. You can track the progress of that effort on issue #424. Currently, we have an external contributor documenting the Snapshot APIs.

As far as an OpenAPI or Smithy spec, Lukas is correct that we're currently making the switch to Smithy, which includes converting that Smithy spec to OpenAPI. You can track that progress here, but the creation of such a spec will likely take a few months. From a documentation perspective, the hope is that we can use the Smithy or OpenAPI specs to autogenerate the API reference documentation and include examples from that spec for other clients.

[hdhalter]

We've made significant improvements to the doc website: navigation improvements, feedback mechanism, and we continue to build content. Most REST APIs are completed. For the ones that are open, we have individual issues for them. (The team has not embraced SMITHY and we continue to look for alternatives.) We are currently working on rebuilding the doc site so it will have a new look and feel. This experience will be across the whole OpenSearch.org website.

I'm going to close this issue, as our changes will be tracked in individual issues or in the project-website repo.

[lukas-vlcek]

Thanks for the effort @hdhalter !