docs overhaul: render flyteidl under the /api/ path

.github/workflows/tests.yml

@@ -69,6 +69,9 @@ jobs:
       - name: Build the documentation
         working-directory: ${{ github.workspace }}/flyte
         shell: bash -el {0}
+        env:
+          # this is a fake key for algolia docsearch to make docs build process pass
+          DOCSEARCH_API_KEY: fake_docsearch_api_key
         run: |
           conda activate monodocs-env
           make docs

.gitignore

@@ -31,8 +31,8 @@ vendor/
 **/bin/
 docs/_tags/
 docs/api/flytectl
-docs/protos
 docs/api/flytekit
+docs/api/flyteidl
 docs/flytesnacks
 docs/examples
 docs/_src

docs/api/index.md

@@ -10,13 +10,20 @@
 Flytekit is the main Python SDK for building workflows.
 :::
 
-:::{grid-item-card} {octicon}`workflow` Flytectl SDK
+:::{grid-item-card} {octicon}`workflow` Flytectl CLI
 :link: flytectl/docs_index
 :link-type: doc
 
 Flytectl is a Golang binary that can be installed on any platform supported by Golang.
 :::
 
+:::{grid-item-card} {octicon}`code` FlyteIDL
+:link: flyteidl/docs_index
+:link-type: doc
+
+Flyte’s core specification language.
+:::
+
 ::::
 
 ```{toctree}
@@ -25,4 +32,5 @@ Flytectl is a Golang binary that can be installed on any platform supported by G
 
 flytekit <flytekit/docs_index>
 flytectl <flytectl/docs_index>
-```
+flyteidl <flyteidl/docs_index>
+```

docs/community/contribute_docs.md

@@ -13,7 +13,7 @@ The Flyte documentation comprises the following types:
 * **API documentation:**
   * {doc}`flytekit <../api/flytekit/docs_index>`
   * {doc}`flytectl <../api/flytectl/docs_index>`
-  * {doc}`flyteidl <../reference_flyteidl>`
+  * {doc}`flyteidl <../api/flyteidl/docs_index>`
 * **{ref}`Tutorials <tutorials>`:** Longer, more advanced guides that use multiple Flyte features to solve real-world problems. Some tutorials may require extra setup, while others can only run on larger clusters.
 * **{ref}`Integrations examples <integrations>`:** These examples showcase how to use the Flyte plugins that integrate with the broader data and machine learning ecosystem.
 * **{ref}`Deployment documentation <deployment>`:** Guidance on deploying and configuring the Flyte backend.

docs/conf.py

@@ -288,6 +288,16 @@
     "flytectl/gen/flytectl_demo_teardown": "../../api/flytectl/gen/flytectl_demo_teardown",
     "flytectl/gen/flytectl_demo_exec": "../../api/flytectl/gen/flytectl_demo_exec",
     "flytectl/gen/flytectl_demo_reload": "../../api/flytectl/gen/flytectl_demo_reload",
+
+    # flyteidl
+    "reference_flyteidl": "../../api/flyteidl/docs_index.html",
+    "protos/docs/core/core": "../../api/flyteidl/docs/core/core.html",
+    "protos/docs/admin/admin": "../../api/flyteidl/docs/admin/admin.html",
+    "protos/docs/service/service": "../../api/flyteidl/docs/service/service.html",
+    "protos/docs/datacatalog/datacatalog": "../../api/flyteidl/docs/datacatalog/datacatalog.html",
+    "protos/docs/event/event": "../../api/flyteidl/docs/event/event.html",
+    "protos/docs/plugins/plugins": "../../api/flyteidl/docs/plugins/plugins.html",
+    "protos/README": "../../api/flyteidl/contributing.html",
 }
 
 
@@ -376,6 +386,9 @@
 html_theme_options = {
     # custom flyteorg pydata theme options
     "github_url": "https://github.com/flyteorg/flyte",
+    "logo": {
+        "text": "Flyte Docs",
+    },
     "icon_links": [
         {
             "name": "GitHub",
@@ -476,9 +489,9 @@
 tags_overview_title = "Pages by tags"
 
 # Algolia Docsearch credentials
-docsearch_app_id = os.getenv("DOCSEARCH_APP_ID")
+docsearch_app_id = "WLG0MZB58Q"
 docsearch_api_key = os.getenv("DOCSEARCH_API_KEY")
-docsearch_index_name = os.getenv("DOCSEARCH_INDEX_NAME")
+docsearch_index_name = "flyte"
 
 # -- Options for intersphinx extension ---------------------------------------
 
@@ -529,7 +542,7 @@
 
 # Pattern for replacing all ref/doc labels that point to protos/docs with /protos/docs
 PROTO_REF_PATTERN = r"([:<])(protos/docs)"
-PROTO_REF_REPLACE = r"\1/protos/docs"
+PROTO_REF_REPLACE = r"\1/api/flyteidl/docs"
 
 # These patterns are used to replace values in source files that are imported
 # from other repos.
@@ -549,7 +562,7 @@
     r"<_tags/DistributedComputing>": r"</_tags/DistributedComputing>",
     r"{ref}`bioinformatics <bioinformatics>`": r"bioinformatics",
     PROTO_REF_PATTERN: PROTO_REF_REPLACE,
-    r"/protos/docs/service/index": r"/protos/docs/service/service"
+    r"/protos/docs/service/index": r"/api/flyteidl/docs/service/service"
 }
 
 # r"<environment_setup>": r"</flytesnacks/environment_setup>",
@@ -626,8 +639,7 @@
         "name": "flyteidl",
         "source": "../flyteidl",
         "docs_path": "protos",
-        "dest": "protos",  # to stay compatible with flyteidl docs path naming
-        "cmd": ["cp", "../flyteidl/README.md", "protos/README.md"],
+        "dest": "api/flyteidl",
         "local": True,
     }
 ]

docs/index.md

@@ -102,7 +102,7 @@ Below are the API reference to the different components of Flyte:
   - Flyte's official Python SDK.
 * - {doc}`FlyteCTL <api/flytectl/docs_index>`
   - Flyte's command-line interface for interacting with a Flyte cluster.
-* - {doc}`FlyteIDL <reference_flyteidl>`
+* - {doc}`FlyteIDL <api/flyteidl/docs_index>`
   - Flyte's core specification language.
 ```
 
@@ -135,6 +135,5 @@ flytesnacks/tutorials/index
 flytesnacks/integrations/index
 deployment/index
 api/index
-reference_flyteidl
 community/index
 ```

flyteidl/README.md

@@ -1,6 +1,6 @@
 # Flyteidl
 
-This is one of the core repositories of Flyte. It contains the Specification of the Flyte Language using protobuf messages, the Backend API specification in gRPC, and Swagger REST. The repo contains the generated clients and protocol message structures in multiple languages. Along with the generated code, the repository also contains the Golang clients for Flyte's backend APIs (the services grouped under FlyteAdmin).
+This is one of the core components of Flyte. It contains the Specification of the Flyte Language using protobuf messages, the Backend API specification in gRPC, and Swagger REST. The repo contains the generated clients and protocol message structures in multiple languages. Along with the generated code, the repository also contains the Golang clients for Flyte's backend APIs (the services grouped under FlyteAdmin).
 
 
 [![Slack](https://img.shields.io/badge/slack-join_chat-white.svg?logo=slack&style=social)](https://slack.flyte.org)
@@ -11,69 +11,4 @@ This is one of the core repositories of Flyte. It contains the Specification of
 
 ## Contributing to Flyteidl
 
-## Tooling for Flyteidl
-
-1. Run ``make download_tooling`` to install generator dependencies.
-
-```bash
-   make download_tooling
-```
-
-2. Ensure Docker is installed locally.
-3. Run ``make generate`` to generate all the code, mock client, and docs for FlyteAdmin Service.
-
-```bash
-    make generate
-```
-
-4. To add new dependencies for documentation generation, modify ``doc-requirements.in`` and run
-
-```bash
-   make doc-requirements.txt
-```
-
-## Docs structure
-
-The index.rst files for protos are arranged in parallel under the ``docs`` folder.
-All the proto definitions are within ``protos/flyteidl`` and their corresponding docs are in ``protos/docs``.
-
-```
-docs
-├── admin
-│   ├── admin.rst
-│   └── index.rst
-├── core
-│   ├── core.rst
-│   └── index.rst
-├── datacatalog
-│   ├── datacatalog.rst
-│   └── index.rst
-├── event
-│   ├── event.rst
-│   └── index.rst
-├── plugins
-│   ├── index.rst
-│   └── plugins.rst
-├── service
-│   ├── index.rst
-│   └── service.rst
-```
-
-Each module in protos has a module in docs with the same name.
-For example: ``protos/flyteidl/core`` has a module ``protos/docs/core`` under the ``docs`` folder which has the corresponding index and documentation files.
-
-
-## Generating Documentation
-
-* If a new module is to be introduced, follow the structure for core files in `generate_protos.sh` file which helps generate the core documentation from its proto files.
-```
-     core_proto_files=`ls protos/flyteidl/core/*.proto |xargs`
-     # Remove any currently generated file
-     ls -d protos/docs/core/* | grep -v index.rst | xargs rm
-     protoc --doc_out=protos/docs/core --doc_opt=restructuredtext,core.rst -I=protos `echo $core_proto_files`
-```
-
-* ``make generate`` generates the modified rst files.
-
-* ``make html`` generates the Sphinx documentation from the docs folder that uses the modified rst files.
-
+See the [contributing docs](protocs/contributing.md) for more information.

flyteidl/protos/docs/contributing.md

@@ -0,0 +1,68 @@
+## Contributing to FlyteIDL
+
+## Install Tooling
+
+1. Run ``make download_tooling`` to install generator dependencies.
+
+```bash
+   make download_tooling
+```
+
+2. Ensure Docker is installed locally.
+3. Run ``make generate`` to generate all the code, mock client, and docs for FlyteAdmin Service.
+
+```bash
+    make generate
+```
+
+4. To add new dependencies for documentation generation, modify ``doc-requirements.in`` and run
+
+```bash
+   make doc-requirements.txt
+```
+
+## Docs structure
+
+The index.rst files for protos are arranged in parallel under the ``docs`` folder.
+All the proto definitions are within ``protos/flyteidl`` and their corresponding docs are in ``protos/docs``.
+
+```
+docs
+├── admin
+│   ├── admin.rst
+│   └── index.rst
+├── core
+│   ├── core.rst
+│   └── index.rst
+├── datacatalog
+│   ├── datacatalog.rst
+│   └── index.rst
+├── event
+│   ├── event.rst
+│   └── index.rst
+├── plugins
+│   ├── index.rst
+│   └── plugins.rst
+├── service
+│   ├── index.rst
+│   └── service.rst
+```
+
+Each module in protos has a module in docs with the same name.
+For example: ``protos/flyteidl/core`` has a module ``protos/docs/core`` under the ``docs`` folder which has the corresponding index and documentation files.
+
+
+## Generating Documentation
+
+* If a new module is to be introduced, follow the structure for core files in `generate_protos.sh` file which helps generate the core documentation from its proto files.
+```
+     core_proto_files=`ls protos/flyteidl/core/*.proto |xargs`
+     # Remove any currently generated file
+     ls -d protos/docs/core/* | grep -v index.rst | xargs rm
+     protoc --doc_out=protos/docs/core --doc_opt=restructuredtext,core.rst -I=protos `echo $core_proto_files`
+```
+
+* ``make generate`` generates the modified rst files.
+
+* ``make html`` generates the Sphinx documentation from the docs folder that uses the modified rst files.
+