doc: add conduit operator

changelog/2024-10-17-introducing-conduit-operator.md

@@ -0,0 +1,18 @@
+---
+slug: '2024-10-17-conduit-operator'
+title: Introducing Conduit Operator
+draft: false
+tags: [conduit, scale, kubernetes]
+---
+
+We're happy to announce that we recently released [Conduit Operator v0.0.1](https://github.com/ConduitIO/conduit-operator/releases/tag/v0.0.1), a Kubernetes operator that extends the Kubernetes API to enhance the management of Conduit instances on our own [Conduit Platform](https://meroxa.io).
+
+<!--truncate-->
+
+Using this operator, we will be able to significantly reduce the time needed to deploy new connectors, along with many other features.
+
+For **more information**, check out [its documentation page](/docs/scaling/conduit-operator).
+
+:::info
+Conduit Operator is open source and available on [GitHub](https://github.com/ConduitIO/conduit-operator).
+:::

docs/3-scaling/0-conduit-operator.mdx

@@ -0,0 +1,157 @@
+---
+title: "Conduit Operator"
+toc_max_heading_level: 6
+---
+
+The **Conduit Operator** is a [Kubernetes operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) designed to simplify the management and orchestration of Conduit instances. The operator extends the [Kubernetes API](https://kubernetes.io/docs/concepts/overview/kubernetes-api/) with a [custom resource](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) for Conduit. 
+
+This custom resource allows users to define and manage Conduit instances declaratively, enabling seamless integration with Kubernetes' native features such as scaling, monitoring, and logging.
+
+### Overview
+
+Conduit pipelines are represented as Conduit custom resources, where each pipeline will be provisioned 
+as a distinct Conduit instance with its own lifecycle. 
+
+The Conduit custom resource definition format is very similar to that of [pipeline configurations](/docs/using/pipelines/configuration-file):
+
+```yaml
+apiVersion: operator.conduit.io/v1
+kind: Conduit
+metadata:
+  name: conduit-generator
+spec:
+  running: true
+  name: generator.log
+  description: generator pipeline
+  connectors:
+    - name: source-connector
+      type: source
+      plugin: builtin:generator
+      settings:
+        - name: format.type
+          value: structured
+        - name: format.options
+          value: "id:int,name:string,company:string,trial:bool"
+        - name: recordCount
+          value: "3"
+    - name: destination-connector
+      type: destination
+      plugin: builtin:log
+```
+
+### Quickstart
+
+:::info
+To get started with the Conduit Operator, follow the steps described on the [Conduit Operator GitHub repository](https://github.com/ConduitIO/conduit-operator).
+:::
+
+### Installing standalone connectors
+
+#### `plugin` and `pluginVersion`
+
+A standalone connector can be installed by providing the connector name via `plugin`, and version via `pluginVersion`. 
+
+The operator will then install the connector and create the necessary resources to run it. These connectors need to be referred by their GitHub `organization/repository` names.
+
+:::info
+Plugin version can optionally be specified or the latest will be used. 
+:::
+
+#### Example
+
+Here's a full example using `conduitio/conduit-connector-generator` as `plugin` and `v0.5.0` as `pluginVersion`:
+
+```yaml
+apiVersion: operator.conduit.io/v1
+kind: Conduit
+metadata:
+  name: conduit-generator
+spec:
+  running: true
+  name: generator.log
+  description: generator pipeline
+  connectors:
+    - name: source-connector
+      type: source
+      plugin: conduitio/conduit-connector-generator
+      pluginVersion: v0.5.0 # optional (default: latest)
+      settings:
+        - name: format.type
+          value: structured
+        - name: format.options
+          value: "id:int,name:string,company:string,trial:bool"
+        - name: recordCount
+          value: "3"
+    - name: destination-connector
+      type: destination
+      plugin: conduitio/conduit-connector-log
+      pluginVersion: v0.3.0
+```
+
+### Schema Support
+
+With schema support added to Conduit on [v0.11.0](/changelog/2024-08-19-conduit-0-11-0-release), it is also possible to use multiple conduit instances sharing their schema via another schema registry.
+
+Details are provided directly via the Conduit resource. `basicAuthUser`, `basicAuthPassword` and `bearerToken` can be provided as secrets but will be copied over into controller owned secrets for management, and ease of use.
+
+#### Example
+
+```yaml
+apiVersion: operator.conduit.io/v1alpha
+kind: Conduit
+metadata:
+  name: conduit-generator-schema-registry
+spec:
+  running: true
+  name: generator.standalone.log
+  description: generator pipeline
+  schemaRegistry:
+    url: http://schema-registry-apicurio-registry.tenant:8080/apis/ccompat/v7
+    # These can be provided as secrets:
+
+    # basicAuthUser: ${BASIC_AUTH_USER}
+    # basicAuthPassword: ${BASIC_AUTH_PASSWORD}
+    # bearerToken: ${BEARER_TOKEN}
+  connectors:
+    - name: source-connector
+      type: source
+      plugin: conduitio/conduit-connector-generator
+      settings:
+        - name: format.type
+          value: structured
+        - name: format.options.id
+          value: "int"
+        - name: format.options.name
+          value: "string"
+        - name: format.options.company
+          value: "string"
+        - name: format.options.trial
+          value: "bool"
+        - name: recordCount
+          value: "3"
+    - name: destination-connector
+      type: destination
+      plugin: conduitio/conduit-connector-log
+      pluginVersion: v0.4.0
+```
+
+### Deploying
+
+The operator provides a [helm chart](https://github.com/ConduitIO/conduit-operator/blob/main/charts/conduit-operator) that can be used for deployment on any cluster. Additional metadata can be injected into each provisioned conduit instance via the `controller.conduitMetadata` configuration.
+
+For example, to instruct a prometheus instance to scrape each Conduit instance metrics:
+
+```yaml
+controller:
+  conduitMetadata:
+    podAnnotations:
+      prometheus.io/scrape: true
+      prometheus.io/path: /metrics
+      prometheus.io/port: 8080
+```
+
+For more configuration options see [charts/conduit-operator/values.yaml](https://github.com/ConduitIO/conduit-operator/blob/main/charts/conduit-operator/values.yaml).
+
+:::tip
+For a low-code experience, and many other Enterprise features, we recommend you use the [Conduit Platform](https://meroxa.io).
+:::

src/sidebars/sidebars.ts

@@ -23,9 +23,8 @@ const sidebars: SidebarsConfig = {
     {type: 'autogenerated', dirName: '2-developing', customProps: { slugBase: 'foo' }},
 
     {type: 'html', value: '<li class="sidebar-header">Scaling Conduit</li>'},
-    {type: "link",label: "Conduit Platform",href: "https://meroxa.io"},
-    {type: "link",label: "Conduit Operator",href: "https://github.com/ConduitIO/conduit-operator"},
     {type: 'autogenerated', dirName: '3-scaling'},
+    {type: "link",label: "Conduit Platform",href: "https://meroxa.io"},
 
     {type: 'html', value: '<li class="sidebar-header">Community</li>'},
     {type: "link", label: "Discord Community",href: "https://discord.meroxa.com"},