-
Notifications
You must be signed in to change notification settings - Fork 7
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: new documentation structure + other issues (#168)
* add headers and refactor * remove unused images * different structure * add link to conduit platform * tweak css * style differently the first element * fix menu items on small viewports * disable until we finish so we have preview deploys * fix flickering * another iteration * leave sidebar tidy * structure pipeline pages * update header nav bar * document pipeline statuses * document CLI flags * structure finished * add link to opencdc * consistent formatting * better urls * update some links and rename pages * upate links * fix more broken links * fix redirect * finish broken links * remove conclusion * fix statuses definition * update again * update links * add some redirects * tweak implication * update specs * sort redirects * fix redirect * fix broken links
- Loading branch information
Showing
140 changed files
with
526 additions
and
438 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
docs/features/pipeline-semantics.mdx → .../1-core-concepts/1-pipeline-semantics.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| { | ||
| "label": "Core concepts" | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| --- | ||
| title: "Core concepts" | ||
| slug: '/core-concepts' | ||
| --- | ||
|
|
||
| ## Pipeline | ||
|
|
||
| A pipeline receives records from one or multiple source connectors, pushes them through zero | ||
| or multiple processors until they reach one or multiple destination connectors. | ||
|
|
||
| ## Connector | ||
|
|
||
| A connector is the internal entity that communicates with a connector plugin and either pushes | ||
| records from the plugin into the pipeline (source connector) or the other way around | ||
| (destination connector). | ||
|
|
||
| ## Connector plugin | ||
|
|
||
| Sometimes also referred to as "plugin", is an external process which communicates with Conduit | ||
| and knows how to read/write records from/to a data source/destination (e.g. a database). | ||
|
|
||
| ## Processor | ||
|
|
||
| A component that executes an operation on a single record that flows through the pipeline. | ||
| It can either change the record or filter it out based on some criteria. | ||
|
|
||
| ## OpenCDC Record | ||
|
|
||
| A record represents a single piece of data that flows through a pipeline (e.g. one database row). | ||
| [More info here](/docs/using/opencdc-record). | ||
|
|
||
| ## Collection | ||
|
|
||
| A generic term used in Conduit to describe an entity in a 3rd party system from which records | ||
| are read from or to which records they are written to. Examples are: topics (in Kafka), tables | ||
| (in a database), indexes (in a search engine), collections (in NoSQL databases), etc. | ||
|
|
||
|  |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| --- | ||
| title: 'How to configure Conduit' | ||
| sidebar_label: 'Configuration' | ||
| slug: '/configuration' | ||
| --- | ||
|
|
||
| Conduit accepts CLI flags, environment variables and a configuration file to | ||
| configure its behavior. Each CLI flag has a corresponding environment variable | ||
| and a corresponding field in the configuration file. Conduit uses the value for | ||
| each configuration option based on the following priorities: | ||
|
|
||
| ## CLI flags | ||
|
|
||
| **CLI flags** (highest priority) - if a CLI flag is provided it will always be | ||
| respected, regardless of the environment variable or configuration file. To | ||
| see a full list of available flags run `conduit --help`: | ||
|
|
||
|
|
||
| ```bash | ||
| $ conduit --help | ||
| Usage of conduit: | ||
| -api.enabled | ||
| enable HTTP and gRPC API (default true) | ||
| -config string | ||
| global config file (default "conduit.yaml") | ||
| -connectors.path string | ||
| path to standalone connectors' directory (default "./connectors") | ||
| -db.badger.path string | ||
| path to badger DB (default "conduit.db") | ||
| -db.postgres.connection-string string | ||
| postgres connection string, may be a database URL or in PostgreSQL keyword/value format | ||
| -db.postgres.table string | ||
| postgres table in which to store data (will be created if it does not exist) (default "conduit_kv_store") | ||
| -db.sqlite.path string | ||
| path to sqlite3 DB (default "conduit.db") | ||
| -db.sqlite.table string | ||
| sqlite3 table in which to store data (will be created if it does not exist) (default "conduit_kv_store") | ||
| -db.type string | ||
| database type; accepts badger,postgres,inmemory,sqlite (default "badger") | ||
| -grpc.address string | ||
| address for serving the gRPC API (default ":8084") | ||
| -http.address string | ||
| address for serving the HTTP API (default ":8080") | ||
| -log.format string | ||
| sets the format of the logging; accepts json, cli (default "cli") | ||
| -log.level string | ||
| sets logging level; accepts debug, info, warn, error, trace (default "info") | ||
| -pipelines.error-recovery.backoff-factor int | ||
| backoff factor applied to the last delay (default 2) | ||
| -pipelines.error-recovery.max-delay duration | ||
| maximum delay before restart (default 10m0s) | ||
| -pipelines.error-recovery.max-retries int | ||
| maximum number of retries (default -1) | ||
| -pipelines.error-recovery.max-retries-window duration | ||
| amount of time running without any errors after which a pipeline is considered healthy (default 5m0s) | ||
| -pipelines.error-recovery.min-delay duration | ||
| minimum delay before restart (default 1s) | ||
| -pipelines.exit-on-degraded | ||
| exit Conduit if a pipeline enters a degraded state | ||
| -pipelines.path string | ||
| path to the directory that has the yaml pipeline configuration files, or a single pipeline configuration file (default "./pipelines") | ||
| -processors.path string | ||
| path to standalone processors' directory (default "./processors") | ||
| -schema-registry.confluent.connection-string string | ||
| confluent schema registry connection string | ||
| -schema-registry.type string | ||
| schema registry type; accepts builtin,confluent (default "builtin") | ||
| -version | ||
| prints current Conduit version | ||
| ``` | ||
|
|
||
| ## Environment variables | ||
|
|
||
| **Environment variables** (lower priority) - an environment variable is only | ||
| used if no CLI flag is provided for the same option. Environment variables | ||
| have the prefix `CONDUIT` and contain underscores instead of dots and | ||
| hyphens (e.g. the flag `-db.postgres.connection-string` corresponds | ||
| to `CONDUIT_DB_POSTGRES_CONNECTION_STRING`). | ||
|
|
||
| ## Configuration file | ||
|
|
||
| **Configuration file** (lowest priority) - Conduit by default loads the | ||
| file `conduit.yaml` placed in the same folder as Conduit. The path to the file | ||
| can be customized using the CLI flag `-config`. It is not required to provide | ||
| a configuration file and any value in the configuration file can be overridden | ||
| by an environment variable or a flag. The file content should be a YAML | ||
| document where keys can be hierarchically split on `.`. For example: | ||
|
|
||
| ```yaml | ||
| db: | ||
| type: postgres # corresponds to flag -db.type and env variable CONDUIT_DB_TYPE | ||
| postgres: | ||
| connection-string: postgres://localhost:5432/conduitdb # -db.postgres.connection-string or CONDUIT_DB_POSTGRES_CONNECTION_STRING | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.