Skip to content

Commit

Permalink
documentation for file connectors (#71)
Browse files Browse the repository at this point in the history
* documentation for file connectors

* addressing reviews
  • Loading branch information
maha-hajja authored Jan 18, 2022
1 parent 7f7808f commit c17b8a3
Showing 1 changed file with 38 additions and 0 deletions.
38 changes: 38 additions & 0 deletions pkg/plugins/file/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
### Overview
The Conduit File plugin provides both source and destination connectors,
allowing for a file to be either a source or a destination in a Conduit
pipeline.

### How it works
The Source connector listens for changes appended to the source file and
sends records with the changes.
The Destination connector receives records and writes them to a file.

#### Source
The Source connector only cares to have a valid path, even if the file
doesn't exist, it will still run and wait until a file with the configured
name is there, then it will start listening to changes and sending records.

#### Destination
The Destination connector will create the file if it doesn't exist, and
records with changes will be appended to the destination file when received.

### How to build it
Run `make build-file-plugin`.

### Configuration

| name | part of | description | required | default value |
|------|--------------------|---------------------|----------|---------------|
|`path`|destination, source |The path to the file |true | |

### Limitations
* The Source connector only detects appended changes to the file, so it
doesn't detect deletes or edits.
* The connectors can only access local files on the machine where Conduit
is running. So, running Conduit on a server means it can't access a file
on your local machine.
* Currently, only works reliably with text files (may work with non-text
files, but not guaranteed)


0 comments on commit c17b8a3

Please sign in to comment.