Merge pull request #7 from observIQ/plugin-terminology

Plugin terminology

CONTRIBUTING.md

@@ -68,13 +68,13 @@ A PR is considered to be **ready to merge** when:
 
 ## Design Choices
 
-Best practices for plugin development are documented below, but for changes to
+Best practices for developing a builtin plugin are documented below, but for changes to
 the core agent, we are happy to discuss proposals in the issue tracker.
 
-### Plugin Development
+### Builtin Plugin Development
 
-In order to build a plugin, follow these three steps:
-1. Build a unique plugin struct that satisfies the [`Plugin`](plugin/plugin.go) interface. This struct will define what your plugin does when executed in the pipeline.
+In order to write a builtin plugin, follow these three steps:
+1. Build a unique struct that satisfies the [`Plugin`](plugin/plugin.go) interface. This struct will define what your plugin does when executed in the pipeline.
 
 ```go
 type ExamplePlugin struct {

README.md

@@ -77,12 +77,15 @@ bplogagent graph --config './config.yaml' | dot -Tsvg -o /tmp/graph.svg && open
 ```
 
 ## What is a plugin?
-A plugin is the most basic unit of log monitoring. Each plugin fulfills a single responsibility, such as reading lines from a file, or parsing JSON from a field. These plugins are then chained together in a pipeline to achieve a desired result.
+A plugin is a unit of log monitoring. Each plugin fulfills a specific responsibility. Builtin plugins perform a single granular responsibility such as reading lines from a file, or parsing JSON from a field. These plugins are then chained together in a pipeline to achieve a desired result.
 
 For instance, a user may read lines from a file using the `file_input` plugin. From there, the results of this operation may be sent to a `regex_parser` plugin that creates fields based on a regex pattern. And then finally, these results may be sent to a `file_output` plugin that writes lines to a file.
 
 ## What plugins are available?
-For more information on what plugins are available and how to configure them, take a look at our [documentation](/docs/README.md).
+For more information on what plugins are available and how to configure them, take a look at our [documentation](/docs/README.md). 
+
+## Can I write my own plugins?
+Yes! You can [compose builtin plugins](/docs/plugins.md).
 
 ## Can I route logs through a proxy server?
 Yes. The agent will respect `http_proxy` and `https_proxy` environment variables, as defined in Golang's [net/http](https://golang.org/pkg/net/http/#ProxyFromEnvironment) package.

docs/README.md

@@ -27,7 +27,7 @@ A plugin is the most basic unit of log monitoring. Each plugin fulfills only a s
 
 For instance, a user may read lines from a file using the `file_input` plugin. From there, the results of this operation may be sent to a `regex_parser` plugin that creates fields based on a regex pattern. And then finally, these results may be sent to a `file_output` plugin that writes lines to a file.
 
-## What plugins are available?
+## What plugins are builtin?
 
 Input plugins:
 - [File input](/docs/plugins/file_input.md)
@@ -54,4 +54,4 @@ General purpose plugins:
 - [Copy to multiple outputs](/docs/plugins/copy.md)
 - [Router](/docs/plugins/router.md)
 
-Or take a look at our [Custom Plugins](/docs/custom_plugins.md) for a technology-specific, drop-in log parsing.
+Or create your own [Plugin](/docs/plugins.md) for a technology-specific use case.

docs/plugins.md

@@ -0,0 +1,49 @@
+# Plugins
+
+Plugins can be defined by using a file that contains a templated set of builtin plugins.
+
+For example, a very simple plugin for monitoring Apache Tomcat access logs could look like this:
+`tomcat.yaml`:
+```yaml
+---
+pipeline:
+  - id: tomcat_access_reader
+    type: file_input
+    include:
+      - {{ .path }}
+    output: tomcat_regex_parser
+
+  - id: tomcat_regex_parser
+    type: regex_parser
+    output: {{ .output }}
+    regex: '(?P<remote_host>[^\s]+) - (?P<remote_user>[^\s]+) \[(?P<timestamp>[^\]]+)\] "(?P<http_method>[A-Z]+) (?P<path>[^\s]+)[^"]+" (?P<http_status>\d+) (?P<bytes_sent>[^\s]+)'
+```
+
+Once a plugin config has been defined, it can be used in the log agent's config file with a `type` matching the filename of the plugin.
+
+`config.yaml`:
+```yaml
+---
+pipeline:
+  - id: tomcat_access
+    type: tomcat
+    output: stdout
+    path: /var/log/tomcat/access.log
+
+  - id: stdout
+    type: stdout
+```
+
+The `tomcat_access` plugin is replaced with the builtin plugins from the rendered config in `tomcat.yaml`.
+
+## Building a plugin
+
+Building a plugin is as easy as pulling out a set of builtin plugins in a working configuration file, then templatizing it with
+any parts of the config that need to be treated as variable. In the example of the Tomcat access log plugin above, that just means
+adding variables for `path` and `output`.
+
+Plugins use Go's [`text/template`](https://golang.org/pkg/text/template/) package for template rendering. All fields from
+the plugin configuration are available as variables in the templates except the `type` field.
+
+For the log agent to discover a plugin, it needs to be in the log agent's `plugin` directory. This can be set with the
+`--plugin_dir` argument. For a default installation, the plugin directory is located at `$BPLOGAGENT_HOME/plugins`.