Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync code comments with published docs #1571

Merged
merged 12 commits into from
Dec 2, 2023
2 changes: 1 addition & 1 deletion docs/configuration/plugins/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ For more information please click on the plugin name
| **[SumoLogic](outputs/sumologic/)** | outputs | Send your logs to Sumologic | GA | [1.8.0](https://github.com/SumoLogic/fluentd-output-sumologic/releases/tag/1.8.0) |
| **[Syslog](outputs/syslog/)** | outputs | Output plugin writes events to syslog | GA | [0.9.0.rc.8](https://github.com/cloudfoundry/fluent-plugin-syslog_rfc5424) |
| **[Syslog-NG Match](syslogng-filters/match/)** | syslogng-filters | Selectively keep records | GA | [more info](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/65#TOPIC-1829159) |
| **[Syslog-NG Parser](syslogng-filters/parser/)** | syslogng-filters | Parse data from records | GA | [more info](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/90) |
| **[Syslog-NG Parser](syslogng-filters/parser/)** | syslogng-filters | Parse data from records | GA | [more info](https://axoflow.com/docs/axosyslog-core/chapter-parsers/) |
| **[Syslog-NG Rewrite](syslogng-filters/rewrite/)** | syslogng-filters | Rewrite parts of the message | GA | [more info](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/77) |
| **[Authentication config for syslog-ng outputs](syslogng-outputs/auth/)** | syslogng-outputs | Authentication config for syslog-ng outputs | Testing | []() |
| **[disk-buffer configuration](syslogng-outputs/disk_buffer/)** | syslogng-outputs | disk-buffer configuration | Testing | [](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/32#kanchor2338) |
Expand Down
91 changes: 45 additions & 46 deletions docs/configuration/plugins/syslogng-filters/match.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,21 @@ generated_file: true

# Match
## Overview
Match filters can be used to select the log records to process. These filters have the same options and syntax as [syslog-ng flow match expressions]({{< relref "/docs/logging-operator/configuration/plugins/syslog-ng-filters/match.md" >}}).

{{< highlight yaml >}}

filters:
- match:
or:
- regexp:
value: json.kubernetes.labels.app.kubernetes.io/name
pattern: apache
type: string
- regexp:
value: json.kubernetes.labels.app.kubernetes.io/name
pattern: nginx
type: string

{{</ highlight >}}
Match filters can be used to select the log records to process. These filters have the same options and syntax as the [syslog-ng flow match expressions]({{< relref "/docs/configuration/plugins/syslog-ng-filters/match.md" >}}).

{{< highlight yaml >}}
filters:
- match:
or:
- regexp:
value: json.kubernetes.labels.app.kubernetes.io/name
pattern: apache
type: string
- regexp:
value: json.kubernetes.labels.app.kubernetes.io/name
pattern: nginx
type: string
{{</ highlight >}}

## Configuration
## MatchExpr
Expand All @@ -48,7 +46,9 @@ Default: -

## Regexp Directive

Specify filtering rule. For details, see the [syslog-ng documentation](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/68#TOPIC-1829171).

Specify filtering rule. For details, see the [AxoSyslog Core documentation](https://axoflow.com/docs/axosyslog-core/chapter-manipulating-messages/customizing-message-format/reference-template-functions/#template-function-list)


### pattern (string, required) {#regexp directive-pattern}

Expand All @@ -70,49 +70,48 @@ Default: -

### flags ([]string, optional) {#regexp directive-flags}

Pattern flags
Pattern flags. For details, see the [AxoSyslog Core documentation](https://axoflow.com/docs/axosyslog-core/chapter-manipulating-messages/regular-expressions/reference-regexp-types/regexp-flags-options/)

Default: -

### type (string, optional) {#regexp directive-type}

Pattern type
Pattern type. For details, see the [AxoSyslog Core documentation](https://axoflow.com/docs/axosyslog-core/chapter-manipulating-messages/regular-expressions/reference-regexp-types/regexp-type-options/)

Default: -


#### Example `Regexp` filter configurations
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:

name: demo-flow

spec:

filters:
- match:
regexp:
- value: first
pattern: ^5\d\d$
#### Example `Regexp` filter configurations
```yaml
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: demo-flow
spec:
filters:
- match:
regexp:
- value: first
pattern: ^5\d\d$
match: {}
localOutputRefs:
- demo-output
localOutputRefs:
- demo-output
```

```

#### Syslog-NG Config Result
```

log {
source(main_input);
filter {
match("^5\d\d$" value("first"));
};
destination(output_default_demo-output);
#### Syslog-NG Config Result
```
log {
source(main_input);
filter {
match("^5\d\d$" value("first"));
};
destination(output_default_demo-output);
};

```

```

---
105 changes: 65 additions & 40 deletions docs/configuration/plugins/syslogng-filters/parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,109 +4,134 @@ weight: 200
generated_file: true
---

# [Parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/82#TOPIC-1829229)
# [Parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/)
## Overview
Parser filters can be used to extract key-value pairs from message data. Logging operator currently supports the following parsers:

- [regexp](#regexp)
- [syslog-parser](#syslog)
Parser filters can be used to extract key-value pairs from message data. Logging operator currently supports the following parsers:

## Regexp parser {#regexp}
- [metrics-probe](#metricsprobe)
- [regexp](#regexp)
- [syslog-parser](#syslog)

The regexp parser can use regular expressions to parse fields from a message.
## Regexp parser {#regexp}

{{< highlight yaml >}}
The regexp parser can use regular expressions to parse fields from a message.

filters:
- parser:
regexp:
patterns:
- ".*test_field -> (?<test_field>.*)$"
prefix: .regexp.
{{< highlight yaml >}}
filters:
- parser:
regexp:
patterns:
- ".*test_field -> (?<test_field>.*)$"
prefix: .regexp.
{{</ highlight >}}

{{</ highlight >}}
For details, see the [documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/).

For details, see the [syslog-ng documentation](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/91#TOPIC-1829263).
## Syslog parser {#syslog}

## Syslog parser {#syslog}
The syslog parser can parse syslog messages. For details, see the [documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-syslog/).

The syslog parser can parse syslog messages. For details, see the [syslog-ng documentation](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide/83#TOPIC-1829231).
{{< highlight yaml >}}
filters:
- parser:
syslog-parser: {}
{{</ highlight >}}

{{< highlight yaml >}}

filters:
- parser:
syslog-parser: {}

{{</ highlight >}}

## Configuration
## [Parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/82#TOPIC-1768819)
## [Parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/)

### regexp (*RegexpParser, optional) {#[parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/)-regexp}

### regexp (*RegexpParser, optional) {#[parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/82#topic-1768819)-regexp}
The regular expression patterns that you want to find a match. `regexp-parser()` supports multiple patterns, and stops the processing at the first successful match. For details, see the [regexp-parser() documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/parser-regexp-options/#patterns).

Default: -

### syslog-parser (*SyslogParser, optional) {#[parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/82#topic-1768819)-syslog-parser}
### syslog-parser (*SyslogParser, optional) {#[parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/)-syslog-parser}

Parse message as a [syslog message](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-syslog/).

Default: -

### metrics-probe (*MetricsProbe, optional) {#[parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/82#topic-1768819)-metrics-probe}
### metrics-probe (*MetricsProbe, optional) {#[parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/)-metrics-probe}

Counts the messages that pass through the flow, and creates labeled stats counters based on the fields of the passing messages. For details, see the [documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/metrics-probe/).

Default: -


## [Regexp parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/90)
## [Regexp parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/)

### patterns ([]string, required) {#[regexp parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/90)-patterns}
### patterns ([]string, required) {#[regexp parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/)-patterns}

The regular expression patterns that you want to find a match. regexp-parser() supports multiple patterns, and stops the processing at the first successful match.
The regular expression patterns that you want to find a match. `regexp-parser()` supports multiple patterns, and stops the processing at the first successful match. For details, see the [regexp-parser() documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/parser-regexp-options/#patterns).

Default: -

### prefix (string, optional) {#[regexp parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/90)-prefix}
### prefix (string, optional) {#[regexp parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/)-prefix}

Insert a prefix before the name part of the parsed name-value pairs to help further processing.
Insert a prefix before the name part of the parsed name-value pairs to help further processing. For details, see the [regexp-parser() documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/parser-regexp-options/#prefix).

Default: -

### template (string, optional) {#[regexp parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/90)-template}
### template (string, optional) {#[regexp parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/)-template}

Specify a template of the record fields to match against.
Specify a template of the record fields to match against. For details, see the [regexp-parser() documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/parser-regexp-options/#template).

Default: -

### flags ([]string, optional) {#[regexp parser](https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.36/administration-guide/90)-flags}
### flags ([]string, optional) {#[regexp parser](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/)-flags}

Pattern flags
Flags to influence the behavior of the [regexp-parser()](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/parser-regexp-options/). For details, see the [regexp-parser() documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-regexp/parser-regexp-options/#flags).

Default: -


## SyslogParser

Parse message as a [syslog message](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-syslog/).

### flags ([]string, optional) {#syslogparser-flags}

Pattern flags
Flags to influence the behavior of the [syslog-parser()](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-syslog/parser-syslog-options/). For details, see the [syslog-parser() documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/parser-syslog/parser-syslog-options/#flags).

Default: -


## MetricsProbe


Counts the messages that pass through the flow, and creates labeled stats counters based on the fields of the passing messages. For details, see the [documentation of the AxoSyslog syslog-ng distribution](https://axoflow.com/docs/axosyslog-core/chapter-parsers/metrics-probe/).

{{< highlight yaml>}}SyslogNGFlow
apiVersion: logging.banzaicloud.io/v1beta1
kind: SyslogNGFlow
metadata:
name: flow-mertrics-probe
namespace: default
spec:
filters:
- parser:
metrics-probe:
key: "flow_events"
labels:
namespace: "${json.kubernetes.namespace_name}"{{< /highlight >}}


### - (struct{}, required) {#metricsprobe--}

Default: -

### key (string, optional) {#metricsprobe-key}

The name of the counter to create. Note that the value of this option is always prefixed with syslogng_, so for example key("my-custom-key") becomes syslogng_my-custom-key.
The name of the counter to create. Note that the value of this option is always prefixed with `syslogng_`, so for example `key("my-custom-key")` becomes `syslogng_my-custom-key`.

Default: -

### labels (ArrowMap, optional) {#metricsprobe-labels}

The labels used to create separate counters, based on the fields of the messages processed by metrics-probe(). The keys of the map are the name of the label, and the values are syslog-ng templates.
The labels used to create separate counters, based on the fields of the messages processed by `metrics-probe()`. The keys of the map are the name of the label, and the values are syslog-ng templates.

Default: -

Expand Down
Loading
Loading