Is it possible to use fluentd rewrite_tag_filter output plugin with logging-operator?

[psyhomb]

Is your feature request related to a problem? Please describe.
I would like to split an application’s logs into multiple streams.

Describe the solution you'd like
https://docs.fluentd.org/output/rewrite_tag_filter

[psyhomb]

I've managed to accomplish this by using existing grep filter and multiple Flow objects with the same Output object.

Flow object 1 - Filter out Java service log events:

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: my-service-cw-flow
  namespace: test
spec:
  filters:
  - tag_normaliser:
      format: ${pod_name}-service.log
  - concat:
      multiline_start_regexp: /^([a-zA-Z]+\s+\[\d+-\d+-\d+\s+\d+:\d+:\d+,\d+\])/
      continuous_line_regexp: /^(?![a-zA-Z]+\s+)(?!\d+\.\d+\.\d+\.\d+)/
      key: message
      flush_interval: 1
  - grep:
      exclude:
        - key: message
          pattern: /^(\d+\.\d+\.\d+\.\d+)/
  match:
  - select:
      labels:
        app: my-service
        env: test
  localOutputRefs:
  - my-service-cw-output

Flow object 2 - Filter out HTTP requests log events (common log format) produced by the same Java service:

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: my-service-requests-cw-flow
  namespace: test
spec:
  filters:
  - tag_normaliser:
      format: ${pod_name}-requests.log
  - grep:
      regexp:
        - key: message
          pattern: /^(\d+\.\d+\.\d+\.\d+)/
  match:
  - select:
      labels:
        app: my-service
        env: test
  localOutputRefs:
  - my-service-cw-output

Output object - Ship all log events to AWS CloudWatch (2 separate streams will be created under the same log group):

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
  name: my-service-cw-output
  namespace: test
spec:
  cloudwatch:
    format:
      type: single_value
      add_newline: false
      message_key: message
    auto_create_stream: true
    buffer:
      timekey: 30s
      timekey_use_utc: true
      timekey_wait: 30s
    log_group_name: /aws/eks/test/my-service
    region: us-east-1
    remove_log_stream_name_key: "true"
    retention_in_days: "1"
    use_tag_as_stream: true

[sebastiangaiser]

The plugin should be installed: https://github.com/banzaicloud/logging-operator/blob/fluentd-v1.14.6-alpine-6/fluentd-image/v1.14/Dockerfile#L48

[Sebor]

The plugin should be installed: https://github.com/banzaicloud/logging-operator/blob/fluentd-v1.14.6-alpine-6/fluentd-image/v1.14/Dockerfile#L48

Should it be added to this page?
And what about documentation with configuration?

[genofire]

@Sebor fluentd support it - but the operator not yet.

The Filter Type of the CRD FlowSpec has to enhance
https://kube-logging.github.io/docs/configuration/crds/v1beta1/flow_types/#filter
and the parsing to fluentd config file ...

but if you like write it, it should be a nice first issue to put here another filter in, like geoip e.g.:
https://github.com/kube-logging/logging-operator/blob/master/pkg/sdk/logging/model/filter/geoip.go

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!