Example/Collection for Flows

[genofire]

Is your feature request related to a problem? Please describe.
more examples for common software

Describe the solution you'd like
A repository there yaml files for Flows can be stored and improve by the community.

Describe alternatives you've considered

Additional context
I like to add some flows for e.g. fluxcd and coredns (for output to elasticsearch)

[fekete-robert]

Hi @genofire, for example files, configurations, and descriptions (on what these do and how to use them) the Logging operator docs might be the best place, like this page: https://kube-logging.github.io/docs/examples/cloudwatch-nginx/

The source of the docs is the https://github.com/kube-logging/kube-logging.github.io/ repository, but if you post the yaml and the description here, I can prepare a PR for the doc repo.

[genofire]

i would like to publish a bunch of flows for different applications (to be equal to filebeat - that there dashboards works well) - current: CoreDNS and Nginx

i am not sure, if it would be nice to have that on page in a documentation.

Here the example for the Nginx Ingress Controller:

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: "ingress-nginx"
spec:
  filters:
    - tag_normaliser: {}
    - parser:
        reserve_data: true
        remove_key_name_field: true
        inject_key_prefix: "nginx."
        parse:
          type: "regexp"
          # from https://raw.githubusercontent.com/fluent/fluentd/master/lib/fluent/plugin/parser_nginx.rb
          # enharance for ingress-controller by e.g.
          # https://github.com/elastic/beats/blob/v8.6.1/filebeat/module/nginx/ingress_controller/ingest/pipeline.yml
          expression: '^(?<remote>[^ ]*) -?(?<host>[^ ]*) -?(?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +(?<httpversion>HTTP\/[0-9\.]+)))" (?<code>[^ ]*) (?<size>[^ ]*)(?: "-?(?<referer>[^\"]*)" "(?<agent>[^\"]*)"\s+(?<reqlength>[0-9]+)\s(?<reqtime>[0-9\.]+)\s\[(?<upstream_name>[^ ]*)\]\s\[(?<alternative_name>[^\]]*)\] -?(?<upstream_address_list>[^ -]*) -?(?<reslength_list>[0-9\,]*) -?(?<restime_list>[0-9\.\,]*) -?(?<rescode_list>[0-9\,]*) (?<reqid>[^ ]+))?$'
          types: 'code:integer,size:integer,reqlength:integer,reqtime:float,upstream_address_list:array,reslenght_list:array,restime_list:array,rescode_list:array'
          time_key: "time"
          time_format: "%d/%b/%Y:%H:%M:%S %z"
    - record_modifier:
        records:
          - destination.domain: '${ !(record["nginx.referer"].nil?) ? URI(record["nginx.referer"]).host : record["nginx.host"] }'
            url.original: '${ record["nginx.referer"] }${ record["nginx.path"] }'
            url.domain: '${ !(record["nginx.referer"].nil?) ? URI(record["nginx.referer"]).host : record["nginx.host"] }'
            url.path: '${ record["nginx.path"] }'
            http.version: '${ record["nginx.httpversion"] }'
            nginx.access.remote_ip_list: '${ record["nginx.remote"] }'
            source.address: '${ record["nginx.remote"] }'
            source.ip: '${ record["nginx.remote"] }'
            related.ip: '${ record["nginx.remote"] }'
            http.request.method: '${ record["nginx.method"] }'
            http.request.referrer: '${ record["nginx.referer"] }'
            user.name: '${ record["nginx.user"] }'
            related.user: '${ record["nginx.user"] }'
            user_agent.original: '${ record["nginx.agent"] }'
            http.response.status_code: '${ record["nginx.code"] }'
            nginx.ingress_controller.http.request.length: '${ record["nginx.reqlength"] }'
            nginx.ingress_controller.http.request.time: '${ record["nginx.reqtime"] }'
            nginx.ingress_controller.upstream.name: '${ record["nginx.upstream_name"] }'
            nginx.ingress_controller.upstream.alternative_name: '${ record["nginx.alternative_name"] }'
            nginx.ingress_controller.upstream_address_list: '${ record["nginx.upstream_address_list"] }'
            # TODO split ip and port
            nginx.ingress_controller.upstream.address.merged: '${ [record["upstream_address_list"]].flatten&.last }'
            nginx.ingress_controller.upstream.response.length_list: '${ [record["nginx.reslength_list"]].flatten&.map(&:to_i) }'
            nginx.ingress_controller.upstream.response.length: '${ [record["nginx.reslength_list"]].flatten&.last&.to_i }'
            nginx.ingress_controller.upstream.response.time_list: '${ [record["nginx.restime_list"]].flatten&.map(&:to_f) }'
            nginx.ingress_controller.upstream.response.time: '${ [record["nginx.restime_list"]].flatten&.last&.to_f }'
            nginx.ingress_controller.upstream.response.status_code_list: '${ [record["nginx.rescode_list"]].flatten&.map(&:to_i) }'
            nginx.ingress_controller.upstream.response.status_code: '${ [record["nginx.rescode_list"]].flatten&.last&.to_i }'
            nginx.ingress_controller.http.request.id: '${ record["nginx.reqid"] }'
            http.request.id: '${ record["nginx.reqid"] }'
            http.response.body.bytes: '${ record["nginx.size"] }'
            event.created: '${ time * 1000 }'
            event.kind: "event"
            event.category: "web"
            event.type: "access"
            event.module: "nginx"
            event.outcome: '${ record["nginx.code"].to_i < 400 ? "success" : "failure" }'
            # for dashboard
            fileset.name: '${[ "ingress_controller", "access" ]}'
        remove_keys: "nginx.remote,nginx.host,nginx.user,nginx.method,nginx.path,nginx.httpversion,nginx.code,nginx.size,nginx.referer,nginx.agent,nginx.reqlength,nginx.reqtime,nginx.upstream_name,nginx.alternative_name,nginx.upstream_address_list,nginx.reslength_list,nginx.restime_list,nginx.rescode_list,nginx.reqid"
    - geoip:
       geoip_lookup_keys: "source.ip"
       backend_library: geoip2_c
       skip_adding_null_record: false
       records:
         - source.geo.country_name: '${country.names.en["source.ip"]}'
           source.geo.country_iso_code: '${country.iso_code["source.ip"]}'
           source.geo.city_name: '${city.names.en["source.ip"]}'
           source.geo.region_iso_code: '${subdivisions.0.iso_code["source.ip"]}'
           source.geo.region_name: '${subdivisions.0.names.en["source.ip"]}'
           source.geo.location: '''{ "lat": ${location.latitude["source.ip"]}, "lon": ${location.longitude["source.ip"]} }'''
    # - geoip:
    #     geoip_lookup_keys: "source.ip"
    #     backend_library: geoip
    #     skip_adding_null_record: false
    #     records:
    #       - source.geo.continent_name: '${continent.names.en["source.ip"]}'
    #       # source.as.number: '${asn["source.ip"]}'
    #       # source.as.organization.name: '${organization["source.ip"]}'
  match:
    - select:
        labels:
          app-kubernetes-io/name: "ingress-nginx"

For CoreDNS:

---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: coredns
  namespace: kube-system
spec:
  filters:
    - tag_normaliser: {}
    - parser:
        reserve_data: true
        remove_key_name_field: true
        parse:
          type: "regexp"
          expression: '^\[(?<log.level>.*)\] \[?(?<source.address>.*)\]?:(?<source.port>.*) - (?<dns.id>.*) "(?<dns.question.type>.*) (?<dns.question.class>.*) (?<dns.question.name>.*)\.? (?<network.transport>.*) (?<coredns.query.size>.*) (?<coredns.dnssec_ok>.*) (?<bufsize>.*)" (?<dns.response_code>.*) (?<dns.header_flags>.*) (?<coredns.response.size>.*) (?<coredns.duration>.*)s'
          types: "source.port:integer,dns.id:integer,coredns.query.size:integer,coredns.dnssec_ok:bool,bufsize:integer,dns.header_flags:array,coredns.response.size:integer,coredns.duration:float"
    - record_modifier:
        records:
          - source.ip: '${ record["source.address"] }'
          - dns.header_flags: '${ record["dns.header_flags"].map(&:upcase) }'
          - event.duration: '${ record["coredns.duration"] * 1000000000 }'
          - event.kind: "event"
          - event.category: "network"
          - event.type: "protocol"
          - event.outcome: '${ record["dns.response_code"] == "NOERROR" ? "success" : "failure" }'
          - event.protocol: "dns"
          - event.module: "coredns"
          - related.ip: '${ record["source.address"] }'
          # for dashboard
          - fileset.name: "kubernetes"
          # alias in elastic
          - coredns.query.name: '${ record["dns.question.name"] }'
        remove_keys: "coredns.duration,coredns.dnssec_ok"
  match:
    - select:
        labels:
          k8s-app: "coredns"

[fekete-robert]

Sorry, I've missed your reply, my apologies. We'll find out how best to add them and I'll create a draft PR with these examples in the docs during the weekend and.

[fekete-robert]

Hi @genofire, I've added a short explanation and the yamls to the documentation in kube-logging/kube-logging.github.io#125 . Please review and let me know what you think.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!

[pepov]

I think we could have a separate repository just for community examples. I think we don't need to add everything to the site, a github repository that they can use directly to base off could work better than docs, since these examples will mostly only be yamls, what do you think?