add ability to generate suricata.yaml such that suricata's eve.json is split/rotated so it doesn't grow as large

[mmguero]

We need to look at adding to suricata_config_populate.py the ability to generate suricata.yaml such that suricata's eve.json is split/rotated so it doesn't grow as large.

There are a few things we could do here as possibilities

• specify rotate intervals (e.g., rotate-interval: hour)
• split out logs by protocol (e.g., see the types in the example below)

# Configure the type of alert (and other) logging you would like.
outputs:
  # a line based alerts log similar to Snort's fast.log
  - fast:
      enabled: no
      filename: fast.log
      append: yes
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  # Extensible Event Format (nicknamed EVE) event log in JSON format
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve-alerts.json
      community-id: true
      community-id-seed: 0
      types:
        - alert:
            metadata: yes     # enable inclusion of app layer metadata with alert. Default yes
            tagged-packets: yes
  # Suricata HTTP Logs
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve-http-%Y-%m-%d-%H.json
      rotate-interval: hour
      community-id: true
      community-id-seed: 0
      types:
        - http:
            extended: yes
  # Suricata DNS Logs
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve-dns-%Y-%m-%d-%H.json
      rotate-interval: hour
      community-id: true
      community-id-seed: 0
      types:
        - dns:
            version: 2
            enabled: yes
            requests: yes
            response: yes
            formats: [detailed, grouped]
            types: [a, aaaa, cname, mx, ns, ptr, txt]
  # Suricata TLS Logs
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve-tls-%Y-%m-%d-%H.json
      rotate-interval: hour
      community-id: true
      community-id-seed: 0
      types:
        - tls:
            extended: yes
            session-resumption: no
            custom: [subject, issuer, session_resumed, serial, fingerprint, sni, version, not_before, not_after, certificate, ja3, ja3s]
  # Suricata Flow Logs
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve-flow-%Y-%m-%d-%H.json
      rotate-interval: hour
      community-id: true
      community-id-seed: 0
      types:
        - flow:
  # Suricata Other (dnp3, ftp, nfs, smb, ssh.....) Logs
  - eve-log:
      enabled: yes
      filetype: regular #regular|syslog|unix_dgram|unix_stream|redis
      filename: eve-other-%Y-%m-%d-%H.json
      rotate-interval: hour
      community-id: true
      community-id-seed: 0
      types:
        - smtp:
            extended: yes
            custom: [received, x-mailer, x-originating-ip, relays, reply-to, bcc]
            md5: [body, subject]
        - files:
            force-magic: no
            force-hash: [mdt]
        - dnp3
        - ftp
        - nfs
        - smb
        - tftp
        - ikev2
        - krb5
        - snmp
        - sip
        - ssh
        - dhcp:
            enabled: yes
            extended: no

  # a line based log of HTTP requests (no alerts)
  - http-log:
      enabled: no
      filename: http.log
      append: yes
      #extended: yes     # enable this for extended logging information
      #custom: yes       # enable the custom logging format (defined by customformat)
      #customformat: "%{%D-%H:%M:%S}t.%z %{X-Forwarded-For}i %H %m %h %u %s %B %a:%p -> %A:%P"
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  # a line based log of TLS handshake parameters (no alerts)
  - tls-log:
      enabled: no  # Log TLS connections.
      filename: tls.log # File to store TLS logs.
      append: yes
      #extended: yes     # Log extended information like fingerprint
      #custom: yes       # enabled the custom logging format (defined by customformat)
      #customformat: "%{%D-%H:%M:%S}t.%z %a:%p -> %A:%P %v %n %d %D"
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
      # output TLS transaction where the session is resumed using a
      # session id
      #session-resumption: no

  # output module to store certificates chain to disk
  - tls-store:
      enabled: no
      #certs-log-dir: certs # directory to store the certificates files

  # Packet log... log packets in pcap format. 3 modes of operation: "normal"
  # "multi" and "sguil".
  #
  # In normal mode a pcap file "filename" is created in the default-log-dir,
  # or as specified by "dir".
  # In multi mode, a file is created per thread. This will perform much
  # better, but will create multiple files where 'normal' would create one.
  # In multi mode the filename takes a few special variables:
  # - %n -- thread number
  # - %i -- thread id
  # - %t -- timestamp (secs or secs.usecs based on 'ts-format'
  # E.g. filename: pcap.%n.%t
  #
  # Note that it's possible to use directories, but the directories are not
  # created by Suricata. E.g. filename: pcaps/%n/log.%s will log into the
  # per thread directory.
  #
  # Also note that the limit and max-files settings are enforced per thread.
  # So the size limit when using 8 threads with 1000mb files and 2000 files
  # is: 8*1000*2000 ~ 16TiB.
  #
  # In Sguil mode "dir" indicates the base directory. In this base dir the
  # pcaps are created in the directory structure Sguil expects:
  #
  # $sguil-base-dir/YYYY-MM-DD/$filename.<timestamp>
  #
  # By default all packets are logged except:
  # - TCP streams beyond stream.reassembly.depth
  # - encrypted streams after the key exchange
  #
  - pcap-log:
      enabled: {{ .Values.enable_conditional_pcap }}
      filename: sig-%n-%t.pcap

      # File size limit.  Can be specified in kb, mb, gb.  Just a number
      # is parsed as bytes.
      limit: 5mb

      # If set to a value, ring buffer mode is enabled. Will keep maximum of
      # "max-files" of size "limit"
      max-files: 2000

      # Compression algorithm for pcap files. Possible values: none, lz4.
      # Enabling compression is incompatible with the sguil mode. Note also
      # that on Windows, enabling compression will *increase* disk I/O.
      compression: none

      # Further options for lz4 compression. The compression level can be set
      # to a value between 0 and 16, where higher values result in higher
      # compression.
      #lz4-checksum: no
      #lz4-level: 0

      mode: multi # normal, multi or sguil.

      # Directory to place pcap files. If not provided the default log
      # directory will be used. Required for "sguil" mode.
      dir: /var/log/suricata/pcaps

      #ts-format: usec # sec or usec second format (default) is filename.sec usec is filename.sec.usec
      use-stream-depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets
      honor-pass-rules: no # If set to "yes", flows in which a pass rule matched will stop being logged.
      # Use "all" to log all packets or use "alerts" to log only alerted packets and flows or "tag"
      # to log only flow tagged via the "tag" keyword
      conditional: alerts

  # a full alert log containing much information for signature writers
  # or for investigating suspected false positives.
  - alert-debug:
      enabled: no
      filename: alert-debug.log
      append: yes
      #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'

  # alert output to prelude (https://www.prelude-siem.org/) only
  # available if Suricata has been compiled with --enable-prelude
  - alert-prelude:
      enabled: no
      profile: suricata
      log-packet-content: no
      log-packet-header: yes

  # Stats.log contains data from various counters of the Suricata engine.
  - stats:
      enabled: yes
      filename: stats.log
      append: yes       # append to file (yes) or overwrite it (no)
      totals: yes       # stats for all threads merged together
      threads: no       # per thread stats
      #null-values: yes  # print counters that have value 0. Default: no

[mmguero]

Here's a recap of the changes for this:

• Added new functionality controlled by environment to suricata_config_populate.py that take effect only for live capture (i.e., when SURICATA_LIVE_CAPTURE is true):
  • SURICATA_EVE_FILENAME_PATTERN - how live eve.json files are named (default eve-%Y%m%d_%H%M%S.json)
  • SURICATA_EVE_THREADED - controls threaded file output (default false)
  • SURICATA_EVE_ROTATE_INTERVAL - controls eve.json file rotation (default 1h)
• Removed the script eve-clean-logs.sh from the Suricata containers for cleaning old eve.json logs. This is now handled by the filebeat container by the clean-processed-folder.py script in the same way the Zeek logs are, with behavior controlled by LOG_CLEANUP_MINUTES env variable for Suricata logs.
• Moved the LOG_CLEANUP_MINUTES and ZIP_CLEANUP_MINUTES from upload-common.env to filebeat.env as filebeat is the only container that uses those.
• Adjusted the filebeat.yml configuration files on Malcolm and hedgehog to look for eve*.json as the matching wildcard for harvesting.
• Renamed the script filebeat-clean-zeeklogs-processed-folder.py to clean-processed-folder.py as it is not used for more than just Zeek scripts. Added logic in there differentiating between Zeek and Suricata logs, and check both the suricata and suricata-live folders for old logs to prune. As the suricata documentation reads,
  • In addition to the SIGHUP style rotation discussed above, some outputs support their own time and date based rotation, however removal of old log files is still the responsibility of external tools. These outputs include: Eve