break Zeek/Suricata into two containers: one for "live" capture and one for uploaded PCAPs

[mmguero]

Hedgehog Linux is the preferred method for live network traffic capture/monitoring with Malcolm because it 1) allows you to have dedicated resources for capture separate from the Malcolm aggregator and 2) allows the traffic to be analyzed as it's "streamed" rather than buffering it to PCAP first and then analyzed on some sort of trigger as the PCAPs are rolled (managed by the PCAP_ROTATE_... environment variables).

This second method I've described is the way "standalone" Malcolm captures live traffic, which results in PCAP files being copied to the ./pcap/upload directory and processed in the same way as any other "offline" PCAP files that the user has previously captured and uploaded manually.

The reason for this different implementation has to do with giving the capture docker container the ability to promiscuously monitor a network interface: this, as far as I am aware, requires the container to run with host networking. This is an issue mainly for the Arkime capture program, which needs to have a local connection to the http://opensearch endpoint, which is NOT running in host networking mode. I'd prefer to avoid connecting to opensearch via port forwarding as we'd have to surface authentication/encryption between the containers and it's just much simpler to have them be able to talk openly on the same docker network.

While this is a limitation for arkime capture, as far as Zeek and Suricata are concerned I don't think we'd have the same issue, since those programs are just generating textual log files into directories which could be picked up and analyzed by the other monitoring containers (this is essentially what we're doing with the pcap-capture container-produced PCAP files anyway).

So my proposal is that if live capture on a local interface is requested for a standalone Malcolm instance (via PCAP_IFACE in the docker-compose file) and Zeek and/or Suricata analysis is requested, I'd have an additional container for running those processes "live" for capture in host networking mode. Those containers wouldn't need to have separate images (they could use the same ones as their "offline PCAP" counterparts use today) but would have different arguments for starting live capture. The filebeat container would have to watch these live containers' output in additional locations aside from the ones it monitors today.

It wouldn't "solve" the same issue for Arkime capture (that would still be on a "rotate" delay) but it would improve the quality and speed up the availability of the Zeek/Suricata logs.

[mmguero]

I think this is done, need to do some final testing but closing pending release.