Merge pull request #249 from cisagov/v23.05.0_merge_cisagov

Malcolm v23.05.0

Malcolm v23.05.0 is a major release with new features, enhancements, component version updates and bug fixes.

IMPORTANT NOTE: Malcolm v23.05.0 has completely changed the way it manages its settings: rather than using environment variables found at the top of the `docker-compose.yml` file, it uses environment variables in `.env` files inside of the `config` directory. The locations of a number of configuration files have also changed. It's not recommended to update to Malcolm v23.05.0 from a previous version of Malcolm. Instead, shut down Malcolm, rename your old Malcolm installation directory to something else, and reconfigure Malcolm using `./scripts/configure` and `./scripts/auth_setup`.

v23.04.0...v23.05.0

* New features
    - integrate [ICSNPP-Synchrophasor](https://github.com/cisagov/icsnpp-synchrophasor/) parser (idaholab#190)
    - [End-to-end Malcolm and Hedgehog Linux ISO Installation((cisagov://cisagov.github.io/Malcolm/docs/malcolm-hedgehog-e2e-iso-install.html#InstallationExample) document (idaholab#181)
    - support Malcolm deployment with Kubernetes (idaholab#149)
        + see [Deploying Malcolm with Kubernetes](https://idaholab.github.io/Malcolm/docs/kubernetes.html#Kubernetes)
        + This could be considered a "beta" release for Malcolm deployment with Kubernetes, as there is still [some work](https://github.com/idaholab/Malcolm/issues?q=is%3Aissue+is%3Aopen+kubernetes) to be done in this area. Please let us know what issues or suggestions you have via the [issue tracker](https://github.com/idaholab/Malcolm/issues) or via email to [[email protected]](mailto:[email protected]?subject=Malcolm).
        + contributing issues:
            * inotify issue (idaholab#168)
            * htadmin/nginx and htpasswd (idaholab#169)
            * opensearch (idaholab#170)
            * uploading large PCAP files (idaholab#171)
            * script consolidation (idaholab#172)
            * documentation (idaholab#173)
            * user-defined persistent volumes (idaholab#174)
            * opensearch keystore (idaholab#176)
            * expose other TCP services (idaholab#183)
            * provide with filebeat access to nginx access and error logs (idaholab#186)
            * use Secrets for some environment variables instead of ConfigMaps (idaholab#189)

* Enhancements and fixes
    * remove name-map-ui container (idaholab#165) in favor of using NetBox for asset identification
    * Python script refactoring, consolidation and cleanup
    * standardization of Docker container entrypoints
    * create `./scripts/configure` alias for `./scripts/install.py --configure`

* Component version updates
    - Arkime to [v4.3.0](https://github.com/arkime/arkime/blob/fa0db2415bdc109be7a4dd8ee2c2838673980b5f/CHANGELOG#L33-L72)
    - Capa to [v5.1.0](https://github.com/mandiant/capa/releases/tag/v5.1.0)
    - Fluent Bit to [v2.1.2](https://fluentbit.io/announcements/v2.1.2/)
    - NetBox to [v3.5.0](https://github.com/netbox-community/netbox/releases/tag/v3.5.0)
    - NGINX to [v1.22.1](http://nginx.org/en/CHANGES-1.22)
    - Supercronic to [v0.2.24](https://github.com/aptible/supercronic/releases/tag/v0.2.24)
    - Suricata to [v6.0.10](https://suricata.io/2023/01/31/suricata-6-0-10-released/)
    - Yara to [v4.3.0](https://github.com/VirusTotal/yara/releases/tag/v4.3.0)
    - Zeek to [v5.2.1](https://github.com/zeek/zeek/releases/tag/v5.2.1)

.dockerignore

@@ -2,11 +2,11 @@
 **/*.crt
 **/*.iso
 **/*.key
+**/*.env
 **/*.pem
 **/*.keystore
 **/.git*
 **/__pycache__
-**/auth.env
 **/.ldap_config_defaults
 **/htpasswd
 **/malcolm_*images.tar.gz
@@ -17,22 +17,26 @@
 .configured
 .trigger_workflow_build
 .tmp
+config.*
 docker-compose*yml
 Dockerfiles
 Gemfile.lock
 opensearch
 opensearch-backup
 arkime-logs
 arkime-raw
+kubernetes
 malcolm-iso
 sensor-iso
-nginx/nginx_ldap.conf
+nginx/nginx_ldap*.conf
 pcap
 _site
 scripts
 !scripts/malcolm_common.py
+!scripts/malcolm_kubernetes.py
+!scripts/malcolm_utils.py
 zeek-logs
 suricata-logs
 netbox/netbox/media
 netbox/netbox/postgres
-netbox/netbox/redis
+netbox/netbox/redis

.github/workflows/api-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'api/**'
       - 'Dockerfiles/api.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/arkime-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'arkime/**'
       - 'Dockerfiles/arkime.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/dashboards-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'dashboards/**'
       - 'Dockerfiles/dashboards.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/dashboards-helper-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'dashboards/**'
       - 'Dockerfiles/dashboards-helper.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/file-monitor-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'file-monitor/**'
       - 'Dockerfiles/file-monitor.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/file-upload-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'file-upload/**'
       - 'Dockerfiles/file-upload.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/filebeat-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'filebeat/**'
       - 'Dockerfiles/filebeat.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/freq-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'freq-server/**'
       - 'Dockerfiles/freq.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/htadmin-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'htadmin/**'
       - 'Dockerfiles/htadmin.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/logstash-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'logstash/**'
       - 'Dockerfiles/logstash.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml

@@ -5,7 +5,6 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'malcolm-iso/**'
       - 'shared/bin/*'

.github/workflows/netbox-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'netbox/**'
       - 'Dockerfiles/netbox.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/nginx-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'nginx/**'
       - 'Dockerfiles/nginx.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/opensearch-build-and-push-ghcr.yml

@@ -5,10 +5,12 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'Dockerfiles/opensearch.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/pcap-capture-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'pcap-capture/**'
       - 'Dockerfiles/pcap-capture.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/pcap-monitor-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'pcap-monitor/**'
       - 'Dockerfiles/pcap-monitor.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/postgresql-build-and-push-ghcr.yml

@@ -5,10 +5,12 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'Dockerfiles/postgresql.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/redis-build-and-push-ghcr.yml

@@ -5,10 +5,12 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'Dockerfiles/redis.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'

.github/workflows/sensor-iso-build-docker-wrap-push-ghcr.yml

@@ -5,7 +5,6 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'sensor-iso/**'
       - 'shared/bin/*'
@@ -92,6 +91,7 @@ jobs:
         name: Build image
         run: |
           cp -r ./shared ./docs ./_config.yml ./_includes ./_layouts ./Gemfile ./README.md ./sensor-iso
+          cp ./scripts/malcolm_utils.py ./sensor-iso/shared/bin/
           cp ./scripts/documentation_build.sh ./sensor-iso/docs/
           cp -r ./arkime/patch ./sensor-iso/shared/arkime_patch
           pushd ./sensor-iso

.github/workflows/suricata-build-and-push-ghcr.yml

@@ -5,11 +5,13 @@ on:
     branches:
       - main
       - development
-      - kubernetes
     paths:
       - 'suricata/**'
       - 'Dockerfiles/suricata.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/agg-init.sh'
+      - '!shared/bin/common-init.sh'
+      - '!shared/bin/sensor-init.sh'
       - '!shared/bin/configure-interfaces.py'
       - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'