Merge pull request #247 from mmguero-dev/v23.04.0_merge_cisagov

Malcolm v23.04.0 is a release with enhancements, component version updates and bug fixes. IMPORTANT NOTE: In March 2023 Docker Inc. announced its decision to sunset the "Docker Free Team" plan, which prompted us to decide to migrate away from Docker Hub to the Github Container Registry or "ghcr" (see idaholab#163). Due to public backlash, Docker Inc. reversed its decision. However, the Malcolm project will continue with the decision to use GHCR beginning with this release (Malcolm v23.04.0) and moving forward. If you're updating an existing instance of Malcolm, it's recommended that you back up your `docker-compose.yml` and `docker-compose-standalone.yml` files, replace them with the ones from this release and re-run `./scripts/install.py --configure` to ensure that you're pointing at the latest images (this is actually always good practice when moving to a new release of Malcolm). v23.03.0...v23.04.0 * Enhancements - autostart `install.py --configure` on Malcolm ISO first boot (idaholab#157) - clarify information about auth_setup's use of external OpenSearch connections (idaholab#160) - migrate away from DockerHub container registry (idaholab#163) - give easier option for transferring SSL client files from Malcolm to forwarder (idaholab#177) + added `tx-rx-secure.sh` script as wrapper around [croc](https://github.com/schollz/croc) automatically creating and using a local-only relay * Component version updates - [Zeek v5.2.0](https://github.com/zeek/zeek/releases) (idaholab#161) - [fluent bit v2.0.10](https://fluentbit.io/announcements/v2.0.10/) - [NetBox v3.4.7](https://github.com/netbox-community/netbox/releases/tag/v3.4.7) * Fixes - XFCE4's "save session on exit" causes conflict with Hedgehog kiosk mode if firefox instance is started upon session restore (idaholab#164) - docker-compose move from go-yaml/v3 breaks Malcolm's docker-compose YAML files (idaholab#178, docker/compose#10411) - increase index.mapping.nested_fields.limit in opensearch index template (idaholab#180)
cisagov · Apr 5, 2023 · 37bb7f7 · 37bb7f7
2 parents de48bbe + 462b7d3
commit 37bb7f7
Show file tree

Hide file tree

Showing 91 changed files with 1,433 additions and 983 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -14,6 +14,7 @@
 **/.envrc
 **/.direnv
 **/.opensearch*.curlrc
+.configured
 .trigger_workflow_build
 .tmp
 docker-compose*yml

diff --git a/.github/workflows/api-build-and-push-ghcr.yml b/.github/workflows/api-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'api/**'
       - 'Dockerfiles/api.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/api.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/api:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/api:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/arkime-build-and-push-ghcr.yml b/.github/workflows/arkime-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'arkime/**'
       - 'Dockerfiles/arkime.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -61,4 +64,4 @@ jobs:
           build-args: |
             MAXMIND_GEOIP_DB_LICENSE_KEY=${{ secrets.MAXMIND_GEOIP_DB_LICENSE_KEY }}
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/arkime:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/arkime:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/dashboards-build-and-push-ghcr.yml b/.github/workflows/dashboards-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'dashboards/**'
       - 'Dockerfiles/dashboards.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/dashboards.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/dashboards:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/dashboards-helper-build-and-push-ghcr.yml b/.github/workflows/dashboards-helper-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'dashboards/**'
       - 'Dockerfiles/dashboards-helper.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/dashboards-helper.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/dashboards-helper:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/dashboards-helper:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/file-monitor-build-and-push-ghcr.yml b/.github/workflows/file-monitor-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'file-monitor/**'
       - 'Dockerfiles/file-monitor.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/file-monitor.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/file-monitor:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/file-monitor:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/file-upload-build-and-push-ghcr.yml b/.github/workflows/file-upload-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'file-upload/**'
       - 'Dockerfiles/file-upload.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/file-upload.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/file-upload:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/file-upload:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/filebeat-build-and-push-ghcr.yml b/.github/workflows/filebeat-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'filebeat/**'
       - 'Dockerfiles/filebeat.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/filebeat.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/filebeat-oss:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/filebeat-oss:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/freq-build-and-push-ghcr.yml b/.github/workflows/freq-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'freq-server/**'
       - 'Dockerfiles/freq.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/freq.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/freq:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/freq:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/htadmin-build-and-push-ghcr.yml b/.github/workflows/htadmin-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'htadmin/**'
       - 'Dockerfiles/htadmin.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/htadmin.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/htadmin:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/htadmin:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/logstash-build-and-push-ghcr.yml b/.github/workflows/logstash-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'logstash/**'
       - 'Dockerfiles/logstash.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/logstash.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/logstash-oss:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/logstash-oss:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml b/.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml
@@ -5,9 +5,11 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'malcolm-iso/**'
       - 'shared/bin/*'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_iso_workflow_build'
       - '.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml'
   workflow_dispatch:
@@ -94,7 +96,7 @@ jobs:
         run: |
           IMAGES=( $(grep image: docker-compose.yml | awk '{print $2}' | sort -u) )
           for IMAGE in "${IMAGES[@]}"; do
-            REPO_IMAGE="$(echo "$IMAGE" | sed "s@^\(malcolmnetsec\)@ghcr.io/${{ github.repository_owner }}/\1@" | sed "s/:.*/:${{ steps.extract_branch.outputs.branch }}/")"
+            REPO_IMAGE="$(echo "$IMAGE" | sed "s@^.*\(malcolm\)@ghcr.io/${{ github.repository_owner }}/\1@" | sed "s/:.*/:${{ steps.extract_branch.outputs.branch }}/")"
             docker pull "$REPO_IMAGE" && \
               docker tag "$REPO_IMAGE" "$IMAGE" && \
               docker rmi "$REPO_IMAGE"
@@ -124,4 +126,4 @@ jobs:
         with:
           context: ./malcolm-iso
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/malcolm:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/malcolm:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/name-map-ui-build-and-push-ghcr.yml b/.github/workflows/name-map-ui-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'name-map-ui/**'
       - 'Dockerfiles/name-map-ui.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/name-map-ui.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/name-map-ui:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/name-map-ui:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/netbox-build-and-push-ghcr.yml b/.github/workflows/netbox-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'netbox/**'
       - 'Dockerfiles/netbox.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/netbox.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/netbox:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/netbox:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/nginx-build-and-push-ghcr.yml b/.github/workflows/nginx-build-and-push-ghcr.yml
@@ -5,15 +5,19 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'nginx/**'
       - 'Dockerfiles/nginx.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
       - '_config.yml'
       - '_includes/**'
       - '_layouts/**'
       - 'docs/**'
+      - '!docs/download.md'
       - 'Gemfile'
       - 'README.md'
   workflow_dispatch:
@@ -73,4 +77,4 @@ jobs:
             VCS_REVISION=${{ steps.extract_commit_sha.outputs.sha }}
             GITHUB_TOKEN=${{ secrets.GITHUB_TOKEN }}
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/nginx-proxy:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/nginx-proxy:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/opensearch-build-and-push-ghcr.yml b/.github/workflows/opensearch-build-and-push-ghcr.yml
@@ -5,9 +5,12 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'Dockerfiles/opensearch.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -58,4 +61,4 @@ jobs:
           context: .
           file: ./Dockerfiles/opensearch.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/opensearch:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/opensearch:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/pcap-capture-build-and-push-ghcr.yml b/.github/workflows/pcap-capture-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'pcap-capture/**'
       - 'Dockerfiles/pcap-capture.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/pcap-capture.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/pcap-capture:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/pcap-capture:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/pcap-monitor-build-and-push-ghcr.yml b/.github/workflows/pcap-monitor-build-and-push-ghcr.yml
@@ -5,10 +5,13 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'pcap-monitor/**'
       - 'Dockerfiles/pcap-monitor.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -59,4 +62,4 @@ jobs:
           context: .
           file: ./Dockerfiles/pcap-monitor.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/pcap-monitor:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/pcap-monitor:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/postgresql-build-and-push-ghcr.yml b/.github/workflows/postgresql-build-and-push-ghcr.yml
@@ -5,9 +5,12 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'Dockerfiles/postgresql.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -58,4 +61,4 @@ jobs:
           context: .
           file: ./Dockerfiles/postgresql.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/postgresql:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/postgresql:${{ steps.extract_branch.outputs.branch }}
diff --git a/.github/workflows/redis-build-and-push-ghcr.yml b/.github/workflows/redis-build-and-push-ghcr.yml
@@ -5,9 +5,12 @@ on:
     branches:
       - main
       - development
+      - kubernetes
     paths:
       - 'Dockerfiles/redis.Dockerfile'
       - 'shared/bin/*'
+      - '!shared/bin/configure-interfaces.py'
+      - '!shared/bin/configure-capture.py'
       - '.trigger_workflow_build'
   workflow_dispatch:
   repository_dispatch:
@@ -58,4 +61,4 @@ jobs:
           context: .
           file: ./Dockerfiles/redis.Dockerfile
           push: true
-          tags: ghcr.io/${{ github.repository_owner }}/malcolmnetsec/redis:${{ steps.extract_branch.outputs.branch }}
+          tags: ghcr.io/${{ github.repository_owner }}/malcolm/redis:${{ steps.extract_branch.outputs.branch }}