Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Malcolm v24.03.1 #301

Merged
merged 82 commits into from
Mar 27, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
cf94ca9
update SHA sums for ISOs
mmguero Mar 5, 2024
ff3b97a
minute tweaks to virter scrit
mmguero Mar 6, 2024
d50767f
version bump for v24.03.1 development
mmguero Mar 6, 2024
7a8882c
fix github_image_helper.sh
mmguero Mar 6, 2024
8c0e0f9
use built-in community-id for zeek instead of external plugin
mmguero Mar 6, 2024
a89c8a3
documentation update
mmguero Mar 6, 2024
02f00e7
point to a testing branch for this feature (idaholab/Malcolm#423) and…
mmguero Mar 7, 2024
73ac02d
point to a testing branch for this feature (idaholab/Malcolm#423) and…
mmguero Mar 7, 2024
436069a
update documentation for MALCOLM_NETWORK_INDEX_SUFFIX
mmguero Mar 11, 2024
e4cd3fe
added ascii art for malcolm banner
mmguero Mar 11, 2024
e99854e
slides update
mmguero Mar 12, 2024
567170c
Merge branch 'development' of https://github.com/mmguero-dev/Malcolm …
mmguero Mar 12, 2024
b5f2d78
for idaholab/Malcolm#437; for capture using the zeek-live container, …
mmguero Mar 12, 2024
ea67d08
fix an issue where using a different value for MALCOLM_OTHER_INDEX_PA…
mmguero Mar 13, 2024
0baf9ca
missed some replacers in a few dashboards
mmguero Mar 13, 2024
4244740
fix packet capture dashboard for suricata statistics to take into acc…
mmguero Mar 13, 2024
d46d6ac
added release_carver.sh, see idaholab/Malcolm#440
mmguero Mar 13, 2024
7c98508
added release_carver.sh, see idaholab/Malcolm#440
mmguero Mar 13, 2024
2392498
added release_carver.ps1, see idaholab/Malcolm#440
mmguero Mar 13, 2024
04a2b39
added release_carver.ps1, see idaholab/Malcolm#440
mmguero Mar 13, 2024
52b0eb5
Added 'twice daily' option to examples in documentation for index rot…
mmguero Mar 14, 2024
93eab42
Added 'twice daily' option to examples in documentation for index rot…
mmguero Mar 14, 2024
df00078
make sure index names are lowercased, which is a requirement for open…
mmguero Mar 14, 2024
51ffa93
added release_carver.ps1, see idaholab/Malcolm#440
mmguero Mar 14, 2024
760f3d2
added release_carver.ps1, see idaholab/Malcolm#440
mmguero Mar 14, 2024
f949546
added release_carver.ps1, see idaholab/Malcolm#440
mmguero Mar 14, 2024
a30f20b
idaholab/Malcolm#440, rearrange download page
mmguero Mar 14, 2024
f676774
fix misspelling
mmguero Mar 14, 2024
f318cae
doc update
mmguero Mar 14, 2024
d314c80
doc update
mmguero Mar 14, 2024
a7d6dfe
move a bunch of environment variables from logstash.env to netbox-com…
mmguero Mar 14, 2024
ec9efd2
removing preloading catch-all prefixes (idaholab/Malcolm#436)
mmguero Mar 14, 2024
217cdb1
work in progress for subnet discovery (idaholab/Malcolm#436)
mmguero Mar 14, 2024
2ab4001
Bump Zeek to v6.2.0
mmguero Mar 15, 2024
df86b97
parsing updates for zeek 6.2.0
mmguero Mar 15, 2024
e2a8430
netbox_enrich.rb work in progress for subnet discovery (idaholab/Malc…
mmguero Mar 15, 2024
9aed3b1
netbox_enrich.rb major refactoring while working on idaholab/Malcolm#436
mmguero Mar 15, 2024
34d31d7
minor fixes for zeek v6.2.0 ldap logs
mmguero Mar 15, 2024
699e75c
work for idaholab/Malcolm#445, supporting suricata eve.json rotation
mmguero Mar 18, 2024
fd6b605
try to do install_required_packages earlier in install.py
mmguero Mar 18, 2024
f3805ea
try to do install_required_packages earlier in install.py
mmguero Mar 18, 2024
12f4802
uniformly increase number of results for table visualizations in Dash…
mmguero Mar 19, 2024
c441b7f
change default for suricatal log rotation to 1h (from 5 minutes), ida…
mmguero Mar 19, 2024
20fdc98
don't trigger all workflows for suricata scripts
mmguero Mar 19, 2024
936bf15
moved common malcolm fields into another composable component rather …
mmguero Mar 19, 2024
ee277a9
Revert "uniformly increase number of results for table visualizations…
mmguero Mar 19, 2024
46666e0
adjust some dashbaords line endings
mmguero Mar 19, 2024
b748390
idaholab/Malcolm#447, make sure otherBuckets is set to true for all t…
mmguero Mar 19, 2024
78c7b6d
idaholab/Malcolm#447, set size:100 for dashboards table
mmguero Mar 19, 2024
9ce8c25
related to 936bf157ca8707775512f0753cc3133cbb021fd8, make sure index-…
mmguero Mar 19, 2024
5ba67e5
related to 936bf157ca8707775512f0753cc3133cbb021fd8, make sure index-…
mmguero Mar 19, 2024
928d999
Fix syntax error in malcolm_common.py
mmguero Mar 20, 2024
b9caae9
minor dashboard fixes
mmguero Mar 20, 2024
ba30292
point TDS plugin back upstream to amazon's github repo; rename log po…
mmguero Mar 20, 2024
dd9b376
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
366c34a
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
2bc9214
ignore stderr in output of docker compose ps -q
mmguero Mar 20, 2024
e207704
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
a035107
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
e7bb0d1
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
ba699c0
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
50df2b6
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
1b18a08
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
94c8235
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
3b94d3c
working on idaholab/Malcolm#266, Malcolm ISO should format bigger dri…
mmguero Mar 20, 2024
139957b
testing idaholab/Malcolm#266, Malcolm ISO should format bigger drives…
mmguero Mar 21, 2024
4ae65fe
testing idaholab/Malcolm#266, Malcolm ISO should format bigger drives…
mmguero Mar 21, 2024
2b11765
testing idaholab/Malcolm#266, Malcolm ISO should format bigger drives…
mmguero Mar 21, 2024
441c060
don't kick off github workflow builds for changes to os-disk-config.py
mmguero Mar 21, 2024
87f5cdf
bump fluent bit to v3.0.0
mmguero Mar 21, 2024
4a72770
replace salesforce with corelight hassh plugin
mmguero Mar 21, 2024
319c3c6
update opensearch-py to v2.5.0 (https://github.com/opensearch-project…
mmguero Mar 25, 2024
4256298
update arkime build for investigate allowing specifying another index…
mmguero Mar 25, 2024
a24781a
move profinet plugin to point upstream at amzn/zeek-plugin-profinet n…
mmguero Mar 25, 2024
84c68b5
ignore warnings from compose
mmguero Mar 25, 2024
05a38d2
notice.log seems to have community ID on it now, so parsing that
mmguero Mar 25, 2024
2ae04f7
update local.zeek for sensor
mmguero Mar 25, 2024
8428a40
update kubernetes manifests to point to correct images
mmguero Mar 26, 2024
ef1f32d
Merge branch 'development' of https://github.com/mmguero-dev/Malcolm …
mmguero Mar 26, 2024
eb58988
fix comma that was added during merge
mmguero Mar 26, 2024
8a18311
fix for idaholab/Malcolm#266, Malcolm ISO should format bigger drives…
mmguero Mar 27, 2024
034655c
Merge branch 'development' of https://github.com/mmguero-dev/Malcolm …
mmguero Mar 27, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/api-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/arkime-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/dashboards-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/dashboards-helper-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/file-monitor-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*.sh'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/file-upload-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/filebeat-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/freq-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/htadmin-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/logstash-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '!shared/bin/zeek*'
- '.trigger_workflow_build'
workflow_dispatch:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ on:
- 'shared/bin/*'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_iso_workflow_build'
- '.github/workflows/malcolm-iso-build-docker-wrap-push-ghcr.yml'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/netbox-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
workflow_dispatch:
repository_dispatch:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/nginx-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,17 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
- '_config.yml'
- '_includes/**'
- '_layouts/**'
- 'docs/**'
- '!docs/download.md'
- 'Gemfile'
- 'README.md'
workflow_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/opensearch-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
workflow_dispatch:
repository_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/pcap-capture-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
workflow_dispatch:
repository_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/pcap-monitor-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,12 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
workflow_dispatch:
repository_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/postgresql-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
workflow_dispatch:
repository_dispatch:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/redis-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/zeek*'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
workflow_dispatch:
repository_dispatch:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/suricata-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/zeek-build-and-push-ghcr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ on:
- '!shared/bin/agg-init.sh'
- '!shared/bin/common-init.sh'
- '!shared/bin/sensor-init.sh'
- '!shared/bin/os-disk-config.py'
- '!shared/bin/preseed_late_user_config.sh'
- '!shared/bin/configure-interfaces.py'
- '!shared/bin/configure-capture.py'
- '!shared/bin/suricata*'
- '.trigger_workflow_build'
workflow_dispatch:
repository_dispatch:
Expand Down
2 changes: 1 addition & 1 deletion Dockerfiles/filebeat.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ RUN for INPUT in nginx tcp; do \
chmod 770 /usr/share/filebeat-$INPUT/data; \
done; \
chmod 755 /usr/local/bin/*.sh /usr/local/bin/*.py && \
(echo "* * * * * /usr/local/bin/filebeat-process-zeek-folder.sh\n*/5 * * * * /usr/local/bin/filebeat-clean-zeeklogs-processed-folder.py" > ${SUPERCRONIC_CRONTAB})
(echo "* * * * * /usr/local/bin/filebeat-process-zeek-folder.sh\n*/5 * * * * /usr/local/bin/clean-processed-folder.py" > ${SUPERCRONIC_CRONTAB})

ENV AUTO_TAG $AUTO_TAG
ENV LOG_CLEANUP_MINUTES $LOG_CLEANUP_MINUTES
Expand Down
12 changes: 0 additions & 12 deletions Dockerfiles/logstash.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -28,24 +28,12 @@ ARG LOGSTASH_PARSE_PIPELINE_ADDRESSES=zeek-parse,suricata-parse,beats-parse
ARG LOGSTASH_OPENSEARCH_PIPELINE_ADDRESS_INTERNAL=internal-os
ARG LOGSTASH_OPENSEARCH_PIPELINE_ADDRESS_EXTERNAL=external-os
ARG LOGSTASH_OPENSEARCH_OUTPUT_PIPELINE_ADDRESSES=internal-os,external-os
ARG LOGSTASH_NETBOX_ENRICHMENT=false
ARG LOGSTASH_NETBOX_ENRICHMENT_VERBOSE=false
ARG LOGSTASH_NETBOX_ENRICHMENT_LOOKUP_SERVICE=true
ARG LOGSTASH_NETBOX_AUTO_POPULATE=false
ARG LOGSTASH_NETBOX_CACHE_SIZE=1000
ARG LOGSTASH_NETBOX_CACHE_TTL=30

ENV LOGSTASH_ENRICHMENT_PIPELINE $LOGSTASH_ENRICHMENT_PIPELINE
ENV LOGSTASH_PARSE_PIPELINE_ADDRESSES $LOGSTASH_PARSE_PIPELINE_ADDRESSES
ENV LOGSTASH_OPENSEARCH_PIPELINE_ADDRESS_INTERNAL $LOGSTASH_OPENSEARCH_PIPELINE_ADDRESS_INTERNAL
ENV LOGSTASH_OPENSEARCH_PIPELINE_ADDRESS_EXTERNAL $LOGSTASH_OPENSEARCH_PIPELINE_ADDRESS_EXTERNAL
ENV LOGSTASH_OPENSEARCH_OUTPUT_PIPELINE_ADDRESSES $LOGSTASH_OPENSEARCH_OUTPUT_PIPELINE_ADDRESSES
ENV LOGSTASH_NETBOX_ENRICHMENT $LOGSTASH_NETBOX_ENRICHMENT
ENV LOGSTASH_NETBOX_ENRICHMENT_VERBOSE $LOGSTASH_NETBOX_ENRICHMENT_VERBOSE
ENV LOGSTASH_NETBOX_ENRICHMENT_LOOKUP_SERVICE $LOGSTASH_NETBOX_ENRICHMENT_LOOKUP_SERVICE
ENV LOGSTASH_NETBOX_AUTO_POPULATE $LOGSTASH_NETBOX_AUTO_POPULATE
ENV LOGSTASH_NETBOX_CACHE_SIZE $LOGSTASH_NETBOX_CACHE_SIZE
ENV LOGSTASH_NETBOX_CACHE_TTL $LOGSTASH_NETBOX_CACHE_TTL

USER root

Expand Down
2 changes: 0 additions & 2 deletions Dockerfiles/netbox.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,13 @@ ARG NETBOX_DEVICETYPE_LIBRARY_IMPORT_PATH="/opt/netbox-devicetype-library-import
ARG NETBOX_DEFAULT_SITE=Malcolm
ARG NETBOX_CRON=true
ARG NETBOX_PRELOAD_PATH="/opt/netbox-preload"
ARG NETBOX_PRELOAD_PREFIXES=false

ENV NETBOX_PATH /opt/netbox
ENV BASE_PATH netbox
ENV NETBOX_DEVICETYPE_LIBRARY_IMPORT_PATH $NETBOX_DEVICETYPE_LIBRARY_IMPORT_PATH
ENV NETBOX_DEFAULT_SITE $NETBOX_DEFAULT_SITE
ENV NETBOX_CRON $NETBOX_CRON
ENV NETBOX_PRELOAD_PATH $NETBOX_PRELOAD_PATH
ENV NETBOX_PRELOAD_PREFIXES $NETBOX_PRELOAD_PREFIXES

ADD netbox/patch/* /tmp/netbox-patches/

Expand Down
5 changes: 1 addition & 4 deletions Dockerfiles/suricata.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ RUN sed -i "s/main$/main contrib non-free/g" /etc/apt/sources.list.d/debian.sour
useradd -M --uid ${DEFAULT_UID} --gid ${DEFAULT_GID} --home /nonexistant ${PUSER} && \
usermod -a -G tty ${PUSER} && \
ln -sfr /usr/local/bin/pcap_processor.py /usr/local/bin/pcap_suricata_processor.py && \
(echo "*/5 * * * * /usr/local/bin/eve-clean-logs.sh\n0 */6 * * * /bin/bash /usr/local/bin/suricata-update-rules.sh\n" > ${SUPERCRONIC_CRONTAB}) && \
(echo "0 */6 * * * /bin/bash /usr/local/bin/suricata-update-rules.sh\n" > ${SUPERCRONIC_CRONTAB}) && \
mkdir -p "$SURICATA_CUSTOM_RULES_DIR" "$SURICATA_DEFAULT_RULES_DIR" "$SURICATA_CUSTOM_CONFIG_DIR" && \
chown -R ${PUSER}:${PGROUP} "$SURICATA_CUSTOM_RULES_DIR" "$SURICATA_DEFAULT_RULES_DIR" "$SURICATA_CUSTOM_CONFIG_DIR" && \
cp "$(dpkg -L suricata-update | grep 'update\.yaml$' | head -n 1)" \
Expand All @@ -136,7 +136,6 @@ COPY --chmod=755 shared/bin/pcap_processor.py /usr/local/bin/
COPY --chmod=644 scripts/malcolm_utils.py /usr/local/bin/
COPY --chmod=755 shared/bin/suricata_config_populate.py /usr/local/bin/
COPY --chmod=755 suricata/scripts/docker_entrypoint.sh /usr/local/bin/
COPY --chmod=755 suricata/scripts/eve-clean-logs.sh /usr/local/bin/
COPY --chmod=755 suricata/scripts/suricata-update-rules.sh /usr/local/bin/
COPY --chmod=u=rwX,go=rX suricata/rules-default/ "$SURICATA_DEFAULT_RULES_DIR"/

Expand All @@ -148,7 +147,6 @@ ARG SURICATA_CRON=true
ARG SURICATA_AUTO_ANALYZE_PCAP_FILES=false
ARG SURICATA_CUSTOM_RULES_ONLY=false
ARG SURICATA_AUTO_ANALYZE_PCAP_THREADS=1
ARG LOG_CLEANUP_MINUTES=30
ARG SURICATA_UPDATE_RULES=false
ARG SURICATA_UPDATE_DEBUG=false
ARG SURICATA_UPDATE_ETOPEN=true
Expand All @@ -168,7 +166,6 @@ ENV SURICATA_CRON $SURICATA_CRON
ENV SURICATA_AUTO_ANALYZE_PCAP_FILES $SURICATA_AUTO_ANALYZE_PCAP_FILES
ENV SURICATA_AUTO_ANALYZE_PCAP_THREADS $SURICATA_AUTO_ANALYZE_PCAP_THREADS
ENV SURICATA_CUSTOM_RULES_ONLY $SURICATA_CUSTOM_RULES_ONLY
ENV LOG_CLEANUP_MINUTES $LOG_CLEANUP_MINUTES
ENV SURICATA_UPDATE_RULES $SURICATA_UPDATE_RULES
ENV SURICATA_UPDATE_DEBUG $SURICATA_UPDATE_DEBUG
ENV SURICATA_UPDATE_ETOPEN $SURICATA_UPDATE_ETOPEN
Expand Down
6 changes: 3 additions & 3 deletions Dockerfiles/zeek.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ ENV SUPERCRONIC_SHA1SUM "cd48d45c4b10f3f0bfdd3a57d054cd05ac96812b"
ENV SUPERCRONIC_CRONTAB "/etc/crontab"

# for download and install
ARG ZEEK_VERSION=6.1.1-0
ARG ZEEK_VERSION=6.2.0-0
ENV ZEEK_VERSION $ZEEK_VERSION

# put Zeek and Spicy in PATH
Expand Down Expand Up @@ -160,8 +160,8 @@ ADD shared/bin/zeekdeploy.sh ${ZEEK_DIR}/bin/

# sanity checks to make sure the plugins installed and copied over correctly
# these ENVs should match the number of third party scripts/plugins installed by zeek_install_plugins.sh
ENV ZEEK_THIRD_PARTY_PLUGINS_COUNT 23
ENV ZEEK_THIRD_PARTY_PLUGINS_GREP "(Zeek::Spicy|ANALYZER_SPICY_DHCP|ANALYZER_SPICY_DNS|ANALYZER_SPICY_HTTP|ANALYZER_SPICY_OSPF|ANALYZER_SPICY_OPENVPN_UDP\b|ANALYZER_SPICY_IPSEC_UDP\b|ANALYZER_SPICY_TFTP|ANALYZER_SPICY_WIREGUARD|ANALYZER_SYNCHROPHASOR_TCP|ANALYZER_GENISYS_TCP|ANALYZER_SPICY_PROFINET_IO_CM|ANALYZER_S7COMM_TCP|Corelight::CommunityID|Corelight::PE_XOR|ICSNPP::BACnet|ICSNPP::BSAP|ICSNPP::ENIP|ICSNPP::ETHERCAT|ICSNPP::OPCUA_Binary|Salesforce::GQUIC|Zeek::PROFINET|Zeek::TDS)"
ENV ZEEK_THIRD_PARTY_PLUGINS_COUNT 22
ENV ZEEK_THIRD_PARTY_PLUGINS_GREP "(Zeek::Spicy|ANALYZER_SPICY_DHCP|ANALYZER_SPICY_DNS|ANALYZER_SPICY_HTTP|ANALYZER_SPICY_OSPF|ANALYZER_SPICY_OPENVPN_UDP\b|ANALYZER_SPICY_IPSEC_UDP\b|ANALYZER_SPICY_TFTP|ANALYZER_SPICY_WIREGUARD|ANALYZER_SYNCHROPHASOR_TCP|ANALYZER_GENISYS_TCP|ANALYZER_SPICY_PROFINET_IO_CM|ANALYZER_S7COMM_TCP|Corelight::PE_XOR|ICSNPP::BACnet|ICSNPP::BSAP|ICSNPP::ENIP|ICSNPP::ETHERCAT|ICSNPP::OPCUA_Binary|Salesforce::GQUIC|Zeek::PROFINET|Zeek::TDS)"
ENV ZEEK_THIRD_PARTY_SCRIPTS_COUNT 25
ENV ZEEK_THIRD_PARTY_SCRIPTS_GREP "(bro-is-darknet/main|bro-simple-scan/scan|bzar/main|callstranger-detector/callstranger|cve-2020-0601/cve-2020-0601|cve-2020-13777/cve-2020-13777|CVE-2020-16898/CVE-2020-16898|CVE-2021-38647/omigod|CVE-2021-31166/detect|CVE-2021-41773/CVE_2021_41773|CVE-2021-42292/main|cve-2021-44228/CVE_2021_44228|cve-2022-22954/main|cve-2022-26809/main|CVE-2022-3602/__load__|hassh/hassh|http-more-files-names/main|ja3/ja3|pingback/detect|ripple20/ripple20|SIGRed/CVE-2020-1350|zeek-EternalSafety/main|zeek-httpattacks/main|zeek-sniffpass/__load__|zerologon/main)\.(zeek|bro)"

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Malcolm is a powerful network traffic analysis tool suite designed with the following goals in mind:

* **Easy to use** – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek (formerly Bro) logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis.
* **Easy to use** – Malcolm accepts network traffic data in the form of full packet capture (PCAP) files and Zeek logs. These artifacts can be uploaded via a simple browser-based interface or captured live and forwarded to Malcolm using lightweight forwarders. In either case, the data is automatically normalized, enriched, and correlated for analysis.
* **Powerful traffic analysis** – Visibility into network communications is provided through two intuitive interfaces: OpenSearch Dashboard, a flexible data visualization plugin with dozens of prebuilt dashboards providing an at-a-glance overview of network protocols; and Arkime (formerly Moloch), a powerful tool for finding and identifying the network sessions comprising suspected security incidents.
* **Streamlined deployment** – Malcolm operates as a cluster of Docker containers – isolated sandboxes that each serve a dedicated function of the system. This Docker-based deployment model, combined with a few simple scripts for setup and run-time management, makes Malcolm suitable to be deployed quickly across a variety of platforms and use cases; whether it be for long-term deployment on a Linux server in a security operations center (SOC) or for incident response on a Macbook for an individual engagement.
* **Secure communications** – All communications with Malcolm, both from the user interface and from remote log forwarders, are secured with industry standard encryption protocols.
Expand Down
2 changes: 1 addition & 1 deletion _config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ title: Malcolm
description: A powerful, easily deployable network traffic analysis tool suite
logo: docs/images/logo/Malcolm_outline_banner_dark.png
remote_theme: pages-themes/[email protected]
external_download_url: https://malcolm.fyi/docs/download.html
youtube_url: https://www.youtube.com/@MalcolmNetworkTrafficAnalysis
mastodon:
id:
Expand All @@ -17,6 +16,7 @@ components_docs_uri: docs/components.html
configuring_docs_uri: docs/malcolm-preparation.html
contributing_docs_uri: docs/contributing-guide.html
dashboards_docs_uri: docs/dashboards.html
download_docs_uri: docs/download.html#DownloadISOs
hardening_docs_uri: docs/hardening.html
hedgehog_docs_uri: docs/hedgehog.html
live_analysis_docs_uri: docs/live-analysis.html
Expand Down
2 changes: 1 addition & 1 deletion _layouts/default.html
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ <h1><a href="{{ "/" | absolute_url }}">{{ site.title | default: site.github.repo
<ul class="downloads">
<li><a href="{{ site.github.repository_url }}/releases">GitHub <strong>Releases</strong></a></li>
<li><a href="{{ site.github.repository_url }}/tarball/{{ site.github.default_branch }}">Source <strong>.tgz</strong></a></li>
<li><a href="{{ site.external_download_url | default: site.github.repository_url }}">Download <strong>ISOs</strong></a></li>
<li><a href="{{ site.download_docs_uri | default: docs | relative_url }}">Download <strong>ISOs</strong></a></li>
</ul>
{% endif %}

Expand Down
2 changes: 1 addition & 1 deletion api/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
pytz==2021.3
Flask==2.3.2
gunicorn==20.1.0
opensearch-py==2.4.2
opensearch-py==2.5.0
requests==2.31.0
regex==2022.3.2
dateparser==1.1.1
Expand Down
Loading