16GB RAM and 16 Core system resources are quickly being exhausted till system locks up #494
Closed
1 task done
Labels
bug
Something isn't working
Comments
./logs
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
htadmin-1 | root
htadmin-1 | uid=0(root) gid=0(root) groups=0(root)
htadmin-1 | 2024-06-19 02:23:32,417 INFO Set uid to user 0 succeeded
htadmin-1 | 2024-06-19 02:23:32,421 INFO RPC interface 'supervisor' initialized
htadmin-1 | 2024-06-19 02:23:32,421 CRIT Server 'unix_http_server' running without any HTTP authentication checking
htadmin-1 | 2024-06-19 02:23:32,422 INFO supervisord started with pid 751
htadmin-1 | 2024-06-19 02:23:33,425 INFO spawned: 'nginx' with pid 756
htadmin-1 | 2024-06-19 02:23:33,429 INFO spawned: 'php' with pid 757
htadmin-1 | 2024-06-19 02:23:34,935 INFO success: php entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
htadmin-1 | 2024-06-19 02:23:48,959 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
netbox-redis-1 | redis
netbox-redis-1 | uid=1000(redis) gid=1000(redis) groups=5(tty),1000(redis),1000(redis)
netbox-redis-1 | 2:23AM INF Listening at http://0.0.0.0:80 /...
netbox-postgres-1 | postgres
netbox-postgres-1 | uid=1000(postgres) gid=1000(postgres) groups=1000(postgres),1000(postgres)
netbox-postgres-1 | 2:23AM INF Listening at http://0.0.0.0:80 /...
nginx-proxy-1 | root
nginx-proxy-1 | uid=0(root) gid=0(root) groups=0(root),0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)
nginx-proxy-1 | 2024-06-19 02:23:33,185 INFO Set uid to user 0 succeeded
nginx-proxy-1 | 2024-06-19 02:23:33,190 INFO RPC interface 'supervisor' initialized
nginx-proxy-1 | 2024-06-19 02:23:33,190 CRIT Server 'unix_http_server' running without any HTTP authentication checking
nginx-proxy-1 | 2024-06-19 02:23:33,190 INFO supervisord started with pid 37
nginx-proxy-1 | 2024-06-19 02:23:34,194 INFO spawned: 'logaccess' with pid 100
nginx-proxy-1 | 2024-06-19 02:23:34,206 INFO spawned: 'logerrors' with pid 101
nginx-proxy-1 | 2024-06-19 02:23:34,214 INFO spawned: 'nginx' with pid 102
netbox-1 | usermod: no changes
netbox-1 | ubuntu
netbox-1 | uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),5(tty),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev)
netbox-1 | 2:23AM INF Listening at http://0.0.0.0:8080 /...
nginx-proxy-1 | 2024-06-19 02:23:44,240 INFO success: logaccess entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
file-monitor-1 | usermod: no changes
file-monitor-1 | monitor
upload-1 | root
file-monitor-1 | uid=1000(monitor) gid=1000(monitor) groups=1000(monitor),5(tty)
upload-1 | uid=0(root) gid=0(root) groups=0(root)
file-monitor-1 | 2024-06-19 02:23:31,939 INFO RPC interface 'supervisor' initialized
upload-1 | Creating SSH2 RSA key; this may take some time ...
logstash-1 | usermod: no changes
nginx-proxy-1 | 2024-06-19 02:23:44,249 INFO success: logerrors entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
upload-1 | 3072 SHA256:xKJYQx1c5pj0+p5haJzH0XzHyRMI3kfLKrVK73RMQW0 root@upload (RSA)
upload-1 | Creating SSH2 ECDSA key; this may take some time ...
logstash-1 | logstash
zeek-1 | usermod: no changes
upload-1 | 256 SHA256:Ta6iMPlrhDjaVar6/WONy9aC0iupDKNKTBLlbPTPGmc root@upload (ECDSA)
nginx-proxy-1 | 2024-06-19 02:23:44,249 INFO success: nginx entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
zeek-1 | root
nginx-proxy-1 | 172.18.0.1 - - [19/Jun/2024:02:24:04 +0000] "GET / HTTP/1.1" 401 179 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
zeek-live-1 | usermod: no changes
logstash-1 | uid=1000(logstash) gid=1000(logstash) groups=1000(logstash),5(tty)
logstash-1 | 2024-06-19 02:23:36,280 INFO RPC interface 'supervisor' initialized
file-monitor-1 | 2024-06-19 02:23:31,940 CRIT Server 'unix_http_server' running without any HTTP authentication checking
file-monitor-1 | 2024-06-19 02:23:31,940 INFO supervisord started with pid 757
file-monitor-1 | 2024-06-19 02:23:32,941 INFO spawned: 'clamd' with pid 762
file-monitor-1 | 2024-06-19 02:23:32,942 INFO spawned: 'cron' with pid 763
file-monitor-1 | 2024-06-19 02:23:32,943 INFO spawned: 'fileserve' with pid 764
file-monitor-1 | 2024-06-19 02:23:32,944 INFO spawned: 'logger' with pid 765
file-monitor-1 | 2024-06-19 02:23:32,945 INFO spawned: 'prune' with pid 766
file-monitor-1 | 2024-06-19 02:23:32,946 INFO spawned: 'clamav' with pid 767
file-monitor-1 | 2024-06-19 02:23:32,947 INFO spawned: 'yara' with pid 768
zeek-live-1 | root
zeek-live-1 | uid=0(root) gid=0(root) groups=0(root)
zeek-live-1 | 2024-06-19 02:23:45,616 INFO Set uid to user 0 succeeded
zeek-live-1 | 2024-06-19 02:23:45,619 INFO RPC interface 'supervisor' initialized
zeek-live-1 | 2024-06-19 02:23:45,619 CRIT Server 'unix_http_server' running without any HTTP authentication checking
zeek-live-1 | 2024-06-19 02:23:45,619 INFO supervisord started with pid 755
zeek-live-1 | 2024-06-19 02:23:46,621 INFO spawned: 'cron' with pid 762
zeek-live-1 | 2024-06-19 02:23:46,622 INFO spawned: 'live-zeek' with pid 763
zeek-live-1 | 2024-06-19T02:23:46Z {"level": "info", "msg": "read crontab: /opt/zeek/crontab"}
zeek-live-1 | Running via "/opt/zeek/bin/zeekctl" (5 processes) ...
zeek-live-1 | checking configurations ...
zeek-live-1 | 2024-06-19 02:23:47,768 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
zeek-live-1 | installing ...
logstash-1 | 2024-06-19 02:23:36,280 CRIT Server 'inet_http_server' running without any HTTP authentication checking
zeek-live-1 | creating policy directories ...
zeek-live-1 | installing site policies ...
zeek-live-1 | generating cluster-layout.zeek ...
zeek-1 | uid=0(root) gid=0(root) groups=0(root)
file-monitor-1 | 2024-06-19 02:23:32,948 INFO spawned: 'capa' with pid 769
file-monitor-1 | 2024-06-19 02:23:32,949 INFO spawned: 'watcher' with pid 770
file-monitor-1 | 2024-06-19T02:23:32Z {"level": "info", "msg": "read crontab: /etc/crontab"}
file-monitor-1 | 2024-06-19 02:23:32,973 INFO success: clamd entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
file-monitor-1 | 2024-06-19 02:23:32,974 INFO success: fileserve entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
file-monitor-1 | 2024-06-19 02:23:32,974 INFO success: prune entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
file-monitor-1 | LibClamAV Warning: **************************************************
file-monitor-1 | LibClamAV Warning: *** The virus database is older than 7 days! ***
file-monitor-1 | LibClamAV Warning: *** Please update it as soon as possible. ***
file-monitor-1 | LibClamAV Warning: **************************************************
file-monitor-1 | serving /zeek/extract_files at port 8440
file-monitor-1 | 2024-06-19 02:23:34,115 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: Global time limit set to 120000 milliseconds.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: Global size limit set to 536870912 bytes.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: File size limit set to 134217728 bytes.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: Recursion level limit set to 16.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: Files limit set to 10000.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxEmbeddedPE limit set to 10485760 bytes.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxHTMLNormalize limit set to 10485760 bytes.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxHTMLNoTags limit set to 2097152 bytes.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxScriptNormalize limit set to 5242880 bytes.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxZipTypeRcg limit set to 1048576 bytes.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxPartitions limit set to 50.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxIconsPE limit set to 100.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: MaxRecHWP3 limit set to 16.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: PCREMatchLimit limit set to 10000.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: PCRERecMatchLimit limit set to 5000.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Limits: PCREMaxFileSize limit set to 26214400.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Archive support enabled.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> AlertExceedsMax heuristic detection disabled.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Heuristic alerts enabled.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Portable Executable support enabled.
logstash-1 | 2024-06-19 02:23:36,280 INFO supervisord started with pid 743
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET / HTTP/1.1" 200 7145 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> ELF support enabled.
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/bootstrap-icons.css HTTP/1.1" 200 65618 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Mail files support enabled.
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/google-fonts.css HTTP/1.1" 200 832 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
zeek-1 | 2024-06-19 02:23:45,613 INFO Set uid to user 0 succeeded
logstash-1 | 2024-06-19 02:23:37,282 INFO spawned: 'logstash' with pid 749
zeek-1 | 2024-06-19 02:23:45,617 INFO RPC interface 'supervisor' initialized
logstash-1 | 2024-06-19 02:23:38,284 INFO success: logstash entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
zeek-live-1 | generating local-networks.zeek ...
upload-1 | Creating SSH2 ED25519 key; this may take some time ...
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> OLE2 support enabled.
zeek-live-1 | generating zeekctl-config.zeek ...
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> PDF support enabled.
logstash-1 |
zeek-live-1 | generating zeekctl-config.sh ...
zeek-live-1 | stopping ...
zeek-live-1 | stopping workers ...
zeek-live-1 | stopping proxy ...
zeek-live-1 | stopping manager ...
zeek-live-1 | stopping logger ...
zeek-live-1 | starting ...
zeek-live-1 | starting logger ...
zeek-live-1 | starting manager ...
zeek-live-1 | starting proxy ...
zeek-live-1 | starting workers ...
zeek-live-1 | 2024-06-19 02:26:59,513 INFO success: live-zeek entered RUNNING state, process has stayed up for > than 180 seconds (startsecs)
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/styles.css HTTP/1.1" 200 238932 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /js/scripts.js HTTP/1.1" 200 324 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /assets/img/arkime.svg HTTP/1.1" 200 9940 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /assets/img/opensearch_mark_default.svg HTTP/1.1" 200 1103 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /assets/img/netbox_icon.svg HTTP/1.1" 200 835 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /assets/img/cyberchef.svg HTTP/1.1" 200 35642 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /js/bootstrap.bundle.min.js HTTP/1.1" 200 80420 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /assets/img/bg-masthead.png HTTP/1.1" 200 81181 "https://localhost/css/styles.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/S6uyw4BMUTPHjx4wWw.ttf HTTP/1.1" 200 60540 "https://localhost/css/google-fonts.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/S6u9w4BMUTPHh6UVSwiPHA.ttf HTTP/1.1" 200 59048 "https://localhost/css/google-fonts.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/S6u9w4BMUTPHh7USSwiPHA.ttf HTTP/1.1" 200 63404 "https://localhost/css/google-fonts.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/bootstrap-icons.woff2 HTTP/1.1" 200 90528 "https://localhost/css/bootstrap-icons.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:19 +0000] "GET /css/S6u8w4BMUTPHjxsAXC-v.ttf HTTP/1.1" 200 60976 "https://localhost/css/google-fonts.css" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> SWF support enabled.
upload-1 | 256 SHA256:AaIACIbAdfOfHRt1reE0z7cXtPFvbyhFSgAGkGzndds root@upload (ED25519)
zeek-1 | 2024-06-19 02:23:45,617 CRIT Server 'unix_http_server' running without any HTTP authentication checking
upload-1 | invoke-rc.d: could not determine current runlevel
zeek-1 | 2024-06-19 02:23:45,617 INFO supervisord started with pid 755
upload-1 | invoke-rc.d: policy-rc.d denied execution of restart.
upload-1 | 2024-06-19 02:23:32,636 INFO Set uid to user 0 succeeded
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> HTML support enabled.
logstash-1 | Using bundled JDK: /usr/share/logstash/jdk
logstash-1 | /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> XMLDOCS support enabled.
nginx-proxy-1 | 172.18.0.1 - tboy [19/Jun/2024:02:24:20 +0000] "GET /assets/favicon.ico HTTP/1.1" 200 34494 "https://localhost/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0"
zeek-1 | 2024-06-19 02:23:46,619 INFO spawned: 'cron' with pid 798
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> HWP3 support enabled.
file-monitor-1 | Wed Jun 19 02:23:45 2024 -> Self checking every 3600 seconds.
file-monitor-1 | 2024-06-19 02:24:03,304 INFO success: logger entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
file-monitor-1 | 2024-06-19 02:24:03,305 INFO success: clamav entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
zeek-1 | 2024-06-19 02:23:46,620 INFO spawned: 'pcap-zeek' with pid 799
upload-1 | 2024-06-19 02:23:32,639 INFO RPC interface 'supervisor' initialized
zeek-1 | 2024-06-19T02:23:46Z {"level": "info", "msg": "read crontab: /opt/zeek/crontab"}
upload-1 | 2024-06-19 02:23:32,639 CRIT Server 'unix_http_server' running without any HTTP authentication checking
zeek-1 | 2024-06-19 02:23:47,640 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
zeek-1 | 2024-06-19 02:24:01,662 INFO success: pcap-zeek entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
logstash-1 | /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
logstash-1 | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash-1 |
logstash-1 | [2024-06-19T02:23:46,185][INFO ][org.logstash.secret.store.backend.JavaKeyStore] Created Logstash keystore at /usr/share/logstash/config/logstash.keystore
logstash-1 | Created Logstash keystore at /usr/share/logstash/config/logstash.keystore
file-monitor-1 | 2024-06-19 02:24:03,305 INFO success: yara entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
upload-1 | 2024-06-19 02:23:32,639 INFO supervisord started with pid 766
file-monitor-1 | 2024-06-19 02:24:03,305 INFO success: capa entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
upload-1 | 2024-06-19 02:23:33,642 INFO spawned: 'cron' with pid 926
file-monitor-1 | 2024-06-19 02:24:03,305 INFO success: watcher entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)
upload-1 | 2024-06-19 02:23:33,651 INFO spawned: 'nginx' with pid 927
upload-1 | 2024-06-19 02:23:33,654 INFO spawned: 'php' with pid 928
upload-1 | 2024-06-19 02:23:33,659 INFO spawned: 'sshd' with pid 929
logstash-1 | opensearch-local is up and healthy at http://opensearch:9200
upload-1 | 2024-06-19T02:23:33Z {"level": "info", "msg": "read crontab: /etc/crontab"}
upload-1 | 2024-06-19 02:23:34,898 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
logstash-1 | Waiting until opensearch-local has index template "malcolm_template"...
upload-1 | 2024-06-19 02:23:34,898 INFO success: php entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
upload-1 | 2024-06-19 02:23:34,898 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
upload-1 | 2024-06-19 02:23:48,919 INFO success: nginx entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
pcap-capture-1 | usermod: no changes
pcap-capture-1 | root
pcap-capture-1 | uid=0(root) gid=0(root) groups=0(root)
pcap-capture-1 | 2024-06-19 02:23:31,030 INFO Included extra file "/etc/supervisor.d/capture-groups.conf" during parsing
pcap-capture-1 | 2024-06-19 02:23:31,030 INFO Included extra file "/etc/supervisor.d/netsniff-ens18.conf" during parsing
pcap-capture-1 | 2024-06-19 02:23:31,030 INFO Included extra file "/etc/supervisor.d/tcpdump-ens18.conf" during parsing
pcap-capture-1 | 2024-06-19 02:23:31,030 INFO Set uid to user 0 succeeded
pcap-capture-1 | 2024-06-19 02:23:31,034 INFO RPC interface 'supervisor' initialized
pcap-capture-1 | 2024-06-19 02:23:31,035 CRIT Server 'unix_http_server' running without any HTTP authentication checking
pcap-capture-1 | 2024-06-19 02:23:31,035 INFO supervisord started with pid 76
pcap-capture-1 | 2024-06-19 02:23:32,037 INFO spawned: 'netsniff-ens18' with pid 77
pcap-capture-1 | 2024-06-19 02:23:32,038 INFO spawned: 'netsniff-roll' with pid 78
pcap-monitor-1 | usermod: no changes
pcap-monitor-1 | root
pcap-monitor-1 | uid=0(root) gid=0(root) groups=0(root)
pcap-monitor-1 | 2024-06-19 02:23:32,582 INFO Set uid to user 0 succeeded
pcap-monitor-1 | 2024-06-19 02:23:32,586 INFO RPC interface 'supervisor' initialized
pcap-monitor-1 | 2024-06-19 02:23:32,586 CRIT Server 'unix_http_server' running without any HTTP authentication checking
pcap-monitor-1 | 2024-06-19 02:23:32,586 INFO supervisord started with pid 754
pcap-monitor-1 | 2024-06-19 02:23:33,594 INFO spawned: 'pcap-publisher' with pid 759
logstash-1 | opensearch-local index template "malcolm_template" exists
logstash-1 | 2024/06/19 02:24:12 Setting 'pipeline.batch.delay' from environment.
logstash-1 | 2024/06/19 02:24:12 Setting 'pipeline.batch.size' from environment.
logstash-1 | 2024/06/19 02:24:12 Setting 'pipeline.workers' from environment.
logstash-1 | Using bundled JDK: /usr/share/logstash/jdk
logstash-1 | /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
logstash-1 | /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
logstash-1 | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash-1 | [2024-06-19T02:24:18,903][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
logstash-1 | [2024-06-19T02:24:18,907][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.13.4", "jruby.version"=>"jruby 9.4.5.0 (3.1.4) 2023-11-02 1abae2700f OpenJDK 64-Bit Server VM 17.0.11+9 on 17.0.11+9 +indy +jit [x86_64-linux]"}
logstash-1 | [2024-06-19T02:24:18,908][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx2500m, -Xms2500m, -Xss1536k, -XX:-HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/./urandom, -Dlog4j.formatMsgNoLookups=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
logstash-1 | [2024-06-19T02:24:18,909][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
logstash-1 | [2024-06-19T02:24:18,909][INFO ][logstash.runner ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
logstash-1 | [2024-06-19T02:24:18,913][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
pcap-monitor-1 | 2024-06-19 02:23:33,598 INFO spawned: 'watch-upload' with pid 760
pcap-capture-1 | 2024-06-19 02:23:37,172 INFO success: netsniff-ens18 entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
pcap-capture-1 | 2024-06-19 02:23:47,182 INFO success: netsniff-roll entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
logstash-1 | [2024-06-19T02:24:18,914][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
pcap-monitor-1 | 2024-06-19 02:24:09,076 INFO success: watch-upload entered RUNNING state, process has stayed up for > than 35 seconds (startsecs)
pcap-monitor-1 | 2024-06-19 02:24:43,536 INFO success: pcap-publisher entered RUNNING state, process has stayed up for > than 65 seconds (startsecs)
logstash-1 | [2024-06-19T02:24:19,014][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"c354e463-42ba-4c9a-8d1a-adfbb41551d2", :path=>"/usr/share/logstash/data/uuid"}
pcap-monitor-1 | 2024-06-19 02:29:53 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:39.776s]
logstash-1 | [2024-06-19T02:24:19,220][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash-1 | [2024-06-19T02:24:19,501][INFO ][org.reflections.Reflections] Reflections took 118 ms to scan 1 urls, producing 132 keys and 468 values
pcap-monitor-1 | socket.gaierror: [Errno -3] Temporary failure in name resolution
pcap-monitor-1 |
pcap-monitor-1 | During handling of the above exception, another exception occurred:
logstash-1 | [2024-06-19T02:24:19,739][INFO ][logstash.javapipeline ] Pipeline `malcolm-input` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
logstash-1 | [2024-06-19T02:24:19,873][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been created for key: send_to. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
logstash-1 | [2024-06-19T02:24:19,892][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"malcolm-input", "pipeline.workers"=>4, "pipeline.batch.size"=>75, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>300, "pipeline.sources"=>["/usr/share/logstash/malcolm-pipelines/input/01_beats_input.conf", "/usr/share/logstash/malcolm-pipelines/input/99_input_forward.conf"], :thread=>"#<Thread:0x6c39482a /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
logstash-1 | [2024-06-19T02:24:20,276][INFO ][logstash.javapipeline ] Pipeline Java execution initialization time {"seconds"=>0.38}
logstash-1 | [2024-06-19T02:24:20,279][INFO ][logstash.inputs.beats ] Starting input listener {:address=>"0.0.0.0:5044"}
logstash-1 | /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/nokogiri-1.16.4-java/lib/nokogiri/xml/node.rb:1007: warning: method redefined; discarding old attr
logstash-1 | [2024-06-19T02:24:20,713][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"malcolm-input"}
logstash-1 | [2024-06-19T02:24:21,318][INFO ][org.logstash.beats.Server] Starting server on port: 5044
suricata-live-1 | usermod: no changes
suricata-live-1 | root
arkime-live-1 | usermod: no changes
suricata-live-1 | uid=0(root) gid=0(root) groups=0(root)
arkime-live-1 | root
suricata-live-1 | 2024-06-19 02:23:45,692 INFO Set uid to user 0 succeeded
suricata-live-1 | 2024-06-19 02:23:45,694 INFO RPC interface 'supervisor' initialized
suricata-1 | usermod: no changes
suricata-1 | root
suricata-live-1 | 2024-06-19 02:23:45,694 CRIT Server 'unix_http_server' running without any HTTP authentication checking
suricata-live-1 | 2024-06-19 02:23:45,694 INFO supervisord started with pid 758
suricata-live-1 | 2024-06-19 02:23:46,697 INFO spawned: 'cron' with pid 769
arkime-live-1 | uid=0(root) gid=0(root) groups=0(root)
suricata-1 | uid=0(root) gid=0(root) groups=0(root)
arkime-live-1 | 2024-06-19 02:23:31,813 INFO Set uid to user 0 succeeded
suricata-1 | 2024-06-19 02:23:31 ERROR: ["Command ['suricata', '-v', '-c', '/tmp/tmpnryhl25g/suricata.yaml', '-l', '/tmp/tmpnryhl25g', '-T'] not found or unable to execute"]
suricata-1 | 2024-06-19 02:23:31,667 INFO Set uid to user 0 succeeded
arkime-live-1 | 2024-06-19 02:23:31,818 INFO RPC interface 'supervisor' initialized
pcap-monitor-1 |
arkime-live-1 | 2024-06-19 02:23:31,818 CRIT Server 'unix_http_server' running without any HTTP authentication checking
arkime-live-1 | 2024-06-19 02:23:31,818 INFO supervisord started with pid 754
arkime-live-1 | 2024-06-19 02:23:32,820 INFO spawned: 'viewer' with pid 798
arkime-live-1 | 2024-06-19 02:23:32,821 INFO spawned: 'wise' with pid 799
arkime-live-1 | 2024-06-19 02:23:33,825 INFO success: viewer entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
suricata-live-1 | 2024-06-19 02:23:46,698 INFO spawned: 'live-suricata' with pid 770
suricata-live-1 | Notice: suricata: This is Suricata version 7.0.5 RELEASE running in SYSTEM mode
arkime-live-1 | 2024-06-19 02:23:33,825 INFO success: wise entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
suricata-live-1 | Info: cpu: CPUs/cores online: 16
suricata-live-1 | Info: suricata: Setting engine mode to IDS mode by default
suricata-1 | 2024-06-19 02:23:31,671 INFO RPC interface 'supervisor' initialized
suricata-1 | 2024-06-19 02:23:31,671 CRIT Server 'unix_http_server' running without any HTTP authentication checking
suricata-1 | 2024-06-19 02:23:31,671 INFO supervisord started with pid 758
pcap-monitor-1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7cfb406ec050>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
suricata-1 | 2024-06-19 02:23:32,674 INFO spawned: 'cron' with pid 769
suricata-1 | 2024-06-19 02:23:32,676 INFO spawned: 'pcap-suricata' with pid 770
suricata-1 | 2024-06-19T02:23:32Z {"level": "info", "msg": "read crontab: /etc/crontab"}
suricata-1 | 2024-06-19 02:23:33,706 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
suricata-live-1 | Info: exception-policy: master exception-policy set to: auto
suricata-1 | 2024-06-19 02:23:47,729 INFO success: pcap-suricata entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
suricata-live-1 | 2024-06-19T02:23:46Z {"level": "info", "msg": "read crontab: /etc/crontab"}
suricata-live-1 | Info: app-layer-ftp: FTP memcap: 67108864
suricata-live-1 | Info: coredump-config: Max dump is 0
pcap-monitor-1 | 2024-06-19 02:30:38 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:33.390s]
pcap-monitor-1 | socket.gaierror: [Errno -3] Temporary failure in name resolution
pcap-monitor-1 |
pcap-monitor-1 | During handling of the above exception, another exception occurred:
pcap-monitor-1 |
pcap-monitor-1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7cfb406ec5d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
pcap-monitor-1 | 2024-06-19 02:31:36 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:44.258s]
pcap-monitor-1 | socket.gaierror: [Errno -3] Temporary failure in name resolution
pcap-monitor-1 |
pcap-monitor-1 | During handling of the above exception, another exception occurred:
pcap-monitor-1 |
pcap-monitor-1 | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7cfb406edfd0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
pcap-monitor-1 | 2024-06-19 02:32:55 WARNING: GET http://opensearch:9200/_cluster/health [status:N/A request:59.069s]
pcap-monitor-1 | socket.gaierror: [Errno -3] Temporary failure in name resolution
suricata-live-1 | Info: coredump-config: Core dump setting attempted is 0
suricata-live-1 | Info: coredump-config: Core dump size set to 0
suricata-live-1 | Info: logopenfile: eve-log output device (regular) initialized: eve-%Y%m%d_%H%M%S.json
suricata-live-1 | 2024-06-19 02:23:47,753 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
suricata-live-1 | Info: detect: 1 rule files processed. 43324 rules successfully loaded, 0 rules failed, 0
suricata-live-1 | Info: threshold-config: Threshold config parsed: 0 rule(s) found
suricata-live-1 | Info: detect: 43327 signatures processed. 1104 are IP-only rules, 4864 are inspecting packet payload, 37147 inspect application layer, 108 are decoder event only
suricata-live-1 | Info: runmodes: ens18: creating 2 threads
suricata-live-1 | Info: unix-manager: unix socket '/var/run/suricata/suricata-command.socket'
suricata-live-1 | Notice: threads: Threads created -> W: 2 FM: 1 FR: 1 Engine started.
suricata-live-1 | 2024-06-19 02:26:51,512 INFO success: live-suricata entered RUNNING state, process has stayed up for > than 180 seconds (startsecs)
opensearch-1 | usermod: no changes
opensearch-1 | opensearch
opensearch-1 | uid=1000(opensearch) gid=1000(opensearch) groups=1000(opensearch)
opensearch-1 |
opensearch-1 | OpenSearch Security Plugin does not exist, disable by default
opensearch-1 | OpenSearch Performance Analyzer Plugin does not exist, disable by default
netbox-redis-cache-1 | redis
netbox-redis-cache-1 | uid=1000(redis) gid=1000(redis) groups=5(tty),1000(redis),1000(redis)
netbox-redis-cache-1 | 2:23AM INF Listening at http://0.0.0.0:80 /...
opensearch-1 | WARNING: Using incubator modules: jdk.incubator.vector
opensearch-1 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.14.0.jar)
opensearch-1 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
opensearch-1 | WARNING: System::setSecurityManager will be removed in a future release
opensearch-1 | Jun 19, 2024 2:23:36 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
opensearch-1 | WARNING: COMPAT locale provider will be removed in a future release
opensearch-1 | WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.14.0.jar)
opensearch-1 | WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
opensearch-1 | WARNING: System::setSecurityManager will be removed in a future release
filebeat-1 | usermod: no changes
filebeat-1 | root
opensearch-1 | [2024-06-19T02:23:41,367][WARN ][o.o.s.p.SQLPlugin ] [opensearch] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
opensearch-1 | [2024-06-19T02:23:42,380][WARN ][o.o.g.DanglingIndicesState] [opensearch] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
freq-1 | usermod: no changes
freq-1 | freq
freq-1 | uid=1000(freq) gid=1000(freq) groups=1000(freq),5(tty)
freq-1 | 2024-06-19 02:23:31,671 INFO RPC interface 'supervisor' initialized
freq-1 | 2024-06-19 02:23:31,671 CRIT Server 'unix_http_server' running without any HTTP authentication checking
freq-1 | 2024-06-19 02:23:31,672 INFO supervisord started with pid 757
freq-1 | 2024-06-19 02:23:32,673 INFO spawned: 'freq' with pid 762
freq-1 | 2024-06-19 02:23:37,681 INFO success: freq entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
filebeat-1 | uid=0(root) gid=0(root) groups=0(root)
opensearch-1 | [2024-06-19T02:24:07,213][WARN ][o.o.c.m.MetadataIndexTemplateService] [opensearch] index template [malcolm_template] has index patterns [arkime_sessions3-*] matching patterns from existing older templates [arkime_sessions3_ecs_template,arkime_sessions3_template] with patterns (arkime_sessions3_ecs_template => [arkime_sessions3-*],arkime_sessions3_template => [arkime_sessions3-*]); this template [malcolm_template] will take precedence during new index creation
filebeat-1 | 2024-06-19 02:23:32,337 INFO Set uid to user 0 succeeded
filebeat-1 | 2024-06-19 02:23:32,349 INFO RPC interface 'supervisor' initialized
filebeat-1 | 2024-06-19 02:23:32,349 CRIT Server 'unix_http_server' running without any HTTP authentication checking
filebeat-1 | 2024-06-19 02:23:32,350 INFO supervisord started with pid 740
filebeat-1 | 2024-06-19 02:23:33,353 INFO spawned: 'cron' with pid 746
filebeat-1 | 2024-06-19 02:23:33,362 INFO spawned: 'filebeat' with pid 747
filebeat-1 | 2024-06-19 02:23:33,369 INFO spawned: 'watch-upload' with pid 749
filebeat-1 | 2024-06-19T02:23:33Z {"level": "info", "msg": "read crontab: /etc/crontab"}
filebeat-1 | 2024-06-19 02:23:33,698 INFO success: filebeat entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
filebeat-1 | 2024-06-19T02:23:34.604Z Home path: [/usr/share/filebeat-logs] Config path: [/usr/share/filebeat-logs] Data path: [/usr/share/filebeat-logs/data] Logs path: [/usr/share/filebeat-logs/logs]
filebeat-1 | 2024-06-19 02:23:34,605 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
filebeat-1 | 2024-06-19T02:23:34.613Z Beat ID: 3d8664d5-7f27-4ba5-b722-b8f22321bea9
filebeat-1 | 2024-06-19T02:23:34.643Z Syscall filter successfully installed
filebeat-1 | 2024-06-19T02:23:34.665Z {"message": "Beat info", "system_info": {"beat": {"path": {"config": "/usr/share/filebeat-logs", "data": "/usr/share/filebeat-logs/data", "home": "/usr/share/filebeat-logs", "logs": "/usr/share/filebeat-logs/logs"}, "type": "filebeat", "uuid": "3d8664d5-7f27-4ba5-b722-b8f22321bea9"}, "ecs.version": "1.6.0"}}
filebeat-1 | 2024-06-19T02:23:34.665Z {"message": "Build info", "system_info": {"build": {"commit": "b24ddd14c936c216817afed0cc7d0b23fd920194", "libbeat": "8.13.4", "time": "2024-05-06T06:35:03.000Z", "version": "8.13.4"}, "ecs.version": "1.6.0"}}
filebeat-1 | 2024-06-19T02:23:34.665Z {"message": "Go runtime info", "system_info": {"go": {"os": "linux", "arch": "amd64", "max_procs": 16, "version": "go1.21.9"}, "ecs.version": "1.6.0"}}
filebeat-1 | 2024-06-19T02:23:34.666Z {"message": "Host info", "system_info": {"host": {"architecture": "x86_64", "boot_time": "2024-06-19T02:22:25Z", "containerized": false, "name": "filebeat", "ip": ["127.0.0.1", "::1", "172.18.0.11"], "kernel_version": "6.5.0-41-generic", "mac": ["02:42:ac:12:00:0b"], "os": {"type": "linux", "family": "debian", "platform": "ubuntu", "name": "Ubuntu", "version": "20.04.6 LTS (Focal Fossa)", "major": 20, "minor": 4, "patch": 6, "codename": "focal"}, "timezone": "UTC", "timezone_offset_sec": 0}, "ecs.version": "1.6.0"}}
filebeat-1 | 2024-06-19T02:23:34.670Z {"message": "Process info", "system_info": {"process": {"capabilities": {"inheritable": null, "permitted": null, "effective": null, "bounding": ["chown", "dac_override", "fowner", "fsetid", "kill", "setgid", "setuid", "setpcap", "net_bind_service", "net_raw", "sys_chroot", "mknod", "audit_write", "setfcap"], "ambient": null}, "cwd": "/usr/share/filebeat-logs", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 759, "ppid": 747, "seccomp": {"mode": "filter", "no_new_privs": true}, "start_time": "2024-06-19T02:23:33.450Z"}, "ecs.version": "1.6.0"}}
filebeat-1 | 2024-06-19T02:23:34.670Z Setup Beat: filebeat; Version: 8.13.4
filebeat-1 | 2024-06-19T02:23:34.704Z Beat name: mnescan
filebeat-1 | 2024-06-19T02:23:34.704Z Enabled modules/filesets:
filebeat-1 | 2024-06-19T02:23:34.706Z Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
filebeat-1 | 2024-06-19T02:23:34.707Z filebeat start running.
filebeat-1 | 2024-06-19T02:23:34.711Z Finished loading transaction log file for '/usr/share/filebeat-logs/data/registry/filebeat'. Active transaction id=0
filebeat-1 | 2024-06-19T02:23:34.713Z Finished loading transaction log file for '/usr/share/filebeat-logs/data/registry/filebeat'. Active transaction id=0
filebeat-1 | 2024-06-19T02:23:34.713Z Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
filebeat-1 | 2024-06-19T02:23:34.713Z States Loaded from registrar: 0
filebeat-1 | 2024-06-19T02:23:34.713Z Loading Inputs: 5
filebeat-1 | 2024-06-19T02:23:34.713Z starting input, keys present on the config: [filebeat.inputs.0.clean_inactive filebeat.inputs.0.clean_removed filebeat.inputs.0.close_eof filebeat.inputs.0.close_inactive filebeat.inputs.0.close_removed filebeat.inputs.0.close_renamed filebeat.inputs.0.compression_level filebeat.inputs.0.exclude_files.0 filebeat.inputs.0.exclude_lines.0 filebeat.inputs.0.fields_under_root filebeat.inputs.0.ignore_older filebeat.inputs.0.paths.0 filebeat.inputs.0.scan_frequency filebeat.inputs.0.symlinks filebeat.inputs.0.tags.0 filebeat.inputs.0.type]
filebeat-1 | 2024-06-19T02:23:34.720Z Configured paths: [/zeek/current/*.log]
filebeat-1 | 2024-06-19T02:23:34.720Z Starting input (ID: 14792403151094495097)
filebeat-1 | 2024-06-19T02:23:34.720Z starting input, keys present on the config: [filebeat.inputs.1.clean_inactive filebeat.inputs.1.clean_removed filebeat.inputs.1.close_eof filebeat.inputs.1.close_inactive filebeat.inputs.1.close_removed filebeat.inputs.1.close_renamed filebeat.inputs.1.compression_level filebeat.inputs.1.exclude_lines.0 filebeat.inputs.1.fields_under_root filebeat.inputs.1.ignore_older filebeat.inputs.1.paths.0 filebeat.inputs.1.scan_frequency filebeat.inputs.1.symlinks filebeat.inputs.1.tags.0 filebeat.inputs.1.type]
filebeat-1 | 2024-06-19T02:23:34.722Z Configured paths: [/zeek/live/spool/logger-*/*.log]
filebeat-1 | 2024-06-19T02:23:34.723Z Starting input (ID: 17424728247939255894)
filebeat-1 | 2024-06-19T02:23:34.723Z starting input, keys present on the config: [filebeat.inputs.2.clean_inactive filebeat.inputs.2.clean_removed filebeat.inputs.2.close_eof filebeat.inputs.2.close_inactive filebeat.inputs.2.close_removed filebeat.inputs.2.close_renamed filebeat.inputs.2.compression_level filebeat.inputs.2.exclude_lines.0 filebeat.inputs.2.fields_under_root filebeat.inputs.2.ignore_older filebeat.inputs.2.paths.0 filebeat.inputs.2.scan_frequency filebeat.inputs.2.symlinks filebeat.inputs.2.tags.0 filebeat.inputs.2.type]
filebeat-1 | 2024-06-19T02:23:34.725Z Configured paths: [/zeek/current/signatures(_carved*).log]
filebeat-1 | 2024-06-19T02:23:34.725Z Starting input (ID: 17708124363748150562)
filebeat-1 | 2024-06-19T02:23:34.725Z starting input, keys present on the config: [filebeat.inputs.3.clean_inactive filebeat.inputs.3.clean_removed filebeat.inputs.3.close_eof filebeat.inputs.3.close_inactive filebeat.inputs.3.close_removed filebeat.inputs.3.close_renamed filebeat.inputs.3.compression_level filebeat.inputs.3.fields_under_root filebeat.inputs.3.ignore_older filebeat.inputs.3.paths.0 filebeat.inputs.3.scan_frequency filebeat.inputs.3.symlinks filebeat.inputs.3.tags.0 filebeat.inputs.3.type]
filebeat-1 | 2024-06-19T02:23:34.727Z Configured paths: [/suricata/eve*.json]
filebeat-1 | 2024-06-19T02:23:34.733Z Starting input (ID: 11290400322194405465)
filebeat-1 | 2024-06-19T02:23:34.733Z starting input, keys present on the config: [filebeat.inputs.4.clean_inactive filebeat.inputs.4.clean_removed filebeat.inputs.4.close_eof filebeat.inputs.4.close_inactive filebeat.inputs.4.close_removed filebeat.inputs.4.close_renamed filebeat.inputs.4.compression_level filebeat.inputs.4.fields_under_root filebeat.inputs.4.ignore_older filebeat.inputs.4.paths.0 filebeat.inputs.4.scan_frequency filebeat.inputs.4.symlinks filebeat.inputs.4.tags.0 filebeat.inputs.4.type]
filebeat-1 | 2024-06-19T02:23:34.733Z Configured paths: [/suricata/live/eve*.json]
filebeat-1 | 2024-06-19T02:23:34.734Z Starting input (ID: 13403698867348159062)
filebeat-1 | 2024-06-19T02:23:34.734Z Loading and starting Inputs completed. Enabled inputs: 5
filebeat-1 | 2024-06-19T02:23:44.797Z Harvester started for paths: [/zeek/current/signatures(_carved*).log]: /zeek/current/signatures(_carved).log
filebeat-1 | 2024-06-19T02:23:54.743Z Harvester started for paths: [/suricata/live/eve*.json]: /suricata/live/eve-20240619_022346.json
filebeat-1 | 2024-06-19T02:23:54.747Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/known_services.log
filebeat-1 | 2024-06-19T02:23:54.749Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/ocsp.log
filebeat-1 | 2024-06-19T02:23:54.749Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/ssl.log
filebeat-1 | 2024-06-19T02:23:54.748Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/loaded_scripts.log
filebeat-1 | 2024-06-19T02:23:54.752Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/files.log
filebeat-1 | 2024-06-19T02:23:54.754Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/stdout.log
filebeat-1 | 2024-06-19T02:23:54.754Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/cluster.log
filebeat-1 | 2024-06-19T02:23:54.754Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/http.log
filebeat-1 | 2024-06-19T02:23:54.755Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/software.log
filebeat-1 | 2024-06-19T02:23:54.757Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/stderr.log
filebeat-1 | 2024-06-19T02:23:54.765Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/packet_filter.log
filebeat-1 | 2024-06-19T02:23:54.766Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/weird.log
filebeat-1 | 2024-06-19T02:23:54.766Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/broker.log
filebeat-1 | 2024-06-19T02:23:54.767Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/known_hosts.log
filebeat-1 | 2024-06-19T02:24:04.770Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/conn.log
filebeat-1 | 2024-06-19T02:24:04.770Z Harvester started for paths: [/zeek/live/spool/logger-*/*.log]: /zeek/live/spool/logger-1/dns.log
filebeat-1 | 2024-06-19T02:24:05.937Z Failed to connect to backoff(async(tcp://logstash:5044)): dial tcp 172.18.0.17:5044: connect: connection refused
filebeat-1 | 2024-06-19T02:24:05.937Z Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 1 reconnect attempt(s)
filebeat-1 | 2024-06-19T02:24:07.956Z Failed to connect to backoff(async(tcp://logstash:5044)): dial tcp 172.18.0.17:5044: connect: connection refused
filebeat-1 | 2024-06-19T02:24:07.956Z Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 2 reconnect attempt(s)
filebeat-1 | 2024-06-19 02:24:08,961 INFO success: watch-upload entered RUNNING state, process has stayed up for > than 35 seconds (startsecs)
filebeat-1 | 2024-06-19T02:24:13.231Z Failed to connect to backoff(async(tcp://logstash:5044)): dial tcp 172.18.0.17:5044: connect: connection refused
filebeat-1 | 2024-06-19T02:24:13.232Z Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 3 reconnect attempt(s)
filebeat-1 | 2024-06-19T02:24:31.418Z Failed to connect to backoff(async(tcp://logstash:5044)): dial tcp 172.18.0.17:5044: connect: connection refused
dashboards-1 | usermod: no changes
dashboards-1 | opensearch-dashboards
dashboards-1 | uid=1000(opensearch-dashboards) gid=1000(opensearch-dashboards) groups=1000(opensearch-dashboards),5(tty)
filebeat-1 | 2024-06-19T02:24:43.911Z Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 4 reconnect attempt(s)
dashboards-helper-1 | usermod: no changes
dashboards-1 | log [02:23:44.062] [info][plugins-service] Plugin "dataSourceManagement" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]
arkime-1 | usermod: no changes
dashboards-helper-1 | helper
dashboards-helper-1 | uid=1000(helper) gid=1000(helper) groups=5(tty),42(shadow),1000(helper),1000(helper)
dashboards-helper-1 | 2024-06-19 02:23:32,631 INFO RPC interface 'supervisor' initialized
dashboards-helper-1 | 2024-06-19 02:23:32,632 CRIT Server 'unix_http_server' running without any HTTP authentication checking
dashboards-helper-1 | 2024-06-19 02:23:32,632 INFO supervisord started with pid 32
filebeat-1 | 2024-06-19T02:25:45.847Z DNS lookup failure "logstash": lookup logstash on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout
filebeat-1 | 2024-06-19T02:27:24.038Z Failed to connect to backoff(async(tcp://logstash:5044)): lookup logstash on 127.0.0.11:53: dial udp 127.0.0.11:53: i/o timeout
filebeat-1 | 2024-06-19T02:28:05.968Z Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 5 reconnect attempt(s)
filebeat-1 | 2024-06-19T02:29:24.418Z DNS lookup failure "logstash": lookup logstash on 127.0.0.11:53: read udp 127.0.0.1:37425->127.0.0.11:53: i/o timeout
filebeat-1 | 2024-06-19T02:30:21.956Z Failed to connect to backoff(async(tcp://logstash:5044)): lookup logstash on 127.0.0.11:53: read udp 127.0.0.1:37425->127.0.0.11:53: i/o timeout
filebeat-1 | 2024-06-19T02:30:23.599Z Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 6 reconnect attempt(s)
filebeat-1 | 2024-06-19T02:30:00Z /usr/local/bin/filebeat-process-zeek-folder.sh: not starting: job is still running since 2024-06-19 02:25:00 +0000 UTC (1m0s elapsed)
filebeat-1 | 2024-06-19T02:30:40.721Z DNS lookup failure "logstash": lookup logstash on 127.0.0.11:53: read udp 127.0.0.1:46548->127.0.0.11:53: i/o timeout
filebeat-1 | 2024-06-19T02:30:10Z /usr/local/bin/clean-processed-folder.py: not starting: job is still running since 2024-06-19 02:25:00 +0000 UTC (5m0s elapsed)
filebeat-1 | 2024-06-19T02:31:40.098Z Failed to connect to backoff(async(tcp://logstash:5044)): lookup logstash on 127.0.0.11:53: read udp 127.0.0.1:46548->127.0.0.11:53: i/o timeout
filebeat-1 | 2024-06-19T02:32:07.036Z Attempting to reconnect to backoff(async(tcp://logstash:5044)) with 7 reconnect attempt(s)
dashboards-helper-1 | 2024-06-19 02:23:33,634 INFO spawned: 'cron' with pid 37
dashboards-helper-1 | 2024-06-19 02:23:33,643 INFO spawned: 'idxinit' with pid 38
dashboards-helper-1 | 2024-06-19 02:23:33,653 INFO spawned: 'maps' with pid 39
dashboards-helper-1 | 2024-06-19T02:23:33Z {"level": "info", "msg": "read crontab: /etc/crontab"}
dashboards-helper-1 | 2024-06-19 02:23:33,958 INFO success: idxinit entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
dashboards-helper-1 | 2024-06-19 02:23:33,958 INFO success: maps entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
dashboards-helper-1 | Starting up http-server, serving /opt/maps
dashboards-helper-1 |
dashboards-helper-1 | http-server version: 14.1.1
dashboards-helper-1 |
dashboards-helper-1 | http-server settings:
dashboards-helper-1 | CORS: *
dashboards-helper-1 | Cache: 3600 seconds
dashboards-helper-1 | Connection Timeout: 120 seconds
dashboards-helper-1 | Directory Listings: not visible
dashboards-helper-1 | AutoIndex: not visible
dashboards-helper-1 | Serve GZIP Files: false
dashboards-helper-1 | Serve Brotli Files: false
dashboards-helper-1 | Default File Extension: none
dashboards-helper-1 |
dashboards-helper-1 | Available on:
dashboards-helper-1 | http://127.0.0.1:28991
dashboards-helper-1 | http://172.18.0.15:28991
dashboards-helper-1 | Hit CTRL-C to stop the server
dashboards-helper-1 |
dashboards-helper-1 | 2024-06-19 02:23:35,630 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: opensearch (primary) is running at "http://opensearch:9200"!
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Registering index snapshot repository...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable templates...
dashboards-1 | log [02:23:44.065] [info][plugins-service] Plugin "applicationConfig" is disabled.
dashboards-1 | log [02:23:44.066] [info][plugins-service] Plugin "cspHandler" is disabled.
dashboards-1 | log [02:23:44.066] [info][plugins-service] Plugin "dataSource" is disabled.
dashboards-1 | log [02:23:44.067] [info][plugins-service] Plugin "workspace" is disabled.
dashboards-1 | log [02:23:44.067] [info][plugins-service] Plugin "visTypeXy" is disabled.
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template agent ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template base ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template client ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
arkime-1 | root
dashboards-1 | log [02:23:44.174] [info][plugins-system] Setting up [52] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,expressions,data,securityAnalyticsDashboards,savedObjects,home,apmOss,reportsDashboards,searchRelevanceDashboards,indexManagementDashboards,management,indexPatternManagement,advancedSettings,console,notificationsDashboards,legacyExport,embeddable,dashboard,mlCommonsDashboards,assistantDashboards,visualizations,visTypeVega,visTypeTimeline,visTypeTable,visTypeMarkdown,tileMap,visAugmenter,alertingDashboards,anomalyDetectionDashboards,visBuilder,regionMap,customImportMapDashboards,inputControlVis,visualize,transformVis,ganttChartDashboards,dataExplorer,charts,visTypeVislib,visTypeTagcloud,visTypeTimeseries,visTypeMetric,observabilityDashboards,queryWorkbenchDashboards,discover,savedObjectsManagement,bfetch]
api-1 | usermod: no changes
api-1 | yeflask
api-1 | uid=1000(yeflask) gid=1000(yeflask) groups=1000(yeflask),5(tty)
api-1 | opensearch-local is up and healthy at http://opensearch:9200
api-1 | [2024-06-19 02:23:44 +0000] [757] [INFO] Starting gunicorn 22.0.0
api-1 | [2024-06-19 02:23:44 +0000] [757] [INFO] Listening at: http://0.0.0.0:5000 (757)
api-1 | [2024-06-19 02:23:44 +0000] [757] [INFO] Using worker: sync
api-1 | [2024-06-19 02:23:44 +0000] [808] [INFO] Booting worker with pid: 808
dashboards-1 | log [02:23:44.473] [info][savedobjects-service] Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations...
dashboards-1 | log [02:23:44.523] [info][savedobjects-service] Starting saved objects migrations
dashboards-1 | log [02:23:44.549] [info][savedobjects-service] Creating index .kibana_1.
dashboards-1 | log [02:23:44.665] [info][savedobjects-service] Pointing alias .kibana to .kibana_1.
dashboards-1 | log [02:23:44.707] [info][savedobjects-service] Finished in 163ms.
dashboards-1 | log [02:23:44.721] [warning][cross-compatibility-service] Starting cross compatibility service
arkime-1 | uid=0(root) gid=0(root) groups=0(root)
arkime-1 | 2024-06-19 02:23:32,306 INFO Set uid to user 0 succeeded
arkime-1 | 2024-06-19 02:23:32,309 INFO RPC interface 'supervisor' initialized
arkime-1 | 2024-06-19 02:23:32,309 CRIT Server 'unix_http_server' running without any HTTP authentication checking
arkime-1 | 2024-06-19 02:23:32,309 INFO supervisord started with pid 754
arkime-1 | 2024-06-19 02:23:33,317 INFO spawned: 'initialize' with pid 791
arkime-1 | 2024-06-19 02:23:33,324 INFO spawned: 'pcap-arkime' with pid 792
arkime-1 | 2024-06-19 02:23:33,330 INFO spawned: 'viewer' with pid 793
arkime-1 | 2024-06-19 02:23:33,333 INFO spawned: 'wise' with pid 795
arkime-1 | 2024-06-19 02:23:34,335 INFO success: initialize entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
arkime-1 | 2024-06-19 02:23:34,335 INFO success: viewer entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
arkime-1 | 2024-06-19 02:23:34,335 INFO success: wise entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
arkime-1 | 2024-06-19 02:23:37 URL:https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.csv [23323/23323] -> "ipv4-address-space.csv_new" [1]
arkime-1 | 2024-06-19 02:23:37 URL:https://www.wireshark.org/download/automated/data/manuf [2780329/2780329] -> "oui.txt_new" [1]
arkime-1 | Giving opensearch-local time to start...
arkime-1 | opensearch-local is up and healthy at http://opensearch:9200
dashboards-1 | log [02:23:44.721] [info][plugins-system] Starting [52] plugins: [usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,expressions,data,securityAnalyticsDashboards,savedObjects,home,apmOss,reportsDashboards,searchRelevanceDashboards,indexManagementDashboards,management,indexPatternManagement,advancedSettings,console,notificationsDashboards,legacyExport,embeddable,dashboard,mlCommonsDashboards,assistantDashboards,visualizations,visTypeVega,visTypeTimeline,visTypeTable,visTypeMarkdown,tileMap,visAugmenter,alertingDashboards,anomalyDetectionDashboards,visBuilder,regionMap,customImportMapDashboards,inputControlVis,visualize,transformVis,ganttChartDashboards,dataExplorer,charts,visTypeVislib,visTypeTagcloud,visTypeTimeseries,visTypeMetric,observabilityDashboards,queryWorkbenchDashboards,discover,savedObjectsManagement,bfetch]
dashboards-1 | log [02:23:44.992] [info][listening] Server running at http://0.0.0.0:5601/dashboards
dashboards-1 | log [02:23:45.245] [info][server][OpenSearchDashboards][http] http server running at http://0.0.0.0:5601/dashboards
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template cloud ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template container ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template data_stream ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template destination ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template device ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template dll ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template dns ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template ecs ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template email ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template error ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template event ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template faas ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template file ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template group ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template host ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template http ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template log ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template network ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template observer ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template orchestrator ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template organization ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template package ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template process ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template registry ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template related ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template rule ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template server ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template service ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template source ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template threat ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template tls ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template tracing ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template url ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template user ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template user_agent ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing ECS composable template vulnerability ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing custom ECS composable templates...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing custom ECS composable template arkime ...
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:06Z /data/shared-object-creation.sh: Importing custom ECS composable template malcolm_common ...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing custom ECS composable template miscbeat ...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing custom ECS composable template suricata ...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing custom ECS composable template suricata_stats ...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing custom ECS composable template zeek ...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing custom ECS composable template zeek_diagnostic ...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing custom ECS composable template zeek_ot ...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing malcolm_template (d3bc16eba39d620995b70c25f7f1696c4d3a8fceb716bf3f302d76e4df50726a)...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing template "malcolm_beats_template"...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Importing index pattern...
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:07Z /data/shared-object-creation.sh: Setting default index pattern...
dashboards-helper-1 | 2024-06-19T02:24:09Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:09Z /data/shared-object-creation.sh: Creating index pattern "malcolm_beats_*"...
dashboards-helper-1 | 2024-06-19T02:24:10Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:10Z /data/shared-object-creation.sh: Importing opensearch Dashboards saved objects...
dashboards-helper-1 | 2024-06-19T02:24:11Z /data/shared-object-creation.sh: Importing dashboard "X.509" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:11Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:11Z /data/shared-object-creation.sh: Importing dashboard "GENISYS" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:12Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:12Z /data/shared-object-creation.sh: Importing dashboard "LDAP" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:13Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:13Z /data/shared-object-creation.sh: Importing dashboard "FTP" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:14Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:14Z /data/shared-object-creation.sh: Importing dashboard "PE" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:15Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:15Z /data/shared-object-creation.sh: Importing dashboard "Overview" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:16Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:16Z /data/shared-object-creation.sh: Importing dashboard "Connections - Destination - Top Connection Duration" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:17Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:17Z /data/shared-object-creation.sh: Importing dashboard "SIP" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
arkime-1 | opensearch-local is running!
arkime-1 | Giving WISE time to start...
arkime-1 | Launch wise...
arkime-1 | 2024-06-19 02:23:48,346 INFO success: pcap-arkime entered RUNNING state, process has stayed up for > than 15 seconds (startsecs)
arkime-1 | curl: (7) Failed to connect to 127.0.0.1 port 8081 after 0 ms: Couldn't connect to server
arkime-1 | Waiting for WISE to start
arkime-1 | curl: (7) Failed to connect to 127.0.0.1 port 8081 after 0 ms: Couldn't connect to server
arkime-1 | Waiting for WISE to start
arkime-1 | [[02:23:50.800]] [LOG] Express server listening on host :: port 8081 in development mode
dashboards-helper-1 | 2024-06-19T02:24:18Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:18Z /data/shared-object-creation.sh: Importing dashboard "Tunnels" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:19Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:19Z /data/shared-object-creation.sh: Importing dashboard "QUIC" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:20Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:20Z /data/shared-object-creation.sh: Importing dashboard "ICS Best Guess" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:21Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:21Z /data/shared-object-creation.sh: Importing dashboard "Modbus" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:24:22Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:24:22Z /data/shared-object-creation.sh: Importing dashboard "OSPF" (2024-04-29T15:49:16.000Z > 1970-01-01T00:00:00.000Z) ...
dashboards-helper-1 | 2024-06-19T02:29:20Z /data/shared-object-creation.sh:
dashboards-helper-1 | 2024-06-19T02:29:23Z /data/shared-object-creation.sh: not starting: job is still running since 2024-06-19 02:24:00 +0000 UTC (2m0s elapsed)
arkime-1 | WISE is running!
arkime-1 |
arkime-1 | Initializing opensearch-local database...
arkime-1 | This is a fresh Arkime install
arkime-1 | Erasing
arkime-1 | Creating
arkime-1 | Finished
arkime-1 | Creating default user...
arkime-1 | WARNING - Using authMode=header since not set, add to config file to silence this warning.
arkime-1 | Added
arkime-1 | Initializing fields...
arkime-1 | Initializing views...
arkime-1 | Creating view "Arkime Sessions"
arkime-1 | Creating view "Public IP Addresses"
arkime-1 | Creating view "Suricata Alerts"
arkime-1 | Creating view "Suricata Logs"
arkime-1 | Creating view "Uninventoried Internal Assets"
arkime-1 | Creating view "Uninventoried Observed Services"
arkime-1 | Creating view "Zeek conn.log"
arkime-1 | Creating view "Zeek Exclude conn.log"
arkime-1 | Creating view "Zeek Logs"
arkime-1 | Setting defaults...
arkime-1 |
arkime-1 | opensearch-local database initialized!
arkime-1 |
arkime-1 | {"_shards":{"total":16,"successful":16,"failed":0}}2024-06-19 02:23:56,551 INFO exited: initialize (exit status 0; expected)
arkime-1 | Launch viewer...
arkime-1 | WARNING - Using authMode=header since not set, add to config file to silence this warning.
arkime-1 | SECURITY WARNING - when userNameHeader is set, viewHost should be localhost or use iptables
arkime-1 | Express server listening on host :: port 8005 in development mode
arkime-1 | This node will process Periodic Queries (CRON) & Hunts, delayed by 85 seconds |
|
16GB ram is the bare minimum for running Malcolm, but I do run it in a VM with that much memory for demo purposes and it runs acceptably. With 16GB of ram though, the system had better not be doing anything else of any consequence, and pretty much all of that memory will be consumed. CPUs will be high during startup, but after things get initialized that should die down. Let's check and just see how much memory is allocated for opensearch and logstash, run this command: $ grep JAVA_OPTS config/*.env
config/logstash.env:LS_JAVA_OPTS=-server -Xms2500m -Xmx2500m -Xss1536k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true
config/opensearch.env:OPENSEARCH_JAVA_OPTS=-server -Xms10g -Xmx10g -Xss256k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=trueand what do we have for the As far as the system locking up, that's never happened to me before. Normally what happens with insufficient resources is opensearch gets killed by the OOM killer and that container stops. We could check some other system settings, what is the output of the following commands:
|
|
It looks like i have the same values as you for Xms and Xmx: # When i run ...
~/Malcolm$ grep JAVA_OPTS config/*.env
# I get this output ....
config/logstash.env:LS_JAVA_OPTS=-server -Xmx2500m -Xms2500m -Xss1536k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=true
config/opensearch.env:OPENSEARCH_JAVA_OPTS=-server -Xmx10g -Xms10g -Xss256k -XX:-HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dlog4j.formatMsgNoLookups=trueFor the other commands... ~/Malcolm$ grep CMDLINE /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""~/Malcolm$ tail -n 30 /etc/sysctl.conf
# for what other values do
#kernel.sysrq=438
# the maximum number of open file handles
fs.file-max=2097152
# the maximum number of user inotify watches
fs.inotify.max_user_watches=131072
# the inotify event queue size
fs.inotify.max_queued_events=131072
# the maximum number of user inotify monitors
fs.inotify.max_user_instances=512
# the maximum number of memory map areas a process may have
vm.max_map_count=262144
# the maximum number of incoming connections
net.core.somaxconn=65535
# decrease "swappiness" (swapping out runtime memory vs. dropping pages)
vm.swappiness=1
# the % of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio=40
# maximum % of dirty system memory before committing everything
vm.dirty_ratio=80~/Malcolm$ grep -v '^#' /etc/security/limits.conf /etc/security/limits.d/limits.conf
/etc/security/limits.conf:
/etc/security/limits.conf:
/etc/security/limits.d/limits.conf:
/etc/security/limits.d/limits.conf:* soft nofile 65535
/etc/security/limits.d/limits.conf:* hard nofile 65535
/etc/security/limits.d/limits.conf:* soft memlock unlimited
/etc/security/limits.d/limits.conf:* hard memlock unlimited~/Malcolm$ hostnamectl
...
Icon name: computer-vm
Chassis: vm
Virtualization: kvm
Operating System: Ubuntu 22.04.4 LTS
Kernel: Linux 6.5.0-41-generic
Architecture: x86-64
Hardware Vendor: QEMU
Hardware Model: Standard PC _i440FX + PIIX, 1996_
...Do those values look normal? During the wizard, do you set live capture to true? In my setup i did because I want real time capture but maybe im misunderstanding something. |
|
There may be another setting that I need to add to install.py to set up for you, which would be an oversight on my part. That's to enable enable cgroup accounting for memory and swap space. In the /etc/default/grub file, ensure that the GRUB_CMDLINE_LINUX= variable includes these values:
Then, update your Grub configuration:
and reboot. Other than that, everything looks good. Give that a shot and see if it behaves differently. |
|
As far as setting live capture to true during the wizard, yeah, that's fine. |
|
Thank you. I was doing more reading after my last reply and realized i also missed the following from here because i went straight to the Ubuntu example and didn't see it there:
So with those changes and your grub update i now have I ran Next time I should probably use Ubuntu Server instead of Ubuntu Desktop since the Desktop environment uses more resources and doesn't seem to be needed. Thanks for your help. |
|
Thank you! I need to document and/or add the tweaks for the cgroup stuff to the install.py script. Thanks for bringing it to my attention. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
16GB RAM and 16 Core system resources are quickly being exhausted till system locks up. I tried this install on a busy network and then on a quiet network with only one other machine generating traffic. I tried the setup in 2 different server environments and had the same problem on both. Did i miss something? Is that not enough resources to demo the tool with?
To Reproduce
Steps to reproduce the behavior:
./scripts/auth_setup./scripts/startand loginExpected behavior
System to run stable, not exhaust all resources when on network without much traffic
**Screenshots and/or Logs **
If applicable, attach screenshots or container logs (e.g., the relevant bits of
./scripts/logs) to help explain your problem....logs exceed max characters
Malcolm Version:
How are you running Malcolm?
Additional context
Running inside Docker inside Ubuntu 22.04LTS vm on Proxmox.
The text was updated successfully, but these errors were encountered: