fully automated configuration and installation

[mmguero]

from cisagov#273

Provide ability to do scriptable fully automated configuration/setup.

See my response there:

Thanks for the issue. Right now there are a few "non-automated" parts to the Malcolm setup we'd have to consider:

• the system-level configuration stuff in ./scripts/install.py (run as superuser)
  • this includes changing system settings in /etc/sysctl.conf, file handle limits in /etc/security/limits.conf, setting recommended kernel parameters for grub for best performance, etc. This may or may not be something you want to have done automatically.
• the malcolm-specific configuration stuff in ./scripts/install.py (not needed as superuser)
  • the questions asked in this wizard modify a bunch of environment variables. There actually is an "accept defaults" mode but it doesn't have flags for all of the questions, so I'd need to modify the script to allow you to specify the answer to every single configuration flag at the command line, because what's enabled for one user might not be the same as what's enabled for another user
• the authentication-related stuff
  • generating self-signed certificates and setting up the username/passwords. I'm not sure how we'd handle the passwords stuff, you wouldn't really want to pass it on the command line I think.

It's a great suggestion, thanks. I think we can make it work. I'm going to move this issue over to the idaholab/Malcolm fork which is where I do issue tracking for the project and track it there.

[mmguero]

$ ./scripts/configure --help
usage: configure <arguments>

Malcolm install script

options:
  -v [true|false], --verbose [true|false]
                        Verbose output
  -d [true|false], --defaults [true|false]
                        Accept defaults to prompts without user interaction
  -c [true|false], --configure [true|false]
                        Only do configuration (not installation)

Configuration files:
  -f <string>, --configure-file <string>
                        YAML file (docker-compose file to configure or kubeconfig file)
  -e <string>, --environment-dir <string>
                        Directory containing Malcolm's .env files

Installation files:
  -m <string>, --malcolm-file <string>
                        Malcolm .tar.gz file for installation
  -i <string>, --image-file <string>
                        Malcolm docker images .tar.gz file for installation

Runtime options:
  --dark-mode [true|false]
                        Enable dark mode for OpenSearch Dashboards

Entryption and authentication options:
  --https [true|false]  Require encrypted HTTPS connections
  --ldap [true|false]   Use Lightweight Directory Access Protocol (LDAP)
  --ldap-mode <openldap|winldap>
                        LDAP server compatibility type
  --ldap-start-tls [true|false]
                        Use StartTLS (rather than LDAPS) for LDAP connection security

Docker options:
  -r [true|false], --restart-malcolm [true|false]
                        Restart Malcolm on system restart (unless-stopped)
  --reverse-proxied [true|false]
                        Malcolm will be running behind another reverse proxy (Traefik, Caddy, etc.)
  --traefik-host <string>
                        Request domain (host header value) for Malcolm interface Traefik router (e.g., malcolm.example.org)
  --traefik-host-opensearch <string>
                        Request domain (host header value) for OpenSearch Traefik router (e.g., opensearch.malcolm.example.org)
  --traefik-entrypoint <string>
                        Traefik router entrypoint (e.g., websecure)
  --traefik-resolver <string>
                        Traefik router resolver (e.g., myresolver)
  --docker-network-name <string>
                        External Docker network name (or leave blank for default networking)

OpenSearch options:
  --opensearch [true|false]
                        Malcolm should use and maintain its own OpenSearch instance
  --opensearch-memory <string>
                        Memory for OpenSearch (e.g., 16g, 9500m, etc.)
  --opensearch-primary-url <string>
                        Primary remote OpenSearch connection URL
  --opensearch-primary-ssl-verify [true|false]
                        Require SSL certificate validation for communication with primary OpenSearch instance
  --opensearch-compress-snapshots [true|false]
                        Compress OpenSearch index snapshots
  --opensearch-secondary-remote [true|false]
                        Forward Logstash logs to a secondary remote OpenSearch instance
  --opensearch-secondary-url <string>
                        Secondary remote OpenSearch connection URL
  --opensearch-secondary-ssl-verify [true|false]
                        Require SSL certificate validation for communication with secondary OpenSearch instance

LogStash options:
  --logstash-memory <string>
                        Memory for LogStash (e.g., 4g, 2500m, etc.)
  --logstash-workers <integer>
                        Number of Logstash workers (e.g., 4, 8, etc.)

Expose ports:
  --logstash-expose [true|false]
                        Expose Logstash port to external hosts
  --opensearch-expose [true|false]
                        Expose OpenSearch port to external hosts
  --filebeat-tcp-expose [true|false]
                        Expose Filebeat TCP port to external hosts
  --sftp-expose [true|false]
                        Expose SFTP server (for PCAP upload) to external hosts

Storage options:
  --pcap-path <string>  PCAP storage directory
  --zeek-path <string>  Zeek log storage directory
  --suricata-path <string>
                        Suricata log storage directory
  --opensearch-path <string>
                        OpenSearch index directory
  --opensearch-snapshot-path <string>
                        OpenSearch snapshot directory
  --delete-old-pcap [true|false]
                        Arkime should delete PCAP files based on available storage (see https://arkime.com/faq#pcap-deletion)
  --delete-index-threshold <string>
                        Delete the oldest indices when the database exceeds this threshold (e.g., 250GB, 1TB, 60٪, etc.)

Analysis options:
  --auto-suricata [true|false]
                        Automatically analyze all PCAP files with Suricata
  --suricata-rule-update [true|false]
                        Automatically analyze all PCAP files with Suricata
  --auto-zeek [true|false]
                        Automatically analyze all PCAP files with Zeek
  --zeek-ics-best-guess [true|false]
                        Use "best guess" to identify potential OT/ICS traffic with Zeek
  --reverse-dns [true|false]
                        Perform reverse DNS lookup locally for source and destination IP addresses in logs
  --auto-oui [true|false]
                        Perform hardware vendor OUI lookups for MAC addresses
  --auto-freq [true|false]
                        Perform string randomness scoring on some fields

File extraction options:
  --file-extraction <none|known|mapped|all|interesting>
                        Zeek file extraction behavior
  --file-preservation <none|quarantined|all>
                        Zeek file preservation behavior
  --extracted-file-server [true|false]
                        Expose web interface for downloading preserved files
  --extracted-file-server-password <string>
                        AES-256-CBC encryption password for downloaded preserved files (blank for unencrypted)
  --extracted-file-clamav [true|false]
                        Scan extracted files with ClamAV
  --extracted-file-yara [true|false]
                        Scan extracted files with Yara
  --extracted-file-capa [true|false]
                        Scan extracted files with Capa
  --virustotal-api-key <string>
                        VirusTotal API key to scan extracted files with VirusTotal
  --file-scan-rule-update [true|false]
                        Download updated file scanner signatures periodically

NetBox options:
  --netbox [true|false]
                        Run and maintain an instance of NetBox
  --netbox-enrich [true|false]
                        Enrich network traffic using NetBox
  --netbox-autopopulate [true|false]
                        Automatically populate NetBox inventory based on observed network traffic
  --netbox-site-name <string>
                        Default NetBox site name

Live traffic capture options:
  --live-capture-iface <string>
                        Capture interface(s) (comma-separated)
  --live-capture-filter <string>
                        Capture filter (tcpdump-like filter expression; leave blank to capture all traffic)
  --live-capture-iface-tweak [true|false]
                        Disable capture interface hardware offloading and adjust ring buffer sizes
  --live-capture-arkime [true|false]
                        Capture live network traffic with netsniff-ng for Arkime
  --live-capture-arkime-tcpdump [true|false]
                        Capture live network traffic with tcpdump for Arkime
  --live-capture-zeek [true|false]
                        Capture live network traffic with Zeek
  --live-capture-suricata [true|false]
                        Capture live network traffic with Suricata

[mmguero]

The ./scripts/configure script can also be run noninteractively which can be useful for scripting Malcolm setup. This behavior can be selected by supplying the -d or --defaults option on the command line. Running with the --help option will list the arguments accepted by the script:

$ ./scripts/configure --help
usage: configure <arguments>

Malcolm install script

options:
  -v [true|false], --verbose [true|false]
                        Verbose output
  -d [true|false], --defaults [true|false]
                        Accept defaults to prompts without user interaction
  -c [true|false], --configure [true|false]
                        Only do configuration (not installation)
…

Note that the value for any argument not specified on the command line will be reset to its default (as if for a new Malcolm installation) regardless of the setting's current value in the corresponding .env file. In other words, users who want to use the --defaults option should carefully review all available command-line options and choose all that apply.

[mmguero]

The ./scripts/auth_setup script can also be run noninteractively which can be useful for scripting Malcolm setup. This behavior can be selected by supplying the --auth-noninteractive option on the command line. Running with the --help option will list the arguments accepted by the script:

$ ./scripts/auth_setup --help
usage: auth_setup <arguments>

Malcolm control script

options:
  -v [DEBUG], --verbose [DEBUG]
                        Verbose output
  -f <string>, --file <string>
                        docker-compose or kubeconfig YML file
  -e <string>, --environment-dir <string>
                        Directory containing Malcolm's .env files

…

Authentication Setup:
  --auth [CMDAUTHSETUP]
                        Configure Malcolm authentication
  --auth-noninteractive [CMDAUTHSETUPNONINTERACTIVE]
                        Configure Malcolm authentication (noninteractive using arguments provided)
  --auth-admin-username <string>
                        Administrator username (for --auth-noninteractive)
  --auth-admin-password-openssl <string>
                        Administrator password hash from "openssl -passwd -1" (for --auth-noninteractive)
  --auth-admin-password-htpasswd <string>
                        Administrator password hash from "htpasswd -n -B username | cut -d: -f2" (for --auth-noninteractive)
  --auth-generate-webcerts [AUTHGENWEBCERTS]
                        (Re)generate self-signed certificates for HTTPS access (for --auth-noninteractive)
  --auth-generate-fwcerts [AUTHGENFWCERTS]
                        (Re)generate self-signed certificates for a remote log forwarder
  --auth-generate-netbox-passwords [AUTHGENNETBOXPASSWORDS]
                        (Re)generate internal passwords for NetBox
…

In order to avoid specifying passwords as command line arguments, when setting the administrator password noninteractively the user must instead provide hashes generated by openssl -passwd -1 and htpasswd -n -B username.