Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices #127

Closed
mmguero opened this issue Nov 18, 2022 · 1 comment
Assignees
Labels
enhancement New feature or request install.py Relating to the install.py configuration script

Comments

@mmguero
Copy link
Collaborator

mmguero commented Nov 18, 2022

For installations that want to have the PCAP at one location, Zeek logs at another, etc., it would be good for install.py to ask about that and modify docker-compose.yml so that the volume bind mounts are set up correctly without having to go tweak the file in lots of places.

@mmguero mmguero added enhancement New feature or request install.py Relating to the install.py configuration script labels Nov 18, 2022
@mmguero mmguero self-assigned this Nov 18, 2022
@mmguero mmguero added this to Malcolm Nov 18, 2022
@mmguero mmguero moved this to Todo (develop) in Malcolm Nov 18, 2022
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Nov 18, 2022
…ations for PCAP, zeek logs and opensearch indices
@mmguero mmguero closed this as completed Nov 18, 2022
Repository owner moved this from Todo (develop) to Done in Malcolm Nov 18, 2022
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Nov 18, 2022
…ations for PCAP, zeek logs and opensearch indices
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Nov 18, 2022
…ations for PCAP, zeek logs and opensearch indices
@mmguero
Copy link
Collaborator Author

mmguero commented Dec 5, 2022

Still one more things that needs to be done for this: it should handle the wipe operation too, which it doesn't yet do.

@mmguero mmguero reopened this Dec 5, 2022
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Dec 5, 2022
…in locations other than under Malcolm install repo. adds 'pyyaml' dependency to install/control scripts
mmguero added a commit that referenced this issue Dec 6, 2022
…version updates and bug fixes.

v6.4.2...v6.4.3

* Enhancements
    - Import the [NetBox Device Type Library](https://github.com/netbox-community/devicetype-library) on NetBox first run to populate manufacturers, device types, models and modules
    - [#127](#127) have `install.py --configure` ask about other storage locations for PCAP, Zeek logs and OpenSearch indices
    - [#128](#128) have `install.py --configure` prompt for Arkime to manage uploaded PCAP files or not

* Component version updates
    - Alpine Linux to [v3.17](https://alpinelinux.org/posts/Alpine-3.17.0-released.html) for some Docker containers' base images
    - Filebeat to [v8.5.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.5.2.html)
    - NetBox to [v3.3.9](https://github.com/netbox-community/netbox/releases/tag/v3.3.9)
    - Zeek to [v5.0.4](https://github.com/zeek/zeek/releases/tag/v5.0.4)
    - opensearch-py to [v2.0.1](https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)
    - Fluent Bit to [v2.0.6](https://www.fluentbit.io/announcements/v2.0.6/)

* Fixes
    - Fix some bad links in the documentation and other minor documentation improvements
    - Fix [#126](#126), suricata logs show up in Arkime as "notip" for the protocol
    - Fix [#129](#129), filtering by rootId in Arkime returns no results
    - Fix Docker health checks for NetBox and supporting containers
    - Fix "read-only" version of nginx.conf
    - Tweaks to `install.py` memory recommendations

Squashed commit of the following:

commit 40c7ea0
Merge: 476d941 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:38:24 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_idaholab

commit 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:37:20 2022 -0700

    update SHA sums for ISOs for release candidate

commit 63282c4
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 14:40:13 2022 -0700

    for #127, apply 'wipe' logic to directories that are mounted in locations other than under Malcolm install repo. adds 'pyyaml' dependency to install/control scripts

commit 476d941
Merge: 62518b1 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:59:44 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_idaholab

commit 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:58:19 2022 -0700

    update SHA sums for ISOs for release candidate

commit 782e9ed
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 15:38:26 2022 -0700

    ensure missing paths get assigned correct ownership if install.py is run by root

commit 44fdcc3
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 13:11:47 2022 -0700

    update SHA sums for ISOs for release candidate

commit 8efa19e
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 10:29:04 2022 -0700

    documentation update

commit 259cd29
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 09:49:39 2022 -0700

    update nginx-proxy to be based on alpine 3.17

commit 935745d
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:25:49 2022 -0700

    remove fediverse link in embedded documentation

commit 81124ed
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:16:26 2022 -0700

    Fix bad download links in embedded documentation

commit 153ddf1
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 14:23:38 2022 -0700

    fix reset with https

commit 5671755
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 13:50:52 2022 -0700

    added flexibility in demo reset/populate data

commit 4e8e695
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 16:20:22 2022 -0700

    Update opensearch-py to v2.0.1 (https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)

commit 5b6ca11
Author: SG <[email protected]>
Date:   Tue Nov 29 10:10:22 2022 -0700

    Update some docker images' base image from alpine 3.16 to 3.17

commit 4ac4381
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 07:02:21 2022 -0700

    Fix 'E722 do not use bare except' warning

commit fb5fa22
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:23:51 2022 -0700

    Bump zeek to v5.0.4

commit cccb709
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:22:43 2022 -0700

    beats v8.5.2

commit 569671c
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:21:34 2022 -0700

    fluent bit 2.0.6 for windows scripts

commit 486ccbb
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:58:59 2022 -0700

    pause netbox in read-only mode

commit bd05c0c
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:17:08 2022 -0700

    fix read-only version of nginx.conf

commit b23bdb7
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 12:12:34 2022 -0700

    tweak script for demo population

commit f13fd12
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 10:57:33 2022 -0700

    load netbox device type library on startup (see #17)

commit 7be8ada
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 09:03:12 2022 -0700

    Fix #126, suricata logs show up in Arkime as 'notip' for protocol

commit b20d0a3
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 08:30:34 2022 -0700

    fix #129, filtering by rootId in Arkime returns no results

commit 4cf2b81
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:58:08 2022 -0700

    don't report netbox services as unhealthy if they are disabled

commit 3d50a52
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:50:54 2022 -0700

    adjust memory recommendations in install.py script

commit 2fc15d4
Author: SG <[email protected]>
Date:   Fri Nov 18 12:28:18 2022 -0700

    #127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 9f73809
Author: SG <[email protected]>
Date:   Fri Nov 18 12:08:21 2022 -0700

    specify keystore location

commit aef542b
Author: SG <[email protected]>
Date:   Fri Nov 18 11:59:52 2022 -0700

    #127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit c40cd48
Author: SG <[email protected]>
Date:   Fri Nov 18 11:53:48 2022 -0700

    #127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 3501539
Author: SG <[email protected]>
Date:   Fri Nov 18 10:08:50 2022 -0700

    implement #128, have install.py prompt for Arkime to manage PCAP or not

commit f0ab2d8
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 12:52:25 2022 -0700

    bump development for v6.4.3

commit a046e77
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 10:16:21 2022 -0700

    update SHA sums for downloads
mmguero added a commit to cisagov/Malcolm that referenced this issue Dec 6, 2022
…version updates and bug fixes.

v6.4.2...v6.4.3

* Enhancements
    - Import the [NetBox Device Type Library](https://github.com/netbox-community/devicetype-library) on NetBox first run to populate manufacturers, device types, models and modules
    - [idaholab#127](idaholab#127) have `install.py --configure` ask about other storage locations for PCAP, Zeek logs and OpenSearch indices
    - [idaholab#128](idaholab#128) have `install.py --configure` prompt for Arkime to manage uploaded PCAP files or not

* Component version updates
    - Alpine Linux to [v3.17](https://alpinelinux.org/posts/Alpine-3.17.0-released.html) for some Docker containers' base images
    - Filebeat to [v8.5.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.5.2.html)
    - NetBox to [v3.3.9](https://github.com/netbox-community/netbox/releases/tag/v3.3.9)
    - Zeek to [v5.0.4](https://github.com/zeek/zeek/releases/tag/v5.0.4)
    - opensearch-py to [v2.0.1](https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)
    - Fluent Bit to [v2.0.6](https://www.fluentbit.io/announcements/v2.0.6/)

* Fixes
    - Fix some bad links in the documentation and other minor documentation improvements
    - Fix [idaholab#126](idaholab#126), suricata logs show up in Arkime as "notip" for the protocol
    - Fix [idaholab#129](idaholab#129), filtering by rootId in Arkime returns no results
    - Fix Docker health checks for NetBox and supporting containers
    - Fix "read-only" version of nginx.conf
    - Tweaks to `install.py` memory recommendations

Squashed commit of the following:

commit 02c3e0e
Merge: 0651e86 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:38:00 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_cisagov

commit 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:37:20 2022 -0700

    update SHA sums for ISOs for release candidate

commit 63282c4
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 14:40:13 2022 -0700

    for idaholab#127, apply 'wipe' logic to directories that are mounted in locations other than under Malcolm install repo. adds 'pyyaml' dependency to install/control scripts

commit 0651e86
Merge: 5565a32 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:59:01 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_cisagov

commit 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:58:19 2022 -0700

    update SHA sums for ISOs for release candidate

commit 782e9ed
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 15:38:26 2022 -0700

    ensure missing paths get assigned correct ownership if install.py is run by root

commit 44fdcc3
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 13:11:47 2022 -0700

    update SHA sums for ISOs for release candidate

commit 8efa19e
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 10:29:04 2022 -0700

    documentation update

commit 259cd29
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 09:49:39 2022 -0700

    update nginx-proxy to be based on alpine 3.17

commit 935745d
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:25:49 2022 -0700

    remove fediverse link in embedded documentation

commit 81124ed
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:16:26 2022 -0700

    Fix bad download links in embedded documentation

commit 153ddf1
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 14:23:38 2022 -0700

    fix reset with https

commit 5671755
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 13:50:52 2022 -0700

    added flexibility in demo reset/populate data

commit 4e8e695
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 16:20:22 2022 -0700

    Update opensearch-py to v2.0.1 (https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)

commit 5b6ca11
Author: SG <[email protected]>
Date:   Tue Nov 29 10:10:22 2022 -0700

    Update some docker images' base image from alpine 3.16 to 3.17

commit 4ac4381
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 07:02:21 2022 -0700

    Fix 'E722 do not use bare except' warning

commit fb5fa22
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:23:51 2022 -0700

    Bump zeek to v5.0.4

commit cccb709
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:22:43 2022 -0700

    beats v8.5.2

commit 569671c
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:21:34 2022 -0700

    fluent bit 2.0.6 for windows scripts

commit 486ccbb
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:58:59 2022 -0700

    pause netbox in read-only mode

commit bd05c0c
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:17:08 2022 -0700

    fix read-only version of nginx.conf

commit b23bdb7
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 12:12:34 2022 -0700

    tweak script for demo population

commit f13fd12
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 10:57:33 2022 -0700

    load netbox device type library on startup (see idaholab#17)

commit 7be8ada
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 09:03:12 2022 -0700

    Fix idaholab#126, suricata logs show up in Arkime as 'notip' for protocol

commit b20d0a3
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 08:30:34 2022 -0700

    fix idaholab#129, filtering by rootId in Arkime returns no results

commit 4cf2b81
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:58:08 2022 -0700

    don't report netbox services as unhealthy if they are disabled

commit 3d50a52
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:50:54 2022 -0700

    adjust memory recommendations in install.py script

commit 2fc15d4
Author: SG <[email protected]>
Date:   Fri Nov 18 12:28:18 2022 -0700

    idaholab#127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 9f73809
Author: SG <[email protected]>
Date:   Fri Nov 18 12:08:21 2022 -0700

    specify keystore location

commit aef542b
Author: SG <[email protected]>
Date:   Fri Nov 18 11:59:52 2022 -0700

    idaholab#127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit c40cd48
Author: SG <[email protected]>
Date:   Fri Nov 18 11:53:48 2022 -0700

    idaholab#127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 3501539
Author: SG <[email protected]>
Date:   Fri Nov 18 10:08:50 2022 -0700

    implement idaholab#128, have install.py prompt for Arkime to manage PCAP or not

commit f0ab2d8
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 12:52:25 2022 -0700

    bump development for v6.4.3

commit a046e77
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 10:16:21 2022 -0700

    update SHA sums for downloads
@mmguero mmguero closed this as completed Dec 7, 2022
@mmguero mmguero moved this from Done to Released in Malcolm Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request install.py Relating to the install.py configuration script
Projects
Status: Released
Development

No branches or pull requests

1 participant