Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

filtering by rootId in Arkime returns no results #129

Closed
mmguero opened this issue Nov 21, 2022 · 0 comments
Closed

filtering by rootId in Arkime returns no results #129

mmguero opened this issue Nov 21, 2022 · 0 comments
Assignees
@mmguero
Labels
arkime Relating to Malcolm's use of Arkime bug Something isn't working opensearch Relating to Malcolm's use of OpenSearch

Comments

@mmguero
Copy link
Collaborator

mmguero commented Nov 21, 2022

Malcolm stores Zeek UID in event.id as well as rootId. However, when specifying a query in Arkime like rootId = XXXXXXXXXXXXX it would not return a value.

Adding the following to the index template seems to fix the problem:

"rootId": { "type": "keyword" },
@mmguero mmguero added bug Something isn't working opensearch Relating to Malcolm's use of OpenSearch arkime Relating to Malcolm's use of Arkime labels Nov 21, 2022
@mmguero mmguero self-assigned this Nov 21, 2022
@mmguero mmguero added this to Malcolm Nov 21, 2022
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Nov 21, 2022
@mmguero
fix idaholab#129, filtering by rootId in Arkime returns no results
b20d0a3
@mmguero mmguero moved this to Done in Malcolm Nov 21, 2022
@mmguero mmguero closed this as completed Nov 21, 2022
mmguero added a commit that referenced this issue Dec 6, 2022
@mmguero
Malcolm v6.4.3 is a minor release containing enhancements, component …
46af12c 
…version updates and bug fixes.

v6.4.2...v6.4.3

* Enhancements
    - Import the [NetBox Device Type Library](https://github.com/netbox-community/devicetype-library) on NetBox first run to populate manufacturers, device types, models and modules
    - [#127](#127) have `install.py --configure` ask about other storage locations for PCAP, Zeek logs and OpenSearch indices
    - [#128](#128) have `install.py --configure` prompt for Arkime to manage uploaded PCAP files or not

* Component version updates
    - Alpine Linux to [v3.17](https://alpinelinux.org/posts/Alpine-3.17.0-released.html) for some Docker containers' base images
    - Filebeat to [v8.5.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.5.2.html)
    - NetBox to [v3.3.9](https://github.com/netbox-community/netbox/releases/tag/v3.3.9)
    - Zeek to [v5.0.4](https://github.com/zeek/zeek/releases/tag/v5.0.4)
    - opensearch-py to [v2.0.1](https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)
    - Fluent Bit to [v2.0.6](https://www.fluentbit.io/announcements/v2.0.6/)

* Fixes
    - Fix some bad links in the documentation and other minor documentation improvements
    - Fix [#126](#126), suricata logs show up in Arkime as "notip" for the protocol
    - Fix [#129](#129), filtering by rootId in Arkime returns no results
    - Fix Docker health checks for NetBox and supporting containers
    - Fix "read-only" version of nginx.conf
    - Tweaks to `install.py` memory recommendations

Squashed commit of the following:

commit 40c7ea0
Merge: 476d941 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:38:24 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_idaholab

commit 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:37:20 2022 -0700

    update SHA sums for ISOs for release candidate

commit 63282c4
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 14:40:13 2022 -0700

    for #127, apply 'wipe' logic to directories that are mounted in locations other than under Malcolm install repo. adds 'pyyaml' dependency to install/control scripts

commit 476d941
Merge: 62518b1 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:59:44 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_idaholab

commit 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:58:19 2022 -0700

    update SHA sums for ISOs for release candidate

commit 782e9ed
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 15:38:26 2022 -0700

    ensure missing paths get assigned correct ownership if install.py is run by root

commit 44fdcc3
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 13:11:47 2022 -0700

    update SHA sums for ISOs for release candidate

commit 8efa19e
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 10:29:04 2022 -0700

    documentation update

commit 259cd29
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 09:49:39 2022 -0700

    update nginx-proxy to be based on alpine 3.17

commit 935745d
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:25:49 2022 -0700

    remove fediverse link in embedded documentation

commit 81124ed
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:16:26 2022 -0700

    Fix bad download links in embedded documentation

commit 153ddf1
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 14:23:38 2022 -0700

    fix reset with https

commit 5671755
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 13:50:52 2022 -0700

    added flexibility in demo reset/populate data

commit 4e8e695
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 16:20:22 2022 -0700

    Update opensearch-py to v2.0.1 (https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)

commit 5b6ca11
Author: SG <[email protected]>
Date:   Tue Nov 29 10:10:22 2022 -0700

    Update some docker images' base image from alpine 3.16 to 3.17

commit 4ac4381
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 07:02:21 2022 -0700

    Fix 'E722 do not use bare except' warning

commit fb5fa22
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:23:51 2022 -0700

    Bump zeek to v5.0.4

commit cccb709
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:22:43 2022 -0700

    beats v8.5.2

commit 569671c
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:21:34 2022 -0700

    fluent bit 2.0.6 for windows scripts

commit 486ccbb
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:58:59 2022 -0700

    pause netbox in read-only mode

commit bd05c0c
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:17:08 2022 -0700

    fix read-only version of nginx.conf

commit b23bdb7
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 12:12:34 2022 -0700

    tweak script for demo population

commit f13fd12
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 10:57:33 2022 -0700

    load netbox device type library on startup (see #17)

commit 7be8ada
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 09:03:12 2022 -0700

    Fix #126, suricata logs show up in Arkime as 'notip' for protocol

commit b20d0a3
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 08:30:34 2022 -0700

    fix #129, filtering by rootId in Arkime returns no results

commit 4cf2b81
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:58:08 2022 -0700

    don't report netbox services as unhealthy if they are disabled

commit 3d50a52
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:50:54 2022 -0700

    adjust memory recommendations in install.py script

commit 2fc15d4
Author: SG <[email protected]>
Date:   Fri Nov 18 12:28:18 2022 -0700

    #127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 9f73809
Author: SG <[email protected]>
Date:   Fri Nov 18 12:08:21 2022 -0700

    specify keystore location

commit aef542b
Author: SG <[email protected]>
Date:   Fri Nov 18 11:59:52 2022 -0700

    #127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit c40cd48
Author: SG <[email protected]>
Date:   Fri Nov 18 11:53:48 2022 -0700

    #127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 3501539
Author: SG <[email protected]>
Date:   Fri Nov 18 10:08:50 2022 -0700

    implement #128, have install.py prompt for Arkime to manage PCAP or not

commit f0ab2d8
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 12:52:25 2022 -0700

    bump development for v6.4.3

commit a046e77
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 10:16:21 2022 -0700

    update SHA sums for downloads
mmguero added a commit to cisagov/Malcolm that referenced this issue Dec 6, 2022
@mmguero
Malcolm v6.4.3 is a minor release containing enhancements, component …
082bae7 
…version updates and bug fixes.

v6.4.2...v6.4.3

* Enhancements
    - Import the [NetBox Device Type Library](https://github.com/netbox-community/devicetype-library) on NetBox first run to populate manufacturers, device types, models and modules
    - [idaholab#127](idaholab#127) have `install.py --configure` ask about other storage locations for PCAP, Zeek logs and OpenSearch indices
    - [idaholab#128](idaholab#128) have `install.py --configure` prompt for Arkime to manage uploaded PCAP files or not

* Component version updates
    - Alpine Linux to [v3.17](https://alpinelinux.org/posts/Alpine-3.17.0-released.html) for some Docker containers' base images
    - Filebeat to [v8.5.2](https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-8.5.2.html)
    - NetBox to [v3.3.9](https://github.com/netbox-community/netbox/releases/tag/v3.3.9)
    - Zeek to [v5.0.4](https://github.com/zeek/zeek/releases/tag/v5.0.4)
    - opensearch-py to [v2.0.1](https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)
    - Fluent Bit to [v2.0.6](https://www.fluentbit.io/announcements/v2.0.6/)

* Fixes
    - Fix some bad links in the documentation and other minor documentation improvements
    - Fix [idaholab#126](idaholab#126), suricata logs show up in Arkime as "notip" for the protocol
    - Fix [idaholab#129](idaholab#129), filtering by rootId in Arkime returns no results
    - Fix Docker health checks for NetBox and supporting containers
    - Fix "read-only" version of nginx.conf
    - Tweaks to `install.py` memory recommendations

Squashed commit of the following:

commit 02c3e0e
Merge: 0651e86 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:38:00 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_cisagov

commit 92e8800
Author: Seth Grover <[email protected]>
Date:   Tue Dec 6 14:37:20 2022 -0700

    update SHA sums for ISOs for release candidate

commit 63282c4
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 14:40:13 2022 -0700

    for idaholab#127, apply 'wipe' logic to directories that are mounted in locations other than under Malcolm install repo. adds 'pyyaml' dependency to install/control scripts

commit 0651e86
Merge: 5565a32 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:59:01 2022 -0700

    Merge branch 'development' of https://github.com/mmguero-dev/Malcolm into v643_merge_cisagov

commit 8f25215
Author: Seth Grover <[email protected]>
Date:   Mon Dec 5 07:58:19 2022 -0700

    update SHA sums for ISOs for release candidate

commit 782e9ed
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 15:38:26 2022 -0700

    ensure missing paths get assigned correct ownership if install.py is run by root

commit 44fdcc3
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 13:11:47 2022 -0700

    update SHA sums for ISOs for release candidate

commit 8efa19e
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 10:29:04 2022 -0700

    documentation update

commit 259cd29
Author: Seth Grover <[email protected]>
Date:   Thu Dec 1 09:49:39 2022 -0700

    update nginx-proxy to be based on alpine 3.17

commit 935745d
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:25:49 2022 -0700

    remove fediverse link in embedded documentation

commit 81124ed
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 15:16:26 2022 -0700

    Fix bad download links in embedded documentation

commit 153ddf1
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 14:23:38 2022 -0700

    fix reset with https

commit 5671755
Author: Seth Grover <[email protected]>
Date:   Wed Nov 30 13:50:52 2022 -0700

    added flexibility in demo reset/populate data

commit 4e8e695
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 16:20:22 2022 -0700

    Update opensearch-py to v2.0.1 (https://github.com/opensearch-project/opensearch-py/releases/tag/v2.0.1)

commit 5b6ca11
Author: SG <[email protected]>
Date:   Tue Nov 29 10:10:22 2022 -0700

    Update some docker images' base image from alpine 3.16 to 3.17

commit 4ac4381
Author: Seth Grover <[email protected]>
Date:   Tue Nov 29 07:02:21 2022 -0700

    Fix 'E722 do not use bare except' warning

commit fb5fa22
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:23:51 2022 -0700

    Bump zeek to v5.0.4

commit cccb709
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:22:43 2022 -0700

    beats v8.5.2

commit 569671c
Author: Seth Grover <[email protected]>
Date:   Sat Nov 26 21:21:34 2022 -0700

    fluent bit 2.0.6 for windows scripts

commit 486ccbb
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:58:59 2022 -0700

    pause netbox in read-only mode

commit bd05c0c
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 13:17:08 2022 -0700

    fix read-only version of nginx.conf

commit b23bdb7
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 12:12:34 2022 -0700

    tweak script for demo population

commit f13fd12
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 10:57:33 2022 -0700

    load netbox device type library on startup (see idaholab#17)

commit 7be8ada
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 09:03:12 2022 -0700

    Fix idaholab#126, suricata logs show up in Arkime as 'notip' for protocol

commit b20d0a3
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 08:30:34 2022 -0700

    fix idaholab#129, filtering by rootId in Arkime returns no results

commit 4cf2b81
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:58:08 2022 -0700

    don't report netbox services as unhealthy if they are disabled

commit 3d50a52
Author: Seth Grover <[email protected]>
Date:   Mon Nov 21 06:50:54 2022 -0700

    adjust memory recommendations in install.py script

commit 2fc15d4
Author: SG <[email protected]>
Date:   Fri Nov 18 12:28:18 2022 -0700

    idaholab#127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 9f73809
Author: SG <[email protected]>
Date:   Fri Nov 18 12:08:21 2022 -0700

    specify keystore location

commit aef542b
Author: SG <[email protected]>
Date:   Fri Nov 18 11:59:52 2022 -0700

    idaholab#127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit c40cd48
Author: SG <[email protected]>
Date:   Fri Nov 18 11:53:48 2022 -0700

    idaholab#127, have install.py --configure ask about other storage locations for PCAP, zeek logs and opensearch indices

commit 3501539
Author: SG <[email protected]>
Date:   Fri Nov 18 10:08:50 2022 -0700

    implement idaholab#128, have install.py prompt for Arkime to manage PCAP or not

commit f0ab2d8
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 12:52:25 2022 -0700

    bump development for v6.4.3

commit a046e77
Author: Seth Grover <[email protected]>
Date:   Thu Nov 17 10:16:21 2022 -0700

    update SHA sums for downloads
@mmguero mmguero moved this from Done to Released in Malcolm Dec 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
arkime Relating to Malcolm's use of Arkime bug Something isn't working opensearch Relating to Malcolm's use of OpenSearch
Projects
Malcolm
Status: Released
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mmguero