Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

more consistently differentiate between uploaded and live-captured traffic #321

Closed
mmguero opened this issue Jan 3, 2024 · 1 comment
Closed

more consistently differentiate between uploaded and live-captured traffic #321

mmguero opened this issue Jan 3, 2024 · 1 comment
Assignees
@mmguero
Labels
arkime Relating to Malcolm's use of Arkime beats Relating to Malcolm's use of Beats capture Relating to pcap-capture container enhancement New feature or request logstash Relating to Malcolm's use of Logstash
Milestone
v24.01.0

Comments

@mmguero
Copy link
Collaborator

mmguero commented Jan 3, 2024 •
edited
Loading

Logs' node value is now going to have -upload appended to it when that traffic was traffic in uploaded PCAP files vs. no suffix for captured live by Malcolm.

This is a necessity when using arkime capture mode (see #281) because traffic from these two sources need to have different node names so that Arkime can know how to retrieve PCAP payload when requested.

This will affect Arkime sessions. Zeek, and Suricata network logs will also take this behavior for consistency's sake.

As traffic from hedgehog sensors is "live", this will also be consistent with that.
@mmguero mmguero added beats Relating to Malcolm's use of Beats capture Relating to pcap-capture container enhancement New feature or request logstash Relating to Malcolm's use of Logstash arkime Relating to Malcolm's use of Arkime labels Jan 3, 2024
@mmguero mmguero added this to the v24.01.0 milestone Jan 3, 2024
@mmguero mmguero self-assigned this Jan 3, 2024
@mmguero mmguero added this to Malcolm Jan 3, 2024
@mmguero mmguero moved this to In Progress in Malcolm Jan 3, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 3, 2024
@mmguero
idaholab#321, more consistently differentiate between uploaded and li…
355601d 
…ve-captured traffic
@mmguero mmguero moved this from In Progress to Testing in Malcolm Jan 3, 2024
@mmguero mmguero moved this from Testing to In Progress in Malcolm Jan 5, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 5, 2024
@mmguero
more consistently differentiate between uploaded and live-captured t…
301ff56 
…raffic idaholab#321 WIP
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 5, 2024
@mmguero
more consistently differentiate between uploaded and live-captured t…
58c94f3 
…raffic idaholab#321 WIP
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 5, 2024
@mmguero
more consistently differentiate between uploaded and live-captured tr…
1187798 
…affic idaholab#321 WIP
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 5, 2024
@mmguero
more consistently differentiate between uploaded and live-captured tr…
3846467 
…affic idaholab#321 WIP
@mmguero mmguero moved this from In Progress to Done in Malcolm Jan 5, 2024
@mmguero mmguero closed this as completed Jan 5, 2024
@mmguero mmguero reopened this Jan 9, 2024
@mmguero mmguero moved this from Done to In Progress in Malcolm Jan 9, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented Jan 9, 2024

TODO: when pcap node and pcap host are unspecified, leave the capture flags for those off altogether

mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 10, 2024
@mmguero
only specify --node and/or --host for live capture arkime if those va…
7757497 
…riables are specified idaholab#321
@mmguero mmguero moved this from In Progress to Done in Malcolm Jan 10, 2024
@mmguero mmguero closed this as completed Jan 10, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 10, 2024
@mmguero
allow blank arkime node host for live capture as it will default to h…
0228de1 
…ostname (idaholab#321)
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Jan 10, 2024
@mmguero
allow blank arkime node name for uploaded capture as it will default …
45f8e96 
…to hostname (idaholab#321)
This was referenced Jan 17, 2024
Merged
Merged
mmguero added a commit that referenced this issue Jan 17, 2024
@mmguero
Merge pull request #383 from idaholab/v24.01.0_merge_idaholab
c78a043 
Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.0.1

* Features and enhancements
    + new Malcolm instance landing page (#252)
    + file carve download with password-protected .zip file (#288)
    + new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (#290)
    + allow customizing indexes for logs written to OpenSearch/Elasticsearch (#313)
    + more consistently differentiate between uploaded and live-captured traffic (#321)
    + make download extracted file context item from Arkime smarter (#330)
    + improve netbox device type library import by using "official" import script (#384)
* Component version updates
    + Alpine Linux to [v3.19](https://alpinelinux.org/posts/Alpine-3.19.0-released.html) as the base for some Docker images
    + Fluent Bit to [v2.2.2](https://github.com/fluent/fluent-bit/releases/tag/v2.2.2)
    + Beats to [v8.11.4](https://www.elastic.co/guide/en/beats/libbeat/8.11/release-notes-8.11.4.html)
    + LogStash to [v8.11.4](https://www.elastic.co/guide/en/logstash/current/logstash-8-11-4.html)
* Bug fixes
    + Suricata Alerts dashboard "Alerts - Tags" visualization is useless (#314)
    + third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (#318)
    + update document lookup APIs to search either network or host data (#322)
    + suricata rule update is broken (#323)
    + time sync from hedgehog to Malcolm opensearch instance not working (#324)
    + fix issue specifying database mode via command-line
    + have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/idaholab/Malcolm/tree/v24.0.1/config))
    + added the following variables with relation to #313
        - added `ARKIME_ROTATE_INDEX` to [`arkime.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/arkime.env.example) with default value of `daily` (see [Arkime docs on rotateIndex](https://arkime.com/settings#rotateIndex))
        - added the following variables and defaults to [`opensearch.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/opensearch.env.example):
        ```
        # OpenSearch index patterns and timestamp fields
        # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
        MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field to use for network traffic logs in Logstash and Dashboards
        MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
        # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
        MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
        MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
        # Default time field to use for other logs in Logstash and Dashboards
        MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
        # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
        MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
        ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field used by for sessions in Arkime viewer
        ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
        ```
    + changed default for `EXTRACTED_FILE_HTTP_SERVER_KEY` to `infected` in [`zeek-secret.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/zeek-secret.env.example)
    + added `EXTRACTED_FILE_HTTP_SERVER_ZIP` with default value of `false` in [`zeek.env`](https://github.com/idaholab/Malcolm/tree/v24.0.1/zeek.env.example), see (#288)
mmguero added a commit to cisagov/Malcolm that referenced this issue Jan 17, 2024
@mmguero
Merge pull request #295 from cisagov/v24.01.0_merge_cisagov
2862d43 
Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.0.1

* Features and enhancements
    + new Malcolm instance landing page (idaholab#252)
    + file carve download with password-protected .zip file (idaholab#288)
    + new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (idaholab#290)
    + allow customizing indexes for logs written to OpenSearch/Elasticsearch (idaholab#313)
    + more consistently differentiate between uploaded and live-captured traffic (idaholab#321)
    + make download extracted file context item from Arkime smarter (idaholab#330)
    + improve netbox device type library import by using "official" import script (idaholab#384)
* Component version updates
    + Alpine Linux to [v3.19](https://alpinelinux.org/posts/Alpine-3.19.0-released.html) as the base for some Docker images
    + Fluent Bit to [v2.2.2](https://github.com/fluent/fluent-bit/releases/tag/v2.2.2)
    + Beats to [v8.11.4](https://www.elastic.co/guide/en/beats/libbeat/8.11/release-notes-8.11.4.html)
    + LogStash to [v8.11.4](https://www.elastic.co/guide/en/logstash/current/logstash-8-11-4.html)
* Bug fixes
    + Suricata Alerts dashboard "Alerts - Tags" visualization is useless (idaholab#314)
    + third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (idaholab#318)
    + update document lookup APIs to search either network or host data (idaholab#322)
    + suricata rule update is broken (idaholab#323)
    + time sync from hedgehog to Malcolm opensearch instance not working (idaholab#324)
    + fix issue specifying database mode via command-line
    + have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
* Configuration changes (in [environment variables](https://idaholab.github.io/Malcolm/docs/malcolm-config.html#MalcolmConfigEnvVars) in [`./config/`](https://github.com/cisagov/Malcolm/tree/v24.0.1/config))
    + added the following variables with relation to idaholab#313
        - added `ARKIME_ROTATE_INDEX` to [`arkime.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/arkime.env.example) with default value of `daily` (see [Arkime docs on rotateIndex](https://arkime.com/settings#rotateIndex))
        - added the following variables and defaults to [`opensearch.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/opensearch.env.example):
        ```
        # OpenSearch index patterns and timestamp fields
        # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
        MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field to use for network traffic logs in Logstash and Dashboards
        MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
        # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
        MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
        MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
        # Default time field to use for other logs in Logstash and Dashboards
        MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
        # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
        MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
        # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
        ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
        # Default time field used by for sessions in Arkime viewer
        ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
        ```
    + changed default for `EXTRACTED_FILE_HTTP_SERVER_KEY` to `infected` in [`zeek-secret.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/zeek-secret.env.example)
    + added `EXTRACTED_FILE_HTTP_SERVER_ZIP` with default value of `false` in [`zeek.env`](https://github.com/cisagov/Malcolm/tree/v24.0.1/zeek.env.example), see (idaholab#288)
@mmguero mmguero moved this from Done to Released in Malcolm Jan 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mmguero mmguero

Labels
arkime Relating to Malcolm's use of Arkime beats Relating to Malcolm's use of Beats capture Relating to pcap-capture container enhancement New feature or request logstash Relating to Malcolm's use of Logstash
Projects
Malcolm
Status: Released
Milestone
v24.01.0
Development

No branches or pull requests
1 participant
@mmguero