Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support json-delimited import for Zeek logs #65

Closed
mmguero opened this issue Nov 12, 2021 · 2 comments
Closed

support json-delimited import for Zeek logs #65

mmguero opened this issue Nov 12, 2021 · 2 comments
Assignees
Labels
enhancement New feature or request logstash Relating to Malcolm's use of Logstash upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek
Milestone

Comments

@mmguero
Copy link
Collaborator

mmguero commented Nov 12, 2021

Currently Malcolm only supports the standard tab-delimited format for Zeek logs. There have been some requests to import JSON format as well.

@mmguero mmguero added enhancement New feature or request logstash Relating to Malcolm's use of Logstash upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek labels Nov 12, 2021
@mmguero mmguero added this to Malcolm May 10, 2022
@mmguero mmguero moved this to Todo (develop) in Malcolm May 10, 2022
@mmguero mmguero moved this from Todo (develop) to Triage in Malcolm May 18, 2023
@mmguero mmguero moved this from Triage to Todo (develop) in Malcolm Dec 19, 2023
@mmguero
Copy link
Collaborator Author

mmguero commented Dec 19, 2023

What this would entail:

  • detecting somewhere towards the beginning of the zeek pipeline the JSON vs. TSV format and parsing it accordingly
  • for individual logs types, renaming fields such that they match what we're generating from the TSV-parsed logs (search for _field_names in that file)

@mmguero mmguero added this to the v24.02.0 milestone Jan 2, 2024
@mmguero mmguero modified the milestones: v24.02.0, staging Jan 15, 2024
@mmguero mmguero modified the milestones: z.staging, v24.03.0 Feb 15, 2024
@mmguero mmguero self-assigned this Feb 16, 2024
@mmguero mmguero moved this from Todo (develop) to In Progress in Malcolm Feb 16, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 16, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 16, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 20, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 20, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 21, 2024
@mmguero mmguero moved this from In Progress to Testing in Malcolm Feb 21, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 22, 2024
mmguero added a commit to mmguero-dev/Malcolm that referenced this issue Feb 22, 2024
@mmguero mmguero moved this from Testing to Done in Malcolm Feb 22, 2024
@mmguero
Copy link
Collaborator Author

mmguero commented Feb 22, 2024

As far as I can tell this is done. Everything seems to be working. Will reopen (or log another issue) if I find anything else.

@mmguero mmguero closed this as completed Feb 22, 2024
This was referenced Mar 4, 2024
@mmguero mmguero moved this from Done to Released in Malcolm Mar 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request logstash Relating to Malcolm's use of Logstash upload Relating to PCAP and/or Zeek log ingestion zeek Relating to Malcolm's use of Zeek
Projects
Status: Released
Development

No branches or pull requests

1 participant