Enrich network traffic metadata via NetBox lookups

[mmguero]

Feature-tracking issue dependent on #131

[mmguero]

existing behavior

Up until now network traffic metadata has been enriched using the Automatic host and subnet name assignment feature, in which network hosts are identified by IP address or MAC address, and subnets are identified by CIDR subnet. This is done via config files (originally a delimited file, later JSON) and can be managed with a web interface.

The way this is handled today is that the net-map.json file is read on logstash startup and a Logstash filter file is created so that it becomes part of the pipeline.

As logs are processed by logstash, the fields source.hostname and destination.hostname are populated, respectively, for matching source.ip, destination.ip, source.mac, and destination.mac fields. source.ip and destination.ip are also checked in subnet definitions, and matches are written into source.segment and destination.segment fields. These are not mutually exclusive: these fields can contain multiple values. See the link to the documentation in the first paragraph of this

proposed new behavior

This feature can be replaced with NetBox, although for now we will leave both in place, if someone wants to do the enrichment without the full complexity netbox provides. (in fact, even now (as of v6.4.3) if netbox is enabled and net-map.json exists, it will be automatically recreated in netbox).

I'll be using some examples on the netbox demo as I investigate the mappings we'll want to use for these lookups.

field changes and mapping

This describes which fields from the NetBox model will be enriched into which fields in the Malcolm opensearch indices.

For storage considerations, currently I've chosen to do NetBox enrichment for all Suricata logs and only Zeek conn, notice, weird, signatures and known_* logs. UID values can be used to pivot from another kind of zeek logs back to the conn.log to see the enriched values, and communityID can be used to pivot to Arkime sessions as usual. I'm not currently doing this enrichment for Arkime sessions (but I will probably figure out how to do so using WISE in a future release).

• See the NetBox API documentation and the NetBox documentation. In this list I'll reference the corresponding NetBox model API for the given fields:
  • destination.hostname
  • source.hostname
  • destination.device.cluster (/virtualization/clusters/) (for Virtual Machine device types)
  • destination.device.device_type (/dcim/device-types/)
  • destination.device.id (/dcim/devices/{id})
  • destination.device.manufacturer (/dcim/manufacturers/)
  • destination.device.name (/dcim/devices/)
  • destination.device.role (/dcim/device-roles/)
  • destination.device.service (/ipam/services/)
  • destination.device.site (/dcim/sites/)
  • destination.device.url (/dcim/devices/)
  • destination.device.details (full JSON object, only with LOGSTASH_NETBOX_ENRICHMENT_VERBOSE: 'true')
  • destination.segment.id (/ipam/vrfs/{id})
  • destination.segment.name (/ipam/vrfs/)
  • destination.segment.site (/dcim/sites/)
  • destination.segment.tenant (/tenancy/tenants/)
  • destination.segment.url (/ipam/vrfs/)
  • destination.segment.details (full JSON object, only with LOGSTASH_NETBOX_ENRICHMENT_VERBOSE: 'true')
  • source.… same as destination.…
  • collected as "related" fields (the same approach used in ECS)
    • related.device_type
    • related.manufacturer
    • related.role
    • related.segment
    • related.service
    • related.site

For Malcolm's purposes, both physical devices and virtualized hosts will be stored as described above: the device_type field can be used to distinguish.

IP address to device mapping

IP addresses are under the IPAM model (demo).

Each device (demo) must be assigned a site, device role, and operational status, and may optionally be assigned to a specific location and/or rack within a site. A platform, serial number, and asset tag may optionally be assigned to each device. Device names must be unique within a site, unless the device has been assigned to a tenant. Devices may also be unnamed. When a device has one or more interfaces with IP addresses assigned, a primary IP for the device can be designated, for both IPv4 and IPv6.

Devices can be assigned IP addresses, even multiple IP addresses: When a device has one or more interfaces with IP addresses assigned, a primary IP for the device can be designated, for both IPv4 and IPv6.

In order to look up a device by IP address, you do it like ip -> interface -> device.

IP address to network segment mapping

IP subnets are described in "prefixes" (demo). The documentation states that: ... each prefix can be assigned to a particular site and virtual routing and forwarding instance (VRF). Each VRF represents a separate IP space or routing table. All prefixes not assigned to a VRF are considered to be in the "global" table.

In other words, for our purposes an IP address prefix will get its "name" from the VRF to which it belongs.

MAC address to host name mapping

After quite a bit of experimentation, I've determined that doing MAC address to device mapping isn't reliable enough to be useful (due to the issue of the MAC address shown for a packet is simply the MAC address of the last network device that it traversed). If we do this mapping, you've almost guaranteed both false positives and false negatives.

other considerations

NetBox has the concept of sites. Sites can have overlapping IP address ranges, of course. For now, the value in theNETBOX_DEFAULT_SITE variable in docker-compose will be used as a query parameter for enrichment lookups. If we need to handle multiple sites in a single malcolm instance we'll have to revisit that decision. At that point there would have to be a way to specify site on PCAP upload OR to associate a particular capture appliance (hedgehog capture node) with a site.

implementation

• Ruby script used by LogStash to do the enrichment and to cache resutls to avoid too-frequent lookups
• Logstash filter for lookups
• Logstash filter for segment comparison
• Dashboard
• index template
• Arkime custom fields
• Arkime custom views
• NetBox URL value actions for Arkime
• NetBox config and
  scripts
• install.py questions and docker-compose population
• auth_setup netbox secret generation
• various configuration in docker-compose*.yml (search case-insensitive for netbox)
• documentation
• proof of concept playing around with the netbox API . It's fairly easy to query the API for VRFs and Devices based on an IP address, as shown in that proof-of-concept script.
• device type library used to populate initial list of device manufacturers and types

[mmguero]

Currently I'm implementing these source and destination enrichment objects in the opensearch index template with the nested field type.

However, I might have to rethink this, due to opensearch-project/OpenSearch-Dashboards#657. Nested fields are not supported currently in visualizations, which makes them basically unusable in a Dashboards context. While they can be used in DQL queries, they can't be used in charts/tables/etc.

[mmguero]

However, I might have to rethink this

See mmguero-dev/Malcolm@29b0806

[mmguero]

Here's a screenshot of the new Network Assets dashboard which highlights the enrichment done to network traffic metadata based on NetBox lookups:

[mmguero]

Here's a screenshot of Arkime showing a conn.log entry with the netbox stuff populated:

[mmguero]

Here's a video illustrating navigating from Arkime directly to a device (or subnet) in the NetBox interface:

output.mp4

[mmguero]

Illustrates using network segment and other enriched data in Arkime's connections view:

[mmguero]

Illustrates using SPIGraph to break down network traffic by netbox-enriched fields:

[mmguero]

Here are some more screenshots from NetBox itself illustrating some of the things Malcolm will use to do enrichment (populated for some fake demo data).

Sites

Devices

Device Roles

Device Types

Manufacturers

Prefixes/VRFs

Virtual Machines