Copyright (C) 2011 Battelle Memorial Institute

Licensed under the GNU General Public License Version 2.
See LICENSE for the full text of the license.
See DISCLAIMER for additional disclaimers.

Author: Brandon Carpenter
Contributors: Alex Malozemoff and Glenn Fink



H   H  OOO  N   N EEEEE
H   H O   O NN  N E
HHHHH O   O N N N EEEE
H   H O   O N  NN E
H   H  OOO  N   N EEEEE


Hone is a unique tool for correlating packets to processes to bridge the
HOst-NEtwork divide.


INTRODUCTION
============

When diagnosing anomalous behavior on a network, a system administrator
has two separate areas to focus on: the packets traveling over the
network (i.e., the network view), and the information contained on the
individual hosts connected to the network (i.e., the host view). The
network view provides a glimpse into the overall communication activity
of the network, but it does not reveal what processes are causing this
activity. On the other hand, the host view contains details on the
processes producing the network traffic, but it does not contain
information on which packets are associated with which process. This
inability to correlate packets with their associated process is a
fundamental (although intentional) shortcoming of the modern network
stack. To bridge this gap we introduce the Hone (Host-network)
correlator, an open-source tool that correlates packets to processes to
diagnose problems seen on a network.

While the idea of correlating packets to processes is a simple one, the
implementation requires kernel modifications that alter the way the
network stack works. Perhaps this complication is responsible for the
fact that no other tool takes this approach. While there have been
several tools that have come close to the packet-process correlation
approach taken here, they differ from Hone in fundamental ways.
Discussions of these tools and their differences from Hone can be found
in:

   FINK, G., DUGGIRALA, V., CORREA, R., AND NORTH, C. Bridging the
       host-network divide: survey, taxonomy, and solution. In
       Proceedings of the 20th Large Installation System
       Administration Conference (2006), pp.  247–262.

We plan to publish a follow-on paper on this work soon.


HOW IT WORKS
============

The Linux Hone correlator uses the netfilter framework, which provides
hooks for packet capture and modification at various places in the Linux
network stack. Hone registers callbacks with the INPUT and OUTPUT chains
of the filter table, providing notification of packets destined for and
leaving the local system.

Packets are correlated with a specific connection based on the socket
associated with the given packet. Outgoing packets already contain a
pointer to the originating socket, whereas incoming packets require
protocol-specific lookup functions from the transport layer to determine
the destination socket.

Packet and socket structures do not contain any information linking them
to a particular process. The owning process is determined by taking
advantage of the fact that sockets are created in the context of the
owning process. By hooking the socket creation routines, the creating
process is reported along with a socket identifier for later lookup.
Unfortunately, a problem arises when one performs a fork() or other
socket-duplicating operation, as it is currently not possible to tell
which process actually receives a given packet under these conditions.
We are currently investigating a solution to this problem.

The Linux correlator is comprised of multiple loadable kernel modules:
process_notify, socket_notify, packet_notify, hone_notify and honeevent.
The process_notify, socket_notify and packet_notify modules provide
notification routines to notify registrants of process, socket and
packet events.  The hone_notify module registers for to receive all
three event types and provides hone event notifications to the honeevent
module.  The honeevent module provides a character special device to
provide hone events to userspace in plain text or PCAP-NG format.

Process events are collected by hooking copy_process(), do_execve(),
compat_do_execve() and do_exit() kernel routines using kprobes.  This
approach has drawbacks, but does allows the Hone driver to be used
without modifying the kernel.


PREREQUISITES
=============

Building the Hone sensor kernel modules requires a few prerequisites:

  1. Build tools, such as the gcc tool chain and GNU make
  2. Kernel source header files for the target distribution
  3. Kernel configuration (.config, typically included with the header files)
  4. System map file from kernel build
  5. Kernel with kprobes support (CONFIG_KPROBES) (would like to eliminate
     this requirement)

If you can't build a Linux kernel, then you likely don't have the tools
to build a kernel module.  Be sure to install the recommended tools for
your distribution.  For instance, on Debian, install the build-essential
package.

By default, the build process will look for the kernel header files and
the configuration file in /lib/modules/$(KVER)/build (default for KSRC),
where KVER defaults to `uname -r`.  If you wish to build for a different
kernel version installed on the system, set KVER to the appropriate
version number when running make.  If the kernel headers live elsewhere,
you can tell make about it via the KSRC variable.

Some of the Hone modules use symbols that are not exported.  Therefore,
their address must be mapped statically using the system map file.  Make
will search for the appropriate map file in the following order:

  1. $(PWD)/System.map
  2. /boot/System.map-$(KERNELRELEASE)
  3. /lib/modules/$(KERNELRELEASE)/build/System.map
  4. /proc/kallsyms (only if the running and target versions are the same)
  5. $(sys)/System.map

You can override this search by setting the SYSMAP variable to the path
of the appropriate system map when calling make.


BUILDING
========

Assuming the above prerequisites are met, building the modules is a
trivial process.  Just change to the Hone source directory and execute
make.  Here are a few examples:

Basic build:

   $ make

 OR

   $ make all

If for some reason the prereq_check target fails, you can try skipping it:

   $ make modules

Building for a different version than the running kernel:

   $ make KVER=2.6.38

Building for a kernel that is not installed

   $ make KSRC=/usr/src/linux-2.6.38 SYSMAP=/usr/src/linux-2.6.38/System.map


INSTALLING
==========

To install the modules, just change to the Hone source directory and
execute `make install`.  If you set KVER, KSRC, SYSMAP, ARCH, or any
other make variables, you should set them for install as well.

If the modules are built outside the kernel tree, it may be necessary to
run depmod manually.  If, after installing the modules using `make
install`, you cannot use modprobe to insert the modules, try running
depmod.

Running depmod for the currently running kernel

   # /sbin/depmod -a

Check out the depmod(8) man page if you wish to do something fancier,
like depmod against a specific System.map file.


BUILDING PACKAGES
=================

There are three package types included in the Hone Makefile: tarball,
deb and rpm.  You can build all three by using the packages target or
build them individually using the tarball, deb, or rpm targets.  The deb
and rpm packages depend on dkms.


USAGE
=====

After installing the Hone sensor, the kernel module can be loaded using
modprobe:

   # modprobe honeevent

If for some reason modprobe fails (i.e. depmod failed or was not run),
the modules can be loaded manually using insmod:

   # insmod process_notify.ko
   # insmod socket_notify.ko
   # insmod packet_notify.ko
   # insmod hone_notify.ko
   # insmod honeevent.ko

This will load a character special device driver.  udev should
automatically create two devices in /dev: /dev/hone and /dev/honet.
/dev/hone provides PCAP-NG output while /dev/honet provides plain text
output, one record per line.

If udev is not being used, the devices can be created using mknod.  The
files can be created on any filesystem without the "nodev" mount option,
however, they are normally made in /dev.  Use the following command
replacing $MAJOR and $MINOR with the device major and minor numbers (see
the next paragraph).

   # mknod /dev/hone c $MAJOR $MINOR

The major number is dynamically assigned when the module is loaded.  It
can be determined by reading /sys/module/honeevent/parameters/major or
by locating the insertion message in syslog.  For PCAP-NG output, use
minor number 0.  For plain text, use minor number 1.

There are several module parameters that can be passed to the module at
load time:

   devname - optional base name to give the device in sysfs; default is
       hone
   major - optional major number to assign the device; default is 0
       meaning assigned automatically by the kernel
   hostid - optional GUID to include in the section header block used
       to identify the system on which the capture occurred.
   comment - optional comment to include in the section header block;
       spaces may be encoded as \040 or \x20
   snaplen - set the maximum packet capture length; default is 0 meaning full
       packet capture
   pageorder - set the pageorder value used when allocating pages for
       the ring buffer where buffer size = PAGE_SIZE * pow(2, pageorder);
       default is 1 (2 pages) for 32-bit systems and 2 (4 pages) on
       64-bit systems