CHANGES

2018-02-28 Jerry Lundström

    Release 1.9.0

    This release adds a new option to change how the Berkeley Packet Filter
    is generated to include the host restrictions for all selections,
    previously this restriction would only apply to specific parts.

    Additional tweaks to the RSSM plugin has been made to conform to the
    RSSAC002v3 specification. One noticeable change is that the plugin now
    requires the DNS to be parsed before counted, any error in the parsing
    will result in the message being left out of the statistics.

    Changes:
    - Fix spacing in BPF filter to look better
    - Fix #146: Add `bpf_hosts_apply_all`, apply any host restriction to all
    - `plugin/rssm`:
      - Remove quoting of `start-period` and correctly handle empty hashes
      - Issue #152, Issue #91: Parse DNS before processing RSSM counters
    - `plugin/rssm/dnscap-rssm-rssac002`: Use `YAML::Dump()` for output

    47d892b Issue #152: RSSM YAML output
    d4f1466 Issue #152, Issue #91: Parse DNS before processing RSSM counters
    68fc1ff BPF, `bpf_hosts_apply_all`

2018-02-07 Jerry Lundström

    Release 1.8.0

    This release updates the TCP stream code in order to be able to look
    at more then just the first query, for handling already ongoing TCP
    connections without having seen SYN/ACK and for reassembly of the TCP
    stream prior of parsing it for DNS with an additional layer of parsing
    (see `reassemble_tcp_bfbparsedns`).

    Updates to the Root Server Scaling Measurement (RSSM) plugin have also
    been made to bring it up to date with RSSAC002v3 specification, be
    able to output the YAML format described and an additional script to
    merge YAML files if the interval is less then the RSSAC002v3 24 hour
    period. See "Updates to the RSSM plugin" below and
    `plugins/rssm/README.md`.

    New extended options:
    - `parse_ongoing_tcp`: Start tracking TCP connections even if SYN/ACK
      has not been seen
    - `allow_reset_tcpstate`: Allow external reset of TCP state
    - `reassemble_tcp`: Use to enable TCP stream reassembly
    - `reassemble_tcp_faultreset`: Number of faults before reseting TCP
      state when reassembly is enabled
    - `reassemble_tcp_bfbparsedns`: Enable an experimental additional layer
      of reassemble that uses `libbind` to parse the payload before accepting
      it. If the DNS is invalid it will move 2 bytes within the payload and
      treat it as a new payload, taking the DNS length again and restart
      the process. Requires `libbind` and `reassemble_tcp`.

    New extension functions for plugins:
    - `DNSCAP_EXT_TCPSTATE_GETCURR`: Function to get a pointer for the
      current TCP state
    - `DNSCAP_EXT_TCPSTATE_RESET`: Function to reset a TCP state

    New features:
    - Parse additional DNS queries in TCP connections
    - `-g` and the `txtout` plugin will reset TCP state (if allowed) on
      failure to parse DNS

    Bugfixes:
    - Fix `-g` output, separate error message with a space
    - Fix TCP packets wrongfully flagged as DNS when using layers.
    - Fix TCP debug output when using layers, `ia_str()` is not safe to call
      twice in the same `printf` because of local buffer.
    - Fix exported extension functions, need to be file local

    New tests for:
    - Multiple DNS queries in one TCP connection
    - Query over TCP without SYN
    - Queries over TCP with first query missing length
    - Queries over TCP with middle payloads missing
    - Add test with TCP stream that missing multiple packets in the middle

    Updates to the RSSM plugin (`plugins/rssm`):
    - Add info about saving counts and sources
    - Fix memory leak on `fopen()` errors
    - Update to RSSAC002v3 specification
    - New options:
      - `-D` to disable forking on close
      - `-Y`: Use RSSAC002v3 YAML format when writing counters, the file
        will contain multiple YAML documents, one for each RSSAC002v3 metric
        Used with; -S adds custom metric `dnscap-rssm-sources` and -A adds
        `dnscap-rssm-aggregated-sources`
      - `-n`: Set the service name to use in RSSAC002v3 YAML
      - `-S`: Write source IPs into counters file with the prefix `source`
      - `-A`: Write aggregated IPv6(/64) sources into counters file with
        the prefix `aggregated-source`
      - `-a`: Write aggregated IPv6(/64) sources to
        `<name>.<timesec>.<timeusec>`
    - Add `dnscap-rssm-rssac002` Perl script for merging RSSAC002v3 YAML files
    - Add README.md for the plugin man-page for `dnscap-rssm-rssac002`
    - Add test for YAML output and merging of YAML files

    c7058c8 Use file local functions for all extensions
    66b352d RSSM RSSAC002v3 YAML Tool
    b09efc2 `plugins/rssm` RSSAC002v3
    709aba6 Fix #89: Add additional reassembly layers that parses the
            payload byte for byte for valid DNS
    04fa013 Fix CID 1463944 (again)
    b1cf623 RSSM saving data and forking
    fb23305 Fix CID 1463944
    0fca1a8 Issue #89: TCP stream reassemble
    bb6428c CID 1463814: Check `ns_initparse()` for errors
    a57066f Fix #88: TCP handling

2017-12-27 Jerry Lundström

    Release 1.7.1

    The library used for parsing DNS (libbind) is unable to parse DNS
    messages when there is padding at the end (the UDP/TCP payload is larger
    then the DNS message). This has been fixed by trying to find the actual
    DNS message size, walking all labels and RR data, and then retry parsing.

    Other changes and bug-fixes:
    - Fix size when there is a VLAN to match output of `use_layers` yes/no
    - Add test of VLAN matching
    - Fix `hashtbl.c` building in `rssm`
    - Add test with padded DNS message

    49e5400 Fix #127: If `ns_initparse()` returns `EMSGSIZE`, try and get
            actual size and reparse
    99bda0b Fix #98: VLAN

2017-12-19 Jerry Lundström

    Release 1.7.0

    This release adds IP fragmentation handling by using layers in pcap-thread
    which also adds a new flag to output and modules. `DNSCAP_OUTPUT_ISLAYER`
    indicates that `pkt_copy` is equal to `payload` since the layers of the
    traffic have already been parsed. IP fragments are reassembled with the
    `pcap_thread_ext_frag` extension that is included in pcap-thread.

    New extended (`-o`) options:
    - `use_layers`: Use pcap-thread layers to handle the traffic
    - `defrag_ipv4`: Enabled IPv4 de-fragmentation
    - `defrag_ipv6`: Enabled IPv6 de-fragmentation
    - `max_ipv4_fragments`: Set maximum fragmented IPv4 packets to track
    - `max_ipv4_fragments_per_packet`: Set the maximum IPv4 fragments per
      tracked packet
    - `max_ipv6_fragments`: Set maximum fragmented IPv6 packets to track
    - `max_ipv6_fragments_per_packet`: Set the maximum IPv6 fragments per
      tracked packet

    Currently `-w` does not work with `use_layers` and the plugins `pcapdump`
    and `royparse` will discard output with the flag `DNSCAP_OUTPUT_ISLAYER`
    because they need access to the original packet.

    The `rzkeychange` plugin now encodes certain flag bits in the data that
    it reports for RFC8145 key tag signaling. The flags of interest are:
    `DO`, `CD`, and `RD`. These are encoded in an bit-mask as a hexadecimal
    value before the `_ta` component of the query name.

    Other changes and bug-fixes:
    - Fix #115: document `-g` output, see `OUTPUT FORMATS` `diagnostic` in
      `dnscap(1)` man-page
    - Add test to match output from non-layers runs with those using layers
    - Add test with fragmented DNS queries
    - Fix #120: CBOR/CDS compiles again, update tinycbor to v0.4.2
    - Fix `ip->ip_len` byte order
    - Fix parsing of IP packets with padding or missing parts of payload

    0347f74 Add AUTHORS section in man-page
    ef1b68c Fix CID 1463073
    8a79f89 Layers
    a404d08 Update pcap-thread to v3.1.0, add test for padding fixes
    08402f1 Fix byte order bug.  ip->ip_len must be evaluated with ntohs().
    d6d2340 CBOR/CDS and formatting
    85ec2d8 Fix #87: IP fragmentation reassembly
    22bfd4a Documentation
    c35f19f Adding flag bits to rzkeychange RFC8145 key tag signaling data.
            This may be useful to find "false" key tag signals from sources
            that don't actually perform DNSSEC validation.

2017-12-01 Jerry Lundström

    Release 1.6.0

    New additions to the plugins:
    - `rzkeychange` can now collect RFC8145 key tag signaling. Signals are
      saved during the collection interval, and then sent to the specified
      `-k <zone>`, one at a time, at the end of the interval. Only root zone
      signals are collected. Added by Duane Wessels (@wessels).
    - `royparse` is a new plugin to splits a PCAP into two streams, queries
      in PCAP format and responses in ASCII format. Created by Roy Arends
      (@RoyArends).
    - `txtout` new option `-s` for short output, only print QTYPE and QNAME
      for IN records. Added by Paul Hoffman (@paulehoffman)
    - The extension interface has been extended with `DNSCAP_EXT_IA_STR` to
      export the `ia_str()` function.

    Bugfixes and other changes:
    - Remove duplicated hashtbl code
    - `rssm`: fix bug where count in table was taken out as `uint16_t` but
      was a `uint64_t`
    - Handle return values from hashtbl functions
    - `txtout`: removed unused `-f` options
    - Change `ia_str()` to use buffers with correct sizes, thanks to
      @RoyArends for spotting this!

    Commits:
    3f78a31 Add copy/author text
    1bd914d Fix CID 1462343, 1462344, 1462345
    f9bb955 Fix `fprintf()` format for message size
    abedf84 Fix #105: `inet_ntop` buffers
    bfdcd0d Addresses the suggestions from Jerry.
    dda0996 royparse :)
    4f6520a royparse plugin finished
    f1aa4f2 Fix #103: Remove `opt_f`
    32355b7 Rearrange code to keep the change smaller and fix indentation
    d6612c1 Added -s to txtout for short output
    9d8d1ef Check return of `snprintf()`
    55f5aba Format code
    9f19ec3 Fixed memory leak in rzkeychange_keytagsignal()
    58b8784 Fix memory leaks and better return value checks in
            rzkeychange_submit_counts()
    b06659f Add server and node to keytag signal query name
    705a866 Always free response packets in rzkeychange plugin.
    e802843 Implement RFC8145 key tag signal collection in rzkeychange plugin
    5fbf6d0 Added extension for ia_str() so it can be used by rzkeychange
            plugin.
    3be8b8f Split `dnscap.c` into more files
    e431d14 Fix #92: hashtbl

2017-08-21 Jerry Lundström

    Release 1.5.1

    Compatibility fixes for FreeBSD 11.1+ which is now packing `struct ip`
    and for OpenBSD.

    Commits:
    17e3c92 FreeBSD is packing `struct ip`, need to `memcpy()`
    f8add66 Code formatting
    38cd585 Add documentation about libbind
    d1dd55b Fix #82: Update dependencies for OpenBSD

2017-06-06 Jerry Lundström

    Release 1.5.0

    Added support for writing gzipped PCAP if the `-W` suffix ends with
    `.gz` and made `-X` work without `-x`. New inteface for plugins to
    tell them what extensions are available and a new plugin `rzkeychange`.

    Plugin extensions:
    - Call `plugin_extension(ext, arg)` to tell plugin what extensions exists
    - Add extension for checking responder (`is_responder()`)

    The rzkeychange plugin was developed by Duane Wessels 2016 in support
    of the root zone ZSK size increase. It is also being used in support of
    the 2017 root KSK rollover and collects the following measurements:
    - total number of responses sent
    - number of responses with TC bit set
    - number of responses over TCP
    - number of DNSKEY responses
    - number of ICMP_UNREACH_NEEDFRAG messages received
    - number of ICMP_TIMXCEED_INTRANS messages received
    - number of ICMP_TIMXCEED_REASS messages received

    Other fixes (author Duane Wessels):
    - 232cbd0: Correct comment description for meaning of IPPROTO_AH
    - 181eaa4: Add #include <sys/time.h> for struct timeval on NetBSD

    Commits:

    1d894e2 Make -x and -X work correctly together and update man-page
    34bc54c Make the -X option work without requiring a -x option.
    f43222e Fix CID 1440488, 1440489, 1440490
    aa54395 Update pcap-thread to v2.1.3
    81174ce Prepare SPEC for OSB/COPR
    21d7468 New plugin rzkeychange and plugin extensions
    38491a3 Config header is generated by autotools
    419a8ab Small tweaks and fixes for gzip support
    1967abc updated for earlier BSD versions
    f135c90 added auto gzip if the -W suffix ends with .gz

    Commits during development of rzkeychange (author Duane Wessels):
    - 620828d: Add rzkeychange -z option to specify resolver IP addresses
    - 1f77987: Add -p and -t options to rzkeychange plugin to configure an
      alternate port and TCP. Useful for ssh tunnels.
    - 2a571f1: Split ICMP time exceeded counter into two counters for time
      exceeded due to TTL and another due to fragmentation
    - e4ee2d3: The rzkeychange data collection plugin uses
      `DNSCAP_EXT_IS_RESPONDER` extension to know if an IP address is a
      "responder" or not, because when dnscap is instructed to collect ICMP
      with -I, it processes all ICMP packets, not just those limited to
      responders (or initiators).
    - cee16b8: Add ICMP Time Exceeded to counters
    - ad8a227: Counting source IPs has performance impacts. #ifdef'd out for
      now add ICMP "frag needed" counts
    - c25e72b: Implemented DNS queries with ldns. First there will be some
      test queries to ensure the zone is reachable and configured to receive
      data. Then a query naming the fields, followed by the periodic queries
      delivering counts.
    - fd23be7: Make report zone, server, node command line argumements mandatory
    - 137789b: Adding rzkeychange plugin files

2017-03-29 Jerry Lundström

    Release 1.4.1

    Fixed an issue that when compiled with libpcap that had a specific
    feature enabled it would result in a runtime error which could not be
    worked around.

    Also fixed various compatibility issues and updated dependency
    documentation for CentOS.

    Commits:

    785d4c4 Fix compiler warnings
    2d4df8d Fix #65: Update pcap-thread to v2.1.2
    26d3fbc Fix #64: Add missing dependency
    55e6741 Update pcap-thread to v2.1.1, fix issue with libpcap timestamp
            type
    c6fdb7a Fix typo and remove unused variables

2017-02-27 Jerry Lundström

    Release 1.4.0

    Until it can be confirmed that the threaded code works as well as the
    non-threaded code it has been made optional and requires a configuration
    option to enable it during compilation.

    New extended option:
    - `-o pcap_buffer_size=<bytes>` can be used to increase the capture
      buffer within pcap-thread/libpcap, this can help mitigate dropped
      packets by the kernel during breaks (like when closing dump file).

    Commits:

    1c6fbb2 Update copyright year
    63ef665 Suppress OpenBSD warnings about symbols
    2c99946 pcap-thread v2.0.0, disable threads, errors handling
    4cade97 Fix #56: Update pcap-thread to v1.2.2 and add test

2016-12-23 Jerry Lundström

    Release 1.3.0

    Rare lockup has been fixed that could happen if a signal was received
    in the wrong thread at the wrong time due to `pcap_thread_stop()`
    canceling and waiting on threads to join again. The handling of signals
    have been improved for threaded and non-threaded operations.

    New features:
    - Experimental CBOR DNS Stream format output, see `CBOR_DNS_STREAM.md`
    - Extended options to specify user and group to use when dropping
      privileges, see EXTENDED OPTIONS in man-page

    Commits:

    a5fa14e Signal and threads
    3868104 Use old style C comments
    7946be5 Clarify building
    d5463b4 RPM spec and various automake fixes
    df206bf Resource data indexing and documentation
    0e2d0fe Fix #22, fix #43: Update README
    5921d73 Add stream option RLABELS and RLABEL_MIN_SIZE
    6dd6ec1 Implement experimental CBOR DNS Stream Format
    4baf695 Fix #37: Extended options to specifty user/group to use when
            dropping privileges
    61d830a Fix #35: Use `AC_HEADER_TIME` and fix warning

2016-10-27 Jerry Lundström

    Release 1.2.0

    Update `pcap-thread` to v1.2.0 to get the new callback queue mode which
    puts that mode into using pthread conditions if all pcaps are offline and
    keeps us from losing packets.

    Use `pcap_thread_dropback()` callback to get the notification when a
    packet was dropped because the queue was full, indicating that we can't
    process all the packets.  Added this stats to the `-S` output as total
    and per interface as `ptdrop`.  Changed the output for each interface
    to not cut of information, for example interface name was cut to
    4 characters.

    Other changes:

    - Add extended options `-o <option>=<value>` because we are running out
      of short options.
    - Better handling of library checks and automake rules
    - New option `-F <format>` to specify the format of the output in `-w`
    - Add experimental CBOR output support
      - LDNS is used to parse the packets
      - Tinycbor is used to construct the CBOR output
      - DNS-in-JSON draft [1] for representing the objects
      - Check CBOR topic in README.md for more information
    - When only reading offline pcap files it will not attempt to drop
      privileges and add new option `-N` to explicitly not drop privileges.

    Commits:

    f42e23f Extended options and CBOR output format
    a28f498 Fix #24: Handle packet drops
    2308eaa Fix #26: Unable to drop GID to nobody, exiting.
    82d65f2 Update pcap-thread to v1.1.2

    [1] https://datatracker.ietf.org/doc/draft-hoffman-dns-in-json/

2016-10-11 Jerry Lundström

    Release 1.1.0

    The ownership of DNSCAP was transferred from ISC to DNS-OARC in
    the summer of 2016 and this is the first release since that.

    This project now uses Semantic Versioning and these are the changes
    since the `dnscap-20160205` release (which can also be found using
    the tag `v0.0.0-20160205`).

    Highlights:
    - Restructure repository and use autotools
    - Compiled and tested on Debian, Ubuntu, CentOS, FreeBSD and OpenBSD
      using Jenkins and Travis-CI
    - Source code static analysis using Coverity Scan
    - Compatibility fixes for FreeBSD, OpenBSD and OS X
    - ABI change to `output()`, previous `isfrag` is now a `flags` that
      represents what the packet is through a bitmask
    - Use helper library `pcap-thread` when capturing to solve missing
      packets during very low traffic

    New command line options:
    - `-V`: Prints version and then exits
    - `-M`: Enable monitor mode on interfaces
    - `-D`: Enable immediate mode on interfaces
    - `-W`: Allow to specify a suffix for the pcap dump file
    - `-C`: Limit/rotate capture after a certain amount of bytes

    Special thanks to:
    - Duane Wessels
    - Paul Vixie
    - Klaus Darilion

    Commits:

    bc7eb22 Update license after ownership transfer from ISC to DNS-OARC,
            update contributors, add build badges and removed SuperFastHash
            since apparently it was not used.
    778e457 Add `-V` for displaying version and the exiting
    71c2d79 Fix #12: Sync man-page and help text
    33576ef Swap option C and D, C for this makes more sense. Also ensure
            that `capturedbytes` is zero on start.
    0077aff Correct dump trace with new `flags`
    f9cbba0 Do not use dump suffix unless it set
    4dd81d6 Update the man page
    7435c49 Change new option C to D because C was already taken
    813dddb Fix -B and -E, these options are supported only once
    76f19d1 fix usage of -W
    519b64f Add -Y option to short usage instructions
    348c738 Fix -C feature: capturedbytes was not increased
    3db6f94 Improve logging
    b567bef New option -C: limit/rotate capture after a certain amount
            of bytes
    341abdf Add -W feature: allow to specify a suffix for the pcap dump
            file, e. g.: '.pcap'
    097a3b4 Count every packet which is sent to output(), not only
            the normal ones.
    75e5968 Close PCAPs after dumper_close() to have statistics still
            available during dumper_close(). Otherwise we get a segfault
            on shutdown.
    c09d61a Add debian/ubuntu package files.
    020f2aa Forgot about the compiler warnings and fix the last
            Coverity Scan issue
    00c834d More Coverity Scan fixes
    ad2f230 Fix various Coverity Scan issues
    606f0cd Update pcap thread to version 1.1.1
    f065cd7 Fix #14: Add options `-M` and `-C` for monitor and
            immediate mode, update help and man-page.
    b872035 Update to pcap-thread version 1.1.0
    1f30637 Update pcap_thread to v1.0.1, add travis check that dnscap
            can run
    b19efaa Building from Git repository instructions
    b5460df Use `calloc()` instead of `malloc()` to be sure the memory
            is zeroed
    ae6a04d Use pcap_thread v1.0.0
    9426a2d Update pcap_thread and add pcap stats
    820b2f2 Update pcap_thread and support offline pcaps
    a47dd67 Update pcap_thread
    237a7a7 CentOS autoreconf complained
    7b5568c Use pcap_thread
    11d0388 Revert the changes on all lines that had NULL, 0 before.
    7d6a7e4 Passing IPv6 fragment payloads may not currently be safe.
            Needs more work. For now pass pkt=NULL to be safe for plugins.
    ea8f9a4 Make the family of output() functions future proof with a flags
            bitmask. Rather than separate 'isfrag' and 'isdns' flags,
            they are now set as bitmasks in a single 'flags' value passed
            to output() f
    472a172 A change to the interface of the family of output() functions.
    95a6e62 timeval.* are not unsigned
    d3f32de Fix #1: Use NS_*SZ
    e555871 Fix compiler warnings
    3ed8f29 Fix #1
    864cbd7 Can you change #ifdef __APPLE__ to check for the
            arpa/nameser_compat.h header and include it if it exists?
    796e8ea plugin/rssm needs to include arpa/nameser_compat.h for OS X
            so that the HEADER struct is declared.
    daf4bd3 In plugin/txtout silence compiler warnings about int vs short
    e5bc24b plugin/pcapdump needs to include arpa/nameser_compat.h for OS X
            so that the HEADER struct is declared.
    0061b57 Work around configure problem detecting libresolv on Mac OS X
            Without some #include files, the configure test won't find
            the symbol res_mkquery() in libresolv on OS X.  It is called
            res_9_mkquery()
    5309655 Mac OS X doesn't have setresuid() and setresgid().
            This patch adds configure checks for setreuid() and setregid()
            and will use those instead if the other versions are
            not available.
    d257a1c Fix compilation on FreeBSD and OpenBSD
    07b2a75 Restructure repository and move to Automake.