kabitools
=========

This is a toolkit that is intended to assist in determining whether a given
symbol affects or is affected by other symbols in a kABI sense.

You must install boost to run this kit.

What is a symbol? From the sparse project:

	An identifier with semantic meaning is a "symbol".

	There's a 1:n relationship: each symbol is always associated with
	one identifier, while each identifier can have one or more semantic
	meanings due to C scope rules.

	The progression is symbol -> token -> identifier. The token
	contains the information on where the symbol was declared.

The kernel can be compiled so that a graph is made for each of the
preprocessor ".i" files using the information supplied by sparse.

The kabitools can be installed from the following website. Choose the rpm
that matches your distro and architecture.

http://people.redhat.com/tcamuso/kabitools/

The following will be installed by the rpm.

/usr/sbin/kabi-data.sh
/usr/sbin/kabi-dump
/usr/sbin/kabi-graph
/usr/sbin/kabi-lookup
/usr/sbin/kabi-parser
/usr/sbin/kabiscan
/usr/sbin/kabiscan.help
/usr/sbin/makei.sh
/usr/share/doc/kabitools
/usr/share/doc/kabitools/README
/usr/share/kabitools-pegas-kernel-make.patch
/usr/share/kabitools-rhel-kernel-make.patch

Use of this toolkit consists primarily of two steps.

1. Create the kernel graph files
2. Use the lookup tool, kabi-lookup, or its menu-driven wrapper, kabiscan,
   to determine whether a given symbol could affect kABI.

   There are other tools in the kit for those who like to get under the hood,
   but the most practical application is comprised of the above two steps.


---------------------------------------------------------------------------
Creating the Kernel Graph Files wiith the kabi-graph script
---------------------------------------------------------------------------

This is the easiest way to create the kernel graph files.
This script must be invoked from the top of the kernel tree.

Time to complete a full kernel build is lengthened by about 10% by invoking
this tool.

kabi-graph -[jnDh] [target]

	Builds a graph from the preprocessor output of each file compiled.
	Must be invoked from the top of the kernel tree.

	The optional target argument can be any valid kernel make target
	including a subdirectory, e.g. "virt/" or a module build, e.g.
	"M=drivers/char/tpm"

	The kABI graph will be limited to the scope of the build of the
	optional target.

	Default action without the optarg:
		make clean	# can be disabled with -n option
		make
		make modules

	Options
	-j	- Optional number of processors to assign to the make task.
		  There must be a space between -j and the number.
		  Default is all processors in this system: 40

	-n	- Do not make clean first
	-R	- Recreate the list of graph files: redhat/kabi/kabi-datafiles.list
	-D	- Delete all the graph files and exit.
	-h	- This help screen.

	NOTE:	The graph file extension will be added to .git/info/exclude,
		but will be selectively removed with "kabi-graph -D" command,
		which removes all the graph files.


There are other methods for those who like to tinker. They are detailed at
the end of this README.

---------------------------------------------------------------------------
Using kabiscan to Find Symbols Affecting or Affected by kABI
---------------------------------------------------------------------------

The easiest way of doing this is with the kabiscan menu-driven wrapper
for kabi-lookup.

 Before using this tool, the following criteria must be met.

 * Install boost.
 * Build the kernel graph using the kabi-graph script.
 * Build the kABI whitelist with "make rh-kabi".
 * This script must be invoked from the top of your kernel tree.

 Here is a view of the default menu after a fresh install of kabitools.

 Scan The Kernel for kABI Sensitive Symbols

 Environment
     Most Recent Tag             : kernel-pegas-4.10.0-7.el7$
  b  Current git branch          : pegas
  f  File to receive output      : /dev/stdout
  n  Search name                 :
  m  Search mask                 :
  t  struct or union             : struct

 Run Parameters
  1  Stop after finding one      : true
  l  Only seek whitelist symbols : true
  w  Whole words only            : true
  v  Verbose output              : false
  s  Search mode                 : kABI Sensitive Symbols

 Control
  r  Run the search
  h  help text
  q  Quit

 Enter one of the above:

 Environment

        b - An enumerated list of git branches will be presented from which
            to select by number.

        f - Send the output to a file, instead of stdout.

        n - The name of the symbol being sought, e.g "device" or "enable_irq".

        m - An optional mask that can help speed searches, if you have an idea
            of where to look, e.g. "acpi" or "fs"

        t - This option does not appear when the Search Mode is set to
            "Exported Functions" or when the "Whole Word" search is disabled.
            By typing the associated key, the value toggles between "struct"
            and "union". This is used for more precise searches, when you
            know exactly what you're seeking.

 Run Parameters

         These parameters are toggled by pressing the associated key.

        1 - Stop after finding the first instance when "Whole Word" mode
            is true. When "Whole Word" is false, the search stops after
            the first file containing strings that match the "Search Name."

        l - Only seek symbols that are in the kABI whitelists. If the symbol
            cannot be found in the white list, the search stops there.
            With this option true, the "Whole Word" option is automatically
            set to "true", since you should know exactly what you're seeking
            in the white list.

        w - Whole words only works similarly to the grep -w option. The
            is more precise with this option.

        v - Verbose mode will drill down through the whole hierarchy of
            the given symbol. This generates a lot of output, so don't
            use it unless you really need it.

        s - Search Mode steps through the four available search modes.
            Each time the key is pressed, the Search Mode advances to
            the next mode in the rotation.

     There are several search modes.

 Exported Functions
        Simple lookup of exported function names. Searching the white
        list requires an exact name.

 kABI Sensitive Symbols
        These are symbols like "struct device" that may or may not be
        kABI sensitive, but this mode will test to see if indeed it is.
        Searches can be inexact by setting the "Whole Word" option to
        false.


 Symbol Definition
        Seeks the actual definition of the symbol. Most useful when you
        just want a quick look at the members of a data structure.
        In verbose mode, the entire hierarchy of the structure is presented,
        right down to the last subordinate element. This is a lot of output,
        but when you need it, it's there.

 Count symbol instances
        Counts the number of times the symbol, or substring, appears in
        the graph.


---------------------------------------------------------------------------
Using kabi-lookup to Find Symbols Affecting or Affected by kABI
---------------------------------------------------------------------------

If you prefer the command line, you can use the kabi-lookup tool directly.

kabi-lookup enables the user to determine whether a compound data type,
i.e. struct or union, has any relation to exported symbols. Conversely, the
tool can search for exported symbols to see their dependency tree. The tool
traverses the graph files looking for the symbol provided at the command
line.

Searches can be limited to whitelists and partial or complete pathnames.

If the given symbol is not a dependency of an exported symbol, a message is
printed to stdout indicating that the symbol cannot be found. This does not
mean that the symbol does not exist, but that the symbol does not appear
anywhere in the hierarchy of any of the exported symbols in the search.

It is also useful for dumping the symbols upon which an exported function
depends. The default output for exported functions lists the name, the
return value, if not void, and the argument list with types, if not void.

You can even dump the members of a union or data structure.

With the -v option, every symbol descendant from the symbol given at the
command line is printed to stdout. This is a LOT of stuff, so it's better
to redirect the output to a file for parsing. There are not many situations
where this volume and depth of information is necessary, but it's there if
you need it.

You can also use the tool to count how many times a given symbol appears in
the dependency trees of the exported symbols in the search.

kabi-lookup [-vwl] -e|s|c|d symbol [-m mask] [-p path]

Options:

	-e symbol
	   Searches for an exported symbol. Strings that omit the -w switch
	   will generate output from any and all symbols that contain the
	   string.

	-s symbol
	   Searches for the symbol of a compound data type that  matches
	   the string.

	-d symbol
	   Search for the declaration of a compound data type and print its
	   members to the screen

	-c symbol
	   Counts the number of times the symbol occurs in the search. This
	   is most useful when you know the exact name of the symbol and
	   are using the -w switch.

	-v Verbose output, prints the hierarchy down to the lowest level.
	   All descendants of nonscalar types are printed. This creates a
	   LOT of data, so it's probably better to direct output to a text
	   file for parsing.

	-w Whole words only. Default is substring match.

	   NOTE: With this switch, if the symbol is a struct or union, the
		 string must include the word "struct" or "union" before
		 the symbol being searched, and the string must be
		 encapsulated with single or double quotes.

	   Examples:
		kabi-lookup -sw "struct pci_dev"
		kabi-lookup -sw 'union ipmi_smi_info_union'

	-l Limit the search to symbols in the white lists.
	   What this means is that the symbol will only be considered found
	   if it appears in the hierarchy of a white listed symbol.

	   Run "make rh-kabi" to create the white lists.

	  NOTE: Must be used with the -w switch, as described above, or the
		symbol will not be found.

	  Examples:
		kabi-lookup -lsw "struct pci_dev"
		kabi-lookup -lsw 'union ipmi_smi_info_union'

	-m string
	   The mask string limits the search to pathnames containing the
	   mask string. For example ...

		 kabi-lookup -m fs -sw 'struct device'

	   ... will search every path having "fs" in its name, e.g.
	    drivers/video/fbsysfs.i and fs/coredump.i

	   If you wish to limit the search to a specific directory, a
	   trailing / must be used, e.g.

		 kabi-lookup -m fs/ -sw 'struct device'
		 kabi-lookup -m drivers/acpi/ -sw 'struct acpi_device'

	   The more unique the -m string, the more limited the search.

	-1 Hyphen-number-1, limits the search to the first instance.

	-p path
	   Path to the top of the kernel tree, if executing in a different
	   directory.

-------------------
Some Usage Examples
-------------------

	$ kabi-lookup -s pci_dev

	Searches for all instances of "pci_dev", including such patterns as
	"struct pci_device_id".


	$ kabi-lookup -sw "struct pci_dev"

	Will search only for exactly "struct pci_dev". Whole word matches
	for compound data types must include the "struct" or "union"
	declaration.

	NOTE: Quotes are required around search patterns having spaces.


	$ kabi-lookup -m pci -s pci_bus

	Searches for all instances of the string "pci_bus" in any and all
	subdirectories and files that have "pci" in their name.


	$ kabi-lookup -e mlx4_cq

	Searches for any and all exported symbols that match "mlx4_cq" and
	their argument lists and return values.

	$ kabi-lookup -ew mlx4_cq_modify

	Will find the one and only unique mlx4_cq_modify exported symbol
	and its argument lists and return values.


---------------------------------------------------------------------------
Other tools in the kabitools kit
---------------------------------------------------------------------------

kabi-parser - Parses preprocessor .i files for exported symbols and the
              compound data types on which they depend. The results for
              each .i file are serialized into a data file. Exported
              symbols and their entire dependency trees are recorded.

              The data files have the same name stem and reside in the
              same directory as their .i sources. The data is organized as
              as a map using a symbol's crc and its place in the hierarchy
              of its dependent exported symbol.

              Compound data types will include their backpointers, which
              introduces a bit of redundancy, but maximizes the data.
              Backpointers are followed to only one level below to avoid
              infinite recursion. Nested duplicates are also detected and
              limited for the same reason.
              /usr/sbin/kabi-parser

kabi-dump    - Dumps the contents of a serialized data file to make it
               humanly readable. It is really only a debugging tool.
               /usr/sbin/kabi-dump

makei.sh    - Uses the kernel make to compile preprocessor .i files.
              To save time, only files that have EXPORT_SYMBOL in them
              are processed.
              /usr/sbin/makei.sh

kabi-data.sh - A shell script wrapper for kabi-parser that invokes it for
               each .i file in the kernel tree. This shell script creates
               a list of files processed to be used by kabi-lookup tool.
               /usr/sbin/kabi-data.sh


---------------------------------------------------------------------------
Other Methods For Creating the Kernel Graph Files
---------------------------------------------------------------------------

1. Patch the kernel Makefiles
-----------------------------

Patches for the kernel Makefiles are also provided in this kit and are
saved in /usr/share in the following files.

/usr/share/kabitools-pegas-kernel-make.patch
/usr/share/kabitools-rhel-kernel-make.patch

The kabitools script actually applies the appropriate patch to build the
kernel graph, and reverses the patch before exiting.

Example:

# Patch the kernel make
patch -p1 < /usr/share/kabitools-rhel-kernel-make.patch

When you invoke a kernel make with K=1, the graph of the kernel will be
created during the compile of the kernel.

Example:

# Compile the kernel and create the kernel graph
make -j16 K=1

From this point, kabi-lookup can be used as described above.


2. makei.sh and kabi-data.sh scripts
------------------------------------

This method of creating the kernel graph requires both these scripts.

makei.sh - Use this script first to create the preprocessor intermediate
	   ".i" files for the kernel or kernel subdirectory.
	   Must be executed from the top of the kernel tree.

	   Examples:

	   # create .i files for the whole kernel
	   makei.sh ./

	   # create .i files for all files under drivers/acpi
	   makei.sh drivers/acpi

kabi-data.sh -	This script creates the graph files from the .i files
		It requires only one argument, the path to the top of the
		kernel tree.

		Other arguments can be seen by entering the command without
		arguments or with -h switch.

		The most useful of these is -S, which allows you to create
		a graph for just a kernel subdirectory.

		Examples:

		# Create the graph files for the whole kernel from the top
		# of the kernel tree.
		kabi-data.sh -d ./

		# Create the graph files for the whole kernel from a
		# different PWD.
		kabi-data.sh -d /path/to/my/kernel/tree

		# Create the graph files for a subdirectory of the kernel.
		kabi-data.sh -d ./ -s drivers/acpi