Linux Kernel Runtime Guard (LKRG)
=================================

LKRG performs runtime integrity checking of the Linux kernel and detection of
security vulnerability exploits against the kernel.

LKRG is a kernel module (not a kernel patch), so it can be built for and loaded
on top of a wide range of mainline and distros' kernels, without needing to
patch those.  We currently support kernel versions ranging from as far back as
RHEL7's (and its many clones/revisions) and Ubuntu 16.04's to latest mainline
and distros' kernels.  Our Continuous Integration setup has tested this version
of LKRG with up to latest mainline kernel 6.8.0-060800rc6daily20240227-generic
as available for Ubuntu on the release date.

LKRG currently supports the x86-64, 32-bit x86, AArch64 (ARM64), and 32-bit ARM
CPU architectures.

Please refer to CONCEPTS for concepts behind LKRG and for information on its
efficacy, and to PERFORMANCE for information on its performance impact.

The following sections describe how to obtain LKRG sources, build LKRG, test
it, install it on the system, and customize its configuration.


Getting the sources
-------------------

For LKRG releases and latest source code, please refer to its homepage:

	https://lkrg.org

To download this release from there and verify it, you would have used commands
like the below:

	wget https://www.openwall.com/signatures/openwall-offline-signatures.asc
	gpg --import openwall-offline-signatures.asc
	wget https://lkrg.org/download/lkrg-0.9.8.tar.gz.sign
	wget https://lkrg.org/download/lkrg-0.9.8.tar.gz
	gpg --verify lkrg-0.9.8.tar.gz.sign lkrg-0.9.8.tar.gz

Please preserve the GnuPG key above and also use it to verify future releases,
which will most likely work in a similar manner.

Latest LKRG development source code is hosted on GitHub, from where you can
clone the git repository to a local directory using the following command:

	git clone https://github.com/lkrg-org/lkrg


Build requirements
------------------

To build LKRG, you will need the following software:

- GNU make

- GCC, ideally the same version of it that was used to build the kernel itself
  (some people manage with clang, but this is unsupported, so expect issues)

- libelf, including its "development" sub-package, in case your target kernel
  was built with CONFIG_UNWINDER_ORC=y

- A kernel build directory corresponding to the Linux kernel image the module
  is to run on.

For example, under Debian and Ubuntu you can install all of these with:

	sudo apt-get install make gcc libelf-dev linux-headers-$(uname -r)

and under Red Hat'ish distributions (e.g. RHEL, CentOS, Fedora) with:

	sudo yum install make gcc elfutils-libelf-devel kernel-devel

(For documentation purposes, we prefix commands requiring root access with
"sudo", but you may of course run them as root by different means.)


Building
--------

With the above requirements satisfied, you should be able to easily build LKRG
by running "make" when you're in LKRG's top level source code directory.
Building LKRG does not require root, and thus shouldn't be done as root.

To speed up the building, we recommend specifying a parallel job count matching
your machine's logical CPU count, e.g. like this:

	make -j8


Testing
-------

We recommend that before you install LKRG on the system such that it would be
started on bootup, you manually test loading the LKRG module into the kernel
without making the setup permanent.  We also recommend that you keep LKRG's
detection of kernel integrity violations enabled for this test, yet change
its enforcement action from kernel panic (the default) to mere logging.
This way, you can safely detect potential system-specific false positives and
only proceed with installation if there are none.

You can do this for a freshly built LKRG (and while you're still in its top
level source code directory) with the following command:

	sudo insmod output/lkrg.ko kint_enforce=1

Then check kernel messages for any potential errors, use the system for a long
while, and check again:

	sudo dmesg

(Depending on kernel version and system configuration, the "dmesg" command
might not require root.)

Unload LKRG from the kernel with:

	sudo rmmod lkrg

so that it can then be loaded using the same procedure that's used on system
bootup and without the parameter override.


Installation
------------

If your Linux distribution uses a supported init system (systemd or OpenRC),
you can install LKRG with:

	sudo make install

while you're still in its top level source code directory.

We don't in any way favor one init system over another, and would gladly add
support for more of them if there's demand, or especially if we receive such
contributions.  Meanwhile, on a distribution without a supported init system
you can let "sudo make install" partially complete (up to the point where it
finds you're not using a supported init system).

Run the following command to start the LKRG service, for systemd:

	sudo systemctl start lkrg

for OpenRC:

	sudo /etc/init.d/lkrg start

for other:

	sudo modprobe -v lkrg


Autoload on bootup
------------------

In order to automatically load LKRG into the Linux kernel on each bootup run
the following command, for systemd:

	sudo systemctl enable lkrg

for OpenRC:

	sudo rc-update add lkrg boot

for other:

	sudo mkdir -p /etc/modules-load.d/ &&
		echo lkrg | sudo tee /etc/modules-load.d/lkrg.conf

Alternatively, you can put the "modprobe lkrg" command into a system startup
script.  Please note that ideally this command would run before sysctl files
(especially /etc/sysctl.d/01-lkrg.conf) are processed, or otherwise the LKRG
settings specified in those would not take effect.


Installing using DKMS
---------------------

DKMS enables kernel modules to be dynamically built for each kernel version.
What this means in effect is that on kernel upgrades the module is rebuilt.
You can install LKRG using DKMS as well.  For instance, on Red Hat'ish
distributions after following the shared download instructions above:

	sudo tar -xzf lkrg-0.9.8.tar.gz -C /usr/src/
	sudo dnf update -y
	sudo dnf install kernel-devel dkms openssl
	sudo dkms add -m lkrg -v 0.9.8
	sudo dkms build -m lkrg -v 0.9.8
	sudo dkms install -m lkrg -v 0.9.8

The only difference on other distributions should be the installation of the
kernel headers, the DKMS utility, and OpenSSL.  Install the headers for the
target kernels.

You can then query the status with:

	dkms status

If everything is right, you should get similar output to the following:

	lkrg/0.9.8, 5.18.9-200.fc36.x86_64, x86_64: installed

Please refer to the previous two sections for how to start the LKRG service or
have it started on system bootup.  If you wish to use the unit/init file, you
must install it manually, e.g., by running the `lkrg-bootup.sh` script
located under `scripts/bootup/` with the `install` subcommand (as root).


Uninstalling
------------

Similarly to installation, you can uninstall LKRG using "make" as well:

	sudo make uninstall

while you're in the top level source code directory of the installed version.

If you installed using DKMS, you'd uninstall with:

	sudo dkms remove -m lkrg/0.9.8 --all

You can also use the following command to temporarily stop the LKRG service
without uninstalling it, for systemd:

	sudo systemctl stop lkrg

for OpenRC:

	sudo /etc/init.d/lkrg stop

for other:

	sudo modprobe -v -r lkrg


Upgrading
---------

Our suggested way to upgrade LKRG is to start by uninstalling the old version.

You can then follow the Testing and Installation steps for the new version.


Recovery
--------

To account for the hopefully unlikely, but really unfortunate event that some
incompatibility between the Linux kernel or other components of the system and
LKRG isn't detected prior to LKRG installation, yet leads to system crash on
bootup, we've included support for the "nolkrg" kernel parameter.  Thus, you
may disable LKRG by specifying "nolkrg" on the kernel command-line via your
bootloader.  The system should then boot up without LKRG, and thus without
triggering the problem, letting you fix it.  You must be aware though, that you
will not be able to manually load the LKRG module if the kernel was booted with
this parameter.


Module parameters
-----------------

The LKRG kernel module supports a number of parameters, including kint_enforce
already mentioned above and many more.

For freshly built LKRG, you can list the parameters with:

	modinfo output/lkrg.ko

while you're still in LKRG's top level source code directory.

With LKRG installed on the system, you can list them with:

	sudo modinfo lkrg

(Depending on system configuration, "modinfo" might not require root.)

Parameters can be specified on command-lines of "insmod", "modprobe", or after
"options lkrg " in a file in the /etc/modprobe.d directory.

For descriptions of the parameters and their default and possible values,
please refer to the following two sections.


Remote logging configuration (load-time only)
---------------------------------------------

LKRG supports the following module parameters (with default values or lack
thereof specified in braces) to enable its optional remote logging.

- net_server_addr (no default)
  Log server IPv4 address (e.g., 127.0.0.1)

- net_server_port (514)
  Log server TCP port number

- net_server_pk (no default)
  Log server public key (64 hexadecimal digits)

If you're starting LKRG via a systemd unit or startup script (such as those
provided in here), our recommended way to specify the above parameters is by
creating the file /etc/modprobe.d/lkrg.conf with something like this in it:

options lkrg net_server_addr=127.0.0.1 net_server_pk=64hexdigitshere

Please refer to LOGGING on how to use the corresponding userspace components.


Load-time and runtime configuration
-----------------------------------

Besides the parameters optionally specified when loading the module into the
kernel, LKRG also supports a number of sysctl's, which can be used to adjust
its behavior when it is already loaded into the kernel.  For each feature that
is configurable at both load time and run time, we have a module parameter and
a sysctl of similar name (the module parameters lack the "lkrg." prefix, but
are otherwise the same), so the below documentation is mostly usable for both.

To list all LKRG sysctl's and their current values, use:

	sudo sysctl -a | grep lkrg

The sysctl's are (with default values specified in braces):

- lkrg.profile_validate (3)
  Quick choice of a pre-defined profile controlling whether, when, and to what
  extent LKRG validates system integrity and detects attacks.  Allowed values
  are 0 (disabled), 1 (light), 2 (balanced), 3 (heavy), and 4 (paranoid).
  Additionally, this setting will read as 9 (custom) if an underlying setting
  is changed directly (potentially deviating from any of the profiles).

  Higher-numbered validation profiles provide higher likelihood of timely
  detection of an attack, but involve higher performance overhead and higher
  risk of incompatibility with other system software.  Profiles 1 to 3 provide
  reasonable tradeoffs.

  lkrg.profile_validate=3 or higher is incompatible with VirtualBox hosts,
  where you need to use at most lkrg.profile_validate=2.  However, there's no
  problem with setting lkrg.profile_validate=3 on Linux+LKRG guest systems in
  VirtualBox VMs.

  lkrg.profile_validate=4 (paranoid) is incompatible with many distributions
  and has unreasonably high performance overhead and poor scalability while not
  necessarily providing a practically relevant improvement in attack detection.

  Choosing a validation profile sets the following underlying settings, which
  are described further below: kint_validate, pint_validate, pcfi_validate,
  umh_validate, smep_validate, smap_validate, and msr_validate.

- lkrg.profile_enforce (2)
  Quick choice of a pre-defined profile controlling whether and how LKRG acts
  on detected integrity violations and attacks.  Allowed values are 0 (log and
  accept), 1 (selective), 2 (strict), and 3 (paranoid).  Additionally, this
  setting will read as 9 (custom) if an underlying setting is changed directly
  (potentially deviating from any of the profiles).

  Higher-numbered enforcement profiles provide higher likelihood of mitigating
  a compromise or stopping an attack, but also a higher risk of interfering
  with normal system behavior and to a worse extent in case of false positives.

  lkrg.profile_enforce=0 can be used for safe testing of LKRG, where any
  detected violations and attacks are logged but no enforcement is performed.
  It can also be useful where LKRG is meant to act as a sensor within a larger
  security monitoring and response setup (e.g., network-wide).

  lkrg.profile_enforce=1 performs selective enforcement - log only for kernel
  integrity violations, varying effective actions ranging from killing a task
  to triggering a kernel panic for other types of violations and attacks.
  This mode is extremely unlikely to panic the kernel on a false positive.

  lkrg.profile_enforce=2 performs strict enforcement - varying effective
  actions for all types of violations and attacks, including triggering a
  kernel panic for kernel integrity violations.

  lkrg.profile_enforce=3 performs the most paranoid enforcement - kernel panic
  for all types of violations and attacks.

  Choosing an enforcement profile sets the following underlying settings, which
  are described further below: kint_enforce, pint_enforce, pcfi_enforce,
  umh_enforce, smep_enforce, and smap_enforce.

  Also relevant is the kernel's kernel.panic sysctl and panic parameter, which
  makes the system reboot on kernel panic.  For example, kernel.panic=60 in
  /etc/sysctl.conf or in a file under the /etc/sysctl.d directory, or panic=60
  on the kernel's command-line, will make the system reboot in 60 seconds after
  a panic.  This provides a brief opportunity to read the panic message on the
  console yet makes an unattended server try to come back up on its own.

  Profiles are currently available via sysctl only - there are no corresponding
  module parameters.  However, the individual underlying settings, which are
  described further below, do have their corresponding module parameters.

- lkrg.heartbeat (0)
  Whether or not to print a heartbeat message ("System is clean!" or "Tasks are
  clean!" depending on other configuration) whenever the global integrity
  checking routine completes with no violations detected.  Allowed values are 0
  (don't print the message) and 1 (print the message if allowed by log_level).

- lkrg.interval (15)
  LKRG's timer interval for periodic invocation of the global integrity
  checking routine, in seconds.  Allowed values are 5 to 1800.

- lkrg.trigger (N/A)
  Force LKRG to invoke the global integrity checking routine.  If you set this
  to 1, the routine is immediately invoked and this sysctl is reset back to 0.

- lkrg.log_level (3)
  LKRG's logging verbosity level.  Allowed values are from 0 to 4 for normal
  builds or from 0 to 6 for debugging builds.

  Values of 4 and higher are meant for debugging only and produce too verbose
  logging for production use.  Moreover, some messages logged at those high
  levels contain information useful for kernel vulnerability exploitation,
  making those log levels potentially mildly insecure (depending on other
  system configuration).

- lkrg.block_modules (0)
  Whether or not to block further loading of kernel modules.  Allowed values
  are 0 (no) and 1 (yes).

  This feature is meant primarily to prevent unintended user-triggered (or
  attacker-triggered) auto-loading of maybe-vulnerable modules provided in a
  distribution after all intended modules have already been loaded.  This
  feature is not effective (nor is meant to be) against attackers who already
  have root privileges and try to load a module explicitly (they could simply
  flip this setting or even unload LKRG first).

  Please note that enabling this setting (too) early (e.g., using the module
  parameter or /etc/sysctl.*) may cause the system to fail to complete bootup
  (if required modules are still being loaded in later stages of bootup, which
  varies between distributions and system configurations).

  Also relevant is the kernel's kernel.modules_disabled sysctl, which fully
  disables module loading until the system is rebooted.

- lkrg.hide (0)
  Whether or not LKRG should hide itself from the lists of loaded modules and
  KOBJs.  Allowed values are 0 (do not hide LKRG, or unhide it if previously
  hidden) and 1 (hide LKRG).

  Please note that LKRG can be easily detected by other means anyway, such as
  through the presence of its sysctl's.

- lkrg.kint_validate (3)
  Whether and when to validate global kernel integrity.  Allowed values are 0
  (disabled), 1 (only when manually triggered by lkrg.trigger), 2 (also
  periodically every lkrg.interval seconds), and 3 (also periodically every
  lkrg.interval seconds and probabilistically on certain other events).

  This currently applies to kernel and modules code and read-only data, global
  SELinux settings, and some CPU status registers/bits (WP, SMEP, SMAP, MSRs).
  (The validation and enforcement of SMEP, SMAP, and MSRs are separately
  controlled by their respective knobs described below, and SMEP and SMAP are
  validated much more frequently, not only as part of global kernel integrity.)

- lkrg.kint_enforce (2)
  How to act on global kernel integrity violations.  Allowed values are 0 (log
  once and accept new likely-compromised state as valid), 1 (log only for most
  violations, log the violation and restore previous state for SELinux and CPU
  WP bit), and 2 (panic the kernel).

  Note that lkrg.kint_enforce=1 is expected to produce repeated log messages on
  most kernel integrity violations, which can be noisy.  Also note that
  lkrg.kint_enforce=2 is unfortunately the only way to make full use of LKRG's
  global kernel integrity validation.  Running with lkrg.kint_validate=2 or
  higher but lkrg.kint_enforce set to 0 or 1 wastes CPU time on costly checks
  without achieving a corresponding security improvement, except that it might
  provide logs for post-mortem detection and analysis of a security compromise.

- lkrg.pint_validate (2)
  Whether and when to validate process credentials integrity.  Allowed values
  are 0 (disabled), 1 (validate a task's credentials just before it'd make use
  of the credentials), 2 (currently, it has the same meaning as 1), and 3
  (validate credentials of all tasks in the system whenever any task is about
  to make use of its credentials).

  Except with lkrg.pint_validate=0, we also validate the credentials of all
  tasks as part of LKRG's global integrity checking routine.

  lkrg.pint_validate=1 is sufficient to provide most of LKRG's potential at
  timely detection of exploits.  lkrg.pint_validate=3 is a paranoid mode with
  high performance overhead yet likely a minuscule gain in security.

- lkrg.pint_enforce (1)
  How to act on process credentials integrity violations.  Allowed values are 0
  (log once and accept new likely-compromised state as valid), 1 (kill the
  task), and 2 (panic the kernel).

  In Linux kernel's terminology, which we also use here, a "task" refers to a
  thread, and threads of a program may technically have different credentials.
  Our enforcement of process credentials integrity is thus per-thread, and e.g.
  it might happen that we kill an individual compromised thread of a program.

- lkrg.pcfi_validate (2)
  Whether and to what extent to validate Control Flow Integrity (CFI) on kernel
  functions that we monitor because of their usefulness for exploits' Return
  Oriented Programming (ROP) chains.  Allowed values are 0 (disabled), 1 (only
  validate the stack pointer), and 2 (also validate all stack frames).

  Because of the very limited extent of validation performed, we call our CFI
  mechanism pCFI, for poor man's CFI.

  lkrg.pcfi_validate=2 is incompatible with VirtualBox hosts, where you need to
  use at most lkrg.pcfi_validate=1.  However, there's no problem with setting
  lkrg.pcfi_validate=2 on Linux+LKRG guest systems in VirtualBox VMs.

- lkrg.pcfi_enforce (1)
  How to act on pCFI violations.  Allowed values are 0 (log only), 1 (kill the
  task), and 2 (panic the kernel).

  Note that lkrg.pcfi_enforce=0 may produce repeated log messages for the same
  violation, which might occasionally be noisy.

- lkrg.umh_validate (1)
  Whether and to what extent to validate uses of usermodehelper (UMH).  Allowed
  values are 0 (validation disabled), 1 (allow only previously known programs),
  and 2 (completely block UMH).

  UMH can also be protected with pCFI regardless of this setting.

  UMH is a kernel-internal interface, which the kernel uses to invoke programs
  such as /sbin/modprobe (to auto-load a module on demand) and many others.
  When left unrestricted, UMH is convenient for kernel vulnerability exploits.

- lkrg.umh_enforce (1)
  How to act on UMH usage violations.  Allowed values are 0 (log only), 1
  (prevent execution), and 2 (panic the kernel).

- lkrg.smep_validate (1)
  Whether or not to validate the Supervisor Mode Execution Protection (SMEP)
  bit on supporting x86-64 CPUs.  Allowed values are 0 (no) and 1 (yes).

- lkrg.smep_enforce (2)
  How to act on unexpected changes of the SMEP bit.  Allowed values are 0 (log
  once and accept new likely-compromised state as valid), 1 (log the violation
  and restore original value), and 2 (panic the kernel).

- lkrg.smap_validate (1)
  Whether or not to validate the Supervisor Mode Access Prevention (SMAP) bit
  on supporting x86-64 CPUs.  Allowed values are 0 (no) and 1 (yes).

- lkrg.smap_enforce (2)
  How to act on unexpected changes of the SMAP bit.  Allowed values are 0 (log
  once and accept new likely-compromised state as valid), 1 (log the violation
  and restore original value), and 2 (panic the kernel).

- lkrg.msr_validate (0)
  Whether or not to validate CPU Model Specific Registers (MSRs) as part of the
  global integrity checking routine.  Allowed values are 0 (no) and 1 (yes).

  This is currently specific to x86(-64) CPUs.

  There are situations where such validation is undesirable, such as if you run
  LKRG on a host machine that manages VMs and dynamically reconfigures MSRs.
  This is known to be the case for KVM and VirtualBox hosts, where this setting
  needs to be disabled.  However, there's no problem with enabling this setting
  on Linux+LKRG guest systems in VMs on those hosts, and indeed on systems that
  don't run KVM and VirtualBox.

That's all for now.  Greetings from the LKRG team!