crash.8

.\"
.de CO
\dB\\$1\fP \fI\\$2\fP
..
.TH CRASH 8
.SH NAME
crash \- Analyze Linux crash dump data or a live system
.SH SYNOPSIS
.B crash
[\fIOPTION\fR]... \fINAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)\fR
.br
.B crash
[\fIOPTION\fR]... \fI[NAMELIST]                         (live system form)\fR
.SH DESCRIPTION
.B Crash
is a tool for interactively analyzing the state of the Linux system
while it is running, or after a kernel crash has occurred and a 
core dump has been created by the 
.I netdump,
.I diskdump,
.I LKCD,
.I kdump,
.I xendump
.I kvmdump
or
.I VMware
facilities.  It is loosely based on the SVR4 UNIX crash 
command, but has been significantly enhanced
by completely merging it with the 
.B gdb(1)
debugger. The marriage of the two effectively combines the 
kernel-specific nature of the traditional UNIX crash utility with the 
source code level debugging capabilities of 
.B gdb(1). 

In the
.I dumpfile form,
both a NAMELIST and a MEMORY-IMAGE argument must be entered.
In the
.I live system form,
the NAMELIST argument must be entered if 
the kernel's
.I vmlinux 
file is not
located in a known location, such as the 
.I /usr/lib/debug/lib/modules/<kernel-version>
directory.  

The
.B crash
utility has also been extended to support the analysis
of dumpfiles generated by a crash of the Xen hypervisor.  In that
case, the NAMELIST argument must be that of the 
.I xen-syms
binary.
Live system analysis is not supported for the Xen hypervisor.

The 
.B crash
utility command set consists of common kernel core analysis tools 
such as kernel stack back traces of all processes, source code disassembly,
formatted kernel structure and variable displays, virtual memory data, 
dumps of linked-lists, etc., along with several commands that delve 
deeper into specific kernel subsystems.  Appropriate 
.B gdb
commands may also be entered, which in
turn are passed on to the 
.B gdb 
module for execution. 
If desired, commands may be placed in either a
.I $HOME/.crashrc
file and/or in a 
.I .crashrc 
file in the current directory.
During initialization, the commands in 
.I $HOME/.crashrc
are executed first, followed by those in the 
.I ./.crashrc
file.

The 
.B crash 
utility is designed to be independent of Linux version 
dependencies. When new kernel source code impacts the
correct functionality of 
.B crash
and its command set, the utility will 
be updated to recognize new kernel code changes, while
maintaining backwards compatibility with earlier releases.
.SH OPTIONS
.de BS
\fB\\$1\fP\ \fR\\$2\fP
..
.TP
.BI NAMELIST
This is a pathname to an uncompressed kernel image
(a 
.I vmlinux 
file), or a Xen hypervisor image (a 
.I xen-syms
file) 
which has been compiled with the "-g" option.
If using the
.I dumpfile form,
a 
.I vmlinux 
file may be compressed in either gzip or bzip2 formats.
.TP
.BI MEMORY-IMAGE[@ADDRESS]
A kernel core dump file created by the
.I netdump,
.I diskdump,
.I LKCD
.I kdump,
.I xendump
.I kvmdump
or
.I VMware
facilities.  

If a MEMORY-IMAGE argument is not entered, the session will be invoked on
the live system, which typically requires root privileges because of
the device file used to access system RAM.  By default,
.I /dev/crash
will be used if it exists.  If it does not exist, then
.I /dev/mem 
will be used; but if the kernel has been configured 
with 
.B CONFIG_STRICT_DEVMEM, 
then
.I /proc/kcore 
will be used.
It is permissible to explicitly enter 
.I /dev/crash, 
.I /dev/mem
or 
.I /proc/kcore.

An @ADDRESS value must be appended to the MEMORY-IMAGE if the dumpfile
is a raw RAM dumpfile that has no header information describing the file
contents.  Multiple MEMORY-IMAGE@ADDRESS ordered pairs may be entered,
with each dumpfile containing a contiguous block of RAM, where the ADDRESS
value is the physical start address of the block expressed in hexadecimal.
The physical address value(s) will be used to create a temporary ELF header
in /var/tmp, which will only exist during the crash session.  If a raw RAM
dumpile represents a live memory source, such as that specified by the QEMU
mem-path argument of a memory-backend-file object, then "live:" must be
prepended to the MEMORY-IMAGE name.

As VMware facility, the
.B crash
utility is able to process VMware VM memory dump generated by VM suspend
or guest core dump. In that case, .vmss or .guest file should be used as
a MEMORY-IMAGE and .vmem file must be located in the same folder.
.TP
.BI mapfile
If the NAMELIST file is not the same kernel that is
running (live system form), or the kernel that was running 
when the system crashed (dumpfile form), then the
.I System.map
file of the original kernel should be entered on the command line.
.P
.BI -h \ [option]
.br
.BI \--help \ [option]
.RS
Without an 
.I option
argument, display a 
.B crash
usage help message.  If the 
.I option
argument is a
.B crash
command name, the help page for that command is displayed.  If it is
the string "input", a page describing the various
.B crash
command line input options is displayed.  If it is the string "output", a
page describing command line output options is displayed.  
If it is the string "all", then all of the possible help messages
are displayed.  After the help message is displayed, 
.B crash
exits.
.RE
.TP
.B \-s
Silently proceed directly to the "crash>" prompt without displaying
any version, GPL, or 
.B crash
initialization data during startup, and by default, runtime command
output is not passed to any scrolling command.
.TP
.BI \-i \ file
Execute the command(s) contained in
.I file
prior to displaying the "crash>" prompt for interactive
user input.
.TP
.BI \-d \ num
Set the internal debug level.
The higher the number, the more debugging data will be printed when
.B crash
initializes and runs.
.TP
.B \-S
Use 
.I /boot/System.map
as the
.I mapfile\fP.
.TP
.B \-e \fIvi | emacs\fR
Set the 
.B readline(3) 
command line editing mode to "vi" or "emacs".  The default
editing mode is "vi".
.TP
.B \-f
Force the usage of a compressed 
.I vmlinux 
file if its original name
does not start with "vmlinux".
.TP
.B \-k
Indicate that the NAMELIST file is an LKCD "Kerntypes" debuginfo file.
.TP
.BI -g \ [namelist]
Determine if a  
.I vmlinux 
or 
.I xen-syms 
namelist file contains debugging data.
.TP
.B \-t
Display the system-crash timestamp and exit. 
.TP
.B \-L
Attempt to lock all of its virtual address space into memory by calling mlockall(MCL_CURRENT|MCL_FUTURE) 
during initialization.  If the system call fails, an error message will be displayed,
but the session continues.
.TP
.BI \-c \ tty-device
Open the 
.I tty-device 
as the console used for debug messages.
.TP
.BI \-p \ page-size
If a processor's page size cannot be determined by the dumpfile,
and the processor default cannot be used, use 
.I page-size.
.TP
.BI \-o \ filename
Only used with the MEMORY-IMAGE@ADDRESS format for raw RAM dumpfiles, 
specifies a filename of a new ELF vmcore that will be created and used
as the dumpfile.  It will be saved to allow future use as a standalone
vmcore, replacing the original raw RAM dumpfile.
.P
.B -m \fIoption=value\fR
.br
.B --machdep \fIoption=value\fR
.RS
Pass an option and value pair to machine-dependent code.  These
architecture-specific option/pairs should only be
required in very rare circumstances:
.P
.nf
X86_64:
  phys_base=<physical-address>
  irq_eframe_link=<value>
  irq_stack_gap=<value>
  max_physmem_bits=<value>
  kernel_image_size=<value>
  vm=orig       (pre-2.6.11 virtual memory address ranges)
  vm=2.6.11     (2.6.11 and later virtual memory address ranges)
  vm=xen        (Xen kernel virtual memory address ranges)
  vm=xen-rhel4  (RHEL4 Xen kernel virtual address ranges)
  vm=5level     (5-level page tables)
  page_offset=<PAGE_OFFSET-value>
PPC64:
  vm=orig
  vm=2.6.14     (4-level page tables)
IA64:
  phys_start=<physical-address>
  init_stack_size=<size>
  vm=4l         (4-level page tables)
ARM:  
  phys_base=<physical-address>
ARM64:  
  phys_offset=<physical-address>
  kimage_voffset=<kimage_voffset-value>
  max_physmem_bits=<value>
  vabits_actual=<value>
X86:
  page_offset=<CONFIG_PAGE_OFFSET-value>
.fi
.RE
.TP
.B \-x
Automatically load extension modules from a particular directory. 
If a directory is specified in the 
.B CRASH_EXTENSIONS
shell environment
variable, then that directory will be used.  Otherwise 
.I /usr/lib64/crash/extensions
(64-bit architectures) or 
.I /usr/lib/crash/extensions
(32-bit architectures) will be used;
if they do not exist, then the 
.I ./extensions directory will be used.
.TP
.BI --active
Track only the active task on each cpu.
.TP
.BI --buildinfo
Display the crash binary's build date, the user ID of the builder,
the hostname of the machine where the build was done, the target 
architecture, the version number, and the compiler version.
.TP
.BI --memory_module \ modname
Use the
.I modname
as an alternative kernel module to the 
.I crash.ko
module that creates the
.I /dev/crash
device.
.TP
.BI --memory_device \ device
Use
.I device
as an alternative device to the 
.I /dev/crash, /dev/mem
or
.I /proc/kcore
devices.
.TP
.BI --log \ dumpfile
Dump the contents of the kernel log buffer.  A kernel namelist
argument is not necessary, but the dumpfile must contain the
VMCOREINFO data taken from the original /proc/vmcore ELF header.
Note: this option is deprecated and will no longer work for
kernel(>=v5.10).
.TP
.B --no_kallsyms
Do not use kallsyms-generated symbol information contained within 
kernel module object files.
.TP
.B --no_modules
Do not access or display any kernel module related information. 
.TP
.B --no_ikconf
Do not attempt to read configuration data that was built into kernels
configured with 
.B CONFIG_IKCONFIG.
.TP
.B --no_data_debug
Do not verify the validity of all structure member offsets and structure 
sizes that it uses.
.TP
.B --no_kmem_cache
Do not initialize the kernel's slab cache infrastructure, and commands that
use kmem_cache-related data will not work.
.TP
.B --no_elf_notes
Do not use the registers from the ELF NT_PRSTATUS notes saved in a compressed kdump header
for backtraces.
.TP
.B --kmem_cache_delay
Delay the initialization of the kernel's slab cache infrastructure until
it is required by a run-time command.
.TP
.B --readnow
Pass this flag to the embedded 
.B gdb
module, which will override its two-stage strategy that it uses for reading
symbol tables from the NAMELIST.
.TP
.B --smp
Specify that the system being analyzed is an SMP kernel.
.P
.B -v
.br
.B --version
.RS
Display the version of the
.B crash
utility, the version of the embedded
.B gdb
module, GPL information, and copyright notices.
.RE
.TP
.BI --cpus \ number
Specify the 
.I number 
of cpus in the SMP system being analyzed.
.TP
.BI --osrelease \ dumpfile
Display the OSRELEASE vmcoreinfo string from a kdump 
.I dumpfile 
header.
.TP
.BI --hyper
Force the session to be that of a Xen hypervisor.
.TP
.BI --p2m_mfn \ pfn
When a Xen Hypervisor or its dom0 kernel crashes, the dumpfile
is typically analyzed with either the Xen hypervisor or the dom0 kernel.
It is also possible to analyze any of the guest domU kernels if 
the pfn_to_mfn_list_list 
.I pfn 
value of the guest kernel is passed on the
command line along with its NAMELIST and the  
dumpfile.
.TP
.BI --xen_phys_start \ physical-address
Supply the base physical address of the Xen hypervisor's text and static data
for older xendump dumpfiles that did not pass that information in the dumpfile
header.
.TP
.B --zero_excluded
If the makedumpfile(8) facility has filtered a compressed kdump dumpfile to
exclude various types of non-essential pages, or has marked a compressed or
ELF kdump dumpfile as incomplete due to an ENOSPC or other error during its
creation, any attempt to read missing pages will fail.  With this flag, reads
from any of those pages will return zero-filled memory.
.TP
.B --no_panic
Do not attempt to find the task that was running when the kernel crashed.
Set the initial context to that of the "swapper" task on cpu 0.
.TP
.B --more
Use 
.I /bin/more 
as the command output scroller, overriding the default of 
.I /usr/bin/less
and any settings in either 
.I ./.crashrc 
or
.I $HOME/.crashrc. 
.TP
.B --less
Use 
.I /usr/bin/less 
as the command output scroller, overriding 
any settings in either 
.I ./.crashrc 
or 
.I $HOME/.crashrc. 
.TP
.B --hex
Set the default command output radix to 16, overriding the default radix of 10,
and any radix settings in either
.I ./.crashrc
or
.I $HOME/.crashrc.
.TP
.B --dec
Set the default command output radix to 10, overriding any 
radix settings in either
.I ./.crashrc
or 
.I $HOME/.crashrc.  This is the default radix setting.
.TP
.B --CRASHPAGER
Use the output paging command defined in the 
.B CRASHPAGER
shell environment
variable, overriding any settings in either 
.I ./.crashrc 
or 
.I $HOME/.crashrc.
.TP
.B --no_scroll
Do not pass run-time command output to any scrolling command. 
.TP
.B --no_strip
Do not strip cloned kernel text symbol names.
.TP
.B --no_crashrc
Do not execute the commands in either
.I $HOME/.crashrc
or 
.I ./.crashrc.
.TP
.BI --mod \ directory
When loading the debuginfo data of kernel modules with the 
.I mod -S
command, search for their object files in
.I directory
instead of in the standard location.
.TP
.BI --src \ directory
Search for the kernel source code in directory instead of in the
standard location that is compiled into the debuginfo data.
.TP
.BI --kaslr \ offset | auto
If an x86, x86_64, s390x or loongarch64 kernel was configured with
.B CONFIG_RANDOMIZE_BASE,
the offset value is equal to the difference between the symbol values 
compiled into the vmlinux file and their relocated KASLR values.  If set to
auto, the KASLR offset value will be automatically calculated.
.TP
.BI --reloc \ size
When analyzing live x86 kernels that were configured with a
.B CONFIG_PHYSICAL_START
value that is larger than its
.B CONFIG_PHYSICAL_ALIGN
value, then it will be necessary to enter
a relocation size equal to the difference between the two values.
.TP
.BI --hash \ count
Set the number of internal hash queue heads used for list gathering
and verification.  The default count is 32768.
.TP
.B --minimal
Bring up a session that is restricted to the 
.I log, dis, rd, sym, eval, set
and 
.I exit
commands.  This option may provide a way to
extract some minimal/quick information from a corrupted or truncated
dumpfile, or in situations where one of the several kernel subsystem
initialization routines would abort the 
.B crash
session.
.TP
.BI --kvmhost \ [32|64]
When examining an x86 KVM guest dumpfile, this option specifies
that the KVM host that created the dumpfile was an x86 (32-bit) 
or an x86_64 (64-bit) machine, overriding the automatically 
determined value.
.TP
.BI --kvmio \ <size>
override the automatically-calculated KVM guest I/O hole size.
.TP
.BI --offline \ [show|hide]
Show or hide command output that is related to offline cpus.  The
default setting is show.
.SH COMMANDS
Each 
.B crash
command generally falls into one of the following categories:
.TP
.I Symbolic display
Displays of kernel text/data, which take full advantage of the power of 
.B gdb
to format and display data structures symbolically.
.TP
.I System state
The majority of 
.B crash
commands consist of a set of "kernel-aware" 
commands, which delve into various kernel subsystems on a system-wide 
or per-task basis. 
.TP
.I Utility functions
A set of useful helper commands serving various purposes, some simple, 
others quite powerful. 
.TP
.I Session control
Commands that control the 
.B crash
session itself.
.PP
The following alphabetical list consists of a very simple overview of each 
.B crash
command.
However, since individual commands often have several options resulting in 
significantly different output, it is suggested that the full description
of each command be viewed by executing
.I crash\ -h\ \fI<command>\fP, 
or during a 
.B crash
session by simply entering
.B \fIhelp command\fP. 
.TP
.I *
"pointer to" is shorthand for either the
.I struct
or
.I union
commands.  It displays the contents of a kernel structure or union.
.TP
.I alias
creates a single-word alias for a command.
.TP
.I ascii
displays an ascii chart or translates a numeric value into its ascii components.
.TP
.I bpf
provides information on currently-loaded eBPF programs and maps.
.TP
.I bt
displays a task's kernel-stack backtrace.  If it is given the
.I \-a
option, it displays the stack traces of the active tasks on all CPUs.
It is often used with the
.I foreach
command to display the backtraces of all tasks with one command.
.TP
.I btop
translates a byte value (physical offset) to its page number.
.TP
.I dev
displays data concerning the character and block device
assignments, I/O port usage, I/O memory usage, and PCI device data. 
.TP
.I dis
disassembles memory, either entire kernel functions, from a
location for a specified number of instructions, or from the start of a
function up to a specified memory location.
.TP
.I eval
evaluates an expression or numeric type and displays the result
in hexadecimal, decimal, octal and binary.
.TP
.I exit
causes
.B crash
to exit.
.TP
.I extend
dynamically loads or unloads 
.B crash
shared object extension modules.
.TP
.I files
displays information about open files in a context.
.TP
.I foreach
repeats a specified command for the specified (or all) tasks
in the system.
.TP
.I fuser
displays the tasks using the specified file or socket.
.TP
.I gdb
passes its argument to the embedded
.B gdb
module.  It is useful for executing
.B gdb 
commands that have the same name as
.B crash
commands.
.TP
.I help
alone displays the command menu; if followed by a command name, a full
description of a command, its options, and examples are displayed.
Its output is far more complete and useful than this man page.
.TP
.I ipcs
displays data about the System V IPC facilities.
.TP
.I irq
displays data concerning interrupt request numbers and
bottom-half interrupt handling. 
.TP
.I kmem
displays information about the use of kernel memory.
.TP
.I list
displays the contents of a linked list.
.TP
.I log
displays the kernel log_buf contents in chronological order.
.TP
.I mach
displays data specific to the machine type.
.TP
.I mod
displays information about the currently installed kernel modules,
or adds or deletes symbolic or debugging information about specified kernel
modules.
.TP
.I mount
displays information about the currently-mounted filesystems.
.TP
.I net
display various network related data.
.TP
.I p
passes its arguments to the
.B gdb
"print" command for evaluation and display.
.TP
.I ps
displays process status for specified, or all, processes
in the system.
.TP
.I pte
translates the hexadecimal contents of a PTE into its physical
page address and page bit settings.
.TP
.I ptob
translates a page frame number to its byte value.
.TP
.I ptov
translates a hexadecimal physical address into a kernel 
virtual address.
.TP
.I q
is an alias for the "exit" command.
.TP
.I rd
displays the contents of memory, with the output formatted
in several different manners.
.TP
.I repeat
repeats a command indefinitely, optionally delaying a given
number of seconds between each command execution.
.TP
.I runq
displays the tasks on the run queue.
.TP
.I sbitmapq
dumps the contents of the sbitmap_queue structure and the used
bits in the bitmap. Also, it shows the dump of a structure array
associated with the sbitmap_queue.
.TP
.I search
searches a range of user or kernel memory space for given value.
.TP
.I set
either sets a new context, or gets the current context for
display.
.TP
.I sig
displays signal-handling data of one or more tasks.
.TP
.I struct
displays either a structure definition or the contents of a
kernel structure at a specified address.
.TP
.I swap
displays information about each configured swap device.
.TP
.I sym
translates a symbol to its virtual address, or a static 
kernel virtual address to its symbol -- or to a symbol-plus-offset value,
if appropriate.
.TP
.I sys
displays system-specific data.
.TP
.I task
displays the contents of a task_struct.
.TP
.I tree
displays the contents of a red-black tree or a radix tree.
.TP
.I timer
displays the timer queue entries, both old- and new-style,
in chronological order.
.TP
.I union
is similar to the
.I struct
command, except that it works on kernel unions.
.TP
.I vm
displays basic virtual memory information of a context.
.TP
.I vtop
translates a user or kernel virtual address to its physical
address.
.TP
.I waitq
walks the wait queue list displaying the tasks which 
are blocked on the specified wait queue.
.TP
.I whatis
displays the definition of structures, unions, typedefs or
text/data symbols.
.TP
.I wr
modifies the contents of memory on a live system. 
It can only be used if
.I /dev/mem
is the device file being used to access system RAM, and should obviously be used with great care.
.PP
When 
.B crash
is invoked with a Xen hypervisor binary as the NAMELIST, the
command set is slightly modified.  The
.I *, alias, ascii, bt, dis, eval, exit, extend,
.I gdb, help, list, log, p, pte, rd, repeat,
.I search, set, struct, sym, sys, union,
.I whatis, wr
and 
.I q
commands are the same as above.  The following commands
are specific to the Xen hypervisor:
.TP
.I domain
displays the contents of the domain structure for selected, or all, domains.
.TP
.I doms
displays domain status for selected, or all, domains.
.TP
.I dumpinfo
displays Xen dump information for selected, or all, cpus.
.TP
.I pcpus
displays physical cpu information for selected, or all, cpus.
.TP
.I vcpus
displays vcpu status for selected, or all, vcpus.
.SH FILES
.TP
.I .crashrc
Initialization commands.  The file can be located in the user's
.B HOME 
directory and/or the current directory.  Commands found in the
.I .crashrc
file in the 
.B HOME
directory are executed before those in the current directory's 
.I .crashrc
file.
.SH ENVIRONMENT
.TP
.B EDITOR
Command input is read using
.BR readline(3).
If
.B EDITOR
is set to
.I emacs
or
.I vi
then suitable keybindings are used.  If 
.B EDITOR
is not set, then
.I vi
is used.  This can be overridden by
.B set vi
or 
.B set emacs
commands located in a
.IR .crashrc 
file, or by entering
.B -e emacs
on the
.B crash
command line.
.TP
.B CRASHPAGER
If
.B CRASHPAGER
is set, its value is used as the name of the program to which command output will be sent. 
If not, then command output is sent to 
.B /usr/bin/less -E -X 
by default.
.TP
.B CRASH_MODULE_PATH
Specifies an alternative directory tree to search for kernel module
object files.
.TP
.B CRASH_EXTENSIONS
Specifies a directory containing extension modules that will be loaded
automatically if the 
.B -x
command line option is used.
.SH NOTES
.PP
If
.B crash
does not work, look for a newer version: kernel evolution frequently makes
.B crash
updates necessary.
.PP
The command
.B set scroll off
will cause output to be sent directly to
the terminal rather than through a paging program.  This is useful,
for example, if you are running
.B crash
in a window of
.BR emacs .
.SH AUTHOR
Dave Anderson <anderson@redhat.com> wrote
.B crash.
.TP
Jay Fenlason <fenlason@redhat.com> and Dave Anderson <anderson@redhat.com> wrote this man page.
.SH "SEE ALSO"
.PP
The
.I help
command within
.B crash
provides more complete and accurate documentation than this man page.
.PP
.I https://github.com/crash-utility
- the home page of the
.B crash
utility.
.PP
.BR netdump (8),
.BR gdb (1),
.BR makedumpfile(8)