Kernel Crash Dump
Introduction
A Kernel Crash Dump refers to a portion of the contents of volatile memory (RAM) that is copied to disk whenever the execution of the kernel is disrupted. The following events can cause a kernel disruption :
Kernel Panic
Interruptions non masquables (NMI)
Machine Check Exceptions (MCE)
Problème matériel
Intervention manuelle
Kernel Crash Dump Mechanism
When a kernel panic occurs, the kernel relies on the kexec mechanism to quickly reboot a new instance of the kernel in a pre-reserved section of memory that had been allocated when the system booted (see below). This permits the existing memory area to remain untouched in order to safely copy its contents to storage.
Installation
The kernel crash dump utility is installed with the following command:
sudo apt install linux-crashdump
Starting with 16.04, the kernel crash dump mechanism is enabled by default. During the installation, you will be prompted with the following dialog. Unless chosen otherwise, the kdump mechanism will be enabled.
|------------------------| Configuring kdump-tools |------------------------|
| |
| |
| If you choose this option, the kdump-tools mechanism will be enabled. A |
| reboot is still required in order to enable the crashkernel kernel |
| parameter. |
| |
| Should kdump-tools be enabled by default? |
| |
| <Yes> <No> |
| |
|---------------------------------------------------------------------------|
If you ever need to manually enable the functionality, you can use the dpkg-reconfigure kdump-tools command and answer Yes to the question. You can also edit /etc/default/kdump-tools by including the following line:
USE_KDUMP=1
If a reboot has not been done since installation of the linux-crashdump package, a reboot will be required in order to activate the crashkernel= boot parameter. Upon reboot, kdump-tools will be enabled and active.
If you enable kdump-tools after a reboot, you will only need to issue the kdump-config load command to activate the kdump mechanism.
Configuration
In addition to local dump, it is now possible to use the remote dump functionality to send the kernel crash dump to a remote server, using either the SSH or NFS protocols.
Local Kernel Crash Dumps
Local dumps are configured automatically and will remain in use unless a remote protocol is chosen. Many configuration options exist and are thoroughly documented in the /etc/default/kdump-tools file.
Remote Kernel Crash Dumps using the SSH protocol
To enable remote dumps using the SSH protocol, the /etc/default/kdump-tools must be modified in the following manner :
# ---------------------------------------------------------------------------
# Remote dump facilities:
# SSH - username and hostname of the remote server that will receive the dump
# and dmesg files.
# SSH_KEY - Full path of the ssh private key to be used to login to the remote
# server. use kdump-config propagate to send the public key to the
# remote server
# HOSTTAG - Select if hostname of IP address will be used as a prefix to the
# timestamped directory when sending files to the remote server.
# 'ip' is the default.
SSH="ubuntu@kdump-netcrash"
The only mandatory variable to define is SSH. It must contain the username and hostname of the remote server using the format {username}@{remote server}.
SSH_KEY may be used to provide an existing private key to be used. Otherwise, the kdump-config propagate command will create a new keypair. The HOSTTAG variable may be used to use the hostname of the system as a prefix to the remote directory to be created instead of the IP address.
The following example shows how kdump-config propagate is used to create and propagate a new keypair to the remote server :
sudo kdump-config propagate
Need to generate a new ssh key...
The authenticity of host 'kdump-netcrash (192.168.1.74)' can't be established.
ECDSA key fingerprint is SHA256:iMp+5Y28qhbd+tevFCWrEXykDd4dI3yN4OVlu3CBBQ4.
Are you sure you want to continue connecting (yes/no)? yes
ubuntu@kdump-netcrash's password:
propagated ssh key /root/.ssh/kdump_id_rsa to server ubuntu@kdump-netcrash
The kdump-config show command can be used to confirm that kdump is correctly configured to use the SSH protocol :
kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: 0x2c000000 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.4.0-10-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.4.0-10-generic SSH: ubuntu@kdump-netcrash SSH_KEY: /root/.ssh/kdump_id_rsa HOSTTAG: ip current state: ready to kdump
Remote Kernel Crash Dumps using the NFS protocol
To enable remote dumps using the NFS protocol, the /etc/default/kdump-tools must be modified in the following manner :
# NFS - Hostname and mount point of the NFS server configured to receive
# the crash dump. The syntax must be {HOSTNAME}:{MOUNTPOINT}
# (e.g. remote:/var/crash)
#
NFS="kdump-netcrash:/var/crash"
As with the SSH protocol, the HOSTTAG variable can be used to replace the IP address by the hostname as the prefix of the remote directory.
The kdump-config show command can be used to confirm that kdump is correctly configured to use the NFS protocol :
kdump-config show DUMP_MODE: kdump USE_KDUMP: 1 KDUMP_SYSCTL: kernel.panic_on_oops=1 KDUMP_COREDIR: /var/crash crashkernel addr: 0x2c000000 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.4.0-10-generic kdump initrd: /var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.4.0-10-generic NFS: kdump-netcrash:/var/crash HOSTTAG: hostname current state: ready to kdump
Vérification
To confirm that the kernel dump mechanism is enabled, there are a few things to verify. First, confirm that the crashkernel boot parameter is present (note: The following line has been split into two to fit the format of this document:
cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.2.0-17-server root=/dev/mapper/PreciseS-root ro crashkernel=384M-2G:64M,2G-:128M
Le paramètre crashkernel a la syntaxe suivante :
crashkernel=<range1>:<size1>[,<range2>:<size2>,...][@offset]
range=start-[end] 'start' is inclusive and 'end' is exclusive.
Donc, pour le paramètre crashkernel trouvé dans /proc/cmdline, nous devrions avoir :
crashkernel=384M-2G:64M,2G-:128M
La valeur ci-dessus signifie :
if the RAM is smaller than 384M, then don't reserve anything (this is the "rescue" case)
si la quantité de RAM se situe entre 386 Mo et 2 Go (exclusif), alors réservez 64 Mo
si la quantité de RAM est supérieure à 2 Go, alors réservez 128 Mo
Deuxièmement, vérifiez que le noyau a réservé la zone de mémoire requise pour le noyau kdump en faisant :
dmesg | grep -i crash ... [ 0.000000] Reserving 64MB of memory at 800MB for crashkernel (System RAM: 1023MB)
Finally, as seen previously, the kdump-config show command displays the current status of the kdump-tools configuration :
kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr: 0x2c000000
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-4.4.0-10-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to /var/lib/kdump/initrd.img-4.4.0-10-generic
current state: ready to kdump
kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/vmlinuz-4.4.0-10-generic root=/dev/mapper/VividS--vg-root ro debug break=init console=ttyS0,115200 irqpoll maxcpus=1 nousb systemd.unit=kdump-tools.service" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
Testing the Crash Dump Mechanism
Testing the Crash Dump Mechanism will cause a system reboot. In certain situations, this can cause data loss if the system is under heavy load. If you want to test the mechanism, make sure that the system is idle or under very light load.
Vérifiez que le mécanisme SysRq est activé en regardant la valeur du paramètre kernel /proc/sys/kernel/sysrq :
cat /proc/sys/kernel/sysrq
Si une valeur de 0 est renvoyée, la fonction est désactivée. Activez-la avec la commande suivante :
sudo sysctl -w kernel.sysrq=1
Once this is done, you must become root, as just using sudo will not be sufficient. As the root user, you will have to issue the command echo c > /proc/sysrq-trigger. If you are using a network connection, you will lose contact with the system. This is why it is better to do the test while being connected to the system console. This has the advantage of making the kernel dump process visible.
Une sortie de test typique devrait ressembler à ce qui suit :
sudo -s [sudo] password for ubuntu: # echo c > /proc/sysrq-trigger [ 31.659002] SysRq : Trigger a crash [ 31.659749] BUG: unable to handle kernel NULL pointer dereference at (null) [ 31.662668] IP: [<ffffffff8139f166>] sysrq_handle_crash+0x16/0x20 [ 31.662668] PGD 3bfb9067 PUD 368a7067 PMD 0 [ 31.662668] Oops: 0002 [#1] SMP [ 31.662668] CPU 1 ....
The rest of the output is truncated, but you should see the system rebooting and somewhere in the log, you will see the following line :
Begin: Saving vmcore from kernel crash ...
ls /var/crash
linux-image-3.0.0-12-server.0.crash
Ressources
Kernel Crash Dump is a vast topic that requires good knowledge of the linux kernel. You can find more information on the topic here :
-
Analyzing Linux Kernel Crash (Based on Fedora, it still gives a good walkthrough of kernel dump analysis)