Using the Magic SysRq keys

One of the less known functionalities implemented in the Linux kernel is the Magic SysRq key. Initially implemented as a debugging feature for kernel development it has also made its way into system administration use and the /proc filesystem.

Basically the Magic SysRq keys are low level commands that can be sent to the kernel regardless of a system’s state. That being said it can be most useful in extreme cases where the system is not responding to do a reboot, unmount partitions or sync the partitions (flush their write buffer to disk).

Before being able to actually use this, you have to enable the SysRq keys:

echo "1" > /proc/sys/kernel/sysrq

All these commands can be sent to the kernel using the combination:

Alt+SysRQ+[key]

or by piping the key to the /proc filesystem:

echo [key] > /proc/sysrq-trigger

The SysRQ key on the keyboard can usually be found on the same key as “PrintScreen”. A full list of the commands that can be sent can be found on Wikipedia.

Now you ask yourself, when would I use this? Let’s take the most common example: a server that has become completely unresponsive and the only solution would be to reboot it. But that server hosts a database with precious information and you do not know if all the information kept in memory by the database has been written to disk or if the sudden reboot would trigger errors and inconsistencies on the filesystem. The Magic SysRQ keys would come in handy now:

Gracefully terminate all running processes except init (PID1):

Alt+SysRq+e

or

echo "e" > /proc/sysrq-trigger

Kill all the stubborn processes:

Alt+SysRq+k

or

echo "k" > /proc/sysrq-trigger

Sync all mounted filesystems - meaning flush all the memory buffers to disk:

Alt+SysRq+s

or

echo "s" > /proc/sysrq-trigger

Re-mount all partitions as read-only:

Alt+SysRq+u

or

echo "u" > /proc/sysrq-trigger

Finally tell the box to immediately reboot:

Alt+SysRq+b

or

echo "b" > /proc/sysrq-trigger

The case above is the most common one, but there are other times on the SysRq keys might come in handy. Let’s say you have accidentally deleted something from an ext2/3/4 partition. We know that on those filesystems the data isn’t immediately removed from the drive, but marked as deleted and will be overwritten the next time a process tries to write somewhere on the disk. Before you can use tools to un-delete - we will cover that in the next episode - we can tell the kernel to force re-mount all partitions as read-only, including the / partition:

Alt+SysRq+u

or

echo "u" > /proc/sysrq-trigger

Now we can be sure that no process will overwrite our precious data and use some tool to recover the metadata.