Kdump and Vmcore: Investigating Linux Kernel Panics in Production

When the Linux kernel panics, ordinary logging may stop precisely when the most valuable evidence appears. Kdump closes that gap by reserving memory, using kexec to boot a minimal capture kernel after failure, preserving the crashed kernel's memory image, and writing a vmcore for postmortem analysis.

This guide follows official Linux kernel Kdump documentation, the Red Hat Enterprise Linux kdump guide, and the makedumpfile manual. Package, service, initramfs, and bootloader tooling differ by distribution; follow documentation for the exact release you operate.

1. How Kdump works

1. The production kernel boots with crashkernel= and reserves a memory region.
2. A minimal capture kernel and initramfs are loaded into that region through kexec.
3. On panic, the system transfers control to the capture kernel instead of following a normal full boot.
4. The crashed kernel's memory image is exposed through /proc/vmcore.
5. Capture tooling writes, filters, or compresses vmcore to local or remote storage.
6. Operators analyze it with the exact matching debug kernel.

The capture kernel must remain small and self-sufficient because it boots after a severe failure. Kdump cannot capture every event: power loss, immediate firmware reset, major hardware damage, or missing capture-kernel memory and drivers can still prevent a dump.

2. Why the journal is not enough

A strong investigation combines all four. Vmcore is powerful for kernel failures but does not replace application logs, distributed traces, or hardware evidence.

3. Check support and current state

uname -r
cat /proc/cmdline
grep -i 'Crash kernel' /proc/iomem
systemctl status kdump --no-pager

# On systems providing kdumpctl
kdumpctl status

# Debian and Ubuntu may expose another tool
kdump-config show

Not every command exists on every distribution. Confirm kernel kexec and crash-dump support, installed tooling, actual reserved memory, and a loaded capture kernel. An active service alone does not prove that storage, DNS, networking, or authentication will work after panic.

4. crashkernel sizing is not universal

crashkernel= removes memory from the primary workload, while an undersized reserve can make the capture kernel run out of memory or fail to write the dump. Requirements depend on CPU architecture, RAM, CPU count, drivers, devices, encryption, dump target, and initramfs.

• Use current distribution defaults or estimation tooling where available.
• Do not copy a sizing table from an older release; syntax and guidance change.
• Systems with many NICs, HBAs, multipath, encrypted storage, or large RAM require realistic tests.
• After changing boot parameters, reboot deliberately and confirm the reservation in /proc/iomem.

A successful load is not enough: the capture kernel may boot yet fail when storage or network drivers are used, or when makedumpfile needs a larger memory bitmap.

5. The capture initramfs needs every dependency

A local target requires the relevant filesystem, block, LVM, RAID, multipath, or encryption support. A remote target needs NIC drivers, network configuration, routes, DNS if hostnames are used, and protocol credentials. The image should be minimal without omitting a link.

• Prefer stable identifiers over probe-order device names.
• For encrypted targets, prove that the supported capture path can unlock them.
• Avoid dependence on a control plane that may be unavailable with the failed node.
• Rebuild kdump initramfs after kernel, driver, target, or credential changes according to distribution guidance.
• Keep the path simple enough to audit and exercise.

6. Select a different failure domain

Do not write vmcore to the filesystem involved in the panic or one unavailable to the capture kernel. Plan for worst case rather than the latest dump size. Failure behavior—reboot, halt, or shell—must match availability and security requirements; an emergency shell must not accidentally expose a privileged console.

7. makedumpfile: smaller while preserving evidence

A raw vmcore may approach RAM size. makedumpfile filters page classes and compresses output. Its dump level is a bitmask that can exclude zero, cache, user-process, and free pages.

# Kernel-documentation example; verify installed compression support
makedumpfile -l --message-level 1 -d 31 \
  /proc/vmcore /var/crash/vmcore.filtered

-d 31 often reduces size substantially, but can remove user-space memory needed for some investigations. Do not choose a level solely for the smallest file: define forensic questions, privacy constraints, and vendor-support requirements. Check exit status; the presence of a file does not prove completeness.

8. A vmcore is highly sensitive

RAM may contain tokens, keys, sessions, request contents, customer data, and plaintext from encrypted volumes. Classify vmcore at least as highly as the most sensitive data that could be resident in memory.

• Encrypt transport and target storage and isolate capture credentials.
• Restrict reading, downloading, and analysis, with audit logging.
• Never attach an unreviewed vmcore to a public ticket.
• Define retention, legal hold, and secure deletion.
• Hash captured files to verify integrity during transport.
• Upload to vendors only through approved channels and data-processing policy.

9. Retain the exact analysis artifacts

crash requires a debug-symbol vmlinux that exactly matches the kernel producing vmcore. Similar release names may still hide different build IDs, configurations, compilers, or modules.

Archive per release and build ID:

• unstripped vmlinux or the exact debuginfo or dbgsym package;
• kernel configuration and System.map when required by the workflow;
• matching module binaries and symbols;
• initramfs, capture configuration, and package manifest;
• provenance and checksums, retained at least as long as vmcore.

Do not wait until an old kernel disappears from repositories before looking for symbols. Artifact retention is part of the Kdump design.

10. Initial analysis with crash

crash /path/to/vmlinux /path/to/vmcore

# Inside the crash shell
sys
log
bt
ps
kmem -i
mod
files

sys identifies kernel and state; log shows the kernel ring buffer; bt displays stacks; ps lists tasks; and kmem -i summarizes memory. Start from panic text, crashing CPU and task, and the backtrace, then inspect taint, recent logs, lock or wait state, and hardware errors.

A module name on the stack does not by itself prove causality. Corruption may have occurred earlier; correlate source, symbols, similar incidents, and hardware telemetry.

11. End-to-end testing is disruptive

Critical warning: writing c to /proc/sysrq-trigger intentionally panics the kernel, immediately stops workloads, and can lose unflushed data. Test only in a lab or an approved maintenance window after draining traffic, verifying backups, and confirming console or out-of-band access.

# CRASHES THE MACHINE — never run on serving production
echo c | sudo tee /proc/sysrq-trigger

A valid drill proves the whole chain: panic → capture-kernel boot → available target → complete vmcore → policy-compliant reboot or halt → protected and hashed file → successful opening with crash and matching symbols. kdumpctl status alone cannot prove this path.

12. Monitoring and operational lifecycle

Revalidate after kernel updates, driver or NIC/HBA replacement, storage or network changes, encryption changes, crashkernel resizing, or base-image updates. Fleet reporting should measure nodes that are armed and recently tested, not merely package installation.

Vmcore incident runbook

1. Scope the incident and record time, node, kernel build, and impact.
2. Verify vmcore completeness, checksum, and access controls.
3. Collect journal, pstore, BMC and hardware events, and deployment timeline.
4. Retrieve exact vmlinux, debuginfo, and modules by build ID.
5. Analyze panic text, CPU and task, backtrace, taint, and memory or I/O state.
6. Compare kernel bugs, driver release notes, and recent changes.
7. Produce an evidence-backed hypothesis, rollback-ready mitigation, and safe reproduction.
8. Retain required artifacts and securely remove temporary copies.

Production checklist

• crashkernel sizing reflects the distribution, hardware, and real dependencies.
• Reservation and capture-kernel state are checked after every boot and kernel update.
• Initramfs contains required drivers, storage, network, and encryption support.
• The target crosses failure domains, has capacity, and defines failure behavior.
• Dump level balances forensic value, size, and privacy.
• Vmcore is encrypted, access-controlled, audited, hashed, and retained as sensitive data.
• Build-matched vmlinux, debuginfo, and modules are archived before incidents.
• Alerts cover reservation, loaded state, target health, and artifact coverage.
• End-to-end drills run only in approved disruptive windows.
• A real captured vmcore has been successfully opened and inspected with crash.

Conclusion

Kdump turns a kernel panic from an opaque reboot into analyzable evidence. Its value is not an active service but an entire proven system: sufficient reserved memory, a self-sufficient capture kernel, a target surviving the failure domain, a filtered and protected vmcore, retained matching symbols, and a drill validating the complete path. Kernel incidents can then be investigated from evidence rather than guesses based on the final few log lines.