Veeam Agent for Linux crash with null pointer dereference on 5.9+ kernels
Challenge
Symptom
Shortly after Veeam Agent for Linux creates a veeamsnap snapshot, the system hangs or crashes with a null pointer dereference:
Example of dmesg:
BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 8000000003a47067 P4D 8000000003a47067 PUD 3a48067 PMD 0 Oops: 0000 [#1] SMP PTI CPU: 0 PID: 396645 Comm: VeeamService se Kdump: loaded Tainted: G OE 5.10.0-17-amd64 #1 Debian 5.10.136-1
Associated Environment Specifications
This issue occurs when all 3 of the following are true:
- Linux kernel 5.9 or higher is in use.
- Veeam Agent for Linux version 5.0.2.4567 is installed and uses the veeamsnap kernel module.
- /proc/kallsyms shows all symbol addresses as zeroes:
$ sudo tail /proc/kallsyms 0000000000000000 t fuse_open_common [fuse] 0000000000000000 t fuse_flush_writepages [fuse] 0000000000000000 t fuse_fill_super_common [fuse] 0000000000000000 t fuse_simple_background [fuse] 0000000000000000 t fuse_getxattr [fuse] 0000000000000000 r fuse_dentry_operations [fuse] 0000000000000000 t fuse_free_conn [fuse] 0000000000000000 t fuse_write_update_size [fuse] 0000000000000000 t fuse_sync_release [fuse] 0000000000000000 t fuse_direct_io [fuse]
Cause
On kernels newer than 5.9, Veeam Agent for Linux 5.0.2.4567 has to retrieve symbols usually exposed through /proc/kallsyms. However, several configuration options may restrict this file, causing zero values to be returned instead of valid addresses. A null pointer dereference occurs when Veeam Agent for Linux attempts to use those zero addresses.
Solution
To resolve this, ensure that /proc/kallsyms returns non-zero addresses.
One of the most common ways those values become restricted is through the use of the following sysctl parameters:
- kernel.perf_event_paranoid
- kernel.kptr_restrict
To avoid this issue, both parameters must be set to either:
- 0 — effectively exposing symbol information to all users
or - 1 — exposing addresses to users with CAP_SYSLOG capability (i.e., root, which usually executes Veeam Agent for Linux service process)
If the values are higher than 1, they must be set to either 0 or 1.
After setting those values, recheck the /proc/kallsyms contents and verify that non-zero addresses are returned:
$ sudo tail /proc/kallsyms ffffffffc01391a0 t fuse_open_common [fuse] ffffffffc013bbd0 t fuse_flush_writepages [fuse] ffffffffc013e930 t fuse_fill_super_common [fuse] ffffffffc012f9f0 t fuse_simple_background [fuse] ffffffffc013fab0 t fuse_getxattr [fuse] ffffffffc01428c0 r fuse_dentry_operations [fuse] ffffffffc013cfb0 t fuse_free_conn [fuse] ffffffffc013a360 t fuse_write_update_size [fuse] ffffffffc0137280 t fuse_sync_release [fuse] ffffffffc01397f0 t fuse_direct_io [fuse]
To make these changes persist after a reboot, you must adjust these values in the sysctl configuration files. For more information, refer to the sysctl manpage.
To submit feedback regarding this article, please click this link: Send Article Feedback
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.
To report a typo on this page, highlight the typo with your mouse and press CTRL + Enter.