Challenge
The test file created by these tests do not contain any diagnostic information and will need to be manually removed after testing has concluded. All diagnostic information regarding the performance results are displayed in the command line. Please do not send the testfile.dat into support, its contents will not help with troubleshooting.This document contains information on how to use Microsoft© DiskSpd to simulate Veeam Backup & Replication disk actions to measure disk performance.
DiskSpd and additional info can be found here: https://gallery.technet.microsoft.com/DiskSpd-a-robust-storage-6cd2f223
Solution
Below are some details on the options and some simulations you can do to measure disk speed independently of Veeam. Please keep in mind as with all synthetic benchmarks real-world results may differ.──────────────────────────────────────────────────────────
Common parameters
──────────────────────────────────────────────────────────Usage: diskspd [options] target1 [ target2 [ target3 ...] ]
Target
Available targets:
· File on a volume with an assigned letter: D:\testfile.dat
· File on a CIFS/SMB share: \\nas\share\testfile.dat
· File on an NFS share, provided you have mounted it to a disk letter with Client for NFS: N:\testfile.dat
· Disk: #X where X is the number of the disk in Disk Management. You can use a local disk or one attached by iSCSI, and it does not matter if they are Online are Offline. In this mode diskspd reads or writes directly from/to the disk ("RAW").
You can specify multiple targets. This way you can simulate several jobs running at the same time.
Block size
-b specifies the size of a read or write operation.
For Veeam, this size depends on the job settings. By default, "Local" storage optimization setting is selected and this corresponds to 1MB block size in backups. However, every block of data is compressed (unless using the Decompress option) before it is written to the backup file, so the size is reduced. It is safe to assume that blocks compress on average down to half the size, so in most cases picking a 512KB block size is a good estimate.
If the job is using a different setting, WAN (256KB), LAN (512KB) or Local+ (8MB), change the -b value accordingly to 128KB, 256KB or 4MB. And if the Decompress option is on don't halve the values.
File size
-c specifies the file size you need to create for testing. Typically 1 GB should be enough. Anything lower can be easily cached by hardware and thus yield incorrect results.
Duration
-d specifies the duration of the test. By default it does 5 seconds of warm up (statistics are not collected), then 10 seconds of the test. This is OK for a short test, but for more conclusive results run the test for at least 10 minutes (-d600).
Caching
-Sh disables Windows and hardware caching.
This flag should always be set. VeeamAgents always explicitly disable caching for I/O operations for greater reliability, even though this results in lower speed. Windows Explorer for example, does use the Cache Manager and in a very simple copy-paste test will get greater speeds than Veeam does, due to cached reads and lazy writes. That is why using Explorer is never a valid test.
──────────────────────────────────────────────────────────
Active full or forward incremental
C:\diskspd> diskspd.exe -c1G -b512K -w100 -Sh -d600 D:\testfile.dat
-w100 indicates 100% writes and 0% reads. Sequential I/O is used by default.
IMPORTANT: Contents of testfile.dat will be destroyed without a warning.
──────────────────────────────────────────────────────────
Reverse incremental
Reverse incremental
C:\diskspd> diskspd.exe -c1G -b512K -w67 -r4K -Sh -d600 D:\testfile.dat
-w67 indicates 67% writes and 33% reads to simulate 2 write and 1 read operations that happen in reverse incremental backup jobs.
-r4K enables random I/O that are 4KB aligned, for a more realistic simulation.
IMPORTANT: Contents of testfile.dat will be destroyed without a warning.
After the test has finished, take Total IO MB/s from the results and divide it by 3. This is because for every processed block Veeam needs to do 3 I/O operations, thus the effective speed is 3 times slower.
──────────────────────────────────────────────────────────
Transforms, merges, and other synthetic operations
Includes transformation of incrementals to rollbacks, merge operations in forever forward incremental backups and backup copy jobs, and creation of synthetic full backup files and GFS points.
C:\diskspd> diskspd.exe -c1G -b512K -w50 -r4K -Sh -d600 D:\testfile.dat
-w50 indicates 50% writes and 50% reads to simulate reading data from one file and writing that data into another (or in the case of transform, reading the same number of blocks from two files as are written to two other files).
-r4K enables random I/O that are 4KB aligned, for a more realistic simulation.
IMPORTANT: Contents of testfile.dat will be destroyed without a warning.
After the test has finished, take Total IO MB/s from the results and divide it by 2 (4 for transform to rollbacks). This is because for every processed block Veeam needs to do 2 I/O operations, thus the effective speed is 2 times slower. For transform to rollbacks, each block must be read out of the full backup file and written into the rollback before the corresponding block can be read out of the incremental and written into the full, which results in 4x I/O.
To estimate an expected time to complete the synthetic operation, in seconds:
For synthetic full backup and GFS points: divide the expected size of the new full backup file (typically the same as previous full backup files) by the effective speed.
For all other synthetic operations, add the sizes of all of the incremental files which will be merged or transformed, and then divide the resulting sum by the effective speed. Typically only the oldest incremental file is merged, whereas all incremental files are transformed to rollbacks.
──────────────────────────────────────────────────────────
Slow restore or Surebackup
Worst case scenario where the backup file is heavily fragmented inside, which implies a lot of random read I/O:
C:\diskspd> diskspd.exe -b512K -r4K -Sh -d600 \\nas\share\VeeamBackups\Job\Job2014-01-23T012345.vbk
-r4K enables random I/O that are 4KB aligned, for a more realistic simulation.
Best case scenario where the backup file is not fragmented inside (no parallel processing), which implies linear read I/O:
C:\diskspd> diskspd.exe -b512K -Sh -d600 \\nas\share\VeeamBackups\Job\Job2014-01-23T012345.vbk
In both cases you need to pick an existing .vbk file as the target. Only read operations will be performed.
──────────────────────────────────────────────────────────
Direct disk access speed
C:\diskspd> diskspd.exe -Sh -d600 #X
Where X is the number of disk that you see in Disk Management.This will not overwrite any data, it is a safe test, and it works for Offline disks too. You can simulate and measure maximum possible reading speed in SAN or hot-add modes, however this of course will not take any VDDK overhead into account.
More Information
FAQQ: Can diskspd be used to stress-test NAS boxes for reliability ("specified network name is no longer available" errors in Veeam)
A: Unfortunately, no. If the SMB share disappears, diskspd will just ignore that issue. It is better to use Wireshark.
Q: Active full starts fast but then gradually becomes slower and slower. Target agent runs on Windows 2008 (not R2).
A: This is a known (yet undocumented) performance degradation issue on Windows 2008. Upgrade to or deploy 2008 R2 (or newer). In case of a CIFS repository you can force some other 2008 R2 server to be used as gateway/proxying. The issue itself can be demonstrated with diskspd if you set it to run for several hours, then observe how the write rate slowly goes down in Resource Monitor.
Q: I am getting extremely high I/O speed like 4 GB/s in any test I try, even though I have set the -Sh flag, what's going on?
A: Most likely you're running diskspd on a Hyper-V VM, testing performance of a virtualised (.vhdx) disk, so the data is cached by the Hyper-V host. Run the test on the datastore where that .vhdx is located instead.