ZFS ashift values for Samsung 990 PRO NVMe SSDs

TL;DR – ashift=12 performed noticeably better than ashift=13.

I recently installed Proxmox on a new build and couldn’t find any information about the best ashift values for my new NVMe SSD drives. Given that I was installing a brand new server, it gave me a chance to do some quick testing. Here’s what I found.

Hardware setup

  • Asus Pro W680-ACE IPME (without the IPME card installed)
  • 64GB of DDR5-4800 ECC memory
  • 2x 2TB Samsung 990 PRO NVMe SSDs

The SSDs are installed on the motherboard’s M.2 slots, one in the slot directly attached to the CPU and the other in a slot connected through the W680 controller.

Software setup

For both tests I did a clean install of Proxmox VE 8.1.3 with both SSDs in a RAID1 configuration. All zpool/vdev settings were default (compression=lzw, checksum=on, copies=1, recordsize=128K, etc.) except for the ashift values changed for the tests. After the installation, I did the standard apt-get updates to get current, and then installed fio to do the testing.

Test setup

The tests I ran were from Jim Salter’s ARStechnica article which gives good detail on how to run fio to test disk performance. Each test was run back-to-back. I ran each test 4 times. Here’s the script I used to run the tests:

echo "Test 1 - Single 4KiB random write process"

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

echo "Test 2 - 16 parallel 64KiB random write processes"

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1

echo "Test 3 - Single 1MiB random write process"

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

Results

ashift=12 was faster (on average) than ashift=13 for every test. For Test 1, it was 10.3% (15MB/s) faster. For Test 2, it was a whopping 22.3% (499MB/s) faster. And for Test 3, it was 8.5% (117MB/s) faster. Those are pretty big differences – I was surprised they weren’t more in the noise. Made it easy to make the call to set my drives to ashift=12, which also happens to be the common wisdom for all drives today.

TestashiftIteration 1Iteration 2Iteration 3Iteration 4Average
11217692.6150163145.4
21228441126386511042234.75
31211391320155014731370.5
11317772.7118154130.425
2133199147399312801736.25
31310301221147612871253.5
All results in MB/s