ZFS ashift values for Samsung 990 PRO NVMe SSDs

TL;DR – ashift=12 performed noticeably better than ashift=13.

I recently installed Proxmox on a new build and couldn’t find any information about the best ashift values for my new NVMe SSD drives. Given that I was installing a brand new server, it gave me a chance to do some quick testing. Here’s what I found.

Hardware setup

• Asus Pro W680-ACE IPME (without the IPME card installed)
• 64GB of DDR5-4800 ECC memory
• 2x 2TB Samsung 990 PRO NVMe SSDs

The SSDs are installed on the motherboard’s M.2 slots, one in the slot directly attached to the CPU and the other in a slot connected through the W680 controller.

Software setup

For both tests I did a clean install of Proxmox VE 8.1.3 with both SSDs in a RAID1 configuration. All zpool/vdev settings were default (compression=lzw, checksum=on, copies=1, recordsize=128K, etc.) except for the ashift values changed for the tests. After the installation, I did the standard apt-get updates to get current, and then installed fio to do the testing.

Test setup

The tests I ran were from Jim Salter’s ARStechnica article which gives good detail on how to run fio to test disk performance. Each test was run back-to-back. I ran each test 4 times. Here’s the script I used to run the tests:

echo "Test 1 - Single 4KiB random write process"

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

echo "Test 2 - 16 parallel 64KiB random write processes"

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1

echo "Test 3 - Single 1MiB random write process"

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1

Results

ashift=12 was faster (on average) than ashift=13 for every test. For Test 1, it was 10.3% (15MB/s) faster. For Test 2, it was a whopping 22.3% (499MB/s) faster. And for Test 3, it was 8.5% (117MB/s) faster. Those are pretty big differences – I was surprised they weren’t more in the noise. Made it easy to make the call to set my drives to ashift=12, which also happens to be the common wisdom for all drives today.

Test	ashift	Iteration 1	Iteration 2	Iteration 3	Iteration 4	Average
1	12	176	92.6	150	163	145.4
2	12	2844	1126	3865	1104	2234.75
3	12	1139	1320	1550	1473	1370.5
1	13	177	72.7	118	154	130.425
2	13	3199	1473	993	1280	1736.25
3	13	1030	1221	1476	1287	1253.5