{"id":3365,"date":"2023-12-10T17:52:44","date_gmt":"2023-12-11T00:52:44","guid":{"rendered":"https:\/\/kmwoley.com\/blog\/?p=3365"},"modified":"2023-12-10T17:52:46","modified_gmt":"2023-12-11T00:52:46","slug":"zfs-ashift-values-for-samsung-990-pro-nvme-ssds","status":"publish","type":"post","link":"https:\/\/kmwoley.com\/blog\/zfs-ashift-values-for-samsung-990-pro-nvme-ssds\/","title":{"rendered":"ZFS ashift values for Samsung 990 PRO NVMe SSDs"},"content":{"rendered":"\n<p>TL;DR &#8211; <code>ashift=12<\/code> performed noticeably better than <code>ashift=13<\/code>.<\/p>\n\n\n\n<p>I recently installed Proxmox on a new build and couldn&#8217;t find any information about the best <code>ashift<\/code> values for my new NVMe SSD drives. Given that I was installing a brand new server, it gave me a chance to do some quick testing. Here&#8217;s what I found.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Hardware setup<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Asus Pro W680-ACE IPME (without the IPME card installed)<\/li>\n\n\n\n<li>64GB of DDR5-4800 ECC memory<\/li>\n\n\n\n<li>2x 2TB Samsung 990 PRO NVMe SSDs<\/li>\n<\/ul>\n\n\n\n<p>The SSDs are installed on the motherboard&#8217;s M.2 slots, one in the slot directly attached to the CPU and the other in a slot connected through the W680 controller.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Software setup<\/h2>\n\n\n\n<p>For both tests I did a clean install of Proxmox VE 8.1.3 with both SSDs in a RAID1 configuration. All zpool\/vdev settings were default (<code>compression=lzw<\/code>, <code>checksum=on<\/code>, <code>copies=1<\/code>, <code>recordsize=128K<\/code>, etc.)  except for the <code>ashift<\/code> values changed for the tests. After the installation, I did the standard <code>apt-get<\/code> updates to get current, and then installed <code>fio<\/code> to do the testing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Test setup<\/h2>\n\n\n\n<p>The tests I ran were from <a href=\"https:\/\/arstechnica.com\/gadgets\/2020\/02\/how-fast-are-your-disks-find-out-the-open-source-way-with-fio\/\" target=\"_blank\" rel=\"noreferrer noopener\">Jim Salter&#8217;s ARStechnica article<\/a> which gives good detail on how to run <code>fio<\/code> to test disk performance. Each test was run back-to-back. I ran each test 4 times. Here&#8217;s the script I used to run the tests:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \"Test 1 - Single 4KiB random write process\"\n\nfio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --size=4g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1\n\necho \"Test 2 - 16 parallel 64KiB random write processes\"\n\nfio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=256m --numjobs=16 --iodepth=16 --runtime=60 --time_based --end_fsync=1\n\necho \"Test 3 - Single 1MiB random write process\"\n\nfio --name=random-write --ioengine=posixaio --rw=randwrite --bs=1m --size=16g --numjobs=1 --iodepth=1 --runtime=60 --time_based --end_fsync=1\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Results<\/h2>\n\n\n\n<p><code>ashift=12<\/code> was faster (on average) than<code> ashift=13<\/code> for <strong>every<\/strong> test. For Test 1, it was 10.3% (15MB\/s) faster. For Test 2, it was a whopping 22.3% (499MB\/s) faster. And for Test 3, it was 8.5% (117MB\/s) faster. Those are pretty big differences &#8211; I was surprised they weren&#8217;t more in the noise. Made it easy to make the call to set my drives to <code>ashift=12<\/code>, which also happens to be <a href=\"https:\/\/jrs-s.net\/2018\/08\/17\/zfs-tuning-cheat-sheet\/\">the common wisdom<\/a> for all drives today.<\/p>\n\n\n\n<figure class=\"wp-block-table is-style-regular\"><table><tbody><tr><td><strong>Test<\/strong><\/td><td><strong>ashift<\/strong><\/td><td><strong>Iteration 1<\/strong><\/td><td><strong>Iteration 2<\/strong><\/td><td><strong>Iteration 3<\/strong><\/td><td><strong>Iteration 4<\/strong><\/td><td><strong>Average<\/strong><\/td><\/tr><tr><td><strong>1<\/strong><\/td><td>12<\/td><td>176<\/td><td>92.6<\/td><td>150<\/td><td>163<\/td><td>145.4<\/td><\/tr><tr><td><strong>2<\/strong><\/td><td>12<\/td><td>2844<\/td><td>1126<\/td><td>3865<\/td><td>1104<\/td><td>2234.75<\/td><\/tr><tr><td><strong>3<\/strong><\/td><td>12<\/td><td>1139<\/td><td>1320<\/td><td>1550<\/td><td>1473<\/td><td>1370.5<\/td><\/tr><tr><td><strong>1<\/strong><\/td><td>13<\/td><td>177<\/td><td>72.7<\/td><td>118<\/td><td>154<\/td><td>130.425<\/td><\/tr><tr><td><strong>2<\/strong><\/td><td>13<\/td><td>3199<\/td><td>1473<\/td><td>993<\/td><td>1280<\/td><td>1736.25<\/td><\/tr><tr><td><strong>3<\/strong><\/td><td>13<\/td><td>1030<\/td><td>1221<\/td><td>1476<\/td><td>1287<\/td><td>1253.5<\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">All results in MB\/s<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"<p>TL;DR &#8211; ashift=12 performed noticeably better than ashift=13. I recently installed Proxmox on a new build and couldn&#8217;t find any information about the best ashift values for my new NVMe SSD drives. Given that I was installing a brand new server, it gave me a chance to do some quick testing. Here&#8217;s what I found. &hellip; <a href=\"https:\/\/kmwoley.com\/blog\/zfs-ashift-values-for-samsung-990-pro-nvme-ssds\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">ZFS ashift values for Samsung 990 PRO NVMe SSDs<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[382,381],"class_list":["post-3365","post","type-post","status-publish","format-standard","hentry","category-geek","tag-proxmox","tag-zfs"],"_links":{"self":[{"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/posts\/3365","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/comments?post=3365"}],"version-history":[{"count":2,"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/posts\/3365\/revisions"}],"predecessor-version":[{"id":3367,"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/posts\/3365\/revisions\/3367"}],"wp:attachment":[{"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/media?parent=3365"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/categories?post=3365"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kmwoley.com\/blog\/wp-json\/wp\/v2\/tags?post=3365"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}