https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13801 Lustre <p>Kernels since 5.1 have implemented the io_uring interface (<a href="https://kernel.dk/io_uring.pdf" class="external-link" target="_blank" rel="nofollow noopener">https://kernel.dk/io_uring.pdf</a>) for efficient asynchronous IO submission to storage. According to <a href="https://lore.kernel.org/linux-block/20190116175003.17880-1-axboe@kernel.dk/" class="external-link" target="_blank" rel="nofollow noopener">posted results</a>, io_uring is on-par with SPDK doing all of the IO in userspace. The io_uring interface is intended to replace the older libaio interface. </p> <p>With the recent performance improvements for libaio AIO/DIO, it should be possible to use the io_uring interfaces in a similar manner.</p> <p>I don't think many applications are using this interface yet, but since it provides a significant improvement over libaio it will likely become used in performance-oriented applications.</p> LU-13801

Enable io_uring interface for Lustre client

Improvement Minor Resolved Done WC Triage Andreas Dilger Sat, 18 Jul 2020 19:39:50 +0000 Fri, 28 Aug 2020 04:28:31 +0000 Thu, 27 Aug 2020 02:05:21 +0000 0 10 <p>Anything special needed for Lustre, I think there is no special thing needed for filesystem to use io_uring?</p> <p>It would be great to verify this when time is available (eg. if <tt>fio</tt> has an <tt>io_uring</tt> backend). I'm not sure if there is a need for specific file_operations, or if llite/CLIO handles them efficiently today. Just like libaio would <em>work</em> with Lustre, but it wasn't until <a href="https://jira.whamcloud.com/browse/LU-4189" title="Strange code in mdc_changelog_send_thread" class="issue-link" data-issue-key="LU-4189"><del>LU-4189</del></a> landed that it started working well (i.e. not just sync IO submission). </p> <p>FIO has supported ioengine=io_uring since 2019, i think we could did a quickly verify this with newer kernel etc 5.1 if Lustre client could compile well.</p> <p>According to patch <a href="https://review.whamcloud.com/39231" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39231</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-13740" title="Ubuntu 20.04 LTS release" class="issue-link" data-issue-key="LU-13740"><del>LU-13740</del></a> build: update changelog for ubuntu kernel</tt>" the master client is able to compile against kernels up to 5.4 at least, so it should be possible to start testing this.</p> <p>As Andreas pointed out you can fire up a Ubuntu 20.04 LTS with Lustre clients to easily use this functionality.</p> <p>Does it work, James? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>  And does it perform?</p> <p>It mounts <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Last LWG call I talked to Peter about getting a Ubuntu 20 LTS client VM up and running. I hope to see this soon. Especially now we are seeing work like this coming down the pipeline.</p> <p>I got first test resutls and comparison of libaio and ut_ring with fio.<br/> Tested Configuration</p> <ul class="alternate" type="square"> <li>AI400 (20 x Samsung NVMe) for OST/MDT</li> <li>1 x Client(2 x Platinum 8160, 192GB RAM, 2 x IB-EDR) <ul> <li>Ubuntu 20.04 (5.4.0-42-generic)</li> <li>Lustre master branch (commit f384a8733c)</li> </ul> </li> </ul> <p>Test workload (1 thread, QD=1 to 256, 4K random read)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>#!/bin/sh for api in libaio io_uring; do for qd in 1 2 4 8 16 32 64 128 256; do ./fio -name=randread -ioengine=${api} -rw=randread -blocksize=4096 -iodepth=$qd -direct=1 -runtime=10 -group_reporting=1 -create_serialize=0 -size=8g -numjobs=1 -directory=/ai400/testdir -filename_format='f.$jobnum.$filenum' done done </pre> </div></div> <p>Here is results.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>QD libaio io_uring 1 4.2 4.4 2 8.5 8.4 4 16.7 16.8 8 31.5 31.4 16 43.8 43.5 32 60.1 60.1 64 90.7 93.3 128 96.1 103.0 256 95.1 100.0 </pre> </div></div> <p>The good news, at least io_uring didn't break Lustre <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>, but I didn't see huge performance benefit single thread standpoint (with many QDs case). I will play a bit and more collect more results.</p> <p>I guess we could get better performances with Patrick's optimized DIO patches(notice even AIO they shared some code path)</p> <p>I’d be curious to know if they help - 4K random read is a workload im</p> <p> not sure about.  It probably will not help that too much...?  But I’m not sure.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sihara" class="user-hover" rel="sihara">sihara</a> One of important stuff we missed for io_uring test is: we might need -sqthread_poll=1 this might make big difference for io uring testing.</p> <p>I did a quickly test of io_uring on local NVME device and ext4 filesystem:</p> <p>This is to NVME device directly:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LIBAIO io_uring io_uring + poll IOPS: 342656 377382 580258 </pre> </div></div> <p>We could see iops burst very well with same fio command as Ihara's above fio.</p> <p>However, if we compared results on ext4:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> LIBAIO io_uring io_uring + poll IOPS: 312950 258104 260594 </pre> </div></div> <p>It looks io_uring did not help on filesystem, it makes me think io_uring might not help on Lustre too, it might be only helpful in<br/> some cases that target device is really low latency, however network filesystem might not be the case?</p> <p>I think the important thing to note is that libaio is only used by a small number of applications, and has actively been discouraged by the kernel developers. I think the goal for the future is that io_uring is supposed to be a widely supported as a non-POSIX IO interface for the kernel. </p> <p>Luckily, it appears that the libaio optimizations for Lustre also benefit io_uring, so it may be that we don't have a lot of work to do in this area. </p> <p>Closing this ticket as it looks like we don't need to do anything to enable io_uring for Lustre. </p> Related LU-4198 LU-13799 Development Rank 1|i015hb: Rank (Obsolete) 9223372036854775807