https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4198 Lustre <p>Attached to this Jira are some numbers from the direct IO tests. Write operations only.</p> <p>It was noticed that setting RPCs in flight to 256 in these tests gives poorer performance. max rpc here is set to 32.</p> <ul> <li>A sample FIO output: <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fio.4k.write.1.23499: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1 fio-2.1.2 Starting 1 process fio.4k.write.1.23499: Laying out IO file(s) (1 file(s) / 10MB) fio.4k.write.1.23499: (groupid=0, jobs=1): err= 0: pid=10709: Fri Nov 1 11:47:29 2013 write: io=10240KB, bw=2619.7KB/s, iops=654, runt= 3909msec clat (usec): min=579, max=5283, avg=1520.43, stdev=1216.20 lat (usec): min=580, max=5299, avg=1521.37, stdev=1216.22 clat percentiles (usec): | 1.00th=[ 604], 5.00th=[ 652], 10.00th=[ 668], 20.00th=[ 708], | 30.00th=[ 732], 40.00th=[ 756], 50.00th=[ 796], 60.00th=[ 844], | 70.00th=[ 1320], 80.00th=[ 3440], 90.00th=[ 3568], 95.00th=[ 3632], | 99.00th=[ 3824], 99.50th=[ 5024], 99.90th=[ 5216], 99.95th=[ 5280], | 99.99th=[ 5280] bw (KB /s): min= 1224, max= 4366, per=97.64%, avg=2557.14, stdev=1375.64 lat (usec) : 750=37.50%, 1000=30.12% lat (msec) : 2=5.00%, 4=26.76%, 10=0.62% cpu : usr=0.92%, sys=8.70%, ctx=2562, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, &gt;=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, &gt;=64=0.0% issued : total=r=0/w=2560/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=10240KB, aggrb=2619KB/s, minb=2619KB/s, maxb=2619KB/s, mint=3909msec, maxt=3909msec </pre> </div></div></li> </ul> Seen in two environments. AWS cloud (Robert R.) and a dual-OSS setup (3 SSD per OST) over 2x10 GbE. LU-4198

Improve IO performance when using DIRECT IO using libaio

Improvement Minor Resolved Fixed Zhenyu Xu Brett Lee clio Fri, 1 Nov 2013 20:19:29 +0000 Wed, 26 Aug 2020 13:15:57 +0000 Wed, 10 Jun 2020 19:40:35 +0000 Lustre 2.4.1 Lustre 2.14.0 0 25 <p>I am not quite sure how to read this output to know if it is good bad.</p> <p>In general I expect Direct I/O to hurt performance. It gets the filesystem read/write caches out of the way of the app. It is commonly used for databases to minimize the risk (some turn off hardware write caches as well).</p> <p>For 4k io I would not use wide stripping. </p> <blockquote> <p>Additional stripes on a file does not increase IO performance when using DIRECT IO</p></blockquote> <p>This is unfortunately expected since we have to wait for I/O completion on the first stripe before firing RPCs to the next one (i.e. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">foreach(stripe) { lock(stripe); do_sync_io(stripe); unlock(stripe); }</pre> </div></div> <p>) to work around the cascading abort issue. On 1.8, some customers were using a patch to use lockless direct I/O by default.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=johann" class="user-hover" rel="johann">johann</a> Ah, that is what I was afraid of. Is there a lockless direct IO patch for 2.x? That would probably be very helpful in this use case. </p> <p>I see in <a href="https://jira.whamcloud.com/browse/LU-238" title="add procfs tunable to enable/disable lockless direct I/O" class="issue-link" data-issue-key="LU-238"><del>LU-238</del></a> that there is a nolock mount option that we can use to enabled lockless direct IO.</p> <p>What is the cascading abort issue? </p> <p>I tried mounting the client with "nolock" and performance actually got about 4x worse than before. </p> <blockquote><p> I see in <a href="https://jira.whamcloud.com/browse/LU-238" title="add procfs tunable to enable/disable lockless direct I/O" class="issue-link" data-issue-key="LU-238"><del>LU-238</del></a> that there is a nolock mount option that we can use to enabled lockless direct IO. </p></blockquote> <p>It seems that this patch enables lockless I/O not only for direct I/O, but also for buffered I/Os which is quite bad.</p> <blockquote><p> What is the cascading abort issue? </p></blockquote> <p>Client holds a lock on resource from server A and waits for RPC completion on server B. This introduces an implicit dependency between servers. If server B is not responsive (e.g. doing failover or just slow because it is overloaded) and server A issues a blocking AST, the client will get evicted from server A since it cannot release the lock in a timely manner.</p> <blockquote><p> I tried mounting the client with "nolock" and performance actually got about 4x worse than before. </p></blockquote> <p>Strange ... we definitely got better results with 1.8. There is probably something wrong with CLIO.<br/> BTW, to be clear, you should only see a benefit if your direct writes cover multiple stripes, otherwise there won't be any parallelism.</p> <p>HTH</p> <p>in 2.x, the only difference between direct IO and cache IO is whether it caches dirty data on the client. They actually share the same IO framework.</p> <p>Even though it's really strange that no lock version was 4x worse; server takes lock for no lock IO - is there anybody else operating this file meanwhile?</p> <p>I was running a single threaded benchmark (FIO) and there was only a single client on the filesystem. So definitely not shared. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>It seems there are other differences between direct and buffered IO, such as direct io is synchronous. I've noticed while testing AIO with various io depths that AIO appears to make no difference with direct IO.</p> <p>AIO used to work with Direct IO only. I don't know what the state in current kernel is, I'll check it out.</p> <p>If we want to use direct IO, two problems have to be addressed:<br/> 1. lock: if the file is being read or written in direct IO, it's unnecessary to take lock from server. can we make the assumption all direct IO should go with lockless?</p> <p>2. universal direct IO support: in current implementation, the address of user buffer has to be aligned to page. Niu has a patch to address this problem but it uses obsoleted interfaces.</p> <p>Both problems should not be difficult to solve.</p> <p>Robert, will you briefly describe the use case scenarios for direct IO?</p> <p>Hello, all.</p> <p>I have performed tests in one of the two environments where Brett worked (the dual-OSS setup - 3 SSD per OST - over 2x10 GE).</p> <p>I made tests writing directly into a local SSD and with other network block storage tool (NBD - Network Block Device), playing with the O_DIRECT and O_SYNC flags:</p> <ul class="alternate" type="square"> <li>no flags + native Async IO (libaio), 8 writes in flight</li> <li>O_DIRECT + native Async IO (libaio), 8 writes in flight</li> <li>O_DIRECT + O_SYNC</li> </ul> <p>Results are that the performance, in both setups (local SSD and NBD) has the following pattern:</p> <p>only libaio &gt;&gt; faster than &gt;&gt; O_DIRECT + libaio &gt;&gt; faster than &gt;&gt; O_DIRECT + O_SYNC</p> <p>whereas, with Lustre, O_DIRECT + libaio and O_DIRECT + O_SYNC show the same performance.</p> <p>I exchanged a couple of emails with Brett and he confirmed that in Lustre, setting O_DIRECT implies always setting also O_SYNC.</p> <p>Also, in theory:</p> <p>O_DIRECT (Since Linux 2.4.10)<br/> Try to minimize cache effects of the I/O to and from this<br/> file. In general this will degrade performance, but it is<br/> useful in special situations, such as when applications do<br/> their own caching. File I/O is done directly to/from user-<br/> space buffers. The O_DIRECT flag on its own makes an effort<br/> to transfer data synchronously, but does not give the<br/> guarantees of the O_SYNC flag that data and necessary metadata<br/> are transferred. To guarantee synchronous I/O, O_SYNC must be<br/> used in addition to O_DIRECT. See NOTES below for further<br/> discussion.</p> <p> A semantically similar (but deprecated) interface for block<br/> devices is described in raw(8).</p> <p>This comes from open(2) man page.</p> <p>So, to my understanding, O_DIRECT "tries" to write synchronously, but does not offer any guarantee. This is specially important in the case when using O_DIRECT + libaio, which is a library that allows non-blocking parallel writes to single-threaded user-space applications.</p> <p>According to the theory (man pages) and my tests with local SSD and NBD, I would personally say that Lustre deviates from the standard POSIX Filesystems.</p> <p>Not only that, this behavior slows down Lustre, apparently for no reason.</p> <p>If you agree with me on that, I would like to request a change in the code to correct this behavior, or at least some help on where to change it by myself, so I can test again and maybe see a bump in performance.</p> <p>Thanks in advance</p> <p>Using packages built for "SLES 11 SP2" OS, am seeing an LBUG when mounting a newly created storage target (MGT in this case). This event is repeatable. Using packages from:</p> <p><a href="https://build.whamcloud.com/job/lustre-reviews/21279/arch=x86_64,build_type=server,distro=sles11sp2,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/21279/arch=x86_64,build_type=server,distro=sles11sp2,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/</a></p> <p>Stack trace seen on the console is:</p> <p>sles11sp2-2:~/work # mount -t lustre /dev/vdb /sap/mgs<br/> [ 159.338678] LustreError: 3532:0:(sec_ctx.c:80:pop_ctxt()) ASSERTION( segment_eq(get_fs(), get_ds()) ) failed: popping non-kernel context!<br/> [ 159.340266] LustreError: 3532:0:(sec_ctx.c:80:pop_ctxt()) LBUG<br/> [ 159.342723] Kernel panic - not syncing: LBUG<br/> [ 159.343356] Pid: 3532, comm: mount.lustre Tainted: G N 3.0.93-0.5_lustre.ge80a1ca-default #1<br/> [ 159.344239] Call Trace:<br/> [ 159.344384] <span class="error">&#91;&lt;ffffffff810048b5&gt;&#93;</span> dump_trace+0x75/0x310<br/> [ 159.344692] <span class="error">&#91;&lt;ffffffff814473a3&gt;&#93;</span> dump_stack+0x69/0x6f<br/> [ 159.344983] <span class="error">&#91;&lt;ffffffff8144743c&gt;&#93;</span> panic+0x93/0x201<br/> [ 159.345255] <span class="error">&#91;&lt;ffffffffa02f0dc3&gt;&#93;</span> lbug_with_loc+0xa3/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> [ 159.345621] <span class="error">&#91;&lt;ffffffffa076087c&gt;&#93;</span> pop_ctxt+0x19c/0x1a0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 159.345984] <span class="error">&#91;&lt;ffffffffa0b0667d&gt;&#93;</span> osd_ost_init+0x23d/0x8d0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> [ 159.346370] <span class="error">&#91;&lt;ffffffffa0b06d39&gt;&#93;</span> osd_obj_map_init+0x29/0x120 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> [ 159.346767] <span class="error">&#91;&lt;ffffffffa0ae4151&gt;&#93;</span> osd_device_init0+0x281/0x5c0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> [ 159.347166] <span class="error">&#91;&lt;ffffffffa0ae47c6&gt;&#93;</span> osd_device_alloc+0x166/0x2c0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> [ 159.347574] <span class="error">&#91;&lt;ffffffffa04d642b&gt;&#93;</span> class_setup+0x61b/0xad0 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.347957] <span class="error">&#91;&lt;ffffffffa04de5f5&gt;&#93;</span> class_process_config+0xc95/0x18f0 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.348393] <span class="error">&#91;&lt;ffffffffa04e3652&gt;&#93;</span> do_lcfg+0x142/0x460 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.348752] <span class="error">&#91;&lt;ffffffffa04e3a04&gt;&#93;</span> lustre_start_simple+0x94/0x210 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.349168] <span class="error">&#91;&lt;ffffffffa051171a&gt;&#93;</span> osd_start+0x4fa/0x7c0 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.349549] <span class="error">&#91;&lt;ffffffffa051b41d&gt;&#93;</span> server_fill_super+0xfd/0xce0 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.349965] <span class="error">&#91;&lt;ffffffffa04e91e8&gt;&#93;</span> lustre_fill_super+0x178/0x530 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.350362] <span class="error">&#91;&lt;ffffffff811556e3&gt;&#93;</span> mount_nodev+0x83/0xc0<br/> [ 159.350668] <span class="error">&#91;&lt;ffffffffa04e1080&gt;&#93;</span> lustre_mount+0x20/0x30 <span class="error">&#91;obdclass&#93;</span><br/> [ 159.351035] <span class="error">&#91;&lt;ffffffff811551ee&gt;&#93;</span> mount_fs+0x4e/0x1a0<br/> [ 159.351318] <span class="error">&#91;&lt;ffffffff811703f5&gt;&#93;</span> vfs_kern_mount+0x65/0xd0<br/> [ 159.351623] <span class="error">&#91;&lt;ffffffff811704e3&gt;&#93;</span> do_kern_mount+0x53/0x110<br/> [ 159.351930] <span class="error">&#91;&lt;ffffffff81171e2d&gt;&#93;</span> do_mount+0x21d/0x260<br/> [ 159.352246] <span class="error">&#91;&lt;ffffffff81171f30&gt;&#93;</span> sys_mount+0xc0/0xf0<br/> [ 159.352529] <span class="error">&#91;&lt;ffffffff81452112&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> [ 159.352867] <span class="error">&#91;&lt;00007f99b62bd1ea&gt;&#93;</span> 0x7f99b62bd1e9</p> <p>Have confirmed that the same installed OS functions properly using SLES packages from:</p> <p><a href="http://build.whamcloud.com/job/lustre-b2_5/arch=x86_64,build_type=server,distro=sles11sp2,ib_stack=inkernel/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/arch=x86_64,build_type=server,distro=sles11sp2,ib_stack=inkernel/</a></p> <p>Hi Brett, it seems not related, please file a new ticket for the problem.</p> <p>Thanks Jinshan - have opened a different ticket for that issue.</p> <p>In testing the build from 21279 on CentOS 6.4, saw two similar issues. Configuration is a single node running a MGS, 1 MDT, 2 OSTs, and 1 client mount. Also have an identical "control" system. Both worked well (no issues seen, all tests completed w/o issue) with RHEL server bits from:<br/> <a href="http://downloads.whamcloud.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">http://downloads.whamcloud.com/public/lustre/latest-feature-release/el6/server/RPMS/x86_64/</a></p> <p>After reconfiguring same system with the new bits:</p> <p><a href="http://build.whamcloud.com/job/lustre-reviews/21279/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/21279/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/</a></p> <p>and running IO, there were two issues seen. First was a hung system that resulted in a /tmp/debug log ~ 6MB. Second was a system that got corrupted - showed 61% capacity utilization on both OSTs, though no files were present from the client perspective - also produced a debug log ~300K. Both debug logs are available for further review but not uploaded.</p> <p>The system hang failure appeared immediate pursuant to the first of 32 IO tests, using synchronous IO. The second failure was on only 2 of the 32 test cases, all of these using AIO. The two tests that failed were 1 GB write and random write, in 64MB bursts. In those two cases, the IO hung but I was able to ctrl-c out of the IO job.</p> <p>16 tests were run using 1 OST, 16 tests using 2 OSTs. Note that in several of the test cases the performance benefit using these patches (vs. the control node) was very pronounced. Will be working to get more samples to increase the reliability of these data, and to further check/troubleshoot any issues with stability.</p> <p>Update:<br/> Continuing to run benchmarks against this build.</p> <p>No further hung system issues. Oddly, the hung system was on the initial IO, and never seen since.</p> <p>The "corrupted" event is reproducible, though I would no longer call it corrupted. Rather, it has to do with stalled fio kernel threads. After killing off the fio user processes, two kernel threads remained. After rebooting to end those threads, the 61% was cleared.</p> <p>Note that all fio writes using block size 64M are not completing (though they are on the 2.5 release, as well as the root <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/check.png" height="16" width="16" align="absmiddle" alt="" border="0"/> ext4 file system.</p> <p>All other reads/writes (sequential and random) are completing successfully and without incident. Performance data comparisons upcoming.</p> <p>Data in the attached spreadsheet seems to make a good case for including the performance improvements. Also, I’ve not seen any further stability issues since the beginning of the test period.</p> <p>Will you please increase iodepth to at least 32 and see if we can get any better results?</p> <p>Better? I thought those results were pretty good already. Will give it a try.</p> <p>Jinshan - an OST failed on me (each OST is one SATA-II or III disk) and have no other suitable disks. Have ordered a pair of WD 10K RPM Velociraptors (200 MB/s) that will support queue depth up to 32 (NCQ). On hold till then.</p> <p>Jinshan, please update this ticket description to include the reason that this ticket is a dependency for <a href="https://jira.whamcloud.com/browse/LU-3259" title="cl_lock refactoring" class="issue-link" data-issue-key="LU-3259"><del>LU-3259</del></a>.</p> <p>This ticket isn't directly related to CLIO Simplification work. The ticket relationships on Jira have been updated to reflect this.</p> <p>Patches in Gerrit for this issue:<br/> <a href="http://review.whamcloud.com/8201" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8201</a><br/> <a href="http://review.whamcloud.com/8612" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8612</a></p> <p>Let's reopen this ticket after we have a more convincible solution for this issue.</p> <p><img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Patch is still in flight. (Hope this is OK.)</p> <p><a href="https://jira.whamcloud.com/browse/LU-247" title="Lustre client slow performance on BG/P IONs: unaligned DIRECT_IO" class="issue-link" data-issue-key="LU-247"><del>LU-247</del></a> is pretty old and probably no one has time to update it... But it could be very useful (with this) for improving &lt; page size write performance.</p> <p>Jinshan,</p> <p>Attached patch is a suggestion for fixing the need for size glimpsing for dio reads. Not 100% sure it's safe, but some local testing suggests it's OK. (Diff was a little big to drop in gerrit)</p> <p>This work is still useful so probably we should keep this ticket open.</p> <p>Jinshan Xiong (jinshan.xiong@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/32415" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32415</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> clio: turn on parallel mode for some kind of IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5afe155acdeaf5fefc230d43d23936f03e0e447b</p> <p>Jinshan Xiong (jinshan.xiong@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/32416" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32416</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> clio: AIO support for direct IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8c480ce9300ad4e4f23f5fac4e5a3c2d038017c7</p> <p>Jinshan Xiong (jinshan.xiong@gmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/32417" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32417</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> llite: no lock match for lockess I/O<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e1c1366b1bd0129cd5276693da8cb1f895db3594</p> <p>Here is test resutls of patch <a href="https://review.whamcloud.com/32416" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32416</a><br/> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31517/31517_LU-4198.png" title="LU-4198.png attached to LU-4198">LU-4198.png^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>Client <br/> 2 x E5-2650v4@2.20GHz, 128GB memory, 1 x EDR<br/> OSS/MDS<br/> DDN AI200(2xOSS/MDS, 20 x NVMe, 1 x EDR, master branch)</p> <p>Without patch, we only get 80K IOPS at 4k random read with DIO even increased number of threads. Here is fio parameters.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[randread] ioengine=sync ;ioengine=libaio rw=randread blocksize=4096 iodepth=32 direct=1 size=1g runtime=120 numjobs=128 group_reporting directory=/cache0/fio.out filename_format=f.$jobnum.$filenum </pre> </div></div> <p>With aio patch <a href="https://review.whamcloud.com/32416" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32416</a>, it could readch more than 600K IOPS per an client.<br/> patch helps not only supporting libaio on Lustre, but also benchmark (e.g. we only very small number of clients to satulate storage IOPS) and libaio supoprted applications. (e.g. database, virtual machine enviorment).<br/> One thing I need to inform here, but patch worked well only at less number of max_rpcs_in_flight. For instance, max_rpcs_in_flight=1 was scaling very well so far, but max_rpcs_in_flight=256 was problematic and didn't scale anything.</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/34774" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34774</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> llite: transient page simplification<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 704390928ce55173d2b2fca0e0fe244907d750b2</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/8201/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/8201/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> clio: turn on lockless for some kind of IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6bce536725efd166d2772f13fe954f271f9c53b8</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32416/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32416/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> clio: AIO support for direct IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d1dded6e28473d889a9b24b47cbc804f90dd2956</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/37621" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37621</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> clio: return error for short direct IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9913a5411bb305d617ed3bba6bd5d000ffc11121</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37824/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37824/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> clio: Remove pl_owner<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e880a6fccbc57ae335949c1dd20335359b1cb220</p> <p>Can someone please summarize what the state of this ticket is? The subject seems to have wandered from "Additional stripes on a file does not increase IO performance when using DIRECT IO" to lockless DIO to AIO. Johann's and Jinshan's comments seem to be at odds as to whether DIO stripes are parallelized or not. </p> <p>Rough testing (DIO, not AIO) seems to indicate they are not. </p> <p>Nathan, I think regardless of how this ticket started, it ended up being used to land the AIO/DIO support for 2.14. If there are still issues that need to be addressed, they should be done in the context of a new ticket.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=nrutman" class="user-hover" rel="nrutman">nrutman</a></p> <p>I think <a href="https://jira.whamcloud.com/browse/LU-13798" title="Improve direct i/o performance with multiple stripes: Submit all stripes of a DIO and then wait" class="issue-link" data-issue-key="LU-13798"><del>LU-13798</del></a> will parallel DIO with stripes enabled, and we see big improvements with that, so i guess you could take a look there.</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/39733" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39733</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4198" title="Improve IO performance when using DIRECT IO using libaio" class="issue-link" data-issue-key="LU-4198"><del>LU-4198</del></a> clio: turn on lockless for some kind of IO<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 2acd73b09c86ca7ee436152274f9d1beab1ad571</p> Duplicate LU-13786 Related LU-12687 LU-13798 LU-13900 LU-247 LU-13697 LU-9409 LU-10278 LU-13801 Development Rank 1|hzw7nb: Rank (Obsolete) 11385 Severity