https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13973 Lustre <p>Here is a tested workload.</p> <p>4k, random write, FPP(File per process)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[randwrite] ioengine=libaio rw=randwrite blocksize=4k iodepth=4 direct=1 size=${SIZE} runtime=60 numjobs=16 group_reporting directory=/ai400x/out create_serialize=0 filename_format=f.$jobnum.$filenum </pre> </div></div> <p>The test case is that 2 clients have each 16 fio processes and each fio process does 4k random write to different files.<br/> However, if file size is large (128GB in this case), it causes the huge performance impacts. Here is two test results.</p> <p>1GB file</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># SIZE=1g /work/ihara/fio.git/fio --client=hostfile randomwrite.fio write: IOPS=16.8k, BW=65.5MiB/s (68.7MB/s)(3930MiB/60004msec); 0 zone resets </pre> </div></div> <p>128GB file</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># SIZE=128g /work/ihara/fio.git/fio --client=hostfile randomwrite.fio write: IOPS=2894, BW=11.3MiB/s (11.9MB/s)(679MiB/60039msec) </pre> </div></div> <p>As far as I observed those two cases and collected cpu profiles on OSS, in 128GB file case, there were big spinlocks in ldiskfs_mb_new_block() and ldiskfs_mb_normalized_request() and it spent 89% time (14085/15823 samples) of total ost_io_xx() against 20% (1895/9296 samples) in 1GB file case. Please see attached framegraph.</p> master LU-13973

4K random write performance impacts on large sparse files

Bug Major Open Unresolved Qian Yingjin Shuichi Ihara Sun, 20 Sep 2020 07:57:49 +0000 Thu, 29 Oct 2020 11:49:47 +0000 Lustre 2.14.0 0 5 <p>Hi Ihara, </p> <p>Could you please first preallocate all space via fallocate?<br/> i.e<br/> fio with fallocate,<br/> or use the command 'fallocate -l ' to preallocate all needed space,<br/> and then do the fio testing?</p> <p>Thanks,<br/> Qian</p> <p>Yingjin, I also thought fallocate might help and tried fallocate with fio (NOTE, fio use fallocate if filesystem supports it) after patch <a href="https://review.whamcloud.com/#/c/39342/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/39342/</a> applied, but it was same problem and fallocate didn't help neither. btw, overwriting files helped. e.g. create 128GB files and allocate all blocks first then randomwrite on them.</p> <p> </p> <p>Hi Ihara,</p> <p>I may find the reason, it should be a problem of fallocate for direct IO (not for buffered IO).</p> <p>Will make a revised patch soon.</p> <p>Regards,<br/> Qian</p> <p>Please try the updated fallocate patch:<br/> <a href="https://review.whamcloud.com/39342" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39342</a> <a href="https://jira.whamcloud.com/browse/LU-13765" title="ldiskfs_mb_mark_diskspace_used:3472: aborting transaction: error 28 in __ldiskfs_handle_dirty_metadata" class="issue-link" data-issue-key="LU-13765"><del>LU-13765</del></a> osd-ldiskfs: Extend credit correctly for fallocate</p> <p>It jus modified one line:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> diff --git a/lustre/osd-ldiskfs/osd_io.c b/lustre/osd-ldiskfs/osd_io.c index 462a462cc9..689471e8a3 100644 --- a/lustre/osd-ldiskfs/osd_io.c +++ b/lustre/osd-ldiskfs/osd_io.c @@ -2009,7 +2009,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> osd_fallocate(<span class="code-keyword">const</span> struct lu_env *env, struct dt_object *dt, <span class="code-keyword">break</span>; rc = ldiskfs_map_blocks(handle, inode, &amp;map, - LDISKFS_GET_BLOCKS_CREATE_UNWRIT_EXT); + LDISKFS_GET_BLOCKS_CREATE); <span class="code-keyword">if</span> (rc &lt;= 0) { CDEBUG(D_INODE, <span class="code-quote">"inode #%lu: block %u: len %u: "</span> <span class="code-quote">"ldiskfs_map_blocks returned %d\n"</span>, </pre> </div></div> <p>Regards,<br/> Qian</p> <p>Btw, could you please measure the fallocate performance with/without the updated patches?</p> <p>i.e.</p> <p>time fallocate -l 128G test1<br/> time fallocate -l 256G test2<br/> I just want to known whether it will affect the fallocate using time.</p> <p>thanks,<br/> Qian</p> <p>In fact, it seems that fallocate is not working in both patch (patchset6 and patchet7) properly..</p> <p>patchset 6</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# time fallocate -l 128g /ai400x/test1 real 0m0.004s user 0m0.001s sys 0m0.000s [root@ec01 ~]# ls -l /ai400x/test1 -rw-r--r-- 1 root root 0 Sep 21 14:47 /ai400x/test1 </pre> </div></div> <p>patchset 7</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# time fallocate -l 128g /ai400x/test1 real 0m0.003s user 0m0.001s sys 0m0.000s [root@ec01 ~]# ls -l /ai400x/test1 -rw-r--r-- 1 root root 0 Sep 21 15:06 /ai400x/test1 </pre> </div></div> <p>Just fixed the problem:<br/> With LDISKFS_GET_BLOCKS_CREATE:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@qvm1 tests]# time fallocate -l 5G /mnt/lustre/test real 0m0.220s user 0m0.002s sys 0m0.003s [root@qvm1 tests]# stat /mnt/lustre/test File: /mnt/lustre/test Size: 5368709120 Blocks: 10485768 IO Block: 4194304 regular file [root@qvm1 tests]# time fallocate -l 1G /mnt/lustre/test real 0m0.175s user 0m0.002s sys 0m0.003s </pre> </div></div> <p>With LDISKFS_GET_BLOCKS_CREATE_UNWRIT_EXT:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@qvm1 tests]# time fallocate -l 5G /mnt/lustre/test real 0m0.268s user 0m0.002s sys 0m0.005s [root@qvm1 tests]# stat /mnt/lustre/test File: /mnt/lustre/test Size: 5368709120 Blocks: 10485768 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502273 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Context: unconfined_u:object_r:unlabeled_t:s0 Access: 2020-09-21 16:36:57.000000000 +0800 Modify: 2020-09-21 16:36:57.000000000 +0800 Change: 2020-09-21 16:36:57.000000000 +0800 Birth: - </pre> </div></div> <p>Please try the updated patch again.</p> <p>BTW, could you please also try with large allocation by using EXT4 allocation flags:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@qvm1 lustre-release]# git diff diff --git a/lustre/osd-ldiskfs/osd_io.c b/lustre/osd-ldiskfs/osd_io.c index 7897fd4082..233ea54c6f 100644 --- a/lustre/osd-ldiskfs/osd_io.c +++ b/lustre/osd-ldiskfs/osd_io.c @@ -1983,7 +1983,7 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> osd_fallocate(<span class="code-keyword">const</span> struct lu_env *env, struct dt_object *dt, boff = start &gt;&gt; inode-&gt;i_blkbits; blen = (ALIGN(end, 1 &lt;&lt; inode-&gt;i_blkbits) &gt;&gt; inode-&gt;i_blkbits) - boff; - flags = LDISKFS_GET_BLOCKS_CREATE; + flags = LDISKFS_GET_BLOCKS_CREATE_UNWRIT_EXT; <span class="code-keyword">if</span> (mode &amp; FALLOC_FL_KEEP_SIZE) flags |= LDISKFS_GET_BLOCKS_KEEP_SIZE; </pre> </div></div> <p>and measure the allocation time and the performance you did via fio again?</p> <p>Thanks,<br/> Qian</p> <p>Hi Yingjin,<br/> yup, I've also confirmed the latest patch (patchset 8 of <a href="https://review.whamcloud.com/39342" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39342</a>) solved problem.<br/> I was back to original problem of <a href="https://jira.whamcloud.com/browse/LU-13973" title="4K random write performance impacts on large sparse files" class="issue-link" data-issue-key="LU-13973">LU-13973</a> and re-tested again, it also solved problem. fallocate works with O_DIRECT well for now.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># cat hostlist ec01 ec02 # SIZE=1g /work/ihara/fio.git/fio --client=hostlist randomwrite.fio write: IOPS=37.4k, BW=146Mi (153M)(8761MiB/60004msec); 0 zone resets # SIZE=128g /work/ihara/fio.git/fio --client=hostlist randomwrite.fio write: IOPS=38.1k, BW=149Mi (156M)(8921MiB/60007msec); 0 zone resets </pre> </div></div> Related LU-13765 Development Rank 1|i01ab3: Rank (Obsolete) 9223372036854775807 Severity