https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11221 Lustre <p>Investigating some customer issues I noticed that currently our read looks roughly like this:<br/> 1. obtain the pages and lock them<br/> 2. prepare and then execute our bulk request<br/> 3. unlock the pages.</p> <p>essentially holding pages locked for network IO. This seems to be not super optimal since parallel reads cannot proceed and must wait for each other read to complete first. This would also help in case of client death/network problems since any hung bulk RPCs would not have as much impact on parallel operations.</p> <p>We already have dlm locks to protect us so we probably should be fine to drop the locked pages at step2 and possibly even for writes as well (need to investigate eviction implications) - we were already operating in this mode in 18. and prior before read only cache on the server was implemented and each RPC had a private pool of pages not connected to any inode mappings.</p> LU-11221

Do not hold pages locked for network IO on the server.

Improvement Major Resolved Fixed Alex Zhuravlev Oleg Drokin Mon, 6 Aug 2018 18:25:52 +0000 Thu, 11 Jun 2020 09:23:11 +0000 Fri, 9 Aug 2019 15:06:01 +0000 Lustre 2.13.0 Lustre 2.12.4 0 14 <p>Alex</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p><a href="https://review.whamcloud.com/33521" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33521</a></p> <p>I'm not sure patch <a href="https://review.whamcloud.com/33521" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33521</a> can behave using page caches if file size &lt; readcache_max_filesize, but no page caches after readcache_max_filesize.</p> <p>At least, I have been seeing very good merged IOs with "read_cache_enable=0 writethrough_cache_enable=0", but "read_cache_enable=1 writethrough_cache_enable=1 and readcache_max_filesize=1048576" seems to be less chance to make merged IOs and less performance.</p> <p>Please see the following test results.</p> <p>Disable Pagecache mode</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@es18k-vm11 ~]# lctl set_param osd-ldiskfs.*.read_cache_enable=0 obdfilter.*.writethrough_cache_enable=0 obdfilter.*.brw_size=16 Run IOR from 32 clients, 256 process into single OST [root@c01 ~]# salloc -N 32 --ntasks-per-node=8 mpirun -np 256 --allow-run-as-root /work/tools/bin/ior -w -t 1m -b 1g -e -F -v -C -Q 24 -o /scratch0/ost0/file Max Write: 8461.94 MiB/sec (8872.99 MB/sec) </pre> </div></div> <p>I saw good merged IOs (16MB IOs on storage side) in this case.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@es18k-vm11 ~]# blktrace /dev/sda -a issue -a complete -o - | blkiomon -I 120 -h - time: Sun Apr 7 14:19:45 2019 device: 8,0 sizes read (bytes): num 0, min -1, max 0, sum 0, squ 0, avg 0.0, var 0.0 sizes write (bytes): num 17619, min 4096, max 16777216, sum 274927271936, squ 4533348865750859776, avg 15604022.5, var 257270866642200.3 d2c read (usec): num 0, min -1, max 0, sum 0, squ 0, avg 0.0, var 0.0 d2c write (usec): num 17619, min 48, max 6425912, sum 8433387602, squ 5102891088905970, avg 478653.0, var 238785697942.2 throughput read (bytes/msec): num 0, min -1, max 0, sum 0, squ 0, avg 0.0, var 0.0 throughput write (bytes/msec): num 17619, min 0, max 4273907, sum 464644931, squ 427646612524675, avg 26371.8, var 23576427970.2 sizes histogram (bytes): 0: 0 1024: 0 2048: 0 4096: 153 8192: 11 16384: 8 32768: 12 65536: 18 131072: 27 262144: 31 524288: 79 1048576: 77 2097152: 21 4194304: 458 8388608: 742 &gt; 8388608: 15982 &lt;--- good 16M IOs here d2c histogram (usec): 0: 0 8: 0 16: 0 32: 0 64: 43 128: 36 256: 57 512: 90 1024: 32 2048: 18 4096: 2 8192: 12 16384: 31 32768: 169 65536: 339 131072: 1528 262144: 1009 524288: 5433 1048576: 8751 2097152: 53 4194304: 13 8388608: 3 16777216: 0 33554432: 0 &gt;33554432: 0 bidirectional requests: 0 </pre> </div></div> <p>Enabling PageCaches </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@es18k-vm11 ~]# lctl set_param osd-ldiskfs.*.read_cache_enable=1 obdfilter.*.writethrough_cache_enable=1 obdfilter.*.brw_size=16 osd-ldiskfs.*.readcache_max_filesize=1048576 [root@c01 ~]# salloc -N 32 --ntasks-per-node=8 mpirun -np 256 --allow-run-as-root /work/tools/bin/ior -w -t 1m -b 1g -e -F -v -C -Q 24 -o /scratch0/ost0/file Max Write: 6778.19 MiB/sec (7107.45 MB/sec) &lt;-- 25% lower performance than non pagecache mode </pre> </div></div> <p>blktrace tells us there are a lot of 8MB IOs rather than 16M IOs here.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@es18k-vm11 ~]# blktrace /dev/sda -a issue -a complete -o - | blkiomon -I 120 -h - time: Sun Apr 7 14:26:55 2019 device: 8,0 sizes read (bytes): num 0, min -1, max 0, sum 0, squ 0, avg 0.0, var 0.0 sizes write (bytes): num 30756, min 4096, max 16777216, sum 274926415872, squ 2835130225468112896, avg 8938952.3, var 92172676586037.2 d2c read (usec): num 0, min -1, max 0, sum 0, squ 0, avg 0.0, var 0.0 d2c write (usec): num 30756, min 31, max 1827037, sum 19563303393, squ 14554226996221365, avg 636080.9, var 458639796658.0 throughput read (bytes/msec): num 0, min -1, max 0, sum 0, squ 0, avg 0.0, var 0.0 throughput write (bytes/msec): num 30756, min 2, max 4413081, sum 841918911, squ 803811743497523, avg 27374.1, var 25385776471.6 sizes histogram (bytes): 0: 0 1024: 0 2048: 0 4096: 114 8192: 6 16384: 6 32768: 8 65536: 7 131072: 22 262144: 44 524288: 110 1048576: 398 2097152: 442 4194304: 1735 8388608: 20107 &gt; 8388608: 7757 d2c histogram (usec): 0: 0 8: 0 16: 0 32: 1 64: 34 128: 21 256: 75 512: 84 1024: 97 2048: 158 4096: 22 8192: 41 16384: 213 32768: 370 65536: 279 131072: 633 262144: 1093 524288: 4855 1048576: 21389 2097152: 1391 4194304: 0 8388608: 0 16777216: 0 33554432: 0 &gt;33554432: 0 bidirectional requests: 0 </pre> </div></div> <p>thanks for the report. I need some time to analyze this. my first guess is that pagecache overhead (need to allocate pages, scanning to release old pages) introduces additional gaps between I/Os, but this is not true as each thread allocates all needed pages and only after that submit I/Os. so it must be something else.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33521/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33521/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11221" title="Do not hold pages locked for network IO on the server." class="issue-link" data-issue-key="LU-11221"><del>LU-11221</del></a> osd: allow concurrent bulks from pagecache<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0a92632538d8c985e024def73512d18d1570d5ca</p> <p>Landed for 2.13</p> <p>Oleg, Alex - why PG_writeback isn't used for this case?<br/> from other side have pages unlocked open race with truncate from same host.</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36570" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36570</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11221" title="Do not hold pages locked for network IO on the server." class="issue-link" data-issue-key="LU-11221"><del>LU-11221</del></a> osd: allow concurrent bulks from pagecache<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7c0b6676a1b1d3b453a3d3888e26d15b7bf07b44</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36570/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36570/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11221" title="Do not hold pages locked for network IO on the server." class="issue-link" data-issue-key="LU-11221"><del>LU-11221</del></a> osd: allow concurrent bulks from pagecache<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: efcdfe9e075fdfa334d16bcb53399f2978c16d42</p> Duplicate LU-9232 Related Development Rank 1|i000db: Rank (Obsolete) 9223372036854775807