https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12325 Lustre <p>The DOM lock bit is taken opportunistically at open to improve possible read/write access but it uses PW lock mode and can be combined with other lock bits. The proposed improvements are:<br/> 1. when file is closed and there were no read/write, drop DOM bit and downgrade lock mode to less strict lock mode (CR/CW)<br/> 2. if there was write then write data on close (<a href="https://jira.whamcloud.com/browse/LU-11428" title="Writeback on close for DoM" class="issue-link" data-issue-key="LU-11428">LU-11428</a>) and also downgrade lock mode to PR<br/> 3. if DOM bit is dropped and there are other bits remains in lock - downgrade lock mode too </p> LU-12325

Downgrade lock mode for DOM files when DOM bit gone

Improvement Major Open Unresolved Mikhail Pershin Mikhail Pershin DoM2 Wed, 22 May 2019 06:55:13 +0000 Fri, 9 Jun 2023 11:36:06 +0000 0 5 <p><del>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch:</del> <a href="https://review.whamcloud.com/35031" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35031</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-12325" title="Downgrade lock mode for DOM files when DOM bit gone" class="issue-link" data-issue-key="LU-12325">LU-12325</a> dom: use LCK_PR with 'trylock' mode</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: fc6661cba7425757ec1a7f5ff5e2f1490fd2edcb</del></p> <p>patch <a href="https://review.whamcloud.com/35031" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35031</a> worked well and it can optimize performance for the following workload.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client1: open-create (zero byte file) client2: stat client1: stat </pre> </div></div> <p>Here is test resutls.<br/> dom_lock=always</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c082 ~]# /work/home/sihara/io-500-dev/bin/mdtest -C -F -d /es90/out -n 10000 -u [root@c082 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.es90-MDT0000-mdc-ffff92bf94e4d800.lock_count=10004 [root@c083 ~]# /work/home/sihara/io-500-dev/bin/mdtest -T -F -d /es90/out -n 10000 -u [root@c082 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.es90-MDT0000-mdc-ffff92bf94e4d800.lock_count=4 [root@c082 ~]# /work/home/sihara/io-500-dev/bin/mdtest -T -F -d /es90/out -n 10000 -u SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 9145.261 9145.261 9145.261 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 </pre> </div></div> <p>dom_lock=never</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c082 ~]# /work/home/sihara/io-500-dev/bin/mdtest -C -F -d /es90/out -n 10000 -u [root@c082 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.es90-MDT0000-mdc-ffff92bf94e4d800.lock_count=10004 [root@c083 ~]# /work/home/sihara/io-500-dev/bin/mdtest -T -F -d /es90/out -n 10000 -u [root@c082 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.es90-MDT0000-mdc-ffff92bf94e4d800.lock_count=10004 &lt;--- c082 can keep locks, but it's still imcompatible for stat [root@c082 ~]# /work/home/sihara/io-500-dev/bin/mdtest -T -F -d /es90/out -n 10000 -u SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 12923.465 12923.465 12923.465 0.000 File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 </pre> </div></div> <p>patch <a href="https://review.whamcloud.com/35031" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35031</a> + dom_lock=trylock</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c082 ~]# /work/home/sihara/io-500-dev/bin/mdtest -C -F -d /es90/out -n 10000 -u [root@c082 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.es90-MDT0000-mdc-ffff92bf94e4d800.lock_count=10004 [root@c083 ~]# /work/home/sihara/io-500-dev/bin/mdtest -T -F -d /es90/out -n 10000 -u [root@c082 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.es90-MDT0000-mdc-ffff92bf94e4d800.lock_count=10004 [root@c082 ~]# /work/home/sihara/io-500-dev/bin/mdtest -T -F -d /es90/out -n 10000 -u SUMMARY rate: (of 1 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 0.000 0.000 0.000 0.000 File stat : 101680.001 101680.001 101680.001 0.000 &lt;-- 10x faster File read : 0.000 0.000 0.000 0.000 File removal : 0.000 0.000 0.000 0.000 Tree creation : 0.000 0.000 0.000 0.000 Tree removal : 0.000 0.000 0.000 0.000 </pre> </div></div> <p>patch <a href="https://review.whamcloud.com/35031" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/35031</a> + dom_lock=trylock works for open-create, stat workload, but locks are still incompatible for "open-write + stat" workload, right?</p> <p>yes, for write we need OPEN to return PW lock, so that is 'always' mode of dom_lock parameter.</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37088" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37088</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12325" title="Downgrade lock mode for DOM files when DOM bit gone" class="issue-link" data-issue-key="LU-12325">LU-12325</a> ldlm: lock convert with mode downgrade<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f872a4cbdaa4f7d52ef8fc9e0cbdcee041e50dad</p> <p>Mike Pershin (mpershin@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37105" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37105</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12325" title="Downgrade lock mode for DOM files when DOM bit gone" class="issue-link" data-issue-key="LU-12325">LU-12325</a> ldlm: mode downgrade, wire changes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 074008cdfc8e3e7efffc30e744896cda0da327d5</p> <p>It might be related, but dom_lock=trylock behaviors chnaged in recent codes?<br/> I'm testing on 2.15.0-RC3.<br/> Dom lock mode "trylock" which is default.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ai400x2-1-vm1 ~]# lctl get_param mdt.*.dom_lock mdt.exafs-MDT0000.dom_lock=trylock </pre> </div></div> <p>Create new directory and configured dom for all &lt; 1MB files.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# mkdir /exafs/md test.out/ [root@ec01 ~]# lfs setdirstrip e -i 0 /exafs/mdtest.out/ [root@ec01 ~]# lfs setstripe -E 1m -L mdt /exafs/mdtest.out </pre> </div></div> <p>Create 10000 files on client c01</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec01 ~]# mpirun -np 1 --allow-run-as-root /work/tools/bin/mdtest -F -d /exafs/mdtest.out/ -n 10000 -u -C Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 5047.056 5047.056 5047.056 0.000 File stat 0.000 0.000 0.000 0.000 File read 0.000 0.000 0.000 0.000 File removal 0.000 0.000 0.000 0.000 Tree creation 900.065 900.065 900.065 0.000 Tree removal 0.000 0.000 0.000 0.000 [root@ec01 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff98e1eefd9800.lock_count=10004 </pre> </div></div> <p>locks were taken for 10000 files which is expected.</p> <p>Another client ec02 issue "stats" to all 10000 files which client ec01 created above.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec02 ~]# mpirun -np 1 --allow-run-as-root /work/tools/bin/mdtest -F -d /exafs/mdtest.out/ -n 10000 -u -T Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation 0.000 0.000 0.000 0.000 File stat 3353.370 3353.370 3353.370 0.000 File read 0.000 0.000 0.000 0.000 File removal 0.000 0.000 0.000 0.000 Tree creation 0.000 0.000 0.000 0.000 Tree removal 0.000 0.000 0.000 0.000 [root@ec01 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff98e1eefd9800.lock_count=4 </pre> </div></div> <p>all locks for 10000 files on ec01 were canceled which is also expected.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ec02 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff9d914c191000.lock_count=30004 </pre> </div></div> <p>But, why it requires 3 x locks against 10000 files for "stats"?</p> <p>This was same scenarios of what I original tested in <a href="https://jira.whamcloud.com/browse/LU-12325" title="Downgrade lock mode for DOM files when DOM bit gone" class="issue-link" data-issue-key="LU-12325">LU-12325</a>, but different results and behaviors..<br/> When I tested before, second client took same number of locks of target files for "stat". But, now it takes 3x.. </p> <p>Hm.. dom_lock=always also changed behaviors</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> [root@ai400x2-1-vm1 ~]# lctl set_param mdt.*.dom_lock=always mdt.exafs-MDT0000.dom_lock=always [root@ec01 ~]# mpirun -np 1 --allow-run-as-root /work/tools/bin/mdtest -F -d /exafs/mdtest.out/ -n 10000 -u -C [root@ec01 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff98e1eefd9800.lock_count=10005 [root@ec01 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff98e1eefd9800.lock_count=10005 [root@ec02 ~]# mpirun -np 1 --allow-run-as-root /work/tools/bin/mdtest -F -d /exafs/mdtest.out/ -n 10000 -u -T [root@ec02 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff9d914c191000.lock_count=30004 [root@ec01 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff98e1eefd9800.lock_count=5 </pre> </div></div> <p>it seems that dom_lock=never didn't change.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ai400x2-1-vm1 ~]# lctl set_param mdt.*.dom_lock=never mdt.exafs-MDT0000.dom_lock=never [root@ec01 ~]# mpirun -np 1 --allow-run-as-root /work/tools/bin/mdtest -F -d /exafs/mdtest.out/ -n 10000 -u -C [root@ec01 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff98e1eefd9800.lock_count=10004 [root@ec02 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff9d914c191000.lock_count=0 [root@ec02 ~]# mpirun -np 1 --allow-run-as-root /work/tools/bin/mdtest -F -d /exafs/mdtest.out/ -n 10000 -u -T [root@ec02 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff9d914c191000.lock_count=10004 [root@ec01 ~]# lctl get_param ldlm.namespaces.*MDT0000*.lock_count ldlm.namespaces.exafs-MDT0000-mdc-ffff98e1eefd9800.lock_count=10004 </pre> </div></div> <p>In the end, dom_lock=never make better performance today and we really need <a href="https://review.whamcloud.com/37088" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37088</a> and <a href="https://review.whamcloud.com/37105" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37105</a> conjunctions with dom_lock=trylock otherwise the performance is worse than dom_lock=never.</p> <p>when it takes 3x locks, could you repeat that with less files (~2-3) and dump locks on client:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl set_param -n ldlm.dump_namespaces 1 lctl dk &gt; locks_dumped.txt</pre> </div></div> <p>and attach <tt>locks_dumped.txt</tt>? I believe these locks differs by inodebits, e.g. some may have XATTR bit and so on. </p> <p>"Mikhail Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51239" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51239</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12325" title="Downgrade lock mode for DOM files when DOM bit gone" class="issue-link" data-issue-key="LU-12325">LU-12325</a> ldlm: mode convert client changes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ca37a7c2ced28d66ca966d73f242b25cf626ff42</p> <p>"Mikhail Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51262" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51262</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12325" title="Downgrade lock mode for DOM files when DOM bit gone" class="issue-link" data-issue-key="LU-12325">LU-12325</a> mdc: reduce DoM lock mode on file close<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 306444123a484729bf6cb435b1c2fb84b6170158</p> Related LU-11428 Development Rank 1|i00gpr: Rank (Obsolete) 9223372036854775807