https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11170 Lustre <p>This issue was created by maloo for James Nunez &lt;james.a.nunez@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/98efeea0-7f0b-11e8-8fe6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/98efeea0-7f0b-11e8-8fe6-52540065bddc</a></p> <p>sanity test 415 fails for DNE with ZFS with the error</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>total: 500 open/close in 0.87 seconds: 572.35 ops/second rename 500 files took 283 sec sanity test_415: @@@@@@ FAIL: rename took 283 sec </pre> </div></div> <p>So far, this test only fails for ZFS. </p> <p>This test started failing on 2018-07-03 with logs at <a href="https://testing.whamcloud.com/test_sets/98efeea0-7f0b-11e8-8fe6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/98efeea0-7f0b-11e8-8fe6-52540065bddc</a></p> <p>Other test failures at<br/> <a href="https://testing.whamcloud.com/test_sets/8de6a208-8945-11e8-9028-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8de6a208-8945-11e8-9028-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/9bb5e252-8e9c-11e8-b0aa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9bb5e252-8e9c-11e8-b0aa-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/029c73ea-8e9e-11e8-87f3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/029c73ea-8e9e-11e8-87f3-52540065bddc</a></p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_415 - rename took 283 sec<br/> sanity test_415 - rename took 154 &gt; 125 sec</p> LU-11170

sanity test 415 fails with 'rename took N > M sec'

Bug Critical Reopened Unresolved Vitaliy Kuznetsov Maloo DNE always_except zfs Tue, 24 Jul 2018 18:34:40 +0000 Fri, 12 May 2023 07:19:41 +0000 Lustre 2.12.0 Lustre 2.13.0 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.6 Lustre 2.15.0 0 18 <p>This is caused by <a href="https://jira.whamcloud.com/browse/LU-11102" title="lock revoke may not take effect" class="issue-link" data-issue-key="LU-11102"><del>LU-11102</del></a>, but the patch <a href="https://review.whamcloud.com/#/c/32738/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32738/</a> is not landed yet. </p> <p>As Lai said, this test is failing only for the patch that adds it. So, this ticket was opened prematurely and closing the ticket. </p> <p>Reopening to test out async ZFS patches.</p> <p>We should consider to disable sync on ZFS using the tunables from patch <a href="https://review.whamcloud.com/7761" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/7761</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10460" title="Evaluate two LLNL patches for upstream" class="issue-link" data-issue-key="LU-10460"><del>LU-10460</del></a> osd-zfs: Add tunables to disable sync</tt>":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Two new tunables are introduced to control the delay, the osd_object_sync_delay_us and osd_txg_sync_delay_us module options. These values default to -1 which preserves the safe full sync pool behavior. Setting these values to zero or larger will replace the pool sync with a delay of N microseconds. </pre> </div></div> <p>We should set <tt>osd-zfs.&#42;.osd_object_sync_delay_us=0</tt> during this test to see if it reduces the runtime to a reasonable level.</p> <p>It may be worthwhile to submit a separate patch to set this in test-framework.sh for all ZFS targets to see if it significantly reduces the overall test time. We <em>should</em> be able to handle this within the test framework, since most failover testing that we do will sync before any recovery barrier that we impose, and a regular unmount will still flush any uncommitted changes to disk.</p> <p>In the past week, sanity test 415 fails 100% of the time for review-dne-zfs-part-1. </p> <p>This test does not fail for review-ldiskfs, all review-dne-part, nor for review-zfs. </p> <p>We should land <a href="https://review.whamcloud.com/#/c/32933" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/32933</a> .</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/32957" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32957</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> tests: add delay sync for sanity test 415<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d5ec750ab15e8fe7d051a32bb486c358ce2970a1</p> <p>Failing patches will need to be rebased to get the fix for this issue.</p> <p>For some reason, we can't find the osd_object_sync_delay_us parameter on the MDS. In the test log <a href="https://testing.whamcloud.com/test_sets/1f79aa3c-9c04-11e8-b0aa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1f79aa3c-9c04-11e8-b0aa-52540065bddc</a>, we see:</p> <p>CMD: onyx-45vm9 /usr/sbin/lctl get_param -n osd-zfs.*.osd_object_sync_delay_us<br/> onyx-45vm9: error: get_param: param_path 'osd-zfs/*/osd_object_sync_delay_us': No such file or directory<br/> CMD: onyx-45vm10,onyx-45vm9 /usr/sbin/lctl set_param osd-zfs.*.osd_object_sync_delay_us=0<br/> onyx-45vm9: error: set_param: param_path 'osd-zfs/*/osd_object_sync_delay_us': No such file or directory<br/> onyx-45vm10: error: set_param: param_path 'osd-zfs/*/osd_object_sync_delay_us': No such file or directory</p> <p>This parameter landed to master, <a href="https://review.whamcloud.com/#/c/7761/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/7761/</a>, in January and thus, should be seen in this release, 2.11.53.54.</p> <p>+1 on master branch: <a href="https://testing.whamcloud.com/test_sets/7eaab0f8-c5db-11e9-98c8-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7eaab0f8-c5db-11e9-98c8-52540065bddc</a></p> <p>There were 35 failures last week. This is blocking patch review testing on master branch.</p> <blockquote> <p>For some reason, we can't find the osd_object_sync_delay_us parameter on the MDS. In the test log <a href="https://testing.whamcloud.com/test_sets/1f79aa3c-9c04-11e8-b0aa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1f79aa3c-9c04-11e8-b0aa-52540065bddc</a>, we see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-45vm9 /usr/sbin/lctl get_param -n osd-zfs.*.osd_object_sync_delay_us onyx-45vm9: error: get_param: param_path 'osd-zfs/*/osd_object_sync_delay_us': No such file or directory CMD: onyx-45vm10,onyx-45vm9 /usr/sbin/lctl set_param osd-zfs.*.osd_object_sync_delay_us=0 onyx-45vm9: error: set_param: param_path 'osd-zfs/*/osd_object_sync_delay_us': No such file or directory onyx-45vm10: error: set_param: param_path 'osd-zfs/*/osd_object_sync_delay_us': No such file or directory </pre> </div></div> <p>This parameter landed to master, <a href="https://review.whamcloud.com/#/c/7761/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/7761/</a>, in January and thus, should be seen in this release, 2.11.53.54.</p></blockquote> <p>It appears that <tt>osd_object_sync_delay_us</tt> was not added as a Lustre parameter, but rather as a module parameter. So instead of using "<tt>lctl set_param</tt>" to set this, it is necessary to use something like "<tt>echo 0 &gt; /sys/modules/osd-zfs/parameters/ osd_object_sync_delay_us</tt>" to set it.</p> <p>Nathaniel</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>James Nunez (jnunez@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36157" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36157</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> tests: stop running sanity 415 for ZFS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 68d374f4f914403d27712acbee9bf8293a2a3731</p> <p>Nathaniel Clark (nclark@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36395" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36395</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> test: Skip sanity/415 for ZFS 0.8.0+<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e3584f574441185dac6cbd0f306597a6713ede26</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sessions/24c45246-045b-4744-81ac-7ebdc79bcf0c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/24c45246-045b-4744-81ac-7ebdc79bcf0c</a></p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/36463" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36463</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> test: try zfs tunable<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 016f2edb1e00232ddd7366d6a54482b2b5025c29</p> <p>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/36491" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36491</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> osd-zfs: test ZFS version update to 0.8.2<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3c59355d08d0aa7a64e1caf534567cd896966dc2</p> <p><del>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch:</del> <a href="https://review.whamcloud.com/36525" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36525</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> osd-zfs: gather data ZFS version update to 0.8.2</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 8f04a216c726881a64887dce3c18d7124c3e90d1</del></p> <p><del>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch:</del> <a href="https://review.whamcloud.com/36742" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36742</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> osd-zfs: test ZFS versino update to 0.8.2</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: de47eb0b013af40a36505a331be912c235bf8a62</del></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/2c84fba2-0fce-11ea-98f1-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2c84fba2-0fce-11ea-98f1-52540065bddc</a></p> <p>More failure instances on master branch:<br/> <a href="https://testing.whamcloud.com/test_sets/86e61e30-4121-11ea-ac52-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/86e61e30-4121-11ea-ac52-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/dcdc6f56-40b8-11ea-83b6-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/dcdc6f56-40b8-11ea-83b6-52540065bddc</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/df8737c2-e377-4dac-9fd7-f0472a3927ad" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/df8737c2-e377-4dac-9fd7-f0472a3927ad</a></p> <p>A recent failure for review-dne-zfs-part-1 for el8.2 server and client at <a href="https://testing.whamcloud.com/test_sets/e788cbb4-2b21-473d-a127-64dcbc8ceec8" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/e788cbb4-2b21-473d-a127-64dcbc8ceec8</a> . </p> <p>I'm seeing this issue for ldiskfs testing; <a href="https://testing.whamcloud.com/test_sets/9e51c284-86f5-4042-86b1-2436e537e34f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9e51c284-86f5-4042-86b1-2436e537e34f</a> and <a href="https://testing.whamcloud.com/test_sets/2febebf4-9666-4f7f-a633-8afc6aeebf22" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2febebf4-9666-4f7f-a633-8afc6aeebf22</a> . Since this ticket is/was for ZFS, should I create a new ticket for the ldiskfs failures?</p> <p>+1 on b2_14: <a href="https://testing.whamcloud.com/test_sets/034e37f7-45d5-4a95-b720-46540caa8ccd" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/034e37f7-45d5-4a95-b720-46540caa8ccd</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/41f2f737-0a6f-438d-b764-ef446639e144" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/41f2f737-0a6f-438d-b764-ef446639e144</a></p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sessions/9af65bc1-a8a0-4805-acde-0f77e8d8abce" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/9af65bc1-a8a0-4805-acde-0f77e8d8abce</a></p> <p>+1 on master:</p> <p><a href="https://testing.whamcloud.com/test_sets/8a693eb7-f6c9-4e44-b663-58a5297fb9b8" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8a693eb7-f6c9-4e44-b663-58a5297fb9b8</a></p> <p>+1 <a href="https://testing.whamcloud.com/test_sets/5999564a-11d5-4543-bffd-74ea8cd2ba60" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5999564a-11d5-4543-bffd-74ea8cd2ba60</a></p> <p>+1 <a href="https://testing.whamcloud.com/test_sets/1f9a4f4e-e6cd-4029-bf4f-5a1b6c144bd1" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/1f9a4f4e-e6cd-4029-bf4f-5a1b6c144bd1</a></p> <p>note, it is not ZFS</p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/993a9a55-1a43-4a47-a976-271c19135ca0" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/993a9a55-1a43-4a47-a976-271c19135ca0</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/03a97eca-b958-4f73-9383-ab34d6143360" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/03a97eca-b958-4f73-9383-ab34d6143360</a></p> <p>+1 on master <a href="https://testing.whamcloud.com/test_sets/8ecb13cd-7b09-4f2b-a410-e3c20eb5c244" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8ecb13cd-7b09-4f2b-a410-e3c20eb5c244</a></p> <p>"John L. Hammond &lt;jhammond@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47341" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47341</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> test: increase time limit in sanity test_415<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e0700d52edf88f2146a1b5380970e6c2cfc55573</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/3adb9c61-eb90-4af3-9e2c-c18365f5dd36" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3adb9c61-eb90-4af3-9e2c-c18365f5dd36</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47341/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47341/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> test: increase time limit in sanity test_415<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b974d7d8a9759601b6f851895ee32714ccf56f28</p> <p>Tried to avoid again</p> <p>+1 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/2e6c4a29-6de8-4e85-bc7c-a986208af412" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2e6c4a29-6de8-4e85-bc7c-a986208af412</a></p> <p>on 2.15 aarch64 client<br/> <a href="https://testing.whamcloud.com/test_sets/771da1f7-4616-4a2a-b3fc-d0028e98a485" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/771da1f7-4616-4a2a-b3fc-d0028e98a485</a></p> <p>master</p> <p><a href="https://testing.whamcloud.com/test_sets/8ff591d9-c8e4-43ab-84d3-357f79417e08" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/8ff591d9-c8e4-43ab-84d3-357f79417e08</a></p> <p>Still hitting this about 1-3% of runs during normal testing. Failures usually complete within 140s, one as long as 269s.</p> <p>The fact that ZFS is also very slow means that the failures may actually be caused by a regression in the COS code, because commits on ZFS can take a long time. I'm going to push a patch to improve the debugability of the patch, hopefully we can identify the source of the problem.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49724" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49724</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> tests: add debugging to sanity/415<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e33e8906febd161a89bb190648e74dcc3b7414cb</p> <p>+1 master</p> <p><a href="https://testing.whamcloud.com/test_sets/55564f7c-ab54-4536-a2e5-1ec9777a2363" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/55564f7c-ab54-4536-a2e5-1ec9777a2363</a></p> <p>+1 on master:<br/> <a href="https://testing.whamcloud.com/test_sets/c37adf87-d38a-4a38-afa4-effadab56dd5" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/c37adf87-d38a-4a38-afa4-effadab56dd5</a></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49724/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49724/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> tests: add debugging to sanity/415<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6594babc73851fab335c514cd1fee018425e7bb3</p> <p>+1 on master: <a href="https://testing.whamcloud.com/test_sets/d8e1c14f-849d-46a5-b422-c77955590561" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/d8e1c14f-849d-46a5-b422-c77955590561</a></p> <p>With the added debugging and changes from my previous patch, it looks like this issue is not <b>only</b> because some VMs do renames slower than others. <a href="https://testing.whamcloud.com/search?test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=a765d5fa-7e82-11e8-acb8-52540065bddc&amp;start_date=2023-03-28&amp;end_date=2023-04-04&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">Some tests</a> pass after almost 500s, while others fail in 130s.</p> <p>It seems possible that there is variable VM performance between the "uncontended" rename test run and the "contended" rename test run that is causing failures. Looking at <a href="https://testing.whamcloud.com/search?horizon=172800&amp;test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&amp;sub_test_script_id=a765d5fa-7e82-11e8-acb8-52540065bddc&amp;source=sub_tests#redirect" class="external-link" target="_blank" rel="nofollow noopener">recent test results in Maloo</a>, most of the subtest runs take between 20-50s to finish, but some of the passing "uncontested" runs take longer to complete than some of the failures, which is due to VM performance and/or external contention:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rename 500 files without 'touch' took 173 sec rename 500 files with 'touch' took 302 sec : rename 500 files without 'touch' took 181 sec rename 500 files with 'touch' took 287 sec : rename 500 files without 'touch' took 137 sec rename 500 files with 'touch' took 204 sec : rename 500 files without 'touch' took 58 sec rename 500 files with 'touch' took 77 sec </pre> </div></div> <p>The main question that this test should be answering is if a <b>single</b> contended rename is slow and blocked due to DLM contention? Or are all of the contended renames about 2.2x slower than the uncontended renames?</p> <p>I was hoping there was a way to make this test more robust by using the rename stats to see if there are "long pole" renames where max &gt;&gt; average that show a test failure, but even that doesn't appear to be very reliable. In my local (single-VM, on otherwise idle system) testing an <b>uncontended</b> rename I see times with <tt>max &gt; 10x min</tt>, and <tt>max &gt; 4x avg</tt> (stats columns are "<tt>stat, count, 'samples', <span class="error">&#91;unit&#93;</span>, min, max, sum</tt>", so <tt>avg=sum/100</tt> can be easily calculated visually):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mkdir /mnt/testfs/dir; touch /mnt/testfs/dir/f{1..100} # lctl set_param llite.*.stats=clear; (cd /mnt/testfs/dir; rename f g f[1-9]*); lctl get_param llite.*.stats | grep -w rename rename 100 samples [usec] 1289 15669 305488 rename 100 samples [usec] 1194 17080 443209 rename 100 samples [usec] 1393 17190 445666 </pre> </div></div> <p>Comparatively, running this on my real client+server hardware shows much closer times between min/max/avg, with <tt>max=1.1-1.3x avg</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rename 100 samples [usec] 556 839 65490 43136492 rename 100 samples [usec] 466 849 68130 46780816 rename 100 samples [usec] 591 826 68434 47090098 rename 100 samples [usec] 633 801 72962 53361444 rename 100 samples [usec] 615 816 73353 53967167 </pre> </div></div> <p>Lai, since you wrote patch <a href="https://review.whamcloud.com/32738" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32738</a> and this test originally, do you have any suggestions for Vitaliy on how to make it more reliable (e.g. grep something from the client or MDS debug logs to detect DLM lock cancel/contention)?</p> <p>Otherwise, if you don't have any immediate suggestions, I don't think that this test can produce reliable results when run in a VM, and should be changed to use <tt>error_not_in_vm</tt> instead of <tt>error</tt>.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50654" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50654</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> tests: don't fail sanity/415 in VM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 01b1f0ec789313c077e3a6aff64ef145efc4d9f4</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50654/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50654/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11170" title="sanity test 415 fails with &#39;rename took N &gt; M sec&#39;" class="issue-link" data-issue-key="LU-11170">LU-11170</a> tests: don't fail sanity/415 in VM<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 73a7b1c2a3f0114618db7781adb56974ed682f24</p> Blocker LU-12336 Duplicate LU-11225 LU-12697 LU-14103 LU-16237 LU-12843 Related LU-11102 Development Rank 1|hzzzpj: Rank (Obsolete) 9223372036854775807 Severity