https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7117 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;andreas.dilger@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/56e1b56e-53ff-11e5-8f2c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/56e1b56e-53ff-11e5-8f2c-5254006e85c2</a>.</p> <p>The sub-test test_70d failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>error on LL_IOC_LMV_SETSTRIPE '/mnt/lustre/d70d.replay-single/test1' (3): stripe already set error: mkdir: create stripe dir '/mnt/lustre/d70d.replay-single/test1' failed mkdir fails /usr/lib64/lustre/tests/replay-single.sh: line 2236: kill: (25189) - No such process mkdir/rmdir 25189 stopped </pre> </div></div> <p>There are several test failures and timeouts for 70d since 2015-09-02 so I suspect a patch landed on that day or the previous day that introduced a regression. </p> <p>Info required for matching: replay-single 70d</p> LU-7117

replay-single test_70d: timeout and mkdir/rmdir stopped

Bug Critical Resolved Fixed Lai Siyao Maloo Tue, 8 Sep 2015 18:15:13 +0000 Thu, 21 Sep 2017 06:57:05 +0000 Mon, 15 Aug 2016 22:21:10 +0000 Lustre 2.8.0 Lustre 2.9.0 Lustre 2.9.0 0 18 <p>Another failure of replay-single test_70d with logs at <a href="https://testing.hpdd.intel.com/test_sets/e4bc4b96-6cb0-11e5-ab7f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e4bc4b96-6cb0-11e5-ab7f-5254006e85c2</a><br/> 2015-10-26 15:56:26 - <a href="https://testing.hpdd.intel.com/test_sets/0f6e4b5c-7c3b-11e5-bc82-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0f6e4b5c-7c3b-11e5-bc82-5254006e85c2</a><br/> 2015-11-04 01:58:10 - <a href="https://testing.hpdd.intel.com/test_sets/40bafc72-82e0-11e5-8b6b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/40bafc72-82e0-11e5-8b6b-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/65711b06-8857-11e5-84a2-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/65711b06-8857-11e5-84a2-5254006e85c2</a></p> <p>Another failure of stopping the mkdir/rmdir process, but this one takes place inside the random_fail_mdt() routine. I suspect the cause of the failure is the same since logs are the similar:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>shadow-17vm9: CMD: shadow-17vm9.shadow.whamcloud.com lctl get_param -n at_max shadow-17vm10: CMD: shadow-17vm10.shadow.whamcloud.com lctl get_param -n at_max touch: cannot touch `/mnt/lustre/d70d.replay-single/test1/a': No such file or directory touch fails shadow-17vm9: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 6 sec shadow-17vm10: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 6 sec /usr/lib64/lustre/tests/replay-single.sh: line 2114: kill: (16877) - No such process </pre> </div></div> <p>Logs at<br/> 2015-10-27 14:12:35 - <a href="https://testing.hpdd.intel.com/test_sets/1637670e-7cf7-11e5-bceb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1637670e-7cf7-11e5-bceb-5254006e85c2</a><br/> 2015-11-12 02:07:27 - <a href="https://testing.hpdd.intel.com/test_sets/f8324f14-8924-11e5-8ba4-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f8324f14-8924-11e5-8ba4-5254006e85c2</a></p> <p>More instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/5fc600d8-9836-11e5-8fa3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5fc600d8-9836-11e5-8fa3-5254006e85c2</a></p> <p>Another failure on master with the 'No such file or directory' error:<br/> 2015-12-27 14:00:02 - <a href="https://testing.hpdd.intel.com/test_sets/6af615f6-acea-11e5-9134-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6af615f6-acea-11e5-9134-5254006e85c2</a><br/> 2016-01-07 03:11:56 - <a href="https://testing.hpdd.intel.com/test_sets/2f511064-b52d-11e5-aa1f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2f511064-b52d-11e5-aa1f-5254006e85c2</a><br/> 2016-01-12 13:29:58 - <a href="https://testing.hpdd.intel.com/test_sets/7aeee7ae-b97d-11e5-825c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7aeee7ae-b97d-11e5-825c-5254006e85c2</a><br/> 2016-02-03 10:29:58 - <a href="https://testing.hpdd.intel.com/test_sets/7cf30a58-caa8-11e5-a610-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7cf30a58-caa8-11e5-a610-5254006e85c2</a></p> <p>More failure instance on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/48380772-c66c-11e5-8cac-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/48380772-c66c-11e5-8cac-5254006e85c2</a></p> <p>All of the instances occurred with DNE configuration. Patch review testing on master branch is affected by this failure.</p> <p>another master branch failure, during review-dne-part-2:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/0a65754c-dd7d-11e5-ab2a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0a65754c-dd7d-11e5-ab2a-5254006e85c2</a></p> <p>Occurred again on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/4d2a6bf2-fb4c-11e5-acc0-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4d2a6bf2-fb4c-11e5-acc0-5254006e85c2</a></p> <p>Another failure on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/9bbba39e-0326-11e6-b5f1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9bbba39e-0326-11e6-b5f1-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/21698126-0fb3-11e6-9b34-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/21698126-0fb3-11e6-9b34-5254006e85c2</a></p> <p>Another failure instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/86f8d552-1c04-11e6-b5f1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/86f8d552-1c04-11e6-b5f1-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/59ca4542-1d4a-11e6-9089-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/59ca4542-1d4a-11e6-9089-5254006e85c2</a></p> <p>Another failure instance on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/e17c313a-3096-11e6-acf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e17c313a-3096-11e6-acf3-5254006e85c2</a></p> <p>This is now the number one cause of review test failures on master.</p> <p>More failure instances on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/478fbade-3180-11e6-bbf5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/478fbade-3180-11e6-bbf5-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/9d99755c-316f-11e6-a0ce-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9d99755c-316f-11e6-a0ce-5254006e85c2</a></p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20853" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20853</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7117" title="replay-single test_70d: timeout and mkdir/rmdir stopped" class="issue-link" data-issue-key="LU-7117"><del>LU-7117</del></a> tests: enable full debug for reply-single test_70d<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 61ec7fa12d28daffa43bc7a4f0e927d4e6550061</p> <p>After the painful debugging from millions of lines logs, I found that the failure should related with the following scenario:<br/> 1) The client sent unlink (PFID/test1) RPC to the MDT1;<br/> 2) The MDT0 failover;<br/> 3) The MDT1 is replaying with the MDT0;<br/> 4) At that time, the step 1) trigger lookup(test1) RPC from the MDT1 to the MDT0 for the test1's FID.<br/> 5) But because the MDT0 recovery is not completed yet, the test1's FID on the MDT0 is still the old one. That means the MDT0 reply the MDT1 with an invalid test1's FID.<br/> 6) The MDT1 uses the MDT0 given test1's FID to locate the target child object to be deleted, found nothing. Then the unlink operation failed.</p> <p>no new RPCS (except update log redo) should be sent until the recovery is completed?</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/20940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20940</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7117" title="replay-single test_70d: timeout and mkdir/rmdir stopped" class="issue-link" data-issue-key="LU-7117"><del>LU-7117</del></a> osp: control RPC to be sent when recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5fb68c49af3fa08189110a4e980ad792efd7128b</p> <blockquote> <p>no new RPCS (except update log redo) should be sent until the recovery is completed?</p></blockquote> <p>The MDT0 is in recovery, but MDT1 is normal, so the RPC from the client to MDT1 is not blocked.</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/6a54094a-3a9e-11e6-a0ce-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6a54094a-3a9e-11e6-a0ce-5254006e85c2</a></p> <p>Lai Siyao (lai.siyao@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21064" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21064</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7117" title="replay-single test_70d: timeout and mkdir/rmdir stopped" class="issue-link" data-issue-key="LU-7117"><del>LU-7117</del></a> mdt: mdt unlink should lock before lookup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0ae3d50ed2745366f677b79586f3fe72645330ee</p> <p>Another occurrence:<br/> <a href="https://testing.hpdd.intel.com/test_sets/823971fa-433e-11e6-acf3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/823971fa-433e-11e6-acf3-5254006e85c2</a></p> <p>+1 on master at <a href="https://testing.hpdd.intel.com/test_sets/79710324-49a3-11e6-a80f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/79710324-49a3-11e6-a80f-5254006e85c2</a></p> <p>A lot more occurrences recently, like this one on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/4d6f9aea-4a09-11e6-8968-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4d6f9aea-4a09-11e6-8968-5254006e85c2</a></p> <p>How do the patches <a href="http://review.whamcloud.com/20940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20940</a> "<a href="https://jira.whamcloud.com/browse/LU-7117" title="replay-single test_70d: timeout and mkdir/rmdir stopped" class="issue-link" data-issue-key="LU-7117"><del>LU-7117</del></a> osp: control RPC to be sent when recovery" and <a href="http://review.whamcloud.com/21064" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21064</a> "<a href="https://jira.whamcloud.com/browse/LU-7117" title="replay-single test_70d: timeout and mkdir/rmdir stopped" class="issue-link" data-issue-key="LU-7117"><del>LU-7117</del></a> mdt: mdt unlink should lock before lookup" relate to each other? Are they both needed? Are they different approaches to fixing the same problem, and only one is needed?</p> <p>Never mind, I see that <a href="http://review.whamcloud.com/21064" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21064</a> "<a href="https://jira.whamcloud.com/browse/LU-7117" title="replay-single test_70d: timeout and mkdir/rmdir stopped" class="issue-link" data-issue-key="LU-7117"><del>LU-7117</del></a> mdt: mdt unlink should lock before lookup" is abandoned since it was landed as <a href="http://review.whamcloud.com/21088" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21088</a> via <a href="https://jira.whamcloud.com/browse/LU-8353" title="mdt unlink should lock parent before lookup child" class="issue-link" data-issue-key="LU-8353"><del>LU-8353</del></a>.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/20940/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20940/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7117" title="replay-single test_70d: timeout and mkdir/rmdir stopped" class="issue-link" data-issue-key="LU-7117"><del>LU-7117</del></a> osp: set ptlrpc_request::rq_allow_replay properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e3d507eec50fc1ff79acf2a9f93d52d698c887d7</p> <p>Landed for 2.9</p> Duplicate LU-7213 LU-7924 Related LU-7172 LU-7775 LU-6844 LU-8353 LU-8617 Development Rank 1|hzxmvj: Rank (Obsolete) 9223372036854775807 Severity