https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6802 Lustre <p>sanity test 208 is failing occasionally. From the test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>==== test 5: lease broken can't be regained by replay multiop: apply lease error: Device or resource busy Failing mds1 on shadow-17vm12 … shadow-17vm2: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec shadow-17vm5: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 9 sec /usr/lib64/lustre/tests/sanity.sh: line 11648: kill: (19839) - No such process sanity test_208: @@@@@@ FAIL: lease not broken over recovery </pre> </div></div> <p>Test logs for this failure and past failures are at:<br/> 2015-07-03 01:45:25 - <a href="https://testing.hpdd.intel.com/test_sets/0965361e-213f-11e5-a388-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/0965361e-213f-11e5-a388-5254006e85c2</a><br/> 2015-07-03 03:06:42 - <a href="https://testing.hpdd.intel.com/test_sets/a04d4718-214b-11e5-bb73-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a04d4718-214b-11e5-bb73-5254006e85c2</a><br/> 2015-07-03 16:23:00 - <a href="https://testing.hpdd.intel.com/test_sets/38d9e8f0-21de-11e5-a388-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/38d9e8f0-21de-11e5-a388-5254006e85c2</a><br/> 2015-07-03 18:18:18 - <a href="https://testing.hpdd.intel.com/test_sets/b163f5f2-21ef-11e5-a388-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b163f5f2-21ef-11e5-a388-5254006e85c2</a><br/> 2015-07-04 19:45:45 - <a href="https://testing.hpdd.intel.com/test_sets/2e5a5992-2372-11e5-aced-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2e5a5992-2372-11e5-aced-5254006e85c2</a><br/> 2015-07-05 12:09:44 - <a href="https://testing.hpdd.intel.com/test_sets/86e51cfa-234a-11e5-9feb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/86e51cfa-234a-11e5-9feb-5254006e85c2</a><br/> 2015-07-05 15:22:59 - <a href="https://testing.hpdd.intel.com/test_sets/a26fb9a2-2341-11e5-aced-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a26fb9a2-2341-11e5-aced-5254006e85c2</a><br/> 2015-07-05 15:58:01 - <a href="https://testing.hpdd.intel.com/test_sets/04aecbb8-236e-11e5-9feb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/04aecbb8-236e-11e5-9feb-5254006e85c2</a></p> Autotest LU-6802

sanity test_208 fail: “lease not broken over recovery"

Bug Critical Resolved Fixed Jinshan Xiong James Nunez Mon, 6 Jul 2015 16:51:01 +0000 Fri, 8 Sep 2017 13:17:56 +0000 Fri, 4 Dec 2015 18:29:46 +0000 Lustre 2.8.0 Lustre 2.8.0 0 13 <p>Lai</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hit this three times while testing two patches <a href="http://review.whamcloud.com/14911" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14911</a> and <a href="http://review.whamcloud.com/14953" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14953</a> .</p> <p>sanity 208 failed on different subtest, and it often failed with <del>EBUSY (lockrep</del>&gt;lock_policy_res2 is -EBUSY) in ll_lease_open -&gt; md_intent_lock -&gt; mdc_finish_enqueue, while this value should be 0 since MDS doesn't set it for lease open.</p> <p>I need more time to review related code.</p> <p>The log shows this -EBUSY comes from the end of mdt_object_open_lock():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (atomic_read(&amp;obj-&gt;mot_open_count) &gt; open_count) GOTO(out, rc = -EBUSY); </pre> </div></div> <p>I don't understand why this failure is random, and I don't see anything special before subtest 5.</p> <p>Jinshan, do you have idea?</p> <p>Some recent failures on master in review-dne-part-1:<br/> 2015-07-14 09:41:51 - <a href="https://testing.hpdd.intel.com/test_sets/7eb0672e-2a4d-11e5-a37e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/7eb0672e-2a4d-11e5-a37e-5254006e85c2</a><br/> 2015-07-14 11:05:48 - <a href="https://testing.hpdd.intel.com/test_sets/8bd84aa2-2a5c-11e5-a37e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8bd84aa2-2a5c-11e5-a37e-5254006e85c2</a><br/> 2015-07-14 11:47:39 - <a href="https://testing.hpdd.intel.com/test_sets/13d7bb3e-2a5f-11e5-ba1d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/13d7bb3e-2a5f-11e5-ba1d-5254006e85c2</a><br/> 2015-07-14 12:43:45 - <a href="https://testing.hpdd.intel.com/test_sets/32ab3cf8-2a68-11e5-a37e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/32ab3cf8-2a68-11e5-a37e-5254006e85c2</a><br/> 2015-07-14 15:43:16 - <a href="https://testing.hpdd.intel.com/test_sets/229bd900-2a7f-11e5-a37e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/229bd900-2a7f-11e5-a37e-5254006e85c2</a><br/> 2015-07-14 20:03:58 - <a href="https://testing.hpdd.intel.com/test_sets/c3304c60-2ac1-11e5-be51-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c3304c60-2ac1-11e5-be51-5254006e85c2</a></p> <p>Hi Lai,</p> <p>The lease failed to apply in subtest 5 so it has probably has nothing to do with this test. It's more like that the file was not really closed in subtest 4. Would you please take a further look to verify it?</p> <p>so far as I see from the log, the file was closed in subtest 4. I'll take a deeper look.</p> <p>Hi Jinshan. Can you look at this issue which his blocking landings?</p> <p>Lai is off next week so he won't be able to work on it.</p> <p>Hi James,</p> <p>Can you please add the log option of 'INODE' on the MDT side when you're running the test? Just as:</p> <p>lctl set_param debug+=inode</p> <p>Here is a for-test-only patch trying to reproduce the failure and gather debug logs with "PTLDEBUG=+inode" in the test parameters: <a href="http://review.whamcloud.com/15649" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15649</a></p> <p>Hi Jinshan,</p> <p>The failure was reproduced by only running sanity test 208: <a href="https://testing.hpdd.intel.com/test_sessions/35e878d2-2f3a-11e5-92dd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/35e878d2-2f3a-11e5-92dd-5254006e85c2</a><br/> Please check the debug logs. Thanks.</p> <p>This turned out to be a recovery problem - though I don't understand why we can only see this problem recently.</p> <p>in ptlrpc_replay_next(), it moves replay cursor before a request is replayed successfully.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* The last request on committed_list hasn't been replayed */</span> <span class="code-keyword">if</span> (req-&gt;rq_transno &gt; last_transno) { /* Since the imp_committed_list is immutable before * all of it<span class="code-quote">'s requests being replayed, it'</span>s safe to * use a cursor to accelerate the search */ imp-&gt;imp_replay_cursor = imp-&gt;imp_replay_cursor-&gt;next; </pre> </div></div> <p>If that request failed to replay, it may not have a chance to be replayed again.</p> <p>In this test, there are two open requests needed replaying, but the first one failed due to ptlrpc timeout, then client reconnected and skipped the first open, this caused an open request was leaked, and then caused lease open failed.</p> <p>It should reset cursor of import every time before replay starts.</p> <p>Jinshan Xiong (jinshan.xiong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15669" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15669</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6802" title="sanity test_208 fail: “lease not broken over recovery&quot;" class="issue-link" data-issue-key="LU-6802"><del>LU-6802</del></a> ptlrpc: reset replay cursor on reconnection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7fdf6c195899388e87d66a2a84757aefedcac636</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15669/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15669/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6802" title="sanity test_208 fail: “lease not broken over recovery&quot;" class="issue-link" data-issue-key="LU-6802"><del>LU-6802</del></a> ptlrpc: reset replay cursor on reconnection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c9f9e9c54f5284093b689dd0d7fa8baf5a5b2c97</p> <p>I saw the fix when I'm working on another ticket, looks the fix is incorrect:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* LU-6802: Reset replay cursor on successful reconnect */</span> spin_lock(&amp;imp-&gt;imp_lock); imp-&gt;imp_replay_cursor = &amp;imp-&gt;imp_committed_list; imp-&gt;imp_last_replay_transno = 0; spin_unlock(&amp;imp-&gt;imp_lock); </pre> </div></div> <p>We shouldn't reset the imp_last_replay_transno on each reconnect, otherwise, the request which has been replayed but not committed will be replayed again after reconnect. Furthermore, reset the imp_replay_cursor on each reconnect isn't optimal, we'd just move one step back for such case. (after reconnect, if imp_replay_cursor != imp_committed_list, set imp_replay_cursor to imp_replay_cursor-&gt;prev).</p> <p>Niu Yawei (yawei.niu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15871" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15871</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6802" title="sanity test_208 fail: “lease not broken over recovery&quot;" class="issue-link" data-issue-key="LU-6802"><del>LU-6802</del></a> recovery: don't skip open replay on reconnect<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2e3a7253ae8537ce35916a218fddf7b861469aef</p> <p>indeed, this can be optimized. The previous patch is logically correct because a) replaying a replayed request should be okay and target must handle this case and b) it's not common to reconnect during replay. Actually this bug has been existing for long time but never hit until now, it would be more interesting in investigating why reconnecting during replay became so common recently.</p> <p>Not sure why the test didn't fail before, but replaying a request twice looks not okay to me, at least the second execution will fail and emit an error. </p> <p>it will reconstruct a reply <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <blockquote> <p>it will reconstruct a reply</p></blockquote> <p>Speaking of resend, the original fix will break resend replay, because it doesn't start replay from the last timedout one on reconnect, so resend flag will be set on wrong request. (actually, on server side, there can be only one reply data available, which is for the last timedout replay, replay data for all other replays must have been freed).</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15871/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15871/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6802" title="sanity test_208 fail: “lease not broken over recovery&quot;" class="issue-link" data-issue-key="LU-6802"><del>LU-6802</del></a> recovery: don't skip open replay on reconnect<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 76040fdbbb739053695386e7ed6d0dcb1ea7539a</p> <p>Landed for 2.8</p> <p>Should the check made in the patch be made every time the imp_replay_cursor is advanced?</p> <p>Hongchao Zhang (hongchao.zhang@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17351" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17351</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6802" title="sanity test_208 fail: “lease not broken over recovery&quot;" class="issue-link" data-issue-key="LU-6802"><del>LU-6802</del></a> ptlrpc: reset imp_replay_cursor<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 24efe8099b751714c6a9bd89745fc6b30b2f542b</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17351/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17351/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6802" title="sanity test_208 fail: “lease not broken over recovery&quot;" class="issue-link" data-issue-key="LU-6802"><del>LU-6802</del></a> ptlrpc: reset imp_replay_cursor<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 78335a98654f7c52dbe820cf575d2f9f87a6f595</p> <p>Landed for 2.8</p> Duplicate LU-6596 Related LU-8459 Development Rank 1|hzxhe7: Rank (Obsolete) 9223372036854775807 Severity