Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-15086] replay-dual: test_10: @@@@@@ FAIL: test_10 failed with 2 https://jira.whamcloud.com/browse/LU-15086 Lustre <p>This issue was created by maloo for Vladimir Saveliev <vlaidimir.saveliev@hpe.com></p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/2f1edc30-7ddf-4a6c-bc7a-fffbf58435b8" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2f1edc30-7ddf-4a6c-bc7a-fffbf58435b8</a></p> <p>This is relatively regular failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Error: 'test_10 failed with 2' Failure Rate: 6.00% of most recent 100 runs, 0 skipped (all branches) </pre> </div></div> LU-15086

replay-dual: test_10: @@@@@@ FAIL: test_10 failed with 2

Bug Minor Resolved Fixed Alex Zhuravlev Maloo Tue, 12 Oct 2021 15:42:55 +0000 Sat, 20 Nov 2021 14:49:22 +0000 Sat, 20 Nov 2021 14:49:22 +0000 Lustre 2.15.0 0 6 <p>It looks like this has been failing intermittently (once per 4-5 days on full sessions only) since at least 2021-01-01 (the earliest I checked). Since 2021-10-10 it has also failed on 4 review test sessions, and as such is failing more frequently (6x in 2 days). It would be worthwhile to check what changes landed on 10-10 that might have triggered this. </p> <p>my bisection points to one patch:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> COMMIT TESTED PASSED FAILED COMMIT DESCRIPTION 8a770616a5 5 4 1 BAD LU-14797 sec: add projid to nodemap bbfdc7c167 17 15 2 BAD LU-14739 quota: fix quota with root squash enabled da1d93513f 7 6 1 BAD LU-14475 log: Rewrite some log messages 885b494632 18 17 1 BAD LU-12362 ptlrpc: use wait_woken() in ptlrpcd() 9ec5e2329b 90 90 0 LU-6142 lod: <span class="code-keyword">return</span> pools_hash_params to being <span class="code-keyword">static</span>. 2a24b6ec67 90 90 0 LU-14734 ldiskfs: improve message <span class="code-keyword">for</span> large_dir f7f31f8f96 90 90 0 LU-12567 ptlrpc: handle reply and resend reorder 1a409a3e6a 90 90 0 LU-14711 osc: Do not attempt sending empty pages </pre> </div></div> <p>I <em>think</em> there are two potential issues in <a href="https://jira.whamcloud.com/browse/LU-12362" title="kernel warning 'do not call blocking ops when !TASK_RUNNING ' in ptlrpcd" class="issue-link" data-issue-key="LU-12362"><del>LU-12362</del></a>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - <span class="code-keyword">if</span> (timeout == 0) - wait_event_idle(set->set_waitq, - ptlrpcd_check(&env, pc)); - <span class="code-keyword">else</span> <span class="code-keyword">if</span> (wait_event_idle_timeout(set->set_waitq, - ptlrpcd_check(&env, pc), - cfs_time_seconds(timeout)) - == 0) + + add_wait_queue(&set->set_waitq, &wait); + <span class="code-keyword">while</span> (!ptlrpcd_check(&env, pc)) { + <span class="code-object">int</span> ret; + + <span class="code-keyword">if</span> (timeout == 0) + ret = wait_woken(&wait, TASK_IDLE, + MAX_SCHEDULE_TIMEOUT); + <span class="code-keyword">else</span> + ret = wait_woken(&wait, TASK_IDLE, + cfs_time_seconds(timeout)); + <span class="code-keyword">if</span> (ret != 0) + <span class="code-keyword">continue</span>; + <span class="code-comment">/* Timed out */</span> ptlrpc_expired_set(set); + <span class="code-keyword">break</span>; + } + remove_wait_queue(&set->set_waitq, &wait); + </pre> </div></div> <p>1) timeout is reused after spurious wakeup, so this can last longer than expected<br/> 2) ptlrpc_expired_set() can be called for the case when timeout=0</p> <p>added a simple check to pltrpcd to catch too long waiting:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 5019:0:(ptlrpcd.c:515:ptlrpcd()) ASSERTION( timeout == 0 || end - start < timeout + 4 ) failed: timeout 10, end 97, start 83, diff 14, count 3</pre> </div></div> <p>i.e. 10 seconds wait was requested, but actually spent 14 seconds waiting, wait_woken() was called 3 times.</p> <p>yet another interesting example from failed replay-dual:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LustreError: 5009:0:(ptlrpcd.c:513:ptlrpcd()) timeout 19, end 711, start 620, diff 91, count 9</pre> </div></div> <p>"Alex Zhuravlev <bzzz@whamcloud.com>" uploaded a new patch: <a href="https://review.whamcloud.com/45308" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45308</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15086" title="replay-dual: test_10: @@@@@@ FAIL: test_10 failed with 2 " class="issue-link" data-issue-key="LU-15086"><del>LU-15086</del></a> ptlrpc: fix timeout after spurious wakeup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a495c00874105caa299fdf4dfaa482df9b24ad2e</p> <p>Would we be better off reverting <a href="https://jira.whamcloud.com/browse/LU-12362" title="kernel warning 'do not call blocking ops when !TASK_RUNNING ' in ptlrpcd" class="issue-link" data-issue-key="LU-12362"><del>LU-12362</del></a> from master?</p> <p>with the patch above I can't reproduce the problem. would like to hear Neil's opinion.</p> <p>"Oleg Drokin <green@whamcloud.com>" merged in patch <a href="https://review.whamcloud.com/45308/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45308/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15086" title="replay-dual: test_10: @@@@@@ FAIL: test_10 failed with 2 " class="issue-link" data-issue-key="LU-15086"><del>LU-15086</del></a> ptlrpc: fix timeout after spurious wakeup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b8383035406a4b7bee2e6d8674eaef480b3e3b35</p> <p>Landed for 2.15</p> Development Rank 1|i026y7: Rank (Obsolete) 9223372036854775807 Severity