Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-6337] threads stuck at ldlm_completion_ast https://jira.whamcloud.com/browse/LU-6337 Lustre <p>Looks like we have this this issue at 3 times in the past 48 hours. Lots of threads stuck at ldlm_completion_ast. We are running with 2.4.3.</p> <p>see attached console logs</p> LU-6337

threads stuck at ldlm_completion_ast

Bug Major Resolved Duplicate Oleg Drokin Mahmoud Hanafi Thu, 5 Mar 2015 22:35:52 +0000 Fri, 16 Oct 2015 18:18:53 +0000 Fri, 16 Oct 2015 18:18:53 +0000 Lustre 2.4.3 0 5 <p>This ticket looks like <a href="https://jira.whamcloud.com/browse/LU-5497" title="Many MDS service threads blocked in ldlm_completion_ast()" class="issue-link" data-issue-key="LU-5497"><del>LU-5497</del></a>. It seems like moving to 2.5.3 is a possible solution, but we had problem upgrading yesterday. While we investigate problems of upgrading (and honestly need more testing before putting it in production), we need working patches for 2.4.3 from Intel.</p> <p>The symptoms you are seeing are too broad. The most likely cause is due to a lock that is not being released by some party.<br/> In the past the major contributor to it was LU2827 also seen as <a href="https://jira.whamcloud.com/browse/LU-5497" title="Many MDS service threads blocked in ldlm_completion_ast()" class="issue-link" data-issue-key="LU-5497"><del>LU-5497</del></a> at LLNL - but in order for it to manifest you need to have over 45 OSTs in your system OR your nerwork must regularly drop RPCs to/from MDS.<br/> If any of those two are true - then applying <a href="http://review.whamcloud.com/#/c/6511/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6511/</a> and <a href="http://review.whamcloud.com/#/c/9488/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9488/</a> should help you.</p> <p>Also please note that 2.5.3 does not contain fixes for this problem (but the tip of b2_5 does).</p> <p>Our nas-2.5.3 branch as of today is very close to the tip of b2_5. We are at <br/> <a href="https://jira.whamcloud.com/browse/LU-5912" title="locking flaw generates logged errors" class="issue-link" data-issue-key="LU-5912"><del>LU-5912</del></a> libcfs: use vfs api for fsync calls</p> <p>However, I do not see <a href="https://jira.whamcloud.com/browse/LU-5497" title="Many MDS service threads blocked in ldlm_completion_ast()" class="issue-link" data-issue-key="LU-5497"><del>LU-5497</del></a> patch in b2_5. Which commit should it be if the tip of b2_5 contains the fix?</p> <p>Now, if I want to cherry pick #6511 and #9488 to nas-2.4.3, do I also need <br/> <a href="http://review.whamcloud.com/#/c/10601/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10601/</a> in addition to those two?</p> <p>in b2_5 there is a proper fix to this issue under the banner of <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> and then a number of follow-on patches, from <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> to <a href="https://jira.whamcloud.com/browse/LU-5579" title="MDS crashed by "mdt_check_resent_lock()) ASSERTION( lock != NULL ) failed"" class="issue-link" data-issue-key="LU-5579"><del>LU-5579</del></a> and everythign in-between.</p> <p>patch 10601 is purely of informational nature and does not really improve actual hanging situation, so it's ok to skip it for 2.4.3, but it alsmo might be a good idea to add it should this matter require more investigations.</p> <p>I had a conflict in applying #9488/9:</p> <p><<<<<<< HEAD<br/> /* If the client does not require open lock, it does not need to</p> <ul> <li>search lock in exp_lock_hash, since the server thread will</li> <li>make sure the lock will be released, and the resend request</li> <li>can always re-enqueue the lock */<br/> if ((opcode != MDT_IT_OPEN) || (opcode == MDT_IT_OPEN &&<br/> info->mti_spec.sp_cr_flags & MDS_OPEN_LOCK)) {<br/> /* In the function below, .hs_keycmp resolves to</li> <li>ldlm_export_lock_keycmp() */<br/> /* coverity<span class="error">[overrun-buffer-val]</span> */<br/> lock = cfs_hash_lookup(exp->exp_lock_hash, &remote_hdl);<br/> if (lock) <div class="error"><span class="error">Unknown macro: { lock_res_and_lock(lock); if (lock != new_lock) { lh->mlh_reg_lh.cookie = lock->l_handle.h_cookie; lh->mlh_reg_mode = lock->l_granted_mode; LDLM_DEBUG(lock, "Restoring lock cookie"); DEBUG_REQ(D_DLMTRACE, req, "restoring lock cookie "LPX64, lh->mlh_reg_lh.cookie); if (old_lock) *old_lock = LDLM_LOCK_GET(lock); cfs_hash_put(exp->exp_lock_hash, &lock->l_exp_hash); unlock_res_and_lock(lock); return; } cfs_hash_put(exp->exp_lock_hash, &lock->l_exp_hash); unlock_res_and_lock(lock); }</span> </div> <p> }<br/> =======<br/> /* In the function below, .hs_keycmp resolves to</p></li> <li>ldlm_export_lock_keycmp() */<br/> /* coverity<span class="error">[overrun-buffer-val]</span> */</li> </ul> <p> /* Look for first lock found in hash for key that is not new_lock.<br/> There should only be 2 upon resend, new_lock and the first/original<br/> one.<br/> */<br/> data.skip_lock = new_lock;<br/> cfs_hash_for_each_key(exp->exp_lock_hash, &remote_hdl,<br/> not_skip_lock, &data);<br/> lock = data.found_lock;<br/> if (lock != NULL) </p> { lh->mlh_reg_lh.cookie = lock->l_handle.h_cookie; lh->mlh_reg_mode = lock->l_granted_mode; LDLM_DEBUG(lock, "Restoring lock cookie"); DEBUG_REQ(D_DLMTRACE, req, "restoring lock cookie "LPX64, lh->mlh_reg_lh.cookie); if (old_lock) *old_lock = LDLM_LOCK_GET(lock); cfs_hash_put(exp->exp_lock_hash, &lock->l_exp_hash); return; } <p>>>>>>>> c695980... <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> mdt: ensure orig lock is found in hash upon resend</p> <p>Does this ring any bell to you? I guess the code in HEAD came from another patch we cherry-picked before or I missed another patch.</p> <p>Geez, the formatting really screwed up in displaying "*" and indentation.</p> <p>The display will be correct when you enter edit mode.</p> <p>Patch 9488 is against b2_4, so there should not really be a conflict. (btw you can use (code)....(code) tags (but use curvy brackets) to disable formatting on a piece of code when making a comment with it).</p> <p>I see that in your case it's <a href="https://jira.whamcloud.com/browse/LU-4403" title="ASSERTION( lock->l_readers > 0 )" class="issue-link" data-issue-key="LU-4403"><del>LU-4403</del></a> patch you are carryign that messed things up.</p> <p>Sadly it has a ton of whitespace change, but if you do git show -b 08b397f5bf2561f2294315a9039b1930ce0695d5 on it, then you can see the real change.<br/> Also I think to have a passing memory how this patch was not really needed and only arouse due to a bug fixed by patch 6511 (it was backed out in b2_5 as part of lu2827 series of patches)</p> <p>Please close this issue</p> <p>ok - thanks Mahmoud</p> Development Rank 1|hzx7t3: Rank (Obsolete) 17749 Severity