https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-59 Lustre <p>The call traces happened on MDS for ldlm_expired_completion_wait() and status was changed to LBUG.<br/> This seems to be similar to bug 21967, but there is no patches for lustre-1.8.x, right now.<br/> I'm attaching the console log on MDS that we saw. Could you please find out whether this is same bug as 21967. And if yes, please back port patch in 21967 for 1.8.x, also.</p> <p>Thanks<br/> Ihara </p> LU-59

call traces on MDS for ldlm_expired_completion_wait()

Bug Minor Resolved Fixed Niu Yawei Shuichi Ihara Fri, 4 Feb 2011 02:47:17 +0000 Tue, 28 Jun 2011 15:01:37 +0000 Mon, 13 Jun 2011 19:32:00 +0000 Lustre 1.8.6 0 1 <p>Niu</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, Ihara</p> <p>It looks similiar to the bug 21967, the difference is the timeout value in this trace is much longer than 21967's. But I didn't see any patches on bug 21967, seems it's an unresolved issue.</p> <p>BTW: In your test, is the dynamic timeout feature enabled?</p> <p>Ah, 21967 was closed by some reasons. I thought the problem is related to bug 22598 which there are no patches for 1.8.x branch.</p> <p>The dynamic timeout feature means Adaptive Timeout? If so, yes, I didn't disable AT intentionally. So, it should be enabled by default.</p> <p>There are some statistic/diagnostic patches and one 'disable COS by default' patch in bug 22598, and I don't think 1.8 has COS, so the patches in 22598 might not helpful for this issue.</p> <p>Yes, I meant adaptive timeout.Thank you. </p> <p>Hi, Ihara</p> <p>"The call traces happened on MDS for ldlm_expired_completion_wait() and status was changed to LBUG"<br/> I don't quite understand the "status was changed to LBUG", could you make further explanation?</p> <p>The log shows that lots of server threads were blocking on local lock enqueue for a long time, which triggered watchdog to dump the stack traces. I suspect the reason is that some client which holding locks was evicted by server (maybe a dead or hang client), and the server local lock enqueue triggered blocking ast to the evicted client, the blocking ast should be timeout soon, however for some reason, the blocking ast didn't expired in time, which making server threads waiting for a long time and the watchdog was triggered at the end.</p> <p>Is it easy to reproduce? If it's easy, could you turn off the Adaptive Timeout to see if the problem is gone? Thanks. </p> <p>Niu, sorry, there are several MDS crashes (or hang) frequently at the customer site. One of reasons might be <a href="https://jira.whamcloud.com/browse/LU-27" title="(mds_open.c:1667:mds_close()) @@@ no handle for file close ino" class="issue-link" data-issue-key="LU-27"><del>LU-27</del></a> which caused LBUG. Also, we might hit bug 23352 on TCP clients. I saw these problems sometimes happened at the same time. So, I thought this case was also LBUG, but you are right. there are nothing LBUG in this console messages. </p> <p>We applied the patch in <a href="https://jira.whamcloud.com/browse/LU-27" title="(mds_open.c:1667:mds_close()) @@@ no handle for file close ino" class="issue-link" data-issue-key="LU-27"><del>LU-27</del></a> and bug 23352 and are keeping an eye if we still see same problem even after apply these patches. Do you think this error messages can be caused after by <a href="https://jira.whamcloud.com/browse/LU-27" title="(mds_open.c:1667:mds_close()) @@@ no handle for file close ino" class="issue-link" data-issue-key="LU-27"><del>LU-27</del></a> or bug 23352?</p> <p>Thank you, Ihara. I think the fix in <a href="https://jira.whamcloud.com/browse/LU-27" title="(mds_open.c:1667:mds_close()) @@@ no handle for file close ino" class="issue-link" data-issue-key="LU-27"><del>LU-27</del></a> doesn't help much on this issue, whereas the at_min fix in b23352 can avoid the unnecessary client eviction, which might reduce the chance of seeing such error message, I think you can try this patch to see if things will be getting better.</p> <p>What I don't understand is that why the lock enqueue didn't timeout in time (and triggered whatchdog at last), will make further investigaion on the Adaptive Timeout and get back to you later.</p> <p>Hi, Ihara</p> <p>What about the test result after b23352 fix applied?</p> <p>Niu, </p> <p>We haven't seen same issue since applied a patch in b23352.<br/> But, don't we have any fixes in adaptive timeout area?</p> <p>With adaptive timeout, the lock callback timeout could be very long (the default maximum is 600 seconds) and consequently the server working thread might wait in ldlm_cli_enqueue_local() for a very long time, which triggered the watchdog to dump the stack trace in the end. So I think it's not necessary a bug.</p> <p>Ihara</p> <p>Do you have any further questions or can we close out this ticket?</p> <p>Thanks</p> <p>Peter</p> <p>fine. as far as we can see, the problem seems to be fixed by patch in b23352.</p> <p>thanks!</p> <p>Great - thanks Ihara!</p> Development Rank 1|hzw01z: Rank (Obsolete) 10088 Severity