https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1085 Lustre <p>We have multiple Lustre 2.1 OSS nodes crashing repeatedly during recovery. This is on our classified Lustre cluster which was updated from 1.8 to 2.1 on Tuesday. The summary is one observed symptom. We have also seen these assertions appearing together.</p> <p>ASSERTION(cfs_list_empty(&amp;imp-&gt;imp_zombie_chain)) failed<br/> ASSERTION(cfs_atomic_read(&amp;exp-&gt;exp_refcount) == 0) failed</p> <p>We don't have backtraces for the assertions because STONITH kicked in before the crash dump completed.</p> <p>Other OSS nodes are crashing in kernel string handling functions with stacks like</p> <p>machine_kexec<br/> crash_kexec<br/> oops_end<br/> die<br/> do_general_protection<br/> general_protection<br/> (exception RIP: strlen+9)<br/> strlen<br/> string<br/> vsnprintf<br/> libcfs_debug_vmsg2<br/> _debug_req<br/> target_send_replay_msg<br/> target_send_reply<br/> ost_handle<br/> ptlrpc_main</p> <p>So it appears we are passing a bad value in a debug message. </p> <p>Another stack trace:</p> <p>BUG: unable to handle kernel NULL ptr dereference at 000...38<br/> IP: <span class="error">&#91;&lt;fffffffa0a8706&gt;&#93;</span> filter_export_stats_init+0x1f1/0x500 <span class="error">&#91;obdfilter&#93;</span></p> <p>machine_kexec<br/> crash_kexec<br/> oops_end<br/> no_context<br/> __bad_area_semaphore<br/> bad_area_semaphore<br/> __do_page_fault<br/> do_page_fault<br/> page_fault<br/> filter_reconnect<br/> target_handle_connect<br/> ost_handle<br/> ptlrpc_main</p> <p>We have multiple symptoms here that may or not be due to the same bug. We may need to open a separate issue to track the root cause. Note that our branch contains <a href="https://jira.whamcloud.com/browse/LU-874" title="Client eviction on lock callback timeout " class="issue-link" data-issue-key="LU-874"><del>LU-874</del></a> patches that touched the ptlrpc queue management code, so we should be on the lookout for any races introduced there. Also note we can't send debug data from this system.</p> RHEL 6.2<br/> Our branch: <a href="https://github.com/chaos/lustre/commits/2.1.0-llnl">https://github.com/chaos/lustre/commits/2.1.0-llnl</a> LU-1085

ASSERTION(cfs_atomic_read(&exp->exp_refcount) == 0) failed

Bug Critical Resolved Duplicate Oleg Drokin Ned Bass paj Thu, 9 Feb 2012 15:43:30 +0000 Mon, 30 Apr 2012 11:56:20 +0000 Mon, 30 Apr 2012 11:56:20 +0000 Lustre 2.1.1 0 6 <p>Hm, I think we had some fixes in the zombie thread area.</p> <p>Can you please see if this helps you by any chance: <a href="http://review.whamcloud.com/#change,1896,patchset=2" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1896,patchset=2</a></p> <p>the export stats init I don't think I have seen before, I think most of that stuff was nailed in <a href="https://jira.whamcloud.com/browse/LU-106" title="unable to handle kernel paging request in lprocfs_stats_collect()" class="issue-link" data-issue-key="LU-106"><del>LU-106</del></a>, see patches here: <a href="http://review.whamcloud.com/#change,326" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,326</a></p> <p>Hi Oleg,</p> <blockquote><p>Can you please see if this helps you by any chance: <a href="http://review.whamcloud.com/#change,1896,patchset=2" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1896,patchset=2</a></p></blockquote> <p>Both nodes that ASSERTed had been up for a few days, so the mount thread had long since completed. Unless there could be some latent corruption from that race, it seems like that patch won't help.</p> <blockquote><p>I think most of that stuff was nailed in <a href="https://jira.whamcloud.com/browse/LU-106" title="unable to handle kernel paging request in lprocfs_stats_collect()" class="issue-link" data-issue-key="LU-106"><del>LU-106</del></a>, see patches here: <a href="http://review.whamcloud.com/#change,326" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,326</a></p></blockquote> <p>Ah, we don't have the <a href="https://jira.whamcloud.com/browse/LU-106" title="unable to handle kernel paging request in lprocfs_stats_collect()" class="issue-link" data-issue-key="LU-106"><del>LU-106</del></a> patch in our branch. We'll pull it in.</p> <p>For the case where we got into trouble in _debug_req(), I did some<br/> digging in crash to see what state the ptlrpc_reqeust was in. I dug up<br/> the pointer address from the backtrace (let's call it &lt;addr1&gt; to save<br/> typing). Then resolving some of the strings that get passed to<br/> libcfs_debug_vmsg2() from _debug_req(), I see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; struct ptlrpc_request.rq_import &lt;addr&gt; rp_import = 0x0 crash&gt; struct ptlrpc_request.rq_export &lt;addr&gt; rp_export = &lt;addr2&gt; crash&gt; struct obd_export.exp_connection &lt;addr2&gt; exp_connection = 0x5a5a5a5a5a5a5a5a crash&gt; struct obd_export.exp_client_uuid &lt;addr2&gt; exp_client_uuid = { uuid = "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" } </pre> </div></div> <p>So the presence of poison value and bogus uuid suggests<br/> this export has already been destroyed.</p> <p>For reference, here a snippet from from _debug_req()<br/> that uses these values:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2271 void _debug_req(struct ptlrpc_request *req, 2272 struct libcfs_debug_msg_data *msgdata, 2273 const char *fmt, ... ) 2274 { 2275 va_list args; 2276 va_start(args, fmt); 2277 libcfs_debug_vmsg2(msgdata, fmt, args, 2278 " req@%p x"LPU64"/t"LPD64"("LPD64") o%d-&gt;%s@%s:%d/%d" 2279 " lens %d/%d e %d to %d dl "CFS_TIME_T" ref %d " 2280 "fl "REQ_FLAGS_FMT"/%x/%x rc %d/%d\n", 2281 req, req-&gt;rq_xid, req-&gt;rq_transno, 2282 req-&gt;rq_reqmsg ? lustre_msg_get_transno(req-&gt;rq_reqmsg) : 0, 2283 req-&gt;rq_reqmsg &amp;&amp; req_ptlrpc_body_swabbed(req) ? 2284 lustre_msg_get_opc(req-&gt;rq_reqmsg) : -1, 2285 req-&gt;rq_import ? obd2cli_tgt(req-&gt;rq_import-&gt;imp_obd) : 2286 req-&gt;rq_export ? 2287 (char*)req-&gt;rq_export-&gt;exp_client_uuid.uuid : "&lt;?&gt;", 2288 req-&gt;rq_import ? 2289 (char *)req-&gt;rq_import-&gt;imp_connection-&gt;c_remote_uuid.uuid : 2290 req-&gt;rq_export ? 2291 (char *)req-&gt;rq_export-&gt;exp_connection-&gt;c_remote_uuid.uuid : "&lt;?&gt;", </pre> </div></div> <p>I found a third incident of a panicked obd_zombid thread from yesterday. No ASSERT message was captured in the logs for this one, but we did get a complete crash dump. Here is the backtrace</p> <p>machine_kexec<br/> crash_kexec<br/> panic<br/> lbug_with_loc<br/> obd_zombie_impexp_cull<br/> obd_zombie_impexp_thread<br/> kernel_thread</p> <p>I am going to open separate issues for the different crashes we ran into yesterday. We can use this issue to track the obd_zombid crash. </p> <p>We disabled the OSS read and writethrough caches and have not had any crashes since then. Nearly every crash was preceded by hundreds of client reconnect attempts and hundreds of log messages of the form:</p> <p>LustreError: 14210:0:(genops.c:1270:class_disconnect_stale_exports()) ls5-OST0349: disconnect stale client <span class="error">&#91;UUID&#93;</span>@&lt;unknown&gt;</p> <p>Opened the following new issues:</p> <ul> <li><a href="https://jira.whamcloud.com/browse/LU-1092" title="NULL pointer dereference in filter_export_stats_init()" class="issue-link" data-issue-key="LU-1092"><del>LU-1092</del></a> NULL pointer dereference in filter_export_stats_init()</li> <li><a href="https://jira.whamcloud.com/browse/LU-1093" title="unable to handle kernel paging request in target_handle_connect()" class="issue-link" data-issue-key="LU-1093"><del>LU-1093</del></a> unable to handle kernel paging request in target_handle_connect()</li> <li><a href="https://jira.whamcloud.com/browse/LU-1094" title="general protection fault in _debug_req()" class="issue-link" data-issue-key="LU-1094"><del>LU-1094</del></a> general protection fault in _debug_req()</li> </ul> <p>for the zombid backtrace, what was the line number from where the lbug_with_loc was called? I know you did not get LASSERT in the logs, but you can use gdb obdclass.ko and then run "l *(obd_zombie_impexp_cull+0x...)" (the +0x... value from backtrace) and it'll tell you the line number.<br/> Hm, I just noticed that neither obd_zombie_impexp_cull nor obd_zombie_impexp_thread have any lbugs nor lasserts.</p> <p>It was lustre/obdclass/genops.c:728:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 711 /* Export management functions */ 712 static void class_export_destroy(struct obd_export *exp) 713 { 714 struct obd_device *obd = exp-&gt;exp_obd; 715 ENTRY; 716 717 LASSERT_ATOMIC_ZERO(&amp;exp-&gt;exp_refcount); 718 719 CDEBUG(D_IOCTL, "destroying export %p/%s for %s\n", exp, 720 exp-&gt;exp_client_uuid.uuid, obd-&gt;obd_name); 721 722 LASSERT(obd != NULL); 723 724 /* "Local" exports (lctl, LOV-&gt;{mdc,osc}) have no connection. */ 725 if (exp-&gt;exp_connection) 726 ptlrpc_put_connection_superhack(exp-&gt;exp_connection); 727 728 LASSERT(cfs_list_empty(&amp;exp-&gt;exp_outstanding_replies)); 729 LASSERT(cfs_list_empty(&amp;exp-&gt;exp_uncommitted_replies)); 730 LASSERT(cfs_list_empty(&amp;exp-&gt;exp_req_replay_queue)); 731 LASSERT(cfs_list_empty(&amp;exp-&gt;exp_hp_rpcs)); 732 obd_destroy_export(exp); 733 class_decref(obd, "export", exp); 734 735 OBD_FREE_RCU(exp, sizeof(*exp), &amp;exp-&gt;exp_handle); 736 EXIT; 737 } </pre> </div></div> <p>Also I guess I must ask how was the original recovery triggered? Some unrelated crash or was this a recovery test of some sort? (purely of purposes of seeing if there is some another bug at play here).</p> <blockquote><p>Also I guess I must ask how was the original recovery triggered? Some unrelated crash or was this a recovery test of some sort? (purely of purposes of seeing if there is some another bug at play here).</p></blockquote> <p>This was on a production system, not a recovery test.</p> <p>In the case of <a href="http://jira.whamcloud.com/browse/LU-1085?focusedCommentId=28387&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-28387" class="external-link" rel="nofollow">this stack trace</a> the server had crashed about twenty minutes earlier with ASSERTION(cfs_atomic_read(&amp;exp-&gt;exp_refcount) == 0).</p> <p>We are still trying to understand what led to that crash, and the others in LU-<span class="error">&#91;1092,1093,1094&#93;</span>, but the factors at play seem to be high server load, slow server response, and many clients dropping their connection to the server and reconnecting. There was a four hour window in which we had 16 crashes involving 5 OSS nodes. As mentioned above, things seemed to stabilize after we disabled the OSS read and writethrough caches.</p> <p>Apparently we hit this over 20 times on various OSS of multiple filesystems over the weekend. Note that last week we re-enabled OSS read cache (write cache is still disabled).</p> <p>The admins tell me that this often hit while in (or perhaps shortly after) recovery. I'll need to look into why we were in recovery in the first place.</p> <p>I wonder if that's related to <a href="https://jira.whamcloud.com/browse/LU-1166" title="recovery never finished" class="issue-link" data-issue-key="LU-1166"><del>LU-1166</del></a> to some degree.</p> <p>Unfortunately the current patch is not fully complete, but does not make things worse.</p> <p>Any other messages before that crash that you can share with us?</p> <p>Ignore the comment about being in recovery. So far I don't think the logs show that.</p> <p>I've looked at a few nodes, and it looks like there is some kind of client timeout/eviction and then reconnection storm going on before the assertions.</p> <p>It looks like generally there are tens of thousands of "haven't heard from client X in 226 seconds. I think it's dead, and I am evicting it" messages. A couple of minutes clients begin reconnecting in droves. There is a mix of ost "connection from" and ost "Not available for connect" messages.</p> <p>The "haven't heard from client" and the client connect messages are both interleaved in the logs, and often repeated 30,000+ times (lustre squashes them into "previous similar messages" lines).</p> <p>And then we hit one of the two assertions listed at the beginning of this bug.</p> <p>Note that for several of the OSS nodes that I have looked at so far, the clients all seem to be from one particular cluster, which is running 1.8. (servers are all 2.1.0-24chaos).</p> <p>So with many more crashes, any luck in getting the backtraces for the first two assertions?<br/> Any succesful crashdumps?</p> <p>There are two imp_zombie_chain assertions, do you know which one of them did you hit? in obd_zombie_import_add or in class_import_put?</p> <p>As for the second assertion, there is no such assertion in the code? I checked out your tree and here's what I see:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Oleg-Drokins-MacBook-Pro-2:lustre green$ grep -r exp_refcount * | grep ASSERT lustre/obdclass/genops.c: LASSERT_ATOMIC_ZERO(&amp;exp-&gt;exp_refcount); lustre/obdclass/genops.c: LASSERT_ATOMIC_GT_LT(&amp;exp-&gt;exp_refcount, 0, 0x5a5a5a); lustre/obdecho/echo_client.c: LASSERT(cfs_atomic_read(&amp;ec-&gt;ec_exp-&gt;exp_refcount) &gt; 0); </pre> </div></div> <p>The first one is actually defined into LASSERTF so it does not exacty match your output I think?<br/> Can you elaborate a bit more on where that might have come from?</p> <blockquote><p>So with many more crashes, any luck in getting the backtraces for the first two assertions? Any succesful crashdumps?</p></blockquote> <p>I will review the logs and crash dumps and let you know.</p> <blockquote><p>There are two imp_zombie_chain assertions, do you know which one of them did you hit? in obd_zombie_import_add or in class_import_put?</p></blockquote> <p>We hit the one in class_import_put.</p> <blockquote><p>As for the second assertion, there is no such assertion in the code? I checked out your tree and here's what I see:</p></blockquote> <p>Here's the second one in our tree:</p> <p><a href="https://github.com/chaos/lustre/blob/2.1.1-3chaos/lustre/obdclass/genops.c#L717" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/chaos/lustre/blob/2.1.1-3chaos/lustre/obdclass/genops.c#L717</a></p> <p>the LASSERT_ATOMIC_ZERO is defined as LASSERTF internally:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#define LASSERT_ATOMIC_EQ(a, v) \ <span class="code-keyword">do</span> { \ LASSERTF(cfs_atomic_read(a) == v, \ <span class="code-quote">"value: %d\n"</span>, cfs_atomic_read((a))); \ } <span class="code-keyword">while</span> (0) #define LASSERT_ATOMIC_ZERO(a) LASSERT_ATOMIC_EQ(a, 0) </pre> </div></div> <p>What was the value reported?</p> <p>Value reported has been either 1, 2, or 3.</p> <p>Here's a log message and backtrace for the exp_refcount assertion.</p> <p>LustreError: 24253:0:(genops.c:717:class_export_destory()) ASSERTION(cfs_atomic_read(&amp;exp-&gt;exp_refounct) == 0) failed: value: 1</p> <p>COMMAND: "obd_zombid"<br/> #0 machine_kexec<br/> #1 crash_kexec<br/> #2 panic<br/> #3 lbug_with_loc<br/> #4 obd_zombie_impexp_cull<br/> #5 obd_zombie_impexp_thread<br/> #6 kernel_thread</p> <p>Here's a log message and backtrace for the cfs_list_empty assertion.</p> <p>LustreError: 24458:0:(genops.c:931:class_import_put()) ASSERTINO(cfs_list_empty(&amp;imp-&gt;imp_zombie_chain)) failed</p> <p>COMMAND: "ll_ost_54"<br/> #0 machine_kexec<br/> #1 crash_kexec<br/> #2 panic<br/> #3 lbug_with_loc<br/> #4 libcfs_assertion_failed<br/> #5 class_import_put<br/> #6 client_destroy_import<br/> #7 target_handle_connect<br/> #8 ost_handle<br/> #9 ptlrpc_main<br/> #10 kernel_thread</p> <p>Oleg,</p> <p>Do you think the <a href="https://jira.whamcloud.com/browse/LU-1092" title="NULL pointer dereference in filter_export_stats_init()" class="issue-link" data-issue-key="LU-1092"><del>LU-1092</del></a> patch will help with these assertions? Mikhail made a comment to that effect in <a href="https://jira.whamcloud.com/browse/LU-1336" title="OSS GPF at ptlrpc_send_reply+0x470" class="issue-link" data-issue-key="LU-1336"><del>LU-1336</del></a>.</p> <p><a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=893cf2014a38c5bd94890d3522fafe55f024a958" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=893cf2014a38c5bd94890d3522fafe55f024a958</a></p> <p>Yes, this looks related.<br/> Any chance you can try it?</p> <p>Yes, we'll pull the patch in to our tree, and it will eventually get rolled out to our production systems.</p> <p>Believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1092" title="NULL pointer dereference in filter_export_stats_init()" class="issue-link" data-issue-key="LU-1092"><del>LU-1092</del></a></p> Development Rank 1|hzvhe7: Rank (Obsolete) 6467 Severity