https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1094 Lustre <p>We had five occurrences of this crash on OSS nodes in our classified Lustre 2.1 cluster. Timeframe coincided with <a href="https://jira.whamcloud.com/browse/LU-1085" title="ASSERTION(cfs_atomic_read(&amp;exp-&gt;exp_refcount) == 0) failed" class="issue-link" data-issue-key="LU-1085"><del>LU-1085</del></a>. Like the other bugs in that window, this crash was preceded by hundreds of messages like:</p> <p>LustreError: 14210:0:(genops.c:1270:class_disconnect_stale_exports()) ls5-OST0349: disconnect stale client <span class="error">&#91;UUID&#93;</span>@&lt;unknown&gt;</p> <p>general protection fault: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP<br/> Pid: 13890, comm: ll_ost_34</p> <p>machine_kexec<br/> crash_kexec<br/> oops_end<br/> die<br/> do_general_protection<br/> general_protection<br/> <span class="error">&#91;exception RIP: strnlen+9&#93;</span><br/> string<br/> vsnprintf<br/> libcfs_debug_vmsg2<br/> _debug_req<br/> target_send_reply_msg<br/> target_send_reply<br/> ost_handle<br/> ptlrpc_main<br/> kernel_thread</p> <a href="https://github.com/chaos/lustre/commits/2.1.0-llnl">https://github.com/chaos/lustre/commits/2.1.0-llnl</a><br/> RHEL 6.2 LU-1094

general protection fault in _debug_req()

Bug Critical Resolved Duplicate Oleg Drokin Ned Bass paj Fri, 10 Feb 2012 17:58:25 +0000 Mon, 30 Apr 2012 11:57:31 +0000 Mon, 30 Apr 2012 11:57:31 +0000 Lustre 2.1.0 0 3 <p>Comment copied from <a href="https://jira.whamcloud.com/browse/LU-1085" title="ASSERTION(cfs_atomic_read(&amp;exp-&gt;exp_refcount) == 0) failed" class="issue-link" data-issue-key="LU-1085"><del>LU-1085</del></a>:</p> <p>I did some digging in crash to see what state the ptlrpc_reqeust was in. I dug up the pointer address from the backtrace (let's call it &lt;addr1&gt; to save typing). Then resolving some of the strings that get passed to libcfs_debug_vmsg2() from _debug_req(), I see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; struct ptlrpc_request.rq_import &lt;addr1&gt; rp_import = 0x0 crash&gt; struct ptlrpc_request.rq_export &lt;addr1&gt; rp_export = &lt;addr2&gt; crash&gt; struct obd_export.exp_connection &lt;addr2&gt; exp_connection = 0x5a5a5a5a5a5a5a5a crash&gt; struct obd_export.exp_client_uuid &lt;addr2&gt; exp_client_uuid = { uuid = "ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" } </pre> </div></div> <p>So the presence of poison value and bogus uuid suggests this export has already been destroyed.</p> <p>For reference, here a snippet from from _debug_req() that uses these values:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2271 void _debug_req(struct ptlrpc_request *req, 2272 struct libcfs_debug_msg_data *msgdata, 2273 const char *fmt, ... ) 2274 { 2275 va_list args; 2276 va_start(args, fmt); 2277 libcfs_debug_vmsg2(msgdata, fmt, args, 2278 " req@%p x"LPU64"/t"LPD64"("LPD64") o%d-&gt;%s@%s:%d/%d" 2279 " lens %d/%d e %d to %d dl "CFS_TIME_T" ref %d " 2280 "fl "REQ_FLAGS_FMT"/%x/%x rc %d/%d\n", 2281 req, req-&gt;rq_xid, req-&gt;rq_transno, 2282 req-&gt;rq_reqmsg ? lustre_msg_get_transno(req-&gt;rq_reqmsg) : 0, 2283 req-&gt;rq_reqmsg &amp;&amp; req_ptlrpc_body_swabbed(req) ? 2284 lustre_msg_get_opc(req-&gt;rq_reqmsg) : -1, 2285 req-&gt;rq_import ? obd2cli_tgt(req-&gt;rq_import-&gt;imp_obd) : 2286 req-&gt;rq_export ? 2287 (char*)req-&gt;rq_export-&gt;exp_client_uuid.uuid : "&lt;?&gt;", </pre> </div></div> <p>I think this one also has a chance of being related to lu-106, so let's see if the runs with the patch would help.</p> <p>FYI, we did in fact hit this again with the <a href="https://jira.whamcloud.com/browse/LU-106" title="unable to handle kernel paging request in lprocfs_stats_collect()" class="issue-link" data-issue-key="LU-106"><del>LU-106</del></a> patch here:</p> <p><a href="http://review.whamcloud.com/326" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/326</a></p> <p>Sorry, disregard previous comment. We hit a new GPF, not this one. </p> <p>Believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1092" title="NULL pointer dereference in filter_export_stats_init()" class="issue-link" data-issue-key="LU-1092"><del>LU-1092</del></a></p> Development Rank 1|hzvhcv: Rank (Obsolete) 6461 Severity