https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6172 Lustre <p>We have several clients panicked yesterday on this problem. There was ptlrpc_expire_one_request message on nbp9 (10.151.26.5):<br/> <span class="error">&#91;1422397399.479345&#93;</span> Lustre: 5076:0:(client.c:1878:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1422397397/real 1422397397&#93;</span> req@ffff88020b91dc00 x1487556472851932/t0(0) o250-&gt;MGC10.151.26.5@o2ib@10.151.26.5@o2ib:26/25 lens 400/544 e 0 to 1 dl 1422397502 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> <span class="error">&#91;1422397399.507345&#93;</span> Lustre: 5076:0:(client.c:1878:ptlrpc_expire_one_request()) Skipped 3507 previous similar messages</p> <p>The client hits LBUG as shown in dmesg below:</p> <p><span class="error">&#91;1422397952.774718&#93;</span> LustreError: 22882:0:(sec_null.c:196:null_free_reqbuf()) ASSERTION( req-&gt;rq_reqmsg == req-&gt;rq_reqbuf ) failed: req ffff880837378000: reqmsg ffff8808373780e8 is not reqbuf ffff8805b759a400 in null sec<br/> <span class="error">&#91;1422397952.794717&#93;</span> LustreError: 22882:0:(sec_null.c:196:null_free_reqbuf()) LBUG<br/> <span class="error">&#91;1422397952.802717&#93;</span> Pid: 22882, comm: kworker/u:1<br/> <span class="error">&#91;1422397952.806717&#93;</span><br/> <span class="error">&#91;1422397952.806717&#93;</span> Call Trace:<br/> <span class="error">&#91;1422397952.810716&#93;</span> <span class="error">&#91;&lt;ffffffff81004935&gt;&#93;</span> dump_trace+0x75/0x310<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa062c82a&gt;&#93;</span> libcfs_debug_dumpstack+0x4a/0x70 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa062cd5e&gt;&#93;</span> lbug_with_loc+0x3e/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa0914030&gt;&#93;</span> null_free_reqbuf+0x260/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa09022e4&gt;&#93;</span> sptlrpc_cli_free_reqbuf+0x44/0x130 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08c77ad&gt;&#93;</span> __ptlrpc_free_req+0x15d/0x590 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08c7d3b&gt;&#93;</span> __ptlrpc_req_finished+0x15b/0x260 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08de527&gt;&#93;</span> request_out_callback+0xb7/0x240 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa06e2da7&gt;&#93;</span> lnet_msg_detach_md+0x57/0xf0 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa06e42b9&gt;&#93;</span> lnet_finalize+0x89/0x3e0 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085b4d7&gt;&#93;</span> kiblnd_tx_done+0x117/0x3e0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085c372&gt;&#93;</span> kiblnd_txlist_done+0x52/0x60 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085c463&gt;&#93;</span> kiblnd_peer_connect_failed+0xe3/0x2e0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08655ed&gt;&#93;</span> kiblnd_cm_callback+0x4ed/0x1260 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa04f1a02&gt;&#93;</span> cma_work_handler+0x72/0xa0 <span class="error">&#91;rdma_cm&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8107b4dc&gt;&#93;</span> process_one_work+0x16c/0x350<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8107e10a&gt;&#93;</span> worker_thread+0x17a/0x410<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff81082476&gt;&#93;</span> kthread+0x96/0xa0<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8147aae4&gt;&#93;</span> kernel_thread_helper+0x4/0x10<br/> <span class="error">&#91;1422397952.818716&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> Kernel panic - not syncing: LBUG<br/> <span class="error">&#91;1422397952.818716&#93;</span> Pid: 22882, comm: kworker/u:1 Tainted: GF NX 3.0.101-0.31.1.20140612-nasa #1<br/> <span class="error">&#91;1422397952.818716&#93;</span> Call Trace:<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff81004935&gt;&#93;</span> dump_trace+0x75/0x310<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8146ec63&gt;&#93;</span> dump_stack+0x69/0x6f<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8146ecfc&gt;&#93;</span> panic+0x93/0x201<br/> Broadcast message from root (Wed Jan 28 12:11:27 2015):+0xa3/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa0914030&gt;&#93;</span> null_free_reqbuf+0x260/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> System is being shutdown to apply patches during scheduled dedicated time. Plea<span class="error">&#91;e logoff now to a&#93;</span>oid losing data. 1912,1 99%<br/> <span class="error">&#91;he system is goin&#93;</span> DOWN for reboot in 9 minutes!<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa09022e4&gt;&#93;</span> sptlrpc_cli_free_reqbuf+0x44/0x130 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08c77ad&gt;&#93;</span> __ptlrpc_free_req+0x15d/0x590 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08c7d3b&gt;&#93;</span> __ptlrpc_req_finished+0x15b/0x260 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08de527&gt;&#93;</span> request_out_callback+0xb7/0x240 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa06e2da7&gt;&#93;</span> lnet_msg_detach_md+0x57/0xf0 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa06e42b9&gt;&#93;</span> lnet_finalize+0x89/0x3e0 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085b4d7&gt;&#93;</span> kiblnd_tx_done+0x117/0x3e0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085c372&gt;&#93;</span> kiblnd_txlist_done+0x52/0x60 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085c463&gt;&#93;</span> kiblnd_peer_connect_failed+0xe3/0x2e0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08655ed&gt;&#93;</span> kiblnd_cm_callback+0x4ed/0x1260 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa04f1a02&gt;&#93;</span> cma_work_handler+0x72/0xa0 <span class="error">&#91;rdma_cm&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8107b4dc&gt;&#93;</span> process_one_work+0x16c/0x350<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8107e10a&gt;&#93;</span> worker_thread+0x17a/0x410<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff81082476&gt;&#93;</span> kthread+0x96/0xa0<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8147aae4&gt;&#93;</span> kernel_thread_helper+0x4/0x10</p> (Git repo at <a href="https://github.com/jlan/lustre-nas)">https://github.com/jlan/lustre-nas)</a><br/> Server: 2.4.3-12nasS, kernel: 2.6.32-358.23.2.el6<br/> Client: 2.4.3-11nasC, kernel: sles11sp3 3.0.101-0.31.1. LU-6172

Client LBUG in null_free_reqbuf

Bug Minor Resolved Duplicate Yang Sheng Jay Lan Wed, 28 Jan 2015 20:15:22 +0000 Thu, 12 Feb 2015 22:24:04 +0000 Thu, 12 Feb 2015 22:24:04 +0000 Lustre 2.4.3 0 5 <p>The Call Trace part in the decription of the problem was cut-n-paste from a screeen that was contaminated by broadcast messages from root. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>The correct stack trace should be as below:</p> <p><span class="error">&#91;1422397952.818716&#93;</span> Kernel panic - not syncing: LBUG<br/> <span class="error">&#91;1422397952.818716&#93;</span> Pid: 22882, comm: kworker/u:1 Tainted: GF NX 3.0.101-0.31.1.20140612-nasa #1<br/> <span class="error">&#91;1422397952.818716&#93;</span> Call Trace:<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff81004935&gt;&#93;</span> dump_trace+0x75/0x310<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8146ec63&gt;&#93;</span> dump_stack+0x69/0x6f<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8146ecfc&gt;&#93;</span> panic+0x93/0x201<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa062cdc3&gt;&#93;</span> lbug_with_loc+0xa3/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa0914030&gt;&#93;</span> null_free_reqbuf+0x260/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa09022e4&gt;&#93;</span> sptlrpc_cli_free_reqbuf+0x44/0x130 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08c77ad&gt;&#93;</span> __ptlrpc_free_req+0x15d/0x590 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08c7d3b&gt;&#93;</span> __ptlrpc_req_finished+0x15b/0x260 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08de527&gt;&#93;</span> request_out_callback+0xb7/0x240 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa06e2da7&gt;&#93;</span> lnet_msg_detach_md+0x57/0xf0 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa06e42b9&gt;&#93;</span> lnet_finalize+0x89/0x3e0 <span class="error">&#91;lnet&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085b4d7&gt;&#93;</span> kiblnd_tx_done+0x117/0x3e0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085c372&gt;&#93;</span> kiblnd_txlist_done+0x52/0x60 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa085c463&gt;&#93;</span> kiblnd_peer_connect_failed+0xe3/0x2e0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa08655ed&gt;&#93;</span> kiblnd_cm_callback+0x4ed/0x1260 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffffa04f1a02&gt;&#93;</span> cma_work_handler+0x72/0xa0 <span class="error">&#91;rdma_cm&#93;</span><br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8107b4dc&gt;&#93;</span> process_one_work+0x16c/0x350<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8107e10a&gt;&#93;</span> worker_thread+0x17a/0x410<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff81082476&gt;&#93;</span> kthread+0x96/0xa0<br/> <span class="error">&#91;1422397952.818716&#93;</span> <span class="error">&#91;&lt;ffffffff8147aae4&gt;&#93;</span> kernel_thread_helper+0x4/0x10</p> <p>Yang Sheng</p> <p>Could you please advise?</p> <p>Peter</p> <p>I suspect this might be an instance of <a href="https://jira.whamcloud.com/browse/LU-3333" title="lustre_msg_get_opc()) incorrect message magic: a0b03b5 LBUG" class="issue-link" data-issue-key="LU-3333"><del>LU-3333</del></a></p> <p>Hi, Oleg. I think you may right. Looks like not other place can cause such issue. But seem we won't landed patch to 2.4. How to handle this ticket?</p> <p>Yang Sheng</p> <p>NASA have their own Lustre distribution and so can pick up the patch. They will advise if they need any assistance in porting the fix back to b2_4. Alternatively, if this is a rare issue, they may wait, knowing that this issue will disappear when they upgrade to 2.5.x clients.</p> <p>Peter</p> <p>Thanks, guys!</p> <p>I cherry-picked the patch to nas-2.4.3 branch with no problem.</p> <p>So can we mark this ticket as duplicated with lu-3333?</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-3333" title="lustre_msg_get_opc()) incorrect message magic: a0b03b5 LBUG" class="issue-link" data-issue-key="LU-3333"><del>LU-3333</del></a>.<br/> ~ jfc.</p> Related LU-3333 Development Rank 1|hzx547: Rank (Obsolete) 17268 Severity