https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3698 Lustre <p>As seen on a Hyperion client iwc1</p> <p>There was serious memory pressure lots of allocation were failing. </p> <p>Build:<br/> <a href="http://build.whamcloud.com/job/lustre-master/1594/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1594/</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: Lustre: Build Version: jenkins-arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel-1594-gbdf591f-PRISTINE-2.6.32-358.11.1.el6.x86_64 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-08-05 10:27:53 LustreError: 84566:0:(osc_request.c:2161:osc_build_rpc()) prep_req failed: -12 2013-08-05 10:27:53 LustreError: 84566:0:(osc_request.c:2161:osc_build_rpc()) Skipped 4 previous similar messages 2013-08-05 10:27:53 LustreError: 84566:0:(osc_cache.c:2091:osc_check_rpcs()) Write request failed with -12 2013-08-05 10:27:53 LustreError: 84566:0:(osc_cache.c:2091:osc_check_rpcs()) Skipped 5 previous similar messages 2013-08-05 10:27:53 LustreError: 84566:0:(sec.c:1060:sptlrpc_cli_unwrap_reply()) ASSERTION( req-&gt;rq_repdata == ((void *)0) ) failed: 2013-08-05 10:27:53 LustreError: 84566:0:(sec.c:1060:sptlrpc_cli_unwrap_reply()) LBUG 2013-08-05 10:27:53 Pid: 84566, comm: ptlrpcd_7 2013-08-05 10:27:53 2013-08-05 10:27:53 Call Trace: 2013-08-05 10:27:53 [&lt;ffffffffa056e895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] 2013-08-05 10:27:53 [&lt;ffffffffa056ee97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] 2013-08-05 10:27:53 [&lt;ffffffffa0c52256&gt;] sptlrpc_cli_unwrap_reply+0x1d6/0x240 [ptlrpc] 2013-08-05 10:27:53 [&lt;ffffffffa0c1735f&gt;] after_reply+0x6f/0xd90 [ptlrpc] 2013-08-05 10:27:53 [&lt;ffffffffa057efb1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] 2013-08-05 10:27:53 [&lt;ffffffffa0c1c941&gt;] ptlrpc_check_set+0xfd1/0x1b20 [ptlrpc] 2013-08-05 10:27:53 [&lt;ffffffffa0c4752b&gt;] ptlrpcd_check+0x53b/0x560 [ptlrpc] 2013-08-05 10:27:53 [&lt;ffffffffa0c47a43&gt;] ptlrpcd+0x223/0x380 [ptlrpc] 2013-08-05 10:27:53 [&lt;ffffffff81063310&gt;] ? default_wake_function+0x0/0x20 2013-08-05 10:27:53 [&lt;ffffffffa0c47820&gt;] ? ptlrpcd+0x0/0x380 [ptlrpc] 2013-08-05 10:27:53 [&lt;ffffffff81096936&gt;] kthread+0x96/0xa0 2013-08-05 10:27:53 [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 2013-08-05 10:27:53 [&lt;ffffffff810968a0&gt;] ? kthread+0x0/0xa0 2013-08-05 10:27:53 [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>There should be crashdump for this that I will take a look at if needed. </p> <p>I will do a little more digging and further updating of this LU it looks like we are not handling the ENOMEM (-12) quite right in this code path. </p> LU-3698

sec.c:1060:sptlrpc_cli_unwrap_reply() ASSERTION( req->rq_repdata == ((void *)0) ) failed

Bug Critical Resolved Fixed Keith Mannthey Keith Mannthey Mon, 5 Aug 2013 18:53:17 +0000 Tue, 24 Sep 2013 04:58:36 +0000 Tue, 24 Sep 2013 04:58:36 +0000 Lustre 2.5.0 Lustre 2.5.0 0 6 <p>The kernel log from the boot. </p> <p>Well I took a quick look at the code and I think this may be a little more complicated than a usual out of memory issue"</p> <p>osc_check_rpcs has the following code:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> osc_object_lock(osc); if (osc_makes_rpc(cli, osc, OBD_BRW_WRITE)) { rc = osc_send_write_rpc(env, cli, osc, pol); if (rc &lt; 0) { CERROR("Write request failed with %d\n", rc); /* osc_send_write_rpc failed, mostly because of * memory pressure. * * It can't break here, because if: * - a page was submitted by osc_io_submit, so * page locked; * - no request in flight * - no subsequent request * The system will be in live-lock state, * because there is no chance to call * osc_io_unplug() and osc_check_rpcs() any * more. pdflush can't help in this case, * because it might be blocked at grabbing * the page lock as we mentioned. * * Anyway, continue to drain pages. */ /* break; */ } } </pre> </div></div> <p>So we don't handle the enomem on write it seems?? </p> <p>It seems like this might be the outcome of this decision? </p> <p>Jinshan: can you comment on Keith's question.</p> <p>We just saw this on one of our test machines with b2_4 HEAD as well.</p> <p>I took a look at related code and it seems that the problem is that in ptl_send_rpc() req is not sent and req flags are not reset, so in ptlrpc_check_set() reply flag checking thinks req has got reply and goes on to process the reply and thus LBUG.</p> <p>I <em>guess</em> something like following should fix the LBUG. But I'm not familiar with ptlrpc code so not entirely sure.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> diff --git a/lustre/ptlrpc/client.c b/lustre/ptlrpc/client.c index 8b96c5d..9008e62 100644 --- a/lustre/ptlrpc/client.c +++ b/lustre/ptlrpc/client.c @@ -1737,6 +1737,7 @@ int ptlrpc_check_set(const struct lu_env *env, struct ptlrpc_request_set *set) spin_lock(&amp;req-&gt;rq_lock); req-&gt;rq_net_err = 1; spin_unlock(&amp;req-&gt;rq_lock); + continue; } /* need to reset the timeout */ force_timer_recalc = 1; </pre> </div></div> <p>Peng Tao, I agree with your idea and have created a patch for it. <a href="http://review.whamcloud.com/7411" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7411</a></p> <p>A rpc expert will need to take a good look at this idea. </p> <p>Landed for 2.5.0</p> Development Rank 1|hzvx5z: Rank (Obsolete) 9544 Severity