https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5537 Lustre <p>The following assertion failure was seen on an OSS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 19 17:32:08 lola-2 kernel: Lustre: ost: This server is not able to keep up with request traffic (cpu-bound). Aug 19 17:32:08 lola-2 kernel: Lustre: 5309:0:(service.c:1509:ptlrpc_at_check_timed()) earlyQ=1 reqQ=0 recA=0, svcEst=30, delay=0(jiff) Aug 19 17:32:08 lola-2 kernel: Lustre: 5309:0:(service.c:1306:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-1s), not sending earl y reply. Consider increasing at_early_margin (5)? req@ffff880415a7b050 x1476487418415744/t0(0) o400-&gt;d8ca812e-ca2b-b357-39ed-b1b134fb6dbd@19 2.168.1.126@o2ib1:0/0 lens 224/0 e 586846 to 0 dl 1408494727 ref 2 fl Complete:H/c0/ffffffff rc 0/-1 Aug 19 17:32:09 lola-2 kernel: Lustre: soaked-OST0000: Client 87e86655-cbf2-ba09-92c2-7853a9b2c942 (at 192.168.1.119@o2ib1) reconnecting, wai ting for 14 clients in recovery for 1:27 Aug 19 17:32:09 lola-2 kernel: LustreError: 5366:0:(ldlm_lib.c:2689:target_bulk_io()) @@@ timeout on bulk GET after 0+0s req@ffff88083a61b40 0 x1476486691018500/t0(4300509964) o4-&gt;8dda3382-83f8-6445-5eea-828fd59e4a06@192.168.1.116@o2ib1:0/0 lens 504/448 e 391470 to 0 dl 1408494729 ref 2 fl Complete:/4/0 rc 0/0 Aug 19 17:32:09 lola-2 kernel: LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) ASSERTION( req-&gt;rq_no_reply == 0 ) failed: Aug 19 17:32:09 lola-2 kernel: Lustre: soaked-OST0000: Bulk IO write error with 8dda3382-83f8-6445-5eea-828fd59e4a06 (at 192.168.1.116@o2ib1) , client will retry: rc -110 Aug 19 17:32:09 lola-2 kernel: LustreError: 5432:0:(niobuf.c:550:ptlrpc_send_reply()) LBUG Aug 19 17:32:09 lola-2 kernel: Pid: 5432, comm: ll_ost_io03_003 Aug 19 17:32:09 lola-2 kernel: Aug 19 17:32:09 lola-2 kernel: Call Trace: Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa0641895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa0641e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa09cda4c&gt;] ptlrpc_send_reply+0x4ec/0x7f0 [ptlrpc] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa09d4aae&gt;] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa09e4d75&gt;] ptlrpc_at_check_timed+0xcd5/0x1370 [ptlrpc] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa09dc1e9&gt;] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa09e66f8&gt;] ptlrpc_main+0x12e8/0x1990 [ptlrpc] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffff81069290&gt;] ? pick_next_task_fair+0xd0/0x130 Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffff81529246&gt;] ? schedule+0x176/0x3b0 Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffffa09e5410&gt;] ? ptlrpc_main+0x0/0x1990 [ptlrpc] Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffff8109abf6&gt;] kthread+0x96/0xa0 Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffff8100c20a&gt;] child_rip+0xa/0x20 Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffff8109ab60&gt;] ? kthread+0x0/0xa0 Aug 19 17:32:09 lola-2 kernel: [&lt;ffffffff8100c200&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>It appears to be a race between a BRW timeout and an attempt to send an early reply.</p> LU-5537

ptlrpc_send_reply(): ASSERTION( req->rq_no_reply == 0 ) failed

Bug Minor Resolved Fixed Li Wei Li Wei Fri, 22 Aug 2014 12:32:28 +0000 Fri, 5 Jun 2015 15:45:48 +0000 Tue, 25 Nov 2014 16:33:03 +0000 Lustre 2.7.0 Lustre 2.7.0 0 10 <p><a href="http://review.whamcloud.com/11740" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11740</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/11740/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11740/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5537" title="ptlrpc_send_reply(): ASSERTION( req-&gt;rq_no_reply == 0 ) failed" class="issue-link" data-issue-key="LU-5537"><del>LU-5537</del></a> ptlrpc: Fix an rq_no_reply assertion failure<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a8d448e4cd5978c546911f98067232bcdd30b651</p> <p>The patch has landed to master; resolving issue.</p> <p>Hi Li Wei,</p> <p>Could you please check whether this issue exists on Lustre b2_5 or not? And if yes, could you please back-port the patch? Thank you!</p> <p>I took a closer look at b2_5 and realized the problem does not exist there.</p> <p>It can happen on b2_5 also</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[2319692.184264] LustreError: 3415:0:(ldlm_lib.c:2724:target_bulk_io()) @@@ Reconnect on bulk PUT req@ffff88043ea04c00 x1499139419601196/t0(0) o3-&gt;85f63be7-8ccc-f8bf-ce43-5c0b15598965@273@gni1:0/0 lens 488/432 e 0 to 0 dl 1431173956 ref 1 fl Interpret:/0/0 rc 0/0 [2319692.209262] LustreError: 3415:0:(ldlm_lib.c:2724:target_bulk_io()) Skipped 4 previous similar messages [2319692.219701] Lustre: snx11128-OST003c: Bulk IO read error with 85f63be7-8ccc-f8bf-ce43-5c0b15598965 (at 273@gni1), client will retry: rc -110 [2319692.548327] LustreError: 65406:0:(niobuf.c:545:ptlrpc_send_reply()) ASSERTION( req-&gt;rq_no_reply == 0 ) failed: [2319692.559511] LustreError: 65406:0:(niobuf.c:545:ptlrpc_send_reply()) LBUG [2319692.566910] Pid: 65406, comm: ll_ost_io02_008 </pre> </div></div> Development Rank 1|hzwud3: Rank (Obsolete) 15412 Severity