https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1934 Lustre <p>OST refuses reconnection from client with 'still busy with 10 active RPCs'. The OST is failing to time out the active RPCs and has been in this state for over a day. Client continually attempts to reconnect. We have a few client nodes in this state that can be used to investigate further.</p> <p>LLNL-bugzilla-ID: 1495</p> <a href="https://github.com/chaos/lustre/commits/2.1.2-3chaos">https://github.com/chaos/lustre/commits/2.1.2-3chaos</a> LU-1934

still busy with active RPCs for days

Bug Minor Resolved Fixed Zhenyu Xu Ned Bass llnl Thu, 13 Sep 2012 14:11:18 +0000 Mon, 29 May 2017 05:40:01 +0000 Mon, 29 May 2017 05:40:01 +0000 Lustre 2.4.0 0 3 <p>Grepping a stuck client NID in /proc/fs/lustre/ost/OSS/ost/req_history shows</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1269062278:172.18.102.20@tcp:12345-172.18.110.123@tcp:x1405404304029454:296:Interpret:1347406306:-1347406306s(-1347407432s) opc 101 </pre> </div></div> <p>I believe this is an LDLM lock enqueue.</p> <p>A couple of service thread on the OSS just completed after about 48 hours and the client reconnected. I dumped all thread backtraces while they were hung, and both looked like this (hand-copied):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>? ptl_send_rpc schedule timeout ? process_timeout cfs_waitq_timedwait ptlrpc_set_wait ? default_wake_function ptlrpc_queue_wait ldlm_server_glimpse_ast filter_intent_policy ldlm_lock_enqueue ldlm_handle_enqueue0 ldlm_handle_enqueue ? ldlm_server_completion_ast ? ost_blocking_ast ? ldlm_server_glimpse_ast ost_handle ? lustre_msg_get_transno ptlrpc_main child_rip </pre> </div></div> <p>When the threads completed I bet there were some messages about how they took too long, and then before completion probably something about failed glimpse AST or some such. Can you please show an example of those messages?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dusk23 2012-09-13 16:47:29 ... ptlrpc_server_handle_request() @@@ Request took longer than estimated (940:172446s); client may timeout. req@... x.../t0(0) o101-&gt;...@172.18.110.123@tcp:0/0 lens 296/352 e 1 to 0 dl 134707603 ref 1 fl Complete:/0/0 rc 301/301 </pre> </div></div> <p>I think I need to paint a more complete picture here. There seems to have been some interaction between multiple clients and OSTs:</p> <p>OSS/OST: dusk23/OST0143 (where the above message was logged)<br/> OSS/OST: dusk20/OST0166<br/> client: graph555 (172.18.110.123 from above message)<br/> client: cslic5</p> <p>graph555 was unable to reconnect to dusk20/OST0143 (busy with 1 active RPCs) and to dusk23/OST0166 (busy with 4 active RPCs)<br/> cslic5 was unable to reconnect to dusk23/OST0166 (busy with 10 active RPCs)</p> <p>This had been going on for some days.</p> <p>At 2012-09-13 16:42:30 dusk20 was powered off by it's failover partner (it seems to have locked up after I did a sysrq-t).</p> <p>Exactly 5 minutes later (recovery window for dusk20's failover partner) the above message was logged, and client connections were restored to all OSTs. So it appears that the processing of cslic5's RPCs by dusk23 was blocked somehow by graph555. I confess I don't quite understand how such a dependency comes about. But I guess the question is, why did dusk20 fail to handle or time out the active RPC from graph555?</p> <p>The req_history entry in my first comment was the active RPC from graph555 on dusk20. I find an entry for that xid in dusk20's logs (preceded several minutes earlier by watchdog stack trace)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2012-09-11 16:41:46 Lustre: Service thread pid 6655 was inactive for 600.00s ... dumping stack trace Pid: 6655, comm: ll_ost_186 schedule_timeout cfs_waitq_timedwait ptlrpc_set_wait ptlrpc_queue_wait ldlm_server_glimpse_ast filter_intent_policy ldlm_lock_enqueue ldlm_handle_enqueue0 ldlm_handle_enqueue ost_handle ptlrpc_main child_rip 2012-09-11 16:50:22 Lustre: 6866:0:(service.c:1035:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (10/-516), not sending early reply req@... x1405404304029454/t0(0) o101-&gt;...@172.18.110.123@tcp:0/0 lens 296/352 e 4 to 0 dl 134707432 ref 2 fl Interpret:/0/0 rc 0/0 </pre> </div></div> <p>Later, when I dumped stack traces at 2012-09-13 16:41:46, thread ll_ost_186 has basically the exact same stack trace.</p> <p>Bobijam</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Client reconnection has been fixed.</p> Related LU-1949 Development End date Fri, 4 Jul 2014 14:11:18 +0000 Rank 1|hzv5fj: Rank (Obsolete) 4414 Severity Start date Thu, 13 Sep 2012 14:11:18 +0000