https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3123 Lustre <p>One of clients could not communicate with one of OSSs due to Timed out RDMA error. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kernel: LustreError: 9948:0:(o2iblnd_cb.c:2914:kiblnd_check_txs()) Timed out tx: active_txs, 2 seconds kernel: LustreError: 9948:0:(o2iblnd_cb.c:2977:kiblnd_check_conns()) Timed out RDMA with 172.26.8.140@o2ib (32) kernel: LustreError: 9948:0:(events.c:198:client_bulk_callback()) event type 1, status -103, desc ffff8104e84f4000 </pre> </div></div> <p>On the server side, </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kernel: LustreError: 19792:0:(o2iblnd_cb.c:2914:kiblnd_check_txs()) Timed out tx: active_txs, 3 seconds kernel: LustreError: 19792:0:(o2iblnd_cb.c:2977:kiblnd_check_conns()) Timed out RDMA with 172.26.10.84@o2ib (18) kernel: LustreError: 19789:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8101b15d4000 kernel: LustreError: 19788:0:(events.c:381:server_bulk_callback()) event type 4, status -5, desc ffff8103505f2b80 kernel: LustreError: 19788:0:(events.c:381:server_bulk_callback()) event type 2, status -5, desc ffff8103505f2b80 kernel: LustreError: 21321:0:(ost_handler.c:1073:ost_brw_write()) @@@ network error on bulk GET 0(16384) req@ffff8101419a3400 x1427389785285422/t0 o4-&gt;b fe63770-9dc7-fabc-fd87-625dec42ca0c@NET_0x50000ac1a0a54_UUID:0/0 lens 448/416 e 0 to 0 dl 1364114285 ref 1 fl Interpret:/0/0 rc 0/0 kernel: Lustre: 21321:0:(ost_handler.c:1224:ost_brw_write()) share3-OST000e: ignoring bulk IO comm error with bfe63770-9dc7-fabc-fd87-625dec42ca0c@NET_0x 50000ac1a0a54_UUID id 12345-172.26.10.84@o2ib - client will retry kernel: Lustre: 19982:0:(ldlm_lib.c:574:target_handle_reconnect()) share3-OST000f: bfe63770-9dc7-fabc-fd87-625dec42ca0c reconnecting kernel: Lustre: 19982:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages kernel: Lustre: 20014:0:(ldlm_lib.c:574:target_handle_reconnect()) share3-OST000c: bfe63770-9dc7-fabc-fd87-625dec42ca0c reconnecting kernel: Lustre: 19981:0:(ldlm_lib.c:874:target_handle_connect()) share3-OST000d: refuse reconnection from bfe63770-9dc7-fabc-fd87-625dec42ca0c@172.26.10. 84@o2ib to 0xffff81040d190c00; still busy with 1 active RPCs kernel: LustreError: 19981:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16) req@ffff810212b87000 x1427389785285592/t0 o8-&gt;bfe63770 -9dc7-fabc-fd87-625dec42ca0c@NET_0x50000ac1a0a54_UUID:0/0 lens 368/264 e 0 to 0 dl 1364114377 ref 1 fl Interpret:/0/0 rc -16/0 </pre> </div></div> <p>Ping (IBoIP) or ibping was okay, but "lctl ping" to this OSS was failing. <br/> "lctl ping" to other OSS was okay. </p> <p>This issue was finally resolved by rebooting the client. </p> <p>Would you please check if we can say this is a network issue or something wrong on the lustre side? </p> <p>Attached is messages and debug log collected from the client and OSS. </p> <p>Regards, </p> LU-3123

A client could not communicate with an OSS due to Timed out RDMA

Bug Minor Resolved Not a Bug Bruno Faccini Mitsuhiro Nishizawa Mon, 8 Apr 2013 09:31:50 +0000 Wed, 2 Apr 2014 11:21:22 +0000 Wed, 2 Apr 2014 11:21:22 +0000 Lustre 1.8.8 0 5 <p>core dump was captured when this client was rebooted. <br/> As core dump itself is too much large, here is simple crash command output. </p> <p>What is the network/IB topology you use at least to get this Clients and OSSs connected ? Also, did you run any pure IB troubleshooting/tool, like to see/extract if some of the error/stats counters increment on involved HCA/switches/boards in the Fabric ??</p> <p>Attached crash-dump extracts show no hung/spinning thread which may indicate some LNET disfunction, and attached Client+OSS dmesg/Lustre-debuglog indicate flaky communications between OSS and Client causing multiple and recurring re-connections and recovery, since at least Sun Mar 24 03:39:09 PDT 2013, as far as I can find in the provided logs/infos.</p> <p>So definitely we need you to also investigate if no problem/errors are reported by your different IB-Fabric elements. BTW, OSS reports network errors with at least 2 Clients (172.26.10.84@o2ib, 172.26.12.181@o2ib) during the same period of time.</p> <p>Hello Mitsuhiro,<br/> Have you been able to troubleshoot you IB fabric as I indicated in my last update ??<br/> Do you still have occurrence of problem/situation ??</p> <p>Bruno, thanks for your comment. The same kind of problem occurred again and our customer has replaced cables. Currently, they are watching if this re-occur. Please proceed to close this ticket at this time. Thank you. </p> Development Rank 1|hzvnav: Rank (Obsolete) 7585 Severity