https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9242 Lustre <p>With my testing of the latest master branch I see jobs failing due to the lose of communication with the OSS servers. I see a reconnect storm but in the end the application error out.</p> Lustre 2.9.54 running on servers using RHEL7 and using ldiskfs. Client side is Cray SLES11SP4 also running lustre 2.9.54 LU-9242

Applications are failing to complete due to connection loss with OSS servers

Bug Blocker Resolved Cannot Reproduce Jian Yu James A Simmons Wed, 22 Mar 2017 16:39:27 +0000 Wed, 26 Apr 2017 16:09:37 +0000 Tue, 25 Apr 2017 16:20:33 +0000 Lustre 2.10.0 0 3 <p>Jian</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi James,</p> <p>From the attached debug log, I saw a lot of the following error messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00000040:20.0:1490199768.961594:0:20430:0:(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff880775b17800 x1562585786929952/t0(0) o104-&gt;sultan-OST000c@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22 00000100:00000040:20.0:1490199768.961736:0:20430:0:(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff880775b11800 x1562585786930480/t0(0) o104-&gt;sultan-OST000c@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22 00000100:00000040:20.0:1490199768.961887:0:20430:0:(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff880775b14200 x1562585786930880/t0(0) o104-&gt;sultan-OST000c@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22 00000100:00000040:19.0:1490199768.961965:0:20412:0:(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff88077b6f8f00 x1562585786934704/t0(0) o104-&gt;sultan-OST0018@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22 00000100:00000040:3.0:1490199768.962021:0:20501:0:(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff88006cc7d700 x1562585786931872/t0(0) o104-&gt;sultan-OST0028@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22 00000100:00000040:6.0:1490199768.962058:0:20511:0:(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff880068527800 x1562585786934880/t0(0) o104-&gt;sultan-OST0014@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22 00000100:00000040:19.0:1490199768.962109:0:20412:0:(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff88077b6f9e00 x1562585786936128/t0(0) o104-&gt;sultan-OST0018@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22 ...... </pre> </div></div> <p>And the context of one of the above error messages is sth like:</p> <div class="panel" style="border-width: 1px;"><div class="panelContent"> <p>00000100:00000001:20.0:1490199768.961593:0:20430:0:(client.c:1251:ptlrpc_check_status()) Process entered<br/> ......<br/> 00000800:00000200:30.0:1490199768.961594:0:16000:0:(o2iblnd_cb.c:194:kiblnd_post_rx()) conn<span class="error">&#91;ffff880fed77f600&#93;</span> (200)++<br/> 00000100:00000040:20.0:1490199768.961594:0:20430:0:<font color="red">(client.c:1271:ptlrpc_check_status()) @@@ status is -22 req@ffff880775b17800 x1562585786929952/t0(0) o104&#45;&gt;sultan-OST000c@3@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Rpc:R/2/0 rc 0/-22</font><br/> ......<br/> 00000100:00000001:1.0:1490199768.961594:0:15994:0:(events.c:91:reply_in_callback()) Process entered<br/> 00000400:00000200:31.0:1490199768.961595:0:27379:0:(lib-move.c:2762:LNetPut()) LNetPut &#45;&gt; 12345-3@gni1<br/> ......<br/> 00000020:00000001:25.0:1490199768.961595:0:27333:0:(tgt_handler.c:1203:tgt_blocking_ast()) Process entered<br/> ......<br/> 00000800:00000200:8.0:1490199768.961595:0:15999:0:<font color="red">(o2iblnd_cb.c:787:kiblnd_post_tx_locked()) 10.37.202.61@o2ib1: no credits</font><br/> 00000100:00000200:1.0:1490199768.961595:0:15994:0:(events.c:93:reply_in_callback()) @@@ type 2, status 0 req@ffff880068525100 x1562585786933744/t0(0) o104&#45;&gt;sultan-OST0014@28@gni1:15/16 lens 296/224 e 0 to 0 dl 1490199781 ref 1 fl Rpc:/2/ffffffff rc 0/-1<br/> 00010000:00010000:26.0:1490199768.961596:0:27337:0:(ldlm_lockd.c:541:ldlm_del_waiting_lock()) ### removed ns: filter-sultan-OST0028_UUID lock: ffff88079d7d4e00/0x43103176140b8536 lrc: 4/0,0 mode: PR/PR res: <span class="error">&#91;0x1204:0x0:0x0&#93;</span>.0x0 rrc: 78 type: EXT <span class="error">&#91;0\-&gt;18446744073709551615&#93;</span> (req 0&#45;&gt;18446744073709551615) flags: 0x50000400010020 nid: 2@gni1 remote: 0xeadfeaea960a0c6d expref: 25 pid: 20407 timeout: 4296946798 lvb_type: 1<br/> 00010000:00000001:25.0:1490199768.961596:0:27333:0:(ldlm_lockd.c:847:ldlm_server_blocking_ast()) Process leaving (rc=0 : 0 : 0)<br/> 00000100:00000040:18.0:1490199768.961596:0:20474:0:(lustre_net.h:2451:ptlrpc_rqphase_move()) @@@ move req "Interpret" &#45;&gt; "Complete" req@ffff88077b61c200 x1562585786932288/t0(0) o104&#45;&gt;sultan-OST0008@28@gni1:15/16 lens 296/192 e 0 to 0 dl 1490199781 ref 1 fl Interpret:R/2/0 rc 0/0<br/> 00000100:00000200:15.0:1490199768.961596:0:17866:0:(niobuf.c:85:ptl_send_buf()) Sending 192 bytes to portal 18, xid 1562506803021520, offset 192<br/> 00000800:00000200:8.0:1490199768.961596:0:15999:0:(o2iblnd_cb.c:223:kiblnd_post_rx()) conn<span class="error">&#91;ffff880fed77f600&#93;</span> (201)--<br/> 00000800:00000200:30.0:1490199768.961597:0:16000:0:<font color="red">(o2iblnd_cb.c:787:kiblnd_post_tx_locked()) 10.37.202.61@o2ib1: no credits</font><br/> 00000020:00000001:25.0:1490199768.961597:0:27333:0:(tgt_handler.c:1258:tgt_blocking_ast()) Process leaving (rc=0 : 0 : 0)<br/> 00000100:00000040:24.0:1490199768.961597:0:27378:0:(lustre_net.h:2451:ptlrpc_rqphase_move()) @@@ move req "Interpret" &#45;&gt; "Complete" req@ffff880fb7677050 x1562506803021552/t0(0) o103&#45;&gt;2d014be5-e7a1-c537-05ce-72f61bb4bd8d@3@gni1:-1/-1 lens 328/192 e 0 to 0 dl 1490200085 ref 1 fl Interpret:H/0/0 rc 0/0<br/> 00000800:00000200:8.0:1490199768.961597:0:15999:0:(o2iblnd_cb.c:3652:kiblnd_scheduler()) conn<span class="error">&#91;ffff880fed77f600&#93;</span> (200)--<br/> 00000100:00000001:31.0:1490199768.961598:0:27379:0:(niobuf.c:100:ptl_send_buf()) Process leaving (rc=0 : 0 : 0)<br/> 00000800:00000200:30.0:1490199768.961598:0:16000:0:(o2iblnd_cb.c:223:kiblnd_post_rx()) conn<span class="error">&#91;ffff880fed77f600&#93;</span> (200)--<br/> 00000400:00000010:15.0:1490199768.961598:0:17866:0:(lib-lnet.h:444:lnet_msg_alloc()) alloc '(msg)': 400 at ffff88077a3e9600 (tot -230794364).<br/> 00000400:00000100:9.0:1490199768.961598:0:15997:0:(lib-move.c:2024:lnet_parse_put()) Dropping PUT from 12345-3@gni1 portal 16 match 1562585786931952 offset 192 length 192: 4<br/> 00000100:00000001:31.0:1490199768.961599:0:27379:0:(connection.c:91:ptlrpc_connection_put()) Process entered<br/> 00000800:00000200:30.0:1490199768.961599:0:16000:0:(o2iblnd_cb.c:3652:kiblnd_scheduler()) conn<span class="error">&#91;ffff880fed77f600&#93;</span> (198)--<br/> 00000020:00000001:26.0:1490199768.961599:0:27337:0:(tgt_handler.c:1203:tgt_blocking_ast()) Process entered<br/> 00010000:00000001:25.0:1490199768.961599:0:27333:0:(ldlm_lock.c:411:ldlm_lock_destroy_nolock()) Process entered<br/> 00000100:00000001:20.0:1490199768.961599:0:20430:0:<font color="red">(client.c:1277:ptlrpc_check_status()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea)</font></p> </div></div> <p>Are there any error messages in the syslog/console logs on the Client and OSS nodes? Could you please also gather the debug log on OSS node? Thank you.</p> <p>The attached debug logs are from the OSS nodes. I'm seeing the console messages on the OSS server</p> <p><span class="error">&#91;16278.899599&#93;</span> Lustre: sultan-OST0000: Client ecf7eb35-3b79-6131-416a-61beb23f4a96 (at 30@gni1) reconnecting<br/> <span class="error">&#91;16278.899639&#93;</span> Lustre: sultan-OST000c: Connection restored to bde05182-67f0-f31e-0491-5a81dc20c8d4 (at 30@gni1)<br/> <span class="error">&#91;16278.899640&#93;</span> Lustre: Skipped 106 previous similar messages<br/> <span class="error">&#91;16278.927976&#93;</span> Lustre: Skipped 17 previous similar messages<br/> <span class="error">&#91;16315.958925&#93;</span> Lustre: sultan-OST0000: Client ecf7eb35-3b79-6131-416a-61beb23f4a96 (at 30@gni1) reconnecting<br/> <span class="error">&#91;16315.969803&#93;</span> Lustre: Skipped 21 previous similar messages<br/> <span class="error">&#91;16352.961881&#93;</span> Lustre: sultan-OST0008: Client ecf7eb35-3b79-6131-416a-61beb23f4a96 (at 30@gni1) reconnecting<br/> <span class="error">&#91;16352.972685&#93;</span> Lustre: Skipped 17 previous similar messages<br/> <span class="error">&#91;16446.989954&#93;</span> Lustre: sultan-OST0000: Client ecf7eb35-3b79-6131-416a-61beb23f4a96 (at 30@gni1) reconnecting<br/> <span class="error">&#91;16446.989955&#93;</span> Lustre: sultan-OST0004: Client ecf7eb35-3b79-6131-416a-61beb23f4a96 (at 30@gni1) reconnecting<br/> <span class="error">&#91;16446.989974&#93;</span> Lustre: Skipped 34 previous similar messages<br/> <span class="error">&#91;16447.018332&#93;</span> Lustre: Skipped 8 previous similar messages<br/> <span class="error">&#91;16517.552863&#93;</span> LNet: 17652:0:(o2iblnd_cb.c:3190:kiblnd_check_conns()) Timed out tx for 10.37.202.59@o2ib1: 0 seconds<br/> <span class="error">&#91;16715.012794&#93;</span> Lustre: sultan-OST0004: Client ecf7eb35-3b79-6131-416a-61beb23f4a96 (at 30@gni1) reconnecting<br/> <span class="error">&#91;16715.023784&#93;</span> Lustre: Skipped 61 previous similar messages<br/> <span class="error">&#91;16986.035621&#93;</span> Lustre: sultan-OST0018: Connection restored to bde05182-67f0-f31e-0491-5a81dc20c8d4 (at 30@gni1)<br/> <span class="error">&#91;16986.046729&#93;</span> Lustre: Skipped 207 previous similar messages<br/> <span class="error">&#91;17385.169047&#93;</span> Lustre: sultan-OST0008: Client b0eed11e-57d2-c40f-9e0b-eadc461d30f5 (at 26@gni1) reconnecting<br/> <span class="error">&#91;17385.179909&#93;</span> Lustre: Skipped 145 previous similar messages<br/> <span class="error">&#91;17627.146131&#93;</span> Lustre: sultan-OST0024: Connection restored to bde05182-67f0-f31e-0491-5a81dc20c8d4 (at 30@gni1)<br/> <span class="error">&#91;17627.157290&#93;</span> Lustre: Skipped 191 previous similar messages<br/> <span class="error">&#91;17994.177003&#93;</span> Lustre: sultan-OST0010: Client ecf7eb35-3b79-6131-416a-61beb23f4a96 (at 30@gni1) reconnecting<br/> <span class="error">&#91;17994.187967&#93;</span> Lustre: Skipped 195 previous similar messages</p> <p>On the client node I see in the console logs:</p> <p>stre: Skipped 55 previous similar messages<br/> Lustre: 6048:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 411 previous similar messages<br/> Lustre: sultan-OST0026-osc-ffff8803bd927800: Connection restored to 10.37.248.83@o2ib1 (at 10.37.248.83@o2ib1)<br/> Lustre: Skipped 55 previous similar messages<br/> Lustre: sultan-OST0032-osc-ffff8803bd927800: Connection to sultan-OST0032 (at 10.37.248.83@o2ib1) was lost; in progress operations using this service will wait for recovery to complete<br/> Lustre: Skipped 54 previous similar messages<br/> Lustre: 6050:0:(client.c:2113:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1490308346/real 1490308346&#93;</span> req@ffff8803b4dedcc0 x1562697286159168/t0(0) o400-&gt;sultan-OST0026-osc-ffff8803bd927800@10.37.248.83@o2ib1:28/4 lens 224/224 e 0 to 1 dl 1490308353 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Lustre: 6050:0:(client.c:2113:ptlrpc_expire_one_request()) Skipped 460 previous similar messages<br/> Lustre: sultan-OST0026-osc-ffff8803bd927800: Connection to sultan-OST0026 (at 10.37.248.83@o2ib1) was lost; in progress operations using this service will wait for recovery to complete<br/> Lustre: Skipped 110 previous similar messages<br/> Lustre: sultan-OST0026-osc-ffff8803bd927800: Connection restored to 10.37.248.83@o2ib1 (at 10.37.248.83@o2ib1)<br/> Lustre: Skipped 167 previous similar messages</p> <p>I will get the client logs as well tomorrow.</p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p> </p> <p>Hi James,<br/> Could you please also run "<tt>lctl set_param printk=+neterror</tt>" on both Client and OSS nodes to see if there are any LNet network errors on console?</p> <p>James</p> <p>Do you still have this test environment setup? Is this simply due to Multi-Rail LNET not being able to work in this configuration?</p> <p>Peter</p> <p>James</p> <p>It does not seem that this is still a live issue for you so I will close for the time being. If you do see this issue on more current builds and can provide enough data to debug then we'll of course reopen </p> <p>Peter</p> <p>I would say its safe to keep this closed. I have updated my test bed to the latest master and I'm not seeing the application failures.</p> <p>thanks for confirming James</p> Development Rank 1|hzz7yn: Rank (Obsolete) 9223372036854775807 Severity