https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2977 Lustre <p>When client mounts the Lustre, we saw following error. What does "consumer defined fatal error" mean and why is this connection rejected?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 17 08:42:52 r3169 kernel: Lustre: Lustre: Build Version: RC2--PRISTINE-2.6.32-279.19.1.el6.x86_64 Mar 17 08:42:52 r3169 kernel: Lustre: Added LNI 10.9.55.1@o2ib3 [8/64/0/180] Mar 17 08:42:53 r3169 kernel: Lustre: Lustre OSC module (ffffffffa0e9b880). Mar 17 08:42:53 r3169 kernel: Lustre: Lustre LOV module (ffffffffa0f2dce0). Mar 17 08:42:53 r3169 kernel: Lustre: Lustre client module (ffffffffa1019020). Mar 17 08:42:53 r3169 kernel: Lustre: MGC10.9.103.1@o2ib3: Reactivating import Mar 17 08:42:53 r3169 kernel: LustreError: 929:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:42:53 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470173/real 1363470173] req@ffff88062749b400 x1429702099075137/t0(0) o8-&gt;images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470178 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mar 17 08:43:43 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:44:08 r3169 kernel: LustreError: 927:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:44:08 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470248/real 1363470248] req@ffff881064484800 x1429702099075267/t0(0) o8-&gt;images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470259 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:44:33 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:44:58 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:44:58 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470298/real 1363470298] req@ffff88100a3e6800 x1429702099078280/t0(0) o8-&gt;images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470314 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:45:23 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:45:48 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error Mar 17 08:45:48 r3169 kernel: Lustre: 3305:0:(client.c:1817:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1363470348/real 1363470348] req@ffff88100a019000 x1429702099078408/t0(0) o8-&gt;images-OST002e-osc-ffff880865647000@10.9.102.38@o2ib3:28/4 lens 368/512 e 0 to 1 dl 1363470369 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 08:46:13 r3169 kernel: LustreError: 11-0: an error occurred while communicating with 10.9.102.37@o2ib3. The ost_connect operation failed with -19 Mar 17 08:46:38 r3169 kernel: LustreError: 935:0:(o2iblnd_cb.c:2569:kiblnd_rejected()) 10.9.102.38@o2ib3 rejected: consumer defined fatal error </pre> </div></div> CentOS6.3 Lustre-2.1.3 LU-2977

network connection rejected due to consumer defined fatal error

Bug Minor Resolved Not a Bug Liang Zhen Shuichi Ihara Sun, 17 Mar 2013 02:28:03 +0000 Sat, 12 Jan 2019 03:49:04 +0000 Sat, 12 Jan 2019 03:49:04 +0000 Lustre 2.1.3 0 4 <p>Liang</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>That's a weird error - client didn't seem to recognize the magic number in reject messages. Was there any error showing up on 10.9.102.38@o2ib3?</p> <p>From lustre-discuss@lists.lustre.org 4/25/2017:</p> <p>Regarding:<br/> &gt; LNetError: 2878:0:(o2iblnd_cb.c:2587:kiblnd_rejected()) 172.23.55.211@o2ib rejected: consumer defined fatal error</p> <p>Andreas Dilger noted:</p> <blockquote> <p>This means that the LND didn't connect at startup time, but I don't know what the cause is.<br/> The error that generates this message is IB_CM_REJ_CONSUMER_DEFINED, but I don't know enough about IB to tell you what that means.</p></blockquote> <p>Doug Oucharek responded:</p> <blockquote> <p>That specific message happens when the “magic” u32 field at the start of a message does not match what we are expecting. We do check if the message was transmitted as a different endian from us so when you see this error, we assume that message has been corrupted or the sender is using an invalid magic value. I don’t believe this value has changed in the history of the LND so this is more likely corruption of some sort.</p></blockquote> <p>FYI: About the lustre-discuss conversation &#8211; it was determined to be a failing IB subnet manager. While sminfo reported good health, a more formal check of the manager proved that it was faulty.</p> Development End date Wed, 20 Mar 2013 02:28:03 +0000 Rank 1|hzvljz: Rank (Obsolete) 7259 Severity Start date Mon, 18 Mar 2013 02:28:03 +0000