https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3417 Lustre <p>We had a server running Lustre over ZFS, Lustre version <a href="https://github.com/chaos/lustre/tree/2.3.63-6chaos" class="external-link" target="_blank" rel="nofollow noopener">2.3.63-6chaos</a> that hit the following assertion. Clients were mostly BG/Q clients, with some 2.1 x86_64 as well.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&lt;ConMan&gt; Console [vesta56] log at 2013-05-24 17:00:00 PDT. 2013-05-24 17:51:58 LustreError: 33230:0:(sec_null.c:320:null_alloc_rs()) vmalloc of 'rs' (-1073741304 bytes) failed 2013-05-24 17:51:58 LustreError: 33230:0:(sec_null.c:320:null_alloc_rs()) 369821362 total bytes allocated by Lustre, 593003541 by LNET 2013-05-24 17:51:58 LustreError: 33230:0:(pack_generic.c:428:lustre_msg_buf_v2()) msg ffff880da3718108 buffer[1] size -1073741792 too small (required 0, opc=0) 2013-05-24 17:51:58 LNetError: 33230:0:(o2iblnd_cb.c:1601:kiblnd_send()) ASSERTION( __builtin_offsetof(kib_msg_t,ibm_u.immediate.ibim_payload[payload_nob]) &lt;= (4&lt;&lt;10) ) failed: 2013-05-24 17:51:58 LNetError: 33230:0:(o2iblnd_cb.c:1601:kiblnd_send()) LBUG </pre> </div></div> <p>When the failover partner started up, it hit the same assertion after recovery completed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2013-05-24 17:56:42 Lustre: fsv-OST0037: Client e985e75f-abf3-b986-3da7-3cd1c4f29af0 (at 172.20.17.95@o2ib500) refused reconnection, still busy with 1 active RPCs 2013-05-24 17:56:42 Lustre: Skipped 128 previous similar messages 2013-05-24 17:57:25 Lustre: fsv-OST0037: Recovery over after 3:45, of 405 clients 405 recovered and 0 were evicted. 2013-05-24 17:57:25 LustreError: 11574:0:(sec_null.c:320:null_alloc_rs()) vmalloc of 'rs' (-1073741304 bytes) failed 2013-05-24 17:57:25 LustreError: 11574:0:(sec_null.c:320:null_alloc_rs()) 408246602 total bytes allocated by Lustre, 1140997813 by LNET 2013-05-24 17:57:25 LustreError: 11574:0:(pack_generic.c:428:lustre_msg_buf_v2()) msg ffff880e344b8108 buffer[1] size -1073741792 too small (required 0, opc=0) 2013-05-24 17:57:25 LNetError: 11574:0:(o2iblnd_cb.c:1601:kiblnd_send()) ASSERTION( __builtin_offsetof(kib_msg_t,ibm_u.immediate.ibim_payload[payload_nob]) &lt;= (4&lt;&lt;10) ) failed: </pre> </div></div> <p>This happened over and over again for some time until a sysadmin intervened and aborted recovery manually. That seemed to allow everything to start up normally.</p> Lustre 2.3.63-6chaos (<a href="https://github.com/chaos/lustre/tree/2.3.63-6chaos">https://github.com/chaos/lustre/tree/2.3.63-6chaos</a>) LU-3417

ASSERTION( __builtin_offsetof(kib_msg_t,ibm_u.immediate.ibim_payload[payload_nob]) <= (4<<10) ) failed:

Bug Major Resolved Cannot Reproduce Bruno Faccini Christopher Morrone llnl Thu, 30 May 2013 00:21:44 +0000 Wed, 13 Oct 2021 03:03:51 +0000 Wed, 13 Oct 2021 03:03:51 +0000 Lustre 2.4.0 0 4 <p>Bruno is looking into this one</p> <p>Humm the negative sizes in the error msgs look strange but seem not significant at all ..., was a crash-dump taken at the time of one of these LBUGs in a loop ??</p> <p>Hello Christopher,<br/> Did you read my previous update+question ??<br/> Since I am not able to explain what happen with these error msgs, if a crash-dump of one occurence could be available it would greatly help.</p> <p>I believe we have a crash dump somewhere, not from the initial assertion, but from one of the failover assertions. Could be a while before we look into it. So far this only hit once, and we have a lot of higher priority issues at the moment.</p> Development End date Thu, 26 Jun 2014 00:21:44 +0000 Rank 1|hzvs8n: Rank (Obsolete) 8472 Severity Start date Thu, 30 May 2013 00:21:44 +0000