https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2811 Lustre <p>After running fine for about 8 weeks:</p> <p>Message from syslogd@pfs1n1 at Thu Feb 14 08:30:05 2013 ...<br/> pfs1n1 kernel: <span class="error">&#91;1452648.918541&#93;</span> LustreError: <br/> 15442:0:(mdt_lib.c:541:mdt_dump_lmm()) ASSERTION( stripe_count &lt;= <br/> (__s16)160 ) failed:</p> <p>Message from syslogd@pfs1n1 at Thu Feb 14 08:30:05 2013 ...<br/> pfs1n1 kernel: <span class="error">&#91;1452648.928990&#93;</span> LustreError: <br/> 15442:0:(mdt_lib.c:541:mdt_dump_lmm()) LBUG</p> <p>Message from syslogd@pfs1n1 at Thu Feb 14 08:30:05 2013 ...<br/> pfs1n1 kernel: <span class="error">&#91;1452649.070201&#93;</span> Kernel panic - not syncing: LBUG</p> Vanilla Kernel 2.6.32.59<br/> 12 OSTs<br/> approx. 980 clients running a mix of 1.8.x and 2.3 LU-2811

LBUG: stripe_count > LOV_MAX_STRIPE_COUNT

Bug Major Resolved Duplicate Jian Yu Roland Fehrenbacher Thu, 14 Feb 2013 06:17:22 +0000 Thu, 27 Aug 2015 14:32:28 +0000 Thu, 27 Aug 2015 14:32:28 +0000 Lustre 2.1.3 0 7 <p>Hello,<br/> Was there a crash-dump taken at the time of the LBUG ?</p> <p>I can't tell, since even if it was, it would have been wiped by the /tmp cleaning script upon reboot <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Roland,<br/> I was not speaking about a Lustre debug log dumped into /tmp, but a full system crash-dump, meaning that Kdump or such kind of tool is installed+configured on your system. Are you aware of that ?</p> <p>Unfortunately not.</p> <p>The system is part of an HA pair and got reset by the peer. So we couldn't even copy anything from the console.</p> <p>Without at least the stack trace, there isn't really enough information available about what triggered this problem. Is it corruption of an on-disk LOV EA? Was it a corrupted setstripe request from a client? Is there some sanity checking missing in some normal code path? Was this a problem with a 1.8 or 2.1 client?</p> <p>Roland, some things that could help debugging in the future:</p> <ul class="alternate" type="square"> <li>connecting a serial cable on each of the MDS nodes and log to a third system (or each other), perhaps in conjunction with "conman" which can manage ethernet-attached serial console servers</li> <li>using "netconsole" to send console logs to a remote node via UDP</li> <li>configure "kdump" and/or "netdump" to capture crash dumps (this gives by far the most debugging information)</li> </ul> <p>I'm going to close this bug for now, since there isn't enough information to figure out what went wrong. Please re-open if it happens again.</p> <p>OK. I already thought that it would be hard to find the cause with only so much info. I see how easy it will be to get kdump on the cluster. Thanks for looking at this anyway.</p> <p>Thanks Roland. The other thing to ask is - have any changes been made to the vanilla Lustre code on either of the releases?</p> <p>Not as far as I know (the SuSE clients are not managed by myself, they are running 2.6.32/1.8.x, 3.0/2.3 respectively). </p> <p>We have a reproducer for this error, basically if you are running a newer client such as 2.4.2 and mount an older 2.1.5 Lustre filesystem, users can set the stripe to greater than the max stripe setting of 160 in 2.1 and trigger this crash as soon as they query or access the directory with stripe count set to more than 160.</p> <p>Yu, jian</p> <p>Could you please take care of this one?</p> <p>Thanks</p> <p>Peter</p> <p>I'll reproduce and investigate the failure.</p> <p>Can you try patch <a href="http://review.whamcloud.com/#/c/9734" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9734</a>.</p> <blockquote><p>Can you try patch <a href="http://review.whamcloud.com/#/c/9734" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9734</a>.</p></blockquote> <p>Sure, thanks!</p> <p>Before validating the patch, I can reproduce the failure on Lustre 2.1.6 server with Lustre 2.4.3 client according to the steps from Tommy.</p> <p>On Lustre 2.4.3 client node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mkdir /mnt/lustre/dir # lfs getstripe -d /mnt/lustre/dir stripe_count: 1 stripe_size: 1048576 stripe_offset: -1 # lfs setstripe -c 200 /mnt/lustre/dir # lfs getstripe -d /mnt/lustre/dir &lt;------------- hung here </pre> </div></div> <p>Console log on Lustre 2.1.6 MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400):0:0 LustreError: 8510:0:(mdt_lib.c:543:mdt_dump_lmm()) ASSERTION( stripe_count &lt;= (__s16)160 ) failed: LustreError: 8510:0:(mdt_lib.c:543:mdt_dump_lmm()) LBUG Pid: 8510, comm: mdt_00 Call Trace: [&lt;ffffffffa0425785&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [&lt;ffffffffa0425d97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [&lt;ffffffffa0d4fb72&gt;] mdt_dump_lmm+0x272/0x280 [mdt] [&lt;ffffffffa0d495f2&gt;] mdt_getattr_internal+0x672/0xe90 [mdt] [&lt;ffffffffa06bb6c0&gt;] ? lustre_swab_mdt_body+0x0/0x150 [ptlrpc] [&lt;ffffffffa0d4a035&gt;] mdt_getattr+0x225/0x920 [mdt] [&lt;ffffffffa0d40772&gt;] mdt_handle_common+0x932/0x1750 [mdt] [&lt;ffffffffa0d41665&gt;] mdt_regular_handle+0x15/0x20 [mdt] [&lt;ffffffffa06c8b9e&gt;] ptlrpc_main+0xc4e/0x1a40 [ptlrpc] [&lt;ffffffffa06c7f50&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa06c7f50&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffffa06c7f50&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 Kernel panic - not syncing: LBUG </pre> </div></div> <p>I'll back-port the above patch to Lustre b2_1 branch and try again.</p> <p>Here is the back-ported patch for Lustre b2_1 branch <a href="http://review.whamcloud.com/9884" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9884</a>. I'll validate it.</p> <p>With the above patch on Lustre b2_1 branch, the same test passed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mkdir /mnt/lustre/dir # lfs getstripe -d /mnt/lustre/dir stripe_count: 1 stripe_size: 1048576 stripe_offset: -1 # lfs setstripe -c 200 /mnt/lustre/dir # lfs getstripe -d /mnt/lustre/dir stripe_count: 200 stripe_size: 1048576 stripe_offset: -1 # touch /mnt/lustre/dir/file # lfs getstripe -i -c -s /mnt/lustre/dir/file lmm_stripe_count: 160 lmm_stripe_size: 1048576 lmm_stripe_offset: 93 </pre> </div></div> <p>Since Lustre 2.1 is no longer supported we should close this ticket. The solution if someone needs it is in this ticket.</p> Duplicate LU-5578 Development Rank 1|hzvj4f: Rank (Obsolete) 6812 Severity