https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-269 Lustre <p>MDS1/2 are running and active/active cluster configuration. mgs resides on MDS1 and mdt resides on MDS2. Is this an OK way to do things or not? In the original architecture of the system Oracle stated this was supported, but the customer just found out from Tyian over at Oracle that this is not a supported configuration for the MDS devices.</p> <p>As of last Thursday customer could see all raid devices by the OS, but for some reason OST11 just simply would not become available. That issue went away with their "bare metal" reboot of the system on Friday morning. What started however, we have yet to fix:</p> <p>OST15 /dev/md41 resident on OSS4<br/> From /var/log/messages<br/> Ldisk-fs error (device md41): ldiskfs_check_descriptors: Block bitma p for group 0 not in group</p> <p>This device is not mounting anymore to the OSS at the Operating System level. The raid device can be constructed, but Lustre will not mount it.</p> <p>Customer contact for questions is tyler.s.wiegers@lmco.com</p> RHEL 5.5 and Lustre 1.8.0.1 LU-269

Ldisk-fs error (device md41): ldiskfs_check_descriptors: Block bitma p for group 0 not in group

Bug Major Resolved Fixed Cliff White Dan Ferber Tue, 3 May 2011 10:59:32 +0000 Tue, 28 Jun 2011 15:01:39 +0000 Mon, 9 May 2011 07:48:01 +0000 Lustre 1.8.6 0 3 <p>Environment should be RHEL 5.3, not 5.5.</p> <p>We did a clean system startup, cleared messages, and systematically assembled the bitmaps, mounted them, assembled the raid devices, and mounted them for each OST. There were no errors for this OST until attempting to mount the raid device. All other OST's mounted successfully. </p> <p>The unique data point for this OST is that it's raid device is missing a disk (md41). The disk was reported as unknown after our CAM/firmware upgrades yesterday so we replaced it, but we did not re-insert it into the raid. Would that situation cause the errors that we currently see?</p> <p>The log output is the following:</p> <p>oss3# mount -t lustre /dev/md41 /mnt/lustre_ost15<br/> mount.lustre: mount /dev/md41 at /mnt/lustre_ost15 failed: Invalid argument<br/> This may have multiple causes.<br/> Are the mount options correct?<br/> Check the syslog for more info.</p> <p>/var/log/messages output (trimmed dates/times, I had to re-type this in from hard copy):</p> <p>LDISKFS-fs error (device md41): ldiskf_check_descriptors: Block bitmap for group 0 not in group (block 134217728)!<br/> LDISKFS-fs: group descriptors corrupted<br/> LustreError: 26127:0:(obd_mount.c:1278:server_kernel_mount()) premount /dev/md41:0x0 ldiskfs failed: -22, ldiskfs2 failed: -19. Is the ldiskfs module available?<br/> LustreError: 26127:0:(obd_mount.c:1278:server_kernel_mount()) Skipped 2 previous similar messages<br/> LustreError: 26127:0:(obd_mount.c:1590:server_fill_super()) Unable to mount device /dev/md41: -22<br/> LustreError: 26127:0:(obd_mount.c:1993:lustre_fill_super()) Unable to mount (-22)</p> <p>Thanks!</p> <p>Well,a missing disk without a spare would definitely mess up the raid, i would think.</p> <p>Was there data on the missing spindle? Has that been recovered? <br/> First, you need to be sure the md41 device is intact and that the local (ldiskfs) filesystem <br/> on that device is intact. If that requires re-inserting the disk, you should do so. </p> <p>After the md side is healthy, you should run 'fsck -fn' on md41 and see what that reports. <br/> Be sure you have the proper Lustre-aware version of e2fsprogs. (you should have downloaded it <br/> along with the rest of your Lustre RPMS) </p> <p>Assuming the md41 device is restored, 'fsck -fy' may fix the bitmap issue, but run '-fn' first<br/> -that is a read-only pass to test for errors. </p> <p>If there are other errors beyond the bitmap, you should attach the results here, but if you only<br/> have the bitmap issue, proceed with the -fy</p> <p>Also, your first question was un-related - If the MGT and MDT are separate partitions, it okay to have one node active for MGS and the other active for MDS in a failover pair - the MGS is really, really lightweight, so after client mount the MGS node should be more or less idle. </p> <p>We inserted the disk back into the raid, it is currently rebuilding. Trying to mount the OST while the disk is rebuilding gives the same error. We've been able to mount OST's while disks are rebuilding in the past, so the core issue doesn't look like it's resolved.</p> <p>No data should have been lost, we are running an 8+2 raid 6 device, so we can run with 8/10 disks without any data loss.</p> <p>Okay, after the rebuild, please run 'fsck -fn'</p> <p>&gt; We inserted the disk back into the raid, it is currently rebuilding. Trying to mount the OST<br/> &gt; while the disk is rebuilding gives the same error. We've been able to mount OST's while disks<br/> &gt; are rebuilding in the past, so the core issue doesn't look like it's resolved.</p> <p>Based on the following comment:<br/> <a href="http://jira.whamcloud.com/browse/LU-270?focusedCommentId=13656&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_13656" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-270?focusedCommentId=13656&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_13656</a></p> <p>Could you please tell us what version of the mptsas driver you use?<br/> I think the bug was fixed in versions &gt;= 4.18.20.02.</p> <p>&gt; No data should have been lost, we are running an 8+2 raid 6 device, so we can run with 8/10 disks without any data loss.</p> <p>Right.</p> <p>This disk finished rebuilding.</p> <p>Once rebuilt, we attemped to run an e2fsck on the disks, which failed due to the MMP flag being set. We cleared the flag using tune2fs (ref <a href="https://jira.whamcloud.com/browse/LU-270" title="LDisk-fs warning (device md30): ldisk_multi_mount_protect: fsck is running on filesystem" class="issue-link" data-issue-key="LU-270"><del>LU-270</del></a>) and tried the e2fsck again, which failed with another error. We ran tune2fs with the uninit-bg option, which allowed us to run e2fsck. The e2fsck ran for about 3.5 hours before completing.</p> <p>Once the e2fsck was complete, we were able to successfully mount this OST, which is extremely good news. There were multiple recovered files in lost+found which we will be attempting to recover.</p> <p>We are in the process of running e2fsck on all of our OSTs. Once complete we are planning a complete power down of all OSS's, MDS's, and disk arrays in order to do a fresh clean startup. I will update no later than tomorrow with hopefully a <b>problem resolved</b> statement.</p> <p>Great, thanks for keeping us updated<br/> cliffw</p> <p>&#8211; <br/> cliffw<br/> Support Guy<br/> WhamCloud, Inc.<br/> www.whamcloud.com</p> <p>Rob Baker of LMCO has confirmed that the critical situation is over and production is stable. Residual issues will be tracked under a new ticket in the future.</p> Development Rank 1|hzw2un: Rank (Obsolete) 10544 Severity