https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6041 Lustre <p>A backend storage issue caused an OST to get mounted read-only. Remounting showed that the journal entry in the superblock was corrupt. The journal was removed and e2fsck continued; fixing every issue moved all entries into lost+found directory (when mounting the OST as ldiskfs). Next the ext3 internal journal was added back using tune2fs -j /dev/sdd; and another e2fsck was run. It completed successfully. Using ll_recover_lost_found_objs, it appears that the object data is back in place, but there is no CONFIGS, quota_slave, or REMOTE_PARENT_DIR entries. Additionally the oi.* entries are missing as well as the health_check and last_rcvd files. </p> <p>Is there any way to recover these entries? In the e2fsck output we saw things that look like <a href="https://jira.whamcloud.com/browse/LU-2638" title="corruption of MDT &quot;..&quot; entry in some ldiskfs directories" class="issue-link" data-issue-key="LU-2638"><del>LU-2638</del></a> and also <a href="https://jira.whamcloud.com/browse/LU-2901" title="Duplicate filename on the same ldiskfs directory on MDS" class="issue-link" data-issue-key="LU-2901"><del>LU-2901</del></a>. </p> <p>At this time e2fsck runs to completion and complains about block bitmap differences (lots of them), and Free blocks count (12519 of these messages). </p> LU-6041

Recover OST specific directories after e2fsck

Bug Minor Resolved Fixed Jian Yu Jason Hill Wed, 17 Dec 2014 19:49:11 +0000 Mon, 22 Dec 2014 19:11:52 +0000 Mon, 22 Dec 2014 19:11:51 +0000 0 5 <p>We're running the following versions of e2fsprogs:<br/> <span class="error">&#91;lost+found&#93;</span># rpm -qa | grep e2fs<br/> e2fsprogs-libs-1.42.9.wc1-7.el6.x86_64<br/> e2fsprogs-devel-1.42.9.wc1-7.el6.x86_64<br/> e2fsprogs-static-1.42.9.wc1-7.el6.x86_64<br/> e2fsprogs-1.42.9.wc1-7.el6.x86_64<br/> e2fsprogs-debuginfo-1.42.9.wc1-7.el6.x86_64</p> <p>Jian</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>There are newer e2fsprogs-1.42.12-wc1, but they won't fix the missing files. </p> <p>That said, they should all be recreated when the OST is mounted again. Most of the files are not needed on OSTs in any case. </p> <p>Andreas,</p> <p>Something I didn't mention above (grave oversight, my fault) is that mount -t lustre /dev/sdd /tmp/lustre/fs/ost11 fails with "This device hasn't been formatted by Lustre". </p> <p>Would something as simple as a tunefs.lustre --writeconf (with the correct parameters) be sufficient to get this correct? Or would it be better to copy the CONFIGS directory from another OST and then do the writeconf be better?</p> <p><span class="error">&#91;oss3 /&#93;</span># mount -t lustre /dev/sdd /mnt<br/> mount.lustre: /dev/sdd has not been formatted with mkfs.lustre or the backend filesystem type is not supported by this tool</p> <p>dmesg shows no errors. </p> <p><span class="error">&#91;oss3 /&#93;</span># dmesg | grep -i lustre<br/> <span class="error">&#91;oss3 /&#93;</span># uptime<br/> 11:04:37 up 2 days, 50 min, 2 users, load average: 0.13, 0.07, 0.01</p> <p>Is the CONFIGS/ directory possibly still in <tt>lost+found</tt>? There shouldn't be much left there after ll_recover_lost_found_objs moved all the objects back to their respective directories. </p> <p>In particular, CONFIGS/mountdata is one file that isn't recreated automatically at mount time, since that contains info on how to mount tge filesystem, and is what mount.lustre is looking for. </p> <p>We've recreated this file in the past by copying it from another OST and binary editing the OST index (two places in struct lustre_disk_data: ldd_svname and ldd_svindex). </p> <p>It might be just as easy to create a small test filesystem on a test node like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>OSTCOUNT=1 FSNAME={name} sh llmount.sh touch /tmp/ostN mkfs.lustre --ost --mgsname=$HOSTNAME --index={index of broken OST} /tmp/ostN losetup /dev/loop4 /tmp/ostN mkdir /mnt/ostN mount -t lustre /dev/loop4 /mnt/ostN umount /mnt/ostN mount -t ldiskfs /dev/loop4 /mnt/ostN cp /mnt/ostN/CONFIGS/mountdata /tmp </pre> </div></div> <p>So I tried the copy from another OST path, modified CONFIGS/mountdata and renamed CONFIGS/sithfs-OST000a to CONFIGS/sithfs-OST000b and then made the modifications there as well. </p> <p>Mounting now I get errors to stdout saying bad MGS specification and dmesg shows the following:</p> <p><span class="error">&#91;root@sith-oss3 CONFIGS&#93;</span># dmesg<br/> <span class="error">&#91;333605.012087&#93;</span> LDISKFS-fs (sdd): mounted filesystem with ordered data mode. quota=off. Opts: <br/> <span class="error">&#91;333607.639176&#93;</span> LustreError: 11-0: MGC10.36.227.244@o2ib: Communicating with 10.36.227.244@o2ib, operation mgs_target_reg failed with -107.<br/> <span class="error">&#91;333607.652202&#93;</span> LustreError: 166-1: MGC10.36.227.244@o2ib: Connection to MGS (at 10.36.227.244@o2ib) was lost; in progress operations using this service will fail<br/> <span class="error">&#91;333607.667622&#93;</span> LustreError: 3510:0:(obd_mount_server.c:1120:server_register_target()) sithfs-OST000b: error registering with the MGS: rc = -107 (not fatal)<br/> <span class="error">&#91;333607.668041&#93;</span> Lustre: Evicted from MGS (at MGC10.36.227.244@o2ib_0) after server handle changed from 0xddbecfbc8c068220 to 0xddbecfbc8c068506<br/> <span class="error">&#91;333607.668349&#93;</span> Lustre: MGC10.36.227.244@o2ib: Connection restored to MGS (at 10.36.227.244@o2ib)<br/> <span class="error">&#91;333607.706663&#93;</span> LustreError: 3510:0:(llog_osd.c:254:llog_osd_read_header()) sithfs-OST000b-osd: bad log sithfs-OST000b <span class="error">&#91;0xa:0xa:0x0&#93;</span> header magic: 0x32303020 (expected 0x10645539)<br/> <span class="error">&#91;333607.723413&#93;</span> LustreError: 3510:0:(llog_osd.c:254:llog_osd_read_header()) Skipped 1 previous similar message<br/> <span class="error">&#91;333607.733782&#93;</span> LustreError: 3510:0:(mgc_request.c:1707:mgc_llog_local_copy()) MGC10.36.227.244@o2ib: failed to copy remote log sithfs-OST000b: rc = -5<br/> <span class="error">&#91;333607.747895&#93;</span> LustreError: 13a-8: Failed to get MGS log sithfs-OST000b and no local copy.<br/> <span class="error">&#91;333607.756602&#93;</span> LustreError: 15c-8: MGC10.36.227.244@o2ib: The configuration from log 'sithfs-OST000b' failed (-2). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> <span class="error">&#91;333607.781476&#93;</span> LustreError: 3510:0:(obd_mount_server.c:1252:server_start_targets()) failed to start server sithfs-OST000b: -2<br/> <span class="error">&#91;333607.793410&#93;</span> LustreError: 3510:0:(obd_mount_server.c:1723:server_fill_super()) Unable to start targets: -2<br/> <span class="error">&#91;333607.803769&#93;</span> LustreError: 3510:0:(obd_mount_server.c:845:lustre_disconnect_lwp()) sithfs-MDT0000-lwp-OST000b: Can't end config log sithfs-client.<br/> <span class="error">&#91;333607.817767&#93;</span> LustreError: 3510:0:(obd_mount_server.c:1420:server_put_super()) sithfs-OST000b: failed to disconnect lwp. (rc=-2)<br/> <span class="error">&#91;333607.830024&#93;</span> LustreError: 3510:0:(obd_mount_server.c:1450:server_put_super()) no obd sithfs-OST000b<br/> <span class="error">&#91;333607.949653&#93;</span> Lustre: server umount sithfs-OST000b complete<br/> <span class="error">&#91;333607.955474&#93;</span> LustreError: 3510:0:(obd_mount.c:1325:lustre_fill_super()) Unable to mount (-2)<br/> <span class="error">&#91;333707.624293&#93;</span> LDISKFS-fs (sdd): mounted filesystem with ordered data mode. quota=off. Opts:</p> <p>I either missed something or need to remove something from the local (OST) CONFIGS directory, correct? It looks like Lustre is trying to do the right thing and copy the remote log down:</p> <p><span class="error">&#91;333607.733782&#93;</span> LustreError: 3510:0:(mgc_request.c:1707:mgc_llog_local_copy()) MGC10.36.227.244@o2ib: failed to copy remote log sithfs-OST000b: rc = -5</p> <p>Is it just a permission issue? (perms look the same as the OST I copied the CONFIGS directory from..</p> <p><span class="error">&#91;root@sith-oss3 CONFIGS&#93;</span># ls -la<br/> total 104<br/> drwxr-xr-x 2 root root 4096 Dec 22 12:29 .<br/> drwxr-xr-x 7 root root 4096 Dec 18 15:38 ..<br/> <del>rw-r</del><del>r</del>- 1 root root 12289 Dec 18 15:38 mountdata<br/> <del>rw-r</del><del>r</del>- 1 root root 0 Dec 18 15:53 params<br/> <del>rw-r</del><del>r</del>- 1 root root 46081 Dec 18 16:01 sithfs-OST000b<br/> <del>rw-r</del><del>r</del>- 1 root root 30384 Dec 18 15:53 sithfs-client</p> <p>I removed the CONFIGS/sithfs-OST000b file and remounted as lustre; got the "correct" version from the MGS. Now to see if the e2fsck completely screwed up the data or not. </p> <p>Thanks for your help!</p> <p>The CONFIGS/$fsname-OSTnnnn file is unique to each OST, so copying it from the other OST wouldn't help as you saw. It is fetched from the MGS at mount.</p> Development Rank 1|hzx2q7: Rank (Obsolete) 16839 Severity