https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11574 Lustre <p>Filesystem is down.</p> <p>We had raid backend issues and oss crashed. We ran run fsck after crash. when trying to remount ost we get this error. I have tried to run fsck several times.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 4993.782431] LustreError: 157-3: Trying to start OBD nbp13-OST000b_UUID using the wrong disk . Were the /dev/ assignments rearranged? [ 4993.825963] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff882dcd3a2f00[0x0, 1, [0x1:0x0:0x0] hash exist]{ [ 4993.825963] [ 4993.864146] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff882dcd3a2f50 [ 4993.864146] [ 4993.896057] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-ldiskfs@ffff882df011ba00osd-ldiskfs-object@ffff882df011ba00(i:ffff882b7af88958:78/2138703796)[plain] [ 4993.896057] [ 4993.947314] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff882dcd3a2f00 [ 4993.947314] [ 4993.978176] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff882dcd3a3140[0x0, 1, [0x200000003:0x0:0x0] hash exist]{ [ 4993.978176] [ 4994.018450] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff882dcd3a3190 [ 4994.018450] [ 4994.050361] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-ldiskfs@ffff882eee0e8d00osd-ldiskfs-object@ffff882eee0e8d00(i:ffff882b7af6f2d0:77/2138703762)[plain] [ 4994.050361] [ 4994.101617] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff882dcd3a3140 [ 4994.101617] [ 4994.137076] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff882dcd3a2c00[0x0, 1, [0xa:0x0:0x0] hash exist]{ [ 4994.137076] [ 4994.175261] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff882dcd3a2c50 [ 4994.175261] [ 4994.207171] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-ldiskfs@ffff882eee0e9100osd-ldiskfs-object@ffff882eee0e9100(i:ffff882b7af90d90:79/2138703830)[plain] [ 4994.207171] [ 4994.258427] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff882dcd3a2c00 [ 4994.258427] [ 4994.299979] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff882dcd3a2e40[0x0, 1, [0x200000001:0x1017:0x0] hash exist]{ [ 4994.299979] [ 4994.341038] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff882dcd3a2e90 [ 4994.341038] [ 4994.372949] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-ldiskfs@ffff882df011a600osd-ldiskfs-object@ffff882df011a600(i:ffff882c098426e0:1090561/1004891530)[plain] [ 4994.372949] [ 4994.425511] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) } header@ffff882dcd3a2e40 [ 4994.425511] [ 4994.464017] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) header@ffff882bfaae7380[0x0, 1, [0xa:0x18:0x0] hash exist]{ [ 4994.464017] [ 4994.502461] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....local_storage@ffff882bfaae73d0 [ 4994.502461] [ 4994.534372] LustreError: 17122:0:(ofd_dev.c:251:ofd_stack_fini()) ....osd-ldiskfs@ffff882edbe28200osd-ldiskfs-object@ffff882edbe28200(i:ffff882b7af91600:80/2939569139)[plain] [ 4994.534372] [ 4994.571667] Lustre: nbp13-OST000b: Not available <span class="code-keyword">for</span> connect from 10.151.25.231@o2ib (not set up) </pre> </div></div> LU-11574

LustreError: 157-3: Trying to start OBD nbp13-OST000b_UUID using the wrong disk

Bug Blocker Resolved Fixed Alex Zhuravlev Mahmoud Hanafi Fri, 26 Oct 2018 08:21:37 +0000 Fri, 4 Sep 2020 17:44:12 +0000 Tue, 30 Oct 2018 12:57:12 +0000 Lustre 2.10.5 0 9 <p>We have engineers looking into the issue and will provide an update ASAP.</p> <p>last_rcvd seem to be corrupted. please wait few minutes, I'll try to reproduce locally and figure out a solution.</p> <p>Looks like the last_rcvd file of the OST target got corrupted, can you backup the target as a fail-safe, and mount the target as ldiskfs, then manually delete the last_rcvd file, umount it, and try to remount the ost normally again?</p> <p>I'd suggest to make a copy of last_rcvd ..<br/> at least locally that worked fine - mount as ldiskfs, save last_rcvd, rm the original one, umount <br/> and mount as Lustre.</p> <p>what do you mean</p> <p>"at least locally that worked fine"? </p> <p>just to verify.</p> <p>1. Mount the OSTs that are having the issue as ldiskfs.</p> <p>2. copy last_rcvd</p> <p>3. rm last_rcvd from each ost</p> <p>4. umount ost</p> <p>5. and remount as lustre.</p> <p> </p> <p>A releated issue is some of the OST have wrong free space.</p> <p>Here we have nbp15_1-OST13  says has 11TB used but this was a unused OST.</p> <p> </p> <p>nbp15-srv2 /mnt/lustre/nbp15_1-OST13 # du -sk <br/> 4311660 .</p> <p>nbp15-srv2 /mnt/lustre/nbp15_1-OST13 # df -h<br/> /dev/mapper/nbp15_1-OST13 72T 11T 62T 15% /mnt/lustre/nbp15_1-OST13</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nbp15-srv2 ~ # e2fsck -vf /dev/mapper/nbp15_1-OST13 e2fsck 1.42.13.wc6 (05-Feb-2017) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information 80857 inodes used (0.87%, out of 9337344) 8 non-contiguous files (0.0%) 0 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 0/0/0 Extent depth histogram: 80847/2 2703993403 blocks used (14.14%, out of 19122880512) 0 bad blocks 0 large files 80709 regular files 139 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 80848 files </pre> </div></div> <p> </p> <p> </p> <blockquote><p>1. Mount the OSTs that are having the issue as ldiskfs.</p> <p>2. copy last_rcvd</p> <p>3. rm last_rcvd from each ost</p></blockquote> <p>Just from the mounted OST that's having the issue</p> <blockquote><p>4. umount ost</p> <p>5. and remount as lustre.</p></blockquote> <p>Correct.</p> <p> </p> <p>With regards to free space.  I would check <tt>lfs df</tt> from a client to see where lustre shows free space.  If that doesn't clear things up, please open a separate ticket.</p> <p>This workaround work. The prio can be lowered</p> <p> </p> <p>Also, have you run a full e2fsck after the RAID problems? If not, then it is good to save the output (i.e. run under "<tt>script</tt>" or similar).</p> <p>yes we did run full fsck.</p> <p> </p> <p>Please close the case.</p> <p> </p> Related Development Rank 1|i0056n: Rank (Obsolete) 9223372036854775807 Severity