https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2018 Lustre <p>We have an OST on one of our scratch file systems that was deactivated and attempts to reactivate it failed with:</p> <p><span class="error">&#91;204919.753933&#93;</span> Lustre: scratch1-MDT0000: scratch1-OST0001_UUID now active, resetting orphans<br/> <span class="error">&#91;204919.753939&#93;</span> Lustre: Skipped 1 previous similar message<br/> <span class="error">&#91;204919.754155&#93;</span> LustreError: 10403:0:(osc_create.c:589:osc_create()) scratch1-OST0001-osc: oscc recovery failed: -22<br/> <span class="error">&#91;204919.754166&#93;</span> Lustre: scratch1-OST0001_UUID: Failed to clear orphan objects on OST: -22<br/> <span class="error">&#91;204919.754170&#93;</span> Lustre: scratch1-OST0001_UUID: Sync failed deactivating: rc -22</p> <p>First: Is lfsck the proper tool to recover from this error?</p> <p>Second: Since this is the first time that I have ever used lfsck I am not sure what to expect. In this particular case there are 32 OSTs on this file system. Following the examples in the manual, I started a read-only run against all OSTs Saturday night and the the following morning from what I could judge it had only gotten through 11 of the 32 OSTs (if it run sequentially.) It reported LOTS of zero length orphaned inodes. Unfortunately in this case I didn't pipe the output to a log file so information regarding the OST (2/32) that couldn't be reactivated was lost when I ran out out line buffer space. So I stopped the run and restarted it against only that OST. Because it has been running listing user files for many hours I am guessing the answer is no bu is this a valid execution option or is it required that all OSTs must be accounted for?</p> <p>Third: The db files were generated when the file system was quiesced and after e2fsck was run against all targets cleanly. Must lfsck be run while in an offline mode or can it be run while the file system is serving clients? Because of my inexperience with using lfsck I don't know what to expect in terms of duration of the run and since I am using the -n option I will need to run again with corrective options. Also when running the corrective options is -l -c the preferred method?</p> Sun hardware running mdadm LU-2018

Questions about using lfsck

Bug Critical Resolved Fixed Cliff White Joe Mervini Mon, 24 Sep 2012 04:53:40 +0000 Thu, 24 Sep 2015 09:46:19 +0000 Thu, 24 Sep 2015 09:46:19 +0000 Lustre 1.8.x (1.8.0 - 1.8.5) 0 8 <p>Just adding some output since what is going into my log is not what is being seen on my console:</p> <p><span class="error">&#91;root@rmmds-scratch2 ~&#93;</span># lfsck -n -v --mdsdb /lustre-tmp/mdsdb --ostdb /lustre-tmp/scratch</p> {1md1db,1md2db,2md3db,2md4db,3md1db,3md2db,4md3db,4md4db,5md1db,5md2db,6md3db,6md4db,7md1db,7md2db,8md3db,8md4db,9md1db,9md2db,10dm3db,10md4db,11md1db,11md2db,12md3db,12md4db,13md1db,13md2db,14md3db,14md4db,15md1db,15md2db,16md3db,16md4db} <p> /scratch 2&gt;&amp;1 &gt;&gt; /lustre-tmp/lfsck.output-allosts<br/> lfsck 1.41.10.sun2-4chaos (23-Jun-2010)<br/> lfsck: ost_idx 0: pass1: check for duplicate objects<br/> lfsck: ost_idx 0: pass1 OK (2550897 files total)<br/> lfsck: ost_idx 0: pass2: check for missing inode objects<br/> lfsck: ost_idx 0: pass2 OK (2550897 objects)<br/> lfsck: ost_idx 0: pass3: check for orphan objects<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709696, 4096 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709728, 1724416 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709729, 4096 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709697, 4096 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709730, 49152 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709698, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709731, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709699, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709700, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709732, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709701, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709670, 4096 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709671, 4096 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709735, 28672 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709703, 69632 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709736, 4096 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709704, 69632 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709672, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709673, 32768 bytes<br/> lfsck: <span class="error">&#91;0&#93;</span>: pass3 orphan found objid 709737, 4096 bytes</p> <p>Log file:</p> <p><span class="error">&#91;0&#93;</span> zero-length orphan objid 6050029<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6049997<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050030<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6049998<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050031<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6049999<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050032<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050000<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050001<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050033<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050002<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050034<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050003<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050035<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050036<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050004<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050005<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050037<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050006<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050038<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050039<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050007<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050008<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050040<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050009<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050041<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050010<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050042<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050043<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050011<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050044<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050012<br/> <span class="error">&#91;0&#93;</span> zero-length orphan objid 6050045</p> <blockquote> <p><span class="error">&#91;204919.753933&#93;</span> Lustre: scratch1-MDT0000: scratch1-OST0001_UUID now active, resetting orphans<br/> <span class="error">&#91;204919.753939&#93;</span> Lustre: Skipped 1 previous similar message<br/> <span class="error">&#91;204919.754155&#93;</span> LustreError: 10403:0:(osc_create.c:589:osc_create()) scratch1-OST0001-osc: oscc recovery failed: -22<br/> <span class="error">&#91;204919.754166&#93;</span> Lustre: scratch1-OST0001_UUID: Failed to clear orphan objects on OST: -22<br/> <span class="error">&#91;204919.754170&#93;</span> Lustre: scratch1-OST0001_UUID: Sync failed deactivating: rc -22<br/> First: Is lfsck the proper tool to recover from this error?</p></blockquote> <p>We usually try to fix problems manually first before running lfsck (which can be time consuming).<br/> Could you please provide us with the logs of OST1, if any?</p> <p>That said, it does seems that you have orphan files on the OSTs which would have to be removed. Andreas, any thoughts on the e2fsck output?</p> <p>Joe, AFAIK it is not possible to run lfsck against only a single OST. If you omit the ostdb files from the other OSTs, lfsck (right or wrong) will assume that those OSTs could not generate an ostdb and assume they have been removed. I would not recommend that you run lfsck until we determine what the problem is.</p> <p>I don't think lfsck will fix the -EINVAL problem you are seeing, but it isn't possible to know for sure what problem you are hitting without more information.</p> <p>If there are useful console error messages from the OST, please post them here. Alternately (or additionally), running with full debug during the attempted mount of OST0001 would be useful, so we can determine exactly where the -22 error is coming from.</p> <p>Typical areas to check include that on the MDS the {{lctl get_param osc.</p> {fsname}-OST0001-osc.next_id}} (I think, can't check a system right now, essentially what is in the <tt>lov_objids</tt> file for this OST) <em>roughly</em> matches the LAST_ID on the OST itself (from {{lctl get_param obdfilter.{fsname} <p>-OST0001.last_id}} on the OSS). These values do not have to be identical, but cannot be too far apart. If this isn't the problem, we'll have to dig further.</p> <p>Just to confirm, the inability for the MDS to start using this OST should not affect the ability of the clients to read existing files from the OST. This would only prevent new files from being created there.</p> <p>Andreas, here is the output from the lctl get_param on the MDS and the affected OSS:</p> <p><span class="error">&#91;root@rmmds-scratch1 ~&#93;</span># lctl get_param osc.scratch1-OST0001-osc/prealloc_next_id<br/> osc.scratch1-OST0001-osc.prealloc_next_id=14540259<br/> <span class="error">&#91;root@rmoss-scratch1 scratch1-OST0001&#93;</span># lctl get_param obdfilter.scratch1-OST0001.last_id<br/> obdfilter.scratch1-OST0001.last_id=14563595</p> <p>I'd say these two numbers are pretty far off. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Here is the dmesg info when the OST was mounted:</p> <p><span class="error">&#91;102794.862025&#93;</span> LDISKFS FS on md2, external journal on md12<br/> <span class="error">&#91;102794.862095&#93;</span> LDISKFS-fs: mounted filesystem with ordered data mode.<br/> <span class="error">&#91;102797.452170&#93;</span> LustreError: 137-5: scratch1-OST0001: Not available for connect from 10.1.35.2@o2ib (no target)<br/> <span class="error">&#91;102797.452177&#93;</span> LustreError: Skipped 4 previous similar messages<br/> <span class="error">&#91;102806.069608&#93;</span> kjournald starting. Commit interval 5 seconds<br/> <span class="error">&#91;102806.142879&#93;</span> LDISKFS FS on md2, external journal on md12<br/> <span class="error">&#91;102806.142955&#93;</span> LDISKFS-fs: mounted filesystem with ordered data mode.<br/> <span class="error">&#91;102806.142958&#93;</span> LDISKFS-fs: file extents enabled<br/> <span class="error">&#91;102806.151589&#93;</span> LDISKFS-fs: mballoc enabled<br/> <span class="error">&#91;102806.416842&#93;</span> Lustre: scratch1-OST0001: Now serving scratch1-OST0001 on /dev/md2 with recovery enabled<br/> <span class="error">&#91;102806.416850&#93;</span> Lustre: scratch1-OST0001: Will be in recovery for at least 5:00, or until 1852 clients reconnect<br/> <span class="error">&#91;102806.463397&#93;</span> Lustre: scratch1-OST0001.ost: Set parameter quota_type=ug2<br/> <span class="error">&#91;102807.547883&#93;</span> LustreError: 137-5: scratch1-OST0002: Not available for connect from 10.1.33.4@o2ib (no target)<br/> <span class="error">&#91;102828.489696&#93;</span> Lustre: scratch1-OST0001: Starting recovery timer for 5:00<br/> <span class="error">&#91;103089.133873&#93;</span> Lustre: scratch1-OST0000: Recovery period over after 5:00, of 1852 clients 8 recovered and 1844 were evicted<br/> <span class="error">&#91;103089.133881&#93;</span> Lustre: scratch1-OST0000: Sending delayed replies to 8 recovered clients<br/> <span class="error">&#91;103127.713576&#93;</span> Lustre: scratch1-OST0001: Recovery period over after 5:00, of 1852 clients 8 recovered and 1844 were evicted<br/> <span class="error">&#91;103127.713586&#93;</span> Lustre: scratch1-OST0001: Sending delayed replies to 8 recovered clients<br/> <span class="error">&#91;103397.269470&#93;</span> Lustre: scratch1-OST0000: received MDS connection from 10.1.35.2@o2ib<br/> <span class="error">&#91;103397.270150&#93;</span> Lustre: scratch1-OST0000: Deleting orphan objects from 15685910 to 15685927, orphan objids won't be reused any more.<br/> <span class="error">&#91;103397.271317&#93;</span> LustreError: 17563:0:(filter.c:3173:filter_handle_precreate()) scratch1-OST0001: ignoring bogus orphan destroy request: obd<br/> id 14540258 last_id 14563595<br/> <span class="error">&#91;204924.148299&#93;</span> Lustre: scratch1-OST0001: Client scratch1-mdtlov_UUID (at 10.1.35.2@o2ib) reconnecting<br/> <span class="error">&#91;204924.148993&#93;</span> Lustre: scratch1-OST0001: received MDS connection from 10.1.35.2@o2ib<br/> <span class="error">&#91;204924.149018&#93;</span> Lustre: Skipped 1 previous similar message<br/> <span class="error">&#91;204924.149414&#93;</span> LustreError: 17598:0:(filter.c:3173:filter_handle_precreate()) scratch1-OST0001: ignoring bogus orphan destroy request: obd<br/> id 14540258 last_id 14563595</p> <p>Is there any other information that I can provide to assist you in helping me resolve my activation problem?</p> <p>Beyond that will it be wise/beneficial to run lfsck -i -c against the file system to clear the dangling and zero length inode?</p> <p>Joe, you are correct - the MDT and the OST have very different expectations on what object should be allocated next on this OST. The maximum object preallocation window is 10000, so these two numbers should not exceed this amount. If they do, then the OST refuses to destroy the large number of "orphan" objects to avoid potentially erasing the whole OST due to a software bug or corruption.</p> <p>In this case, the difference is over 23000, though it doesn't appear that the two values are corrupted. Is it possible that the MDS was restored from backup at some point since this OST was taken offline? Alternately (this shouldn't happen) was the OST accidentally attached to some other Lustre filesystem?</p> <p>The fix for this is to either binary edit the MDT "lov_objids" file to have the same value as the OST (which will potentially leave some orphan objects on the OST, or (slightly more risky, but less effort) to delete the lov_objids file entirely and have the MDT recreate it from the values on each of the OSTs.</p> <p>I think that I would opt for the less risky approach. If I do the binary edit, and later run lfsck would that clear the orphaned objects? I am not particularly concerned it they wind up in lost+found if I use the -l option just as long as the file system is in a better state than it currently is or was prior to this recent deactivation problem. </p> <p>But on the other hand, what is the risk with deleting the lov_objids? I'm a little bit nervous about doing the binary edit and making a mistake. Just to be clear, if I do edit lov_objids file I am only concern with scratch1-OST0001 right? And in doing the edit I want to modify the value to be 14563595, correct?</p> <p>Also, as I mentioned in my previous comment, is there a benefit to running lfsck on the file system once I am able to activate the OST? Currently the file system is offline with only the backup MDS acting as a client for running lfsck. If there is something to be gained by running lfsck we are will to keep the file system unavailable until it completes if necessary. </p> <p>If you can respond this evening it would be greatly appreciated so I can get these processes rolling over night.</p> <p>Thanks.</p> <p>Also WRT binary edit, the manual discuss modifying the data on an OST. Do I just apply the same principles to the MDT?</p> <p>I have taken down the file system and mounted both the MDT and affected OST ldiskfs. Running the od on the MDT this is the output:</p> <p><span class="error">&#91;root@rmmds-scratch1 CONFIGS&#93;</span># od -Ax -td8 /lustre/md1/lov_objid <br/> 000000 15685909 14540258<br/> 000010 15932110 14947247<br/> 000020 14515004 14128711<br/> 000030 15000526 15162675<br/> 000040 13640425 14099966<br/> 000050 14681958 14342756<br/> 000060 15165350 14397848<br/> 000070 14549423 14439112<br/> 000080 14908468 14520235<br/> 000090 14317909 15447697<br/> 0000a0 14506040 14566356<br/> 0000b0 14878948 14560476<br/> 0000c0 14593685 14742015<br/> 0000d0 14934824 13734107<br/> 0000e0 14365307 14647258<br/> 0000f0 14255774 14431566<br/> 000100</p> <p>Granted I don't really understand what I'm looking at but 000010 (if that is scratch1-OST0001) doesn't line up with any of the numbers above.</p> <p>With that in mind, I really don't know how to proceed. Guidance would definitely be appreciated.</p> <p>Niu</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Joe,<br/> Apologies for not replying sooner - I'm currently travelling. </p> <p>OST0001 is the second number displayed (14540258) at offset 08. Editing this value should not affect the ability to run lfsck later on. </p> <p>As for running lfsck afterward, this could also be done on the running system, so I'm reluctant to have you keep the system down longer than needed. I didn't know that the system was down thus while time - the inability of the MDS to complete recovery with the OST should not have affected the ability of clients to use the rest of the filesystem. </p> <p>So following Andreas' latest comment I was able to understand what I was seeing and successfully edited the lov_objid file on the MDS to get the OST active again. The problem has be fixed and the file system is operational again.</p> <p>For the benefit of anyone reading this that may have had the same problems understanding the documentation in the manual regarding binary edits (Appendix B: How to fix bad LAST_ID on an OST) I am detailing it here with the specifics of my case as example. (Note that this is for editing the MDT and not the OST so some of the information provided in the manual doesn't necessarily apply.)</p> <p>The ids that were identified as being out of sync for the OST were based on error information in dmesg on the deactivated OST:</p> <p><span class="error">&#91;204924.149414&#93;</span> LustreError: 17598:0:(filter.c:3173:filter_handle_precreate()) scratch1-OST0001: ignoring bogus orphan destroy request: obdid 14540258 last_id 14563595</p> <p>In the above the obdid 14540258 represents what the MDS had recorded as the last_id, and the last_id 14563595 is what the OST reported. As Andreas mentioned above the difference between those two numbers is greater than 23000 (only by 337 - but enough to cause the problem!)</p> <p>The first step I took was to mount the MDT as type ldiskfs:</p> <p><span class="error">&#91;root@rmoss-scratch1 /root&#93;</span># mount -t ldiskfs /dev/md1 /lustre/md1</p> <p>I then ran the od command in Step 1 to get the objids on the MDT:</p> <p><span class="error">&#91;root@rmmds-scratch1 CONFIGS&#93;</span># od -Ax -td8 /lustre/md1/lov_objid <br/> 000000 15685909 14540258<br/> 000010 15932110 14947247<br/> 000020 14515004 14128711<br/> 000030 15000526 15162675<br/> 000040 13640425 14099966<br/> 000050 14681958 14342756<br/> 000060 15165350 14397848<br/> 000070 14549423 14439112<br/> 000080 14908468 14520235<br/> 000090 14317909 15447697<br/> 0000a0 14506040 14566356<br/> 0000b0 14878948 14560476<br/> 0000c0 14593685 14742015<br/> 0000d0 14934824 13734107<br/> 0000e0 14365307 14647258<br/> 0000f0 14255774 14431566<br/> 000100</p> <p>The thing that confused me initially was that although I could see the number 14540258 on the first line, I didn't understand the offset (and I guess I still have a question about it: Does each line represent an OSS?)</p> <p>Then I mounted the affected OST ldiskfs:</p> <p><span class="error">&#91;root@rmoss-scratch1 scratch1-OST0001&#93;</span># mount -t ldiskfs /dev/md2 /lustre/md2</p> <p>I then ran Step 2. I don't really understand the purpose of the command and it is not clear whether it is supposed to be run on the MDS or the OSS since the documentation instructs you to mount the OST in Step 4. In any event the output was meaningless to me. (It would be nice if someone could explain it with an example.)</p> <p>Then following the instructions in the manual I ran the debugfs command (Step 3) which gave me the last_id of the OST that was consistent with the dmesg entry:</p> <p><span class="error">&#91;root@rmoss-scratch1 scratch1-OST0001&#93;</span># debugfs.ldiskfs -c -R 'dump O/0/LAST_ID /tmp/LAST_ID' /dev/md2;od -Ax -td8 /tmp/LAST_ID<br/> debugfs.ldiskfs 1.41.10.sun2-4chaos (23-Jun-2010)<br/> /dev/md2: catastrophic mode - not reading inode or group bitmaps<br/> 000000 14563595<br/> 000008</p> <p>I then ran Step 4:</p> <p><span class="error">&#91;root@rmoss-scratch1 scratch1-OST0001&#93;</span># mount -t ldiskfs /dev/md2 /lustre/md2</p> <p><span class="error">&#91;root@rmoss-scratch1 scratch1-OST0001&#93;</span># ls -1s /lustre/md2/<br/> CONFIGS/ health_check last_rcvd lost+found/ lquota.group lquota.user O/</p> <p><span class="error">&#91;root@rmoss-scratch1 scratch1-OST0001&#93;</span># ls -1s /lustre/md2/O/0/d*|grep -v <span class="error">&#91;a-z&#93;</span>|sort -k2 -n &gt; /tmp/obj.md2<br/> (The above command takes some time to run.)</p> <p><span class="error">&#91;root@rmoss-scratch1 ~&#93;</span># tail -30 /tmp/obj.md2 <br/> 0 14563566<br/> 0 14563567<br/> 0 14563568<br/> 0 14563569<br/> 0 14563570<br/> 0 14563571<br/> 0 14563572<br/> 0 14563573<br/> 0 14563574<br/> 0 14563575<br/> 0 14563576<br/> 0 14563577<br/> 0 14563578<br/> 0 14563579<br/> 0 14563580<br/> 0 14563581<br/> 0 14563582<br/> 0 14563583<br/> 0 14563584<br/> 0 14563585<br/> 0 14563586<br/> 0 14563587<br/> 0 14563588<br/> 0 14563589<br/> 0 14563590<br/> 0 14563591<br/> 0 14563592<br/> 0 14563593<br/> 0 14563594<br/> 0 14563595</p> <p>As pointed out in the manual, the value of the last_id matched the existing objects confirming that the problem was on the MDS, and that the problem could be resolved by removing the lov_objid file. However, Andreas' comment about removing the lov_objid file on the MDS being a little more risky, I opted to edit the file on the MDT.</p> <p>One thing that is very poorly documented in the manual was the purpose for the HEX to decimal translation and in my option should be placed in the section where the editing is actually discussed. </p> <p>The last_id is represented as a decimal number when that value is obtained via the "od -Ax -td8" command. However the when you convert the file from binary to ascii the contents are in HEX. (This is not explained at all.)</p> <p>I copy the lov_objid file from the MDT to /tmp as describe in the manual. Below is the output without redirection to an output file.</p> <p><span class="error">&#91;root@rmmds-scratch1 tmp&#93;</span># xxd lov_objid<br/> 0000000: 1559 ef00 0000 0000 e2dd dd00 0000 0000 .Y..............<br/> 0000010: ce1a f300 0000 0000 af13 e400 0000 0000 ................<br/> 0000020: 3c7b dd00 0000 0000 4796 d700 0000 0000 &lt;{......G.......<br/> 0000030: cee3 e400 0000 0000 335d e700 0000 0000 ........3]......<br/> 0000040: e922 d000 0000 0000 fe25 d700 0000 0000 .".......%......<br/> 0000050: 6607 e000 0000 0000 64da da00 0000 0000 f.......d.......<br/> 0000060: a667 e700 0000 0000 98b1 db00 0000 0000 .g..............<br/> 0000070: af01 de00 0000 0000 c852 dc00 0000 0000 .........R......<br/> 0000080: 347c e300 0000 0000 ab8f dd00 0000 0000 4|..............<br/> 0000090: 5579 da00 0000 0000 91b6 eb00 0000 0000 Uy..............<br/> 00000a0: 3858 dd00 0000 0000 d443 de00 0000 0000 8X.......C......<br/> 00000b0: e408 e300 0000 0000 dc2c de00 0000 0000 .........,......<br/> 00000c0: 95ae de00 0000 0000 fff1 e000 0000 0000 ................<br/> 00000d0: 28e3 e300 0000 0000 db90 d100 0000 0000 (...............<br/> 00000e0: 7b32 db00 0000 0000 da7f df00 0000 0000 {2..............<br/> 00000f0: 9e86 d900 0000 0000 4e35 dc00 0000 0000 ........N5......</p> <p>Given that the last_id the MDS thought was correct for the deactivated OST (14540258) I first wanted to make sure I knew what I was looking for. So I took 14540248 and converted it to HEX.</p> <p><span class="error">&#91;root@rmmds-scratch1 tmp&#93;</span># echo "obase=16; 14540258"|bc<br/> DDDDE2</p> <p>This is the line that represents the first and second OSTs in the system: <br/> 0000000: 1559 ef00 0000 0000 e2dd dd00 0000 0000 .Y..............</p> <p>But if you notice, there is a byte swap that is not mentioned in the manual so the DDDDE2 number that is generated by the bc command must be convert to E2DD DD00 0000 0000 to be properly inserted into the file.</p> <p>I then converted the correct last_id number to HEX:<br/> <span class="error">&#91;root@rmmds-scratch1 tmp&#93;</span># echo "obase=16; 14563595"|bc<br/> DE390B</p> <p>Then created the ascii file by running the command "xxd lov_objid lov_objid.asc" and edited as described in the manual taking extra care to do the byte swap to "0b39 de00 0000 0000", followed by recreating the HEX file with "xxd -r lov_objid.asc lov_objid.new".</p> <p><span class="error">&#91;root@rmmds-scratch1 tmp&#93;</span># vi /tmp/lov_objid.asc <br/> &lt;snip&gt;<br/> 0000000: 1559 ef00 0000 0000 0b39 de00 0000 0000 .Y..............<br/> 0000010: ce1a f300 0000 0000 af13 e400 0000 0000 ................<br/> &lt;/snip&gt;</p> <p>I confirmed that the file was what I expected:</p> <p><span class="error">&#91;root@rmmds-scratch1 tmp&#93;</span># od -Ax -td8 /tmp/lov_objid.new <br/> 000000 15685909 14563595<br/> 000010 15932110 14947247<br/> 000020 14515004 14128711<br/> 000030 15000526 15162675<br/> 000040 13640425 14099966<br/> 000050 14681958 14342756<br/> 000060 15165350 14397848<br/> 000070 14549423 14439112<br/> 000080 14908468 14520235<br/> 000090 14317909 15447697<br/> 0000a0 14506040 14566356<br/> 0000b0 14878948 14560476<br/> 0000c0 14593685 14742015<br/> 0000d0 14934824 13734107<br/> 0000e0 14365307 14647258<br/> 0000f0 14255774 14431566<br/> 000100</p> <p>I then moved the new file into place, unmount both targets on both the MDS and OSS, then mounted only those 2 devices as type lustre and confirmed that the OST came online and was activated. Once that was done I remounted the rest of the OSTs and am presently running lfsck.</p> <p>Joe, your commentary here is right on the mark. You are correct that these steps are not adequately described for someone "not skilled in the art". I take these kind of things for granted, but not many people are as closely familiar with the code and on-disk formats as I am.</p> <p>It definitely makes sense to update the user manual in this case with a better description of the required steps, as you have documented here.</p> <p>Also, it would be good to have Lustre handle this case more transparently than it does today. This is partially addressed with the patch in <a href="https://bugzilla.lustre.org/show_bug.cgi?id=24128" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=24128</a>, but this needs to be refreshed for the latest Lustre code, and landed to a release branch.</p> <p>Reassigning to Cliff to see what improvements can be made to the manual</p> <p>The LAST_ID reconstruction described here is now handled automatically by the OSS and LFSCK.</p> Related LU-14 Development Rank 1|hzv3ov: Rank (Obsolete) 4120