Just adding some output since what is going into my log is not what is being seen on my console:
[root@rmmds-scratch2 ~]# lfsck -n -v --mdsdb /lustre-tmp/mdsdb --ostdb /lustre-tmp/scratch
{1md1db,1md2db,2md3db,2md4db,3md1db,3md2db,4md3db,4md4db,5md1db,5md2db,6md3db,6md4db,7md1db,7md2db,8md3db,8md4db,9md1db,9md2db,10dm3db,10md4db,11md1db,11md2db,12md3db,12md4db,13md1db,13md2db,14md3db,14md4db,15md1db,15md2db,16md3db,16md4db} /scratch 2>&1 >> /lustre-tmp/lfsck.output-allosts
lfsck 1.41.10.sun2-4chaos (23-Jun-2010)
lfsck: ost_idx 0: pass1: check for duplicate objects
lfsck: ost_idx 0: pass1 OK (2550897 files total)
lfsck: ost_idx 0: pass2: check for missing inode objects
lfsck: ost_idx 0: pass2 OK (2550897 objects)
lfsck: ost_idx 0: pass3: check for orphan objects
lfsck: [0]: pass3 orphan found objid 709696, 4096 bytes
lfsck: [0]: pass3 orphan found objid 709728, 1724416 bytes
lfsck: [0]: pass3 orphan found objid 709729, 4096 bytes
lfsck: [0]: pass3 orphan found objid 709697, 4096 bytes
lfsck: [0]: pass3 orphan found objid 709730, 49152 bytes
lfsck: [0]: pass3 orphan found objid 709698, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709731, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709699, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709700, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709732, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709701, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709670, 4096 bytes
lfsck: [0]: pass3 orphan found objid 709671, 4096 bytes
lfsck: [0]: pass3 orphan found objid 709735, 28672 bytes
lfsck: [0]: pass3 orphan found objid 709703, 69632 bytes
lfsck: [0]: pass3 orphan found objid 709736, 4096 bytes
lfsck: [0]: pass3 orphan found objid 709704, 69632 bytes
lfsck: [0]: pass3 orphan found objid 709672, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709673, 32768 bytes
lfsck: [0]: pass3 orphan found objid 709737, 4096 bytes
Log file:
[0] zero-length orphan objid 6050029
[0] zero-length orphan objid 6049997
[0] zero-length orphan objid 6050030
[0] zero-length orphan objid 6049998
[0] zero-length orphan objid 6050031
[0] zero-length orphan objid 6049999
[0] zero-length orphan objid 6050032
[0] zero-length orphan objid 6050000
[0] zero-length orphan objid 6050001
[0] zero-length orphan objid 6050033
[0] zero-length orphan objid 6050002
[0] zero-length orphan objid 6050034
[0] zero-length orphan objid 6050003
[0] zero-length orphan objid 6050035
[0] zero-length orphan objid 6050036
[0] zero-length orphan objid 6050004
[0] zero-length orphan objid 6050005
[0] zero-length orphan objid 6050037
[0] zero-length orphan objid 6050006
[0] zero-length orphan objid 6050038
[0] zero-length orphan objid 6050039
[0] zero-length orphan objid 6050007
[0] zero-length orphan objid 6050008
[0] zero-length orphan objid 6050040
[0] zero-length orphan objid 6050009
[0] zero-length orphan objid 6050041
[0] zero-length orphan objid 6050010
[0] zero-length orphan objid 6050042
[0] zero-length orphan objid 6050043
[0] zero-length orphan objid 6050011
[0] zero-length orphan objid 6050044
[0] zero-length orphan objid 6050012
[0] zero-length orphan objid 6050045
[204919.753933] Lustre: scratch1-MDT0000: scratch1-OST0001_UUID now active, resetting orphans
[204919.753939] Lustre: Skipped 1 previous similar message
[204919.754155] LustreError: 10403:0:(osc_create.c:589:osc_create()) scratch1-OST0001-osc: oscc recovery failed: -22
[204919.754166] Lustre: scratch1-OST0001_UUID: Failed to clear orphan objects on OST: -22
[204919.754170] Lustre: scratch1-OST0001_UUID: Sync failed deactivating: rc -22
First: Is lfsck the proper tool to recover from this error?
We usually try to fix problems manually first before running lfsck (which can be time consuming).
Could you please provide us with the logs of OST1, if any?
That said, it does seems that you have orphan files on the OSTs which would have to be removed. Andreas, any thoughts on the e2fsck output?
Joe, AFAIK it is not possible to run lfsck against only a single OST. If you omit the ostdb files from the other OSTs, lfsck (right or wrong) will assume that those OSTs could not generate an ostdb and assume they have been removed. I would not recommend that you run lfsck until we determine what the problem is.
I don't think lfsck will fix the -EINVAL problem you are seeing, but it isn't possible to know for sure what problem you are hitting without more information.
If there are useful console error messages from the OST, please post them here. Alternately (or additionally), running with full debug during the attempted mount of OST0001 would be useful, so we can determine exactly where the -22 error is coming from.
Typical areas to check include that on the MDS the {{lctl get_param osc.
{fsname}-OST0001-osc.next_id}} (I think, can't check a system right now, essentially what is in the lov_objids file for this OST) roughly matches the LAST_ID on the OST itself (from {{lctl get_param obdfilter.{fsname}-OST0001.last_id}} on the OSS). These values do not have to be identical, but cannot be too far apart. If this isn't the problem, we'll have to dig further.
Just to confirm, the inability for the MDS to start using this OST should not affect the ability of the clients to read existing files from the OST. This would only prevent new files from being created there.
Andreas, here is the output from the lctl get_param on the MDS and the affected OSS:
[root@rmmds-scratch1 ~]# lctl get_param osc.scratch1-OST0001-osc/prealloc_next_id
osc.scratch1-OST0001-osc.prealloc_next_id=14540259
[root@rmoss-scratch1 scratch1-OST0001]# lctl get_param obdfilter.scratch1-OST0001.last_id
obdfilter.scratch1-OST0001.last_id=14563595
I'd say these two numbers are pretty far off. 
Here is the dmesg info when the OST was mounted:
[102794.862025] LDISKFS FS on md2, external journal on md12
[102794.862095] LDISKFS-fs: mounted filesystem with ordered data mode.
[102797.452170] LustreError: 137-5: scratch1-OST0001: Not available for connect from 10.1.35.2@o2ib (no target)
[102797.452177] LustreError: Skipped 4 previous similar messages
[102806.069608] kjournald starting. Commit interval 5 seconds
[102806.142879] LDISKFS FS on md2, external journal on md12
[102806.142955] LDISKFS-fs: mounted filesystem with ordered data mode.
[102806.142958] LDISKFS-fs: file extents enabled
[102806.151589] LDISKFS-fs: mballoc enabled
[102806.416842] Lustre: scratch1-OST0001: Now serving scratch1-OST0001 on /dev/md2 with recovery enabled
[102806.416850] Lustre: scratch1-OST0001: Will be in recovery for at least 5:00, or until 1852 clients reconnect
[102806.463397] Lustre: scratch1-OST0001.ost: Set parameter quota_type=ug2
[102807.547883] LustreError: 137-5: scratch1-OST0002: Not available for connect from 10.1.33.4@o2ib (no target)
[102828.489696] Lustre: scratch1-OST0001: Starting recovery timer for 5:00
[103089.133873] Lustre: scratch1-OST0000: Recovery period over after 5:00, of 1852 clients 8 recovered and 1844 were evicted
[103089.133881] Lustre: scratch1-OST0000: Sending delayed replies to 8 recovered clients
[103127.713576] Lustre: scratch1-OST0001: Recovery period over after 5:00, of 1852 clients 8 recovered and 1844 were evicted
[103127.713586] Lustre: scratch1-OST0001: Sending delayed replies to 8 recovered clients
[103397.269470] Lustre: scratch1-OST0000: received MDS connection from 10.1.35.2@o2ib
[103397.270150] Lustre: scratch1-OST0000: Deleting orphan objects from 15685910 to 15685927, orphan objids won't be reused any more.
[103397.271317] LustreError: 17563:0:(filter.c:3173:filter_handle_precreate()) scratch1-OST0001: ignoring bogus orphan destroy request: obd
id 14540258 last_id 14563595
[204924.148299] Lustre: scratch1-OST0001: Client scratch1-mdtlov_UUID (at 10.1.35.2@o2ib) reconnecting
[204924.148993] Lustre: scratch1-OST0001: received MDS connection from 10.1.35.2@o2ib
[204924.149018] Lustre: Skipped 1 previous similar message
[204924.149414] LustreError: 17598:0:(filter.c:3173:filter_handle_precreate()) scratch1-OST0001: ignoring bogus orphan destroy request: obd
id 14540258 last_id 14563595
Is there any other information that I can provide to assist you in helping me resolve my activation problem?
Beyond that will it be wise/beneficial to run lfsck -i -c against the file system to clear the dangling and zero length inode?
Joe, you are correct - the MDT and the OST have very different expectations on what object should be allocated next on this OST. The maximum object preallocation window is 10000, so these two numbers should not exceed this amount. If they do, then the OST refuses to destroy the large number of "orphan" objects to avoid potentially erasing the whole OST due to a software bug or corruption.
In this case, the difference is over 23000, though it doesn't appear that the two values are corrupted. Is it possible that the MDS was restored from backup at some point since this OST was taken offline? Alternately (this shouldn't happen) was the OST accidentally attached to some other Lustre filesystem?
The fix for this is to either binary edit the MDT "lov_objids" file to have the same value as the OST (which will potentially leave some orphan objects on the OST, or (slightly more risky, but less effort) to delete the lov_objids file entirely and have the MDT recreate it from the values on each of the OSTs.
I think that I would opt for the less risky approach. If I do the binary edit, and later run lfsck would that clear the orphaned objects? I am not particularly concerned it they wind up in lost+found if I use the -l option just as long as the file system is in a better state than it currently is or was prior to this recent deactivation problem.
But on the other hand, what is the risk with deleting the lov_objids? I'm a little bit nervous about doing the binary edit and making a mistake. Just to be clear, if I do edit lov_objids file I am only concern with scratch1-OST0001 right? And in doing the edit I want to modify the value to be 14563595, correct?
Also, as I mentioned in my previous comment, is there a benefit to running lfsck on the file system once I am able to activate the OST? Currently the file system is offline with only the backup MDS acting as a client for running lfsck. If there is something to be gained by running lfsck we are will to keep the file system unavailable until it completes if necessary.
If you can respond this evening it would be greatly appreciated so I can get these processes rolling over night.
Thanks.
Also WRT binary edit, the manual discuss modifying the data on an OST. Do I just apply the same principles to the MDT?
I have taken down the file system and mounted both the MDT and affected OST ldiskfs. Running the od on the MDT this is the output:
[root@rmmds-scratch1 CONFIGS]# od -Ax -td8 /lustre/md1/lov_objid
000000 15685909 14540258
000010 15932110 14947247
000020 14515004 14128711
000030 15000526 15162675
000040 13640425 14099966
000050 14681958 14342756
000060 15165350 14397848
000070 14549423 14439112
000080 14908468 14520235
000090 14317909 15447697
0000a0 14506040 14566356
0000b0 14878948 14560476
0000c0 14593685 14742015
0000d0 14934824 13734107
0000e0 14365307 14647258
0000f0 14255774 14431566
000100
Granted I don't really understand what I'm looking at but 000010 (if that is scratch1-OST0001) doesn't line up with any of the numbers above.
With that in mind, I really don't know how to proceed. Guidance would definitely be appreciated.
Niu
Could you please advise on this one?
Thanks
Peter
Joe,
Apologies for not replying sooner - I'm currently travelling.
OST0001 is the second number displayed (14540258) at offset 08. Editing this value should not affect the ability to run lfsck later on.
As for running lfsck afterward, this could also be done on the running system, so I'm reluctant to have you keep the system down longer than needed. I didn't know that the system was down thus while time - the inability of the MDS to complete recovery with the OST should not have affected the ability of clients to use the rest of the filesystem.
So following Andreas' latest comment I was able to understand what I was seeing and successfully edited the lov_objid file on the MDS to get the OST active again. The problem has be fixed and the file system is operational again.
For the benefit of anyone reading this that may have had the same problems understanding the documentation in the manual regarding binary edits (Appendix B: How to fix bad LAST_ID on an OST) I am detailing it here with the specifics of my case as example. (Note that this is for editing the MDT and not the OST so some of the information provided in the manual doesn't necessarily apply.)
The ids that were identified as being out of sync for the OST were based on error information in dmesg on the deactivated OST:
[204924.149414] LustreError: 17598:0:(filter.c:3173:filter_handle_precreate()) scratch1-OST0001: ignoring bogus orphan destroy request: obdid 14540258 last_id 14563595
In the above the obdid 14540258 represents what the MDS had recorded as the last_id, and the last_id 14563595 is what the OST reported. As Andreas mentioned above the difference between those two numbers is greater than 23000 (only by 337 - but enough to cause the problem!)
The first step I took was to mount the MDT as type ldiskfs:
[root@rmoss-scratch1 /root]# mount -t ldiskfs /dev/md1 /lustre/md1
I then ran the od command in Step 1 to get the objids on the MDT:
[root@rmmds-scratch1 CONFIGS]# od -Ax -td8 /lustre/md1/lov_objid
000000 15685909 14540258
000010 15932110 14947247
000020 14515004 14128711
000030 15000526 15162675
000040 13640425 14099966
000050 14681958 14342756
000060 15165350 14397848
000070 14549423 14439112
000080 14908468 14520235
000090 14317909 15447697
0000a0 14506040 14566356
0000b0 14878948 14560476
0000c0 14593685 14742015
0000d0 14934824 13734107
0000e0 14365307 14647258
0000f0 14255774 14431566
000100
The thing that confused me initially was that although I could see the number 14540258 on the first line, I didn't understand the offset (and I guess I still have a question about it: Does each line represent an OSS?)
Then I mounted the affected OST ldiskfs:
[root@rmoss-scratch1 scratch1-OST0001]# mount -t ldiskfs /dev/md2 /lustre/md2
I then ran Step 2. I don't really understand the purpose of the command and it is not clear whether it is supposed to be run on the MDS or the OSS since the documentation instructs you to mount the OST in Step 4. In any event the output was meaningless to me. (It would be nice if someone could explain it with an example.)
Then following the instructions in the manual I ran the debugfs command (Step 3) which gave me the last_id of the OST that was consistent with the dmesg entry:
[root@rmoss-scratch1 scratch1-OST0001]# debugfs.ldiskfs -c -R 'dump O/0/LAST_ID /tmp/LAST_ID' /dev/md2;od -Ax -td8 /tmp/LAST_ID
debugfs.ldiskfs 1.41.10.sun2-4chaos (23-Jun-2010)
/dev/md2: catastrophic mode - not reading inode or group bitmaps
000000 14563595
000008
I then ran Step 4:
[root@rmoss-scratch1 scratch1-OST0001]# mount -t ldiskfs /dev/md2 /lustre/md2
[root@rmoss-scratch1 scratch1-OST0001]# ls -1s /lustre/md2/
CONFIGS/ health_check last_rcvd lost+found/ lquota.group lquota.user O/
[root@rmoss-scratch1 scratch1-OST0001]# ls -1s /lustre/md2/O/0/d*|grep -v [a-z]|sort -k2 -n > /tmp/obj.md2
(The above command takes some time to run.)
[root@rmoss-scratch1 ~]# tail -30 /tmp/obj.md2
0 14563566
0 14563567
0 14563568
0 14563569
0 14563570
0 14563571
0 14563572
0 14563573
0 14563574
0 14563575
0 14563576
0 14563577
0 14563578
0 14563579
0 14563580
0 14563581
0 14563582
0 14563583
0 14563584
0 14563585
0 14563586
0 14563587
0 14563588
0 14563589
0 14563590
0 14563591
0 14563592
0 14563593
0 14563594
0 14563595
As pointed out in the manual, the value of the last_id matched the existing objects confirming that the problem was on the MDS, and that the problem could be resolved by removing the lov_objid file. However, Andreas' comment about removing the lov_objid file on the MDS being a little more risky, I opted to edit the file on the MDT.
One thing that is very poorly documented in the manual was the purpose for the HEX to decimal translation and in my option should be placed in the section where the editing is actually discussed.
The last_id is represented as a decimal number when that value is obtained via the "od -Ax -td8" command. However the when you convert the file from binary to ascii the contents are in HEX. (This is not explained at all.)
I copy the lov_objid file from the MDT to /tmp as describe in the manual. Below is the output without redirection to an output file.
[root@rmmds-scratch1 tmp]# xxd lov_objid
0000000: 1559 ef00 0000 0000 e2dd dd00 0000 0000 .Y..............
0000010: ce1a f300 0000 0000 af13 e400 0000 0000 ................
0000020: 3c7b dd00 0000 0000 4796 d700 0000 0000 <{......G.......
0000030: cee3 e400 0000 0000 335d e700 0000 0000 ........3]......
0000040: e922 d000 0000 0000 fe25 d700 0000 0000 .".......%......
0000050: 6607 e000 0000 0000 64da da00 0000 0000 f.......d.......
0000060: a667 e700 0000 0000 98b1 db00 0000 0000 .g..............
0000070: af01 de00 0000 0000 c852 dc00 0000 0000 .........R......
0000080: 347c e300 0000 0000 ab8f dd00 0000 0000 4|..............
0000090: 5579 da00 0000 0000 91b6 eb00 0000 0000 Uy..............
00000a0: 3858 dd00 0000 0000 d443 de00 0000 0000 8X.......C......
00000b0: e408 e300 0000 0000 dc2c de00 0000 0000 .........,......
00000c0: 95ae de00 0000 0000 fff1 e000 0000 0000 ................
00000d0: 28e3 e300 0000 0000 db90 d100 0000 0000 (...............
00000e0: 7b32 db00 0000 0000 da7f df00 0000 0000 {2..............
00000f0: 9e86 d900 0000 0000 4e35 dc00 0000 0000 ........N5......
Given that the last_id the MDS thought was correct for the deactivated OST (14540258) I first wanted to make sure I knew what I was looking for. So I took 14540248 and converted it to HEX.
[root@rmmds-scratch1 tmp]# echo "obase=16; 14540258"|bc
DDDDE2
This is the line that represents the first and second OSTs in the system:
0000000: 1559 ef00 0000 0000 e2dd dd00 0000 0000 .Y..............
But if you notice, there is a byte swap that is not mentioned in the manual so the DDDDE2 number that is generated by the bc command must be convert to E2DD DD00 0000 0000 to be properly inserted into the file.
I then converted the correct last_id number to HEX:
[root@rmmds-scratch1 tmp]# echo "obase=16; 14563595"|bc
DE390B
Then created the ascii file by running the command "xxd lov_objid lov_objid.asc" and edited as described in the manual taking extra care to do the byte swap to "0b39 de00 0000 0000", followed by recreating the HEX file with "xxd -r lov_objid.asc lov_objid.new".
[root@rmmds-scratch1 tmp]# vi /tmp/lov_objid.asc
<snip>
0000000: 1559 ef00 0000 0000 0b39 de00 0000 0000 .Y..............
0000010: ce1a f300 0000 0000 af13 e400 0000 0000 ................
</snip>
I confirmed that the file was what I expected:
[root@rmmds-scratch1 tmp]# od -Ax -td8 /tmp/lov_objid.new
000000 15685909 14563595
000010 15932110 14947247
000020 14515004 14128711
000030 15000526 15162675
000040 13640425 14099966
000050 14681958 14342756
000060 15165350 14397848
000070 14549423 14439112
000080 14908468 14520235
000090 14317909 15447697
0000a0 14506040 14566356
0000b0 14878948 14560476
0000c0 14593685 14742015
0000d0 14934824 13734107
0000e0 14365307 14647258
0000f0 14255774 14431566
000100
I then moved the new file into place, unmount both targets on both the MDS and OSS, then mounted only those 2 devices as type lustre and confirmed that the OST came online and was activated. Once that was done I remounted the rest of the OSTs and am presently running lfsck.
Joe, your commentary here is right on the mark. You are correct that these steps are not adequately described for someone "not skilled in the art". I take these kind of things for granted, but not many people are as closely familiar with the code and on-disk formats as I am.
It definitely makes sense to update the user manual in this case with a better description of the required steps, as you have documented here.
Also, it would be good to have Lustre handle this case more transparently than it does today. This is partially addressed with the patch in https://bugzilla.lustre.org/show_bug.cgi?id=24128, but this needs to be refreshed for the latest Lustre code, and landed to a release branch.
Reassigning to Cliff to see what improvements can be made to the manual
The LAST_ID reconstruction described here is now handled automatically by the OSS and LFSCK.