This is a relatively benign error message. It means some object was being truncated, the OST crashed, and the truncate didn't complete correctly on recovery. Running e2fsck will repair the object to a "safe" size (12kB in this case), which might result in NUL padding at the end of the file.
You can locate which MDS file this belongs to with the following commands (this is much easier on 2.x). On the OST run:
ost> debugfs -c -R "stat <393896873>" /dev/mapper/ost_lustre01_1 debugfs 1.41.90.wc3 (28-May-2011) /dev/vgmyth/lvmythost0: catastrophic mode - not reading inode or group bitmaps Inode: 63681 Type: regular Mode: 0666 Flags: 0x80000 Generation: 2393149953 Version: 0x0000002a:00005f81 User: 1000 Group: 1000 Size: 25165824 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 49152 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4bad91f0:00000000 -- Fri Mar 26 23:04:48 2010 atime: 0x4bad91f0:00000000 -- Fri Mar 26 23:04:48 2010 mtime: 0x4bad91d1:00000000 -- Fri Mar 26 23:04:17 2010 crtime: 0x4bad87f0:d2d42b84 -- Fri Mar 26 22:22:08 2010 Size of extra inode fields: 28 Extended attributes stored in inode body: fid = "b9 da 24 00 00 00 00 00 6a fa 0d 3f 01 00 00 00 eb 5b 0b 00 00 00 00 00 00 00 00 00 00 00 00 00 " (32) fid: objid=744427 seq=0 parent=[0x24dab9:0x3f0dfa6a:0x0] stripe=1 EXTENTS: (0-255):4620544-4620799, (256-6143):4621312-4627199
Of interest is the "fid:" line, where it reports the parent inode number (the first number inside the square brackets, 0x24dab9 in this example).
On the MDT run:
mdt> debugfs -c -R "ncheck 0x24dab9" /dev/{MDT}
debugfs 1.41.90.wc3 (28-May-2011)
/dev/vgmyth/lvmythmdt0.ssd: catastrophic mode - not reading inode or group bitmaps
Inode Pathname
2415289 /ROOT/tmp/4stripe
If your filesystem is large, ncheck may take some time. If there are multiple inodes affected from multiple OSTs, you can list them all on the same "ncheck" line to avoid scanning the filesystem multiple times. In Lustre 2.x this "ncheck" step can be avoided, and the pathname can be resolved directly from the FID (e.g. "lfs fid2path [0x24dab9:0x3f0dfa6a:0x0]").
Andreas,
thanks for your comment. Based on this, I have managed to identify 5 (out of 6) files, for on FID ncheck did not return any file on the MDT. I suspect lfsck would report that as orphaned object?
In addition to this, I ran e2fsck -v -f -p on the affected OSTs, e2fsck managed to repair only one off them. In all other cases it stopped without repairing the corrupted orphan link list:
[bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -p /dev/mapper/ost_lustre01_1
lustre01-OST0001: Inodes that were part of a corrupted orphan linked list found.
lustre01-OST0001: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
[bnh65367@cs04r-sc-oss01-05 ~]$
[bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -n /dev/mapper/ost_lustre01_1
e2fsck 1.41.90.wc4 (01-Sep-2011)
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found. Fix? no
Inode 393896873 was part of the orphaned inode list. IGNORED.
Inode 393896873 is in use, but has dtime set. Fix? no
Inode 393896873, i_size is 0, should be 12288. Fix? no
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
lustre01-OST0001: ********** WARNING: Filesystem still has errors **********
890452 inodes used (0.18%)
257084 non-contiguous files (28.9%)
32 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 32/0/0
Extent depth histogram: 778159/111004/1240
1437482870 blocks used (73.59%)
0 bad blocks
55 large files
890404 regular files
39 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
890443 files
[bnh65367@cs04r-sc-oss01-05 ~]$
After checking the file names of the affected files, I'm not too concerned as they are either no longer important or should be relatively easy to recreate. I would still prefer to fix the OSTs.
(Sorry forgot to add this
I'm also usually very cautious when it comes to running fsck manually without knowing what it does.
One one OST e2fsck manage to fix it:
[bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -n /dev/mapper/ost_lustre01_0e2fsck 1.41.90.wc4 (01-Sep-2011)
Pass 1: Checking inodes, blocks, and sizes
Inode 2 creation time (Thu Jan 1 02:08:16 1970) invalid.
Clear? no
Inodes that were part of a corrupted orphan linked list found. Fix? no
Inode 22242677 was part of the orphaned inode list. IGNORED.
Inode 22242677 is in use, but has dtime set. Fix? no
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
lustre01-OST0000: ********** WARNING: Filesystem still has errors **********
930294 inodes used (0.19%)
271214 non-contiguous files (29.2%)
32 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 32/0/0
Extent depth histogram: 813541/115276/1428
1398216469 blocks used (71.58%)
0 bad blocks
36 large files
930246 regular files
39 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
930285 files
[bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -p /dev/mapper/ost_lustre01_0
lustre01-OST0000: Truncating orphaned inode 22242677 (uid=499, gid=499, mode=0100666, size=0)
lustre01-OST0000: Truncating orphaned inode 55623793 (uid=499, gid=101902, mode=0100666, size=0)
lustre01-OST0000: Inode 2 creation time (Thu Jan 1 02:08:16 1970) invalid.
CLEARED.
930294 inodes used (0.19%)
271214 non-contiguous files (29.2%)
32 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 32/0/0
Extent depth histogram: 813541/115276/1428
1398216467 blocks used (71.58%)
0 bad blocks
36 large files
930246 regular files
39 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
930285 files
Running e2fsck -fp will repair only straight forward errors. That is a reasonable approach until one knows what the error is. In this case, you should run e2fsck -fy do that it answers "yes" to repairing these errors.
Thanks for your reply. Unfortunately I had to mount the file system before I received the reply. I've identified all affected files and will schedule another maintenance window to fix it properly.
Are there any questions related to this issue, or can this bug be closed?
Apologies for the delay I missed the update.
Yes, the ticket can be closed. Thanks.
Thanks Frederik!