Details
-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
Lustre 1.8.8
-
e2fsprogs 1.41.90.wc2
-
3
-
11017
Description
After a power loss, an older e2fsck (e2fsprogs 1.41.90.wc2) was run on the OSTs. It found tons of multiply-claimed blocks, including for the /O directory. Here's an example of one of the inodes:
File ... (inode #17825793, mod time Wed Aug 15 19:02:25 2012) has 1 multiply-claimed block(s), shared with 1 file(s): /O (inode #84934657, mod time Wed Aug 15 19:02:25 2012) Clone multiply-claimed blocks? yes Inode 17825793 doesn't have an associated directory entry, it eventually gets put into lost+found.
So the questions are:
- how could this have happened? My slightly-informed-probably-wrong theory is that the journal got corrupted and it replayed some old inodes back into existence. I noticed there were a lot of patches dealing with journal checksums committed after 1.41.90.
- what's the best way to deal with these? Cloning takes forever when you are talking about TB sized files. I tested the delete extended option, and it looks like it deletes both sides of the file. It would be nice if it just deleted the unlinked side. Right now my plan is to create a debugfs script from the read-only e2fsck output, but if there is a better way, that would be good.
Thanks.
Hi Andreas,
One of the targets just went RO:
Oct 28 17:09:45 lfs-oss-2-6 kernel: LDISKFS-fs error (device dm-50): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 577corrupted: 24074 blocks free in bitmap,
24075 - in gd
It looks like I missed it when disabling targets. I had forgotten that I used the shared=ignore flag when cleaning it up, so the clean bill of health from e2fsck was an illusion.
I've marked it deactivated on the MDT. Hopefully it can hold until Wednesday.