Details
-
Bug
-
Resolution: Fixed
-
Critical
-
None
-
Lustre 2.1.2
-
None
-
lustre-source-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
lustre-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
kernel-2.6.32-220.17.1.el6_lustre.x86_64
kernel-ib-1.5.3-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
kernel-firmware-2.6.32-220.17.1.el6_lustre.x86_64
kernel-ib-devel-1.5.3-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
lustre-tests-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
kernel-headers-2.6.32-220.17.1.el6_lustre.x86_64
lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
kernel-mft-2.7.1-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
kernel-devel-2.6.32-220.17.1.el6_lustre.x86_64
lustre-modules-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64
e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64
e2fsprogs-devel-1.42.3.wc1-7.el6.x86_64
e2fsprogs-1.42.3.wc1-7.el6.x86_64
2.6.32-220.17.1.el6_lustre.x86_64
lustre-source-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 lustre-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 kernel-2.6.32-220.17.1.el6_lustre.x86_64 kernel-ib-1.5.3-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 kernel-firmware-2.6.32-220.17.1.el6_lustre.x86_64 kernel-ib-devel-1.5.3-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 lustre-tests-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 kernel-headers-2.6.32-220.17.1.el6_lustre.x86_64 lustre-ldiskfs-3.3.0-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 kernel-mft-2.7.1-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 kernel-devel-2.6.32-220.17.1.el6_lustre.x86_64 lustre-modules-2.1.2-2.6.32_220.17.1.el6_lustre.x86_64.x86_64 e2fsprogs-libs-1.42.3.wc1-7.el6.x86_64 e2fsprogs-devel-1.42.3.wc1-7.el6.x86_64 e2fsprogs-1.42.3.wc1-7.el6.x86_64 2.6.32-220.17.1.el6_lustre.x86_64
-
2
-
4008
Description
I have been seeing a large number of messages like the one below on the production /scratch FS.
Aug 17 17:54:52 mds07 mds07 kernel: LDISKFS-fs warning (device dm-2): ldiskfs_dx_add_entry: Directory index full!
the /scratch FS temporarily holds user /home directories until I install new hardware for separate lustre /home FS . The area of /scratch that is holding user /home directories is backed up on daily basis
Device dm-2 is the mdt for our production scratch FS. The file system has around 160M files at the moment and from what I found by reading various posts the LDISKFS message above suggests that we may have a very large directories in our/scrtach FS. I decided to run fsck -fD which supposedly should optimize directory structures and get rid of the above problem (at least temporarily)
Unfortunately this turned out to be a bad idea. The first pass of fsck found over 3200 invalid Symlinks and decided to clear them, for example
Symlink /ROOT/new_home/dws29/sandy/InstallArea/XML/CamMapCut64.pie (inode #66608225) is invalid.
Clear<y>? yes
I have checked those supposedly invalid symlinks with our /home backup and the symlink are actually correct, so fsck just removed over 3k valid symlinks.
/mnt/backup/home/dws29/sandy/InstallArea/XML/CamMapCut64.pie -> /home/dws29/sandy/Task_pkg/HL2_PowellSnakes/v00-00-020000_CVSHEAD/cmt/../XMLMODULESCHECKED//CamMapCut64.pie
Obviously in a whole /scratch FS we have much more than 3K of Symlinks so I am puzzled by what criteria fsck decided to clear these particular Symlinks.
I was able to recover some of them from our /home backup but the ones that were in not backed up area of /scratch were cleared forever (not cool).
I ran second pass of fsck and then mounted MDT back. Everything seemed ok until the overnight rsync backup process started to copy files and found many I/O errors when trying to enter some directories, for example
ls /home/ad491/progs/gromacs-3.3.3/src/gmxlib/gmx_blas
ls: reading directory /home/ad491/progs/gromacs-3.3.3/src/gmxlib/gmx_blas: Input/output error
I can see inside this directories from mds by using debugfs, so I am hoping that the data are not completely gone.
debugfs -c -R 'ls -l ROOT/new_home/ad491/progs/gromacs-3.3.3/src/gmxlib/gmx_blas/' /dev/mapper/mds08_scratch_mdt
debugfs 1.42.3.wc1 (28-May-2012)
/dev/mapper/mds08_scratch_mdt: catastrophic mode - not reading inode or group bitmaps
51920683 40775 (18) 9040 9043 8192 19-Dec-2009 16:23 .
51920609 40775 (18) 9040 9043 16384 19-Dec-2009 16:26 ..
51921092 40775 (18) 9040 9043 4096 19-Dec-2009 16:23 .deps
51921094 40775 (18) 9040 9043 4096 19-Dec-2009 16:23 .libs
51923088 100664 (17) 9040 9043 0 19-Dec-2009 16:19 Makefile
51923090 100644 (17) 9040 9043 0 2-Feb-2005 13:05 Makefile.am
51923092 100644 (17) 9040 9043 0 28-Feb-2008 15:41 Makefile.in
51923093 100644 (17) 9040 9043 0 24-Aug-2005 01:41 dasum.c
51923094 100664 (17) 9040 9043 0 19-Dec-2009 16:23 dasum.lo
51923095 100664 (17) 9040 9043 0 19-Dec-2009 16:23 dasum.o
51923096 100644 (17) 9040 9043 0 2-Feb-2005 13:05 daxpy.c
51923097 100664 (17) 9040 9043 0 19-Dec-2009 16:23 daxpy.lo
51923098 100664 (17) 9040 9043 0 19-Dec-2009 16:23 daxpy.o
Again I am able to recover directories that are on backed up area of scratch but this is not a lot and many of the corrupted directories are not backed up. Is there any way to reverse/fix what -D optimisation did and reconstruct the data?
I am attaching a log from fsck
and also few days worth of syslog messages from mds and oss servers. Please not that on the 17Aug around 6pm we had an IB network aoutage and there will be some noise related to these problems in the logs.
Also maybe worth mentioning the FS is less than couple of months old and it was created using e2fsprogs-1.42.3.wc1-7.el6.x86_64 which already had some fixes for fsck -D issues.
