|
Niu
Please could you look into this one?
Kit
Might the version of Lustre being run at NOAA be relevant for Niu to be aware of?
Peter
|
|
Sorry about that, it's Lustre 1.8.8.
|
|
And the original mkfs line for one of the OSTs from mkfs.lustre:
mke2fs -j -b 4096 -L lfs2-OST000b -m1 -t ext4 -i 262144 -J size=400 -I 256 -q -O flex_bg,dir_index,extents,uninit_groups,mmp,uninit_bg,dir_nlink,huge_file,64bit -G 256 -E lazy_journal_init -F /dev/mapper/ost_lfs2_11 5735710720
|
|
We need to increase the priority on this. A couple of the OSTs have corrupted checksums on the block groups and will not mount without completing an e2fsck run. On one of the OSTs, there are ~77k inodes with duplicate blocks, which would take weeks to clone. So effectively we are down until we get a solution.
|
|
OST11 is the one with the most problems. It also points strongly to journal corruption.
Read-only e2fsck right after power outage:
e2fsck -v -f -n /dev/mapper/ost_lfs2_11e2fsck 1.41.90.wc2 (14-May-2011)
MMP interval is 10 seconds and total wait time is 42 seconds. Please wait...
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached zero-length inode 69504226. Clear? no
Unattached inode 69504226
Connect to /lost+found? no
Unattached zero-length inode 69504227. Clear? no
Unattached inode 69504227
Connect to /lost+found? no
Unattached zero-length inode 69504228. Clear? no
Unattached inode 69504228
Connect to /lost+found? no
Unattached zero-length inode 69504229. Clear? no
Unattached inode 69504229
Connect to /lost+found? no
Pass 5: Checking group summary information
Free blocks count wrong (2461151422, counted=3112390974).
Fix? no
Inode bitmap differences: -69504142 -69504145 -69504147 -(69504149--69504150) -69504152 -(69504154--69504159) -69504161 -(69504163--69504164) -69504172 -(69504174--69504175) -69504177 -(69504182--69504187)
Fix? no
Free inodes count wrong for group #135750 (143, counted=114).
Fix? no
Free inodes count wrong (88155565, counted=88073017).
Fix? no
lfs2-OST000b: ********** WARNING: Filesystem still has errors **********
1464915 inodes used (1.63%)
151384 non-contiguous files (10.3%)
32 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 0/0/0
Extent depth histogram: 1486591/60821/16
3274559298 blocks used (57.09%)
0 bad blocks
271 large files
1547390 regular files
39 directories
0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
1547425 files
e2fsck -fp:
e2fsck -v -f -p /dev/mapper/ost_lfs2_11MMP interval is 10 seconds and total wait time is 42 seconds. Please wait...
lfs2-OST000b: recovering journal
lfs2-OST000b: Note: if several inode or block bitmap blocks or part
of the inode table require relocation, you may wish to try
running e2fsck with the '-b 32768' option first. The problem
may lie only with the primary block group descriptors, and
the backup block group descriptors may be OK.
lfs2-OST000b: Block bitmap for group 32704 is not in group. (block 2251800507482550)
lfs2-OST000b: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
e2fsck -fy (never completed):
e2fsck 1.41.90.wc2 (14-May-2011)
e2fsck: Group descriptors look bad... trying backup blocks...
MMP interval is 10 seconds and total wait time is 42 seconds. Please wait...
One or more block group descriptor checksums are invalid. Fix? yes
Group descriptor 0 checksum is invalid. FIXED.
Group descriptor 1 checksum is invalid. FIXED.
...
Resize inode not valid. Recreate? yes
Pass 1: Checking inodes, blocks, and sizes
Running additional passes to resolve blocks claimed by more than one inode...
Pass 1B: Rescanning for multiply-claimed blocks
Multiply-claimed block(s) in inode 131074: 1268866046
...
(repeats 77k times)
I built a version of e2fsck that skips pass 1b-1d (dupe checks). Would it be a terrible idea to run this against the filesystem now and deal with the duplicate blocks later? We need to get the filesystem up and running as soon as prudently possible.
|
|
Kit,
There is a later version of e2fsprogs available. If it's not difficult, you could try to run the above e2fsck commands with the new e2fsprogs. I'm not claiming that this will solve your problem.
|
|
Hi James,
We are running a -fn test with the new e2fsprogs now, but unfortunately the output looks pretty similar.
|
|
I've attached the logs of another ost, ost15. It actually doesn't have the dupe blocks issue, it has more traditional looking corruption. It still is taking forever to finish running e2fsck against though, and will not mount due to the group descriptor corruption. I'm not sure what the next step should be with it. I guess just running e2fsck -fy and hoping for the best?
|
I've attached the logs of another ost, ost15. It actually doesn't have the dupe blocks issue, it has more traditional looking corruption. It still is taking forever to finish running e2fsck against though, and will not mount due to the group descriptor corruption. I'm not sure what the next step should be with it. I guess just running e2fsck -fy and hoping for the best?
Yes, I think we can try to fix it with new e2fsprogs.
|
|
Ok, we will get that going. Any thoughts on the dupe blocks issue? Is it ok to just skip the 1b-d passes?
|
Ok, we will get that going. Any thoughts on the dupe blocks issue? Is it ok to just skip the 1b-d passes?
Is the OST have multiply-calimed block issues based on RAID array? Could you check if the RAID array is in healty state? (I'm not expert on RAID, not sure how to check and fix the RAID array).
The "multiply-claimed block" issue should not affect mount device and read on the files, however, it could cause problem once you write to the file, so I think you can just skip the fix of "multiply-claimed block" (1b-d?) phase.
|
|
The RAID is in good shape. There was a power outage, but I think the RAID cache was flushed (it has battery backup). To me it's hard to see how a read-only fsck only finds a small number of issues, but the read-write finds 70k duplicates. It seems like it has to be journal corruption since that is the only thing affecting the disk between the read-only run and then the read-write run.
All of the duplicate inodes are unlinked though, so I think as long as they are never used it should be ok, right? Is there an easy way to clear out unlinked inodes? I guess e2fsck will link them in lost+found, and an rm -rf of those would mark the blocks as free?
|
All of the duplicate inodes are unlinked though, so I think as long as they are never used it should be ok, right? Is there an easy way to clear out unlinked inodes? I guess e2fsck will link them in lost+found, and an rm -rf of those would mark the blocks as free?
Yes, correct.
|
|
Ok, do you have any suggestions on how to safely remove the inodes that aren't linked to dentries?
|
|
I would strongly recommend upgrading to the newest e2fsck instead of continuing to use the old one.
For the one problem reported:
lfs2-OST000b: Block bitmap for group 32704 is not in group. (block 2251800507482550)
you could try running e2fsck with a backup superblock "e2fsck -b 32768" to see if this has fewer problems.
If the O/0/d* directories are corrupted, then it will appear that inodes are not in use, and they would be linked into lost+found at the end of scanning. They can be moved from lost+found to the right location under /O/0/d*/objid using the ll_recover_lost_found_objs tool on the ldiskfs-mounted OST.
If you are sure you want to unlink the inodes with shared blocks, you could use debugfs to mark the inode deleted in the bitmap (note that the angle brackets '<' and '>' are required to distinguish this as an inode number instead of a filename):
Also note that it is NOT possible to specify multiple inode numbers on the one "freei" line - the second argument is taken as the NUMBER OF INODES TO DELETE!!!
If some manual testing shows that this succeeds to fix files for you, you could write a text file containing the "freei <inode_number>" commands (one per line) and have debugfs execute them using "debugfs -R input_file" to run it.
|
|
Hi Andreas,
We have upgraded e2fsck already, I'll see if it helps out any. The results from the e2fsck -fn using it don't look promising, so I think the corruption is already set on disk. e2fsck appears to already be using the backups:
e2fsck: Group descriptors look bad... trying backup blocks...
Do you think using other backups might help or is it likely to be the same?
The issue is that the inodes in the O/0/d* dirs are duplicated with unlinked inodes (with a smaller inode # so I assume they are older?). So there are presumably good inodes linked in the O/0/d* dirs, but then there are bad inodes marked as allocated (though not linked), but are identical (as seen through stat) to the allocated, linked ("good") inodes in the O/0/d* dirs.
I guess there is no automated way to tell e2fsck to not link these "bad" inodes to l+f? Doing freei on 30k+ inodes makes me nervous.
Thanks,
Kit
|
|
I just did some testing (on a test system) and was only able to fix it using clri as opposed to freei. It looks like e2fsck silently resets the inode as in-use if you use freei. Any gotchas with using clri vs freei?
|
|
Using clri will zero the inode itself instead of just the bit in the bitmap. That is not worse if you really want to delete it, but makes the inode unrecoverable. In this case it probably doesn't matter.
As for e2fsck doing this automatically, the "inode badness" patch should be haning the clearing of "corrupt looking" inodes itself, and duplicate blocks is definitely a sign of badness. However, if the duplication is in the extent tree, this is a sign of corruption outside the inode itself and is not counted.
Have you checked that some other data files contain valid data? Sometimes in cases like this it is a bad RAID rebuild or similar, and while a single superblock copy can be found to give the appearance of a valid filesystem to repair, in fact all of the data files are all broken.
|
|
I checked one file at random, and it seems to be ok:
File /O/0/???/29616418 (inode #85460360, mod time Tue Mar 26 14:14:13 2013)
has 1 multiply-claimed block(s), shared with 1 file(s):
... (inode #18351496, mod time Tue Mar 26 14:14:13 2013)
Clone multiply-claimed blocks? no
debugfs: dump /O/0/d2/29616418 /tmp/test1
[root@lfs-oss-2-4 fsck]# file /tmp/test1
/tmp/test1: ASCII text
[root@lfs-oss-2-4 fsck]# less /tmp/test1
47.45889 -69.81590 7560.000
I don't know if that proves it one way or the other, but at least there is some hope that the data is ok.
What do you think about the idea that a corrupted journal could have caused it? Are there any safeguards that were perhaps missing in the older version of e2fsprogs, or bugs in the older version affecting that? The differences between the -fn and -fp/fy results are really pretty striking.
FWIW, it looks like barriers are disabled:
Sep 17 18:16:41 lfs-oss-2-8 kernel: JBD: barrier-based sync failed on dm-52-8 - disabling barriers
|
|
I was hoping that a larger file might be checked (1MB+), since I've seen lots of cases in the past where only a single block per RAID stripe is consistently corrupted. Maybe my concern is unfounded.
IIRC, barriers being disabled happens automatically if the underlying disk refuses to honor the FUA request to flush its cache. That means the kernel has to wait longer for each request.
It is definitely possible if the journal is corrupted that it will have a multiplier when it gets checkpointed back to the filesystem. We implemented journal checksums in ext4 to combat this problem, but this isn't yet as robust as we'd hoped. The current risk is that a single corrupt journal block would stop journal recovery and prevent checkpointing a large number of blocks back to the filesystem, resulting in more inconsistencies than it prevents. The solution is per-block journal checksums, but that is only possibly in the latest upstream kernels (not sure if it landed yet).
|
|
I found a larger bzip file >1M and ran bzip test on it, it seems clean.
Ah interesting. I'll talk to our disk people to see if we can get FUA going.
Are journal checksums enabled by default? I don't see it in the dumpe2fs output.
I'm uploading a series of logs that show a progression of our activities on one of the osts (ost15). We ran into a couple issues. One is that some of the corrupted inodes looked like they were giant non-extent based files, meaning e2fsck would hang there trying to check all the blocks individually. Another is that when e2fsck hits an inode with invalid extents, it tries to load the bitmap to correct it, but if the bitmap is corrupted, it just dies.
I added code to just skip checking non-extent based files and files with invalid extents and just print them so we could clear them out. That's the combined.cmd.
Anyway this should give you some idea of the level and types of corruption we're seeing.
|
|
http://ddntsr.com/ftp/2013-10-18-lustre_ost15_logs2.tar.gz
Too large to upload (30MB)
|
|
http://ddntsr.com/ftp/2013-10-18-lfs2_e2fsck_prepare_lfsck_2013-10-17.tgz (41MB)
The latest read-only from all the OSS, showing the duplicate blocks that still remain.
|
|
patch to add shared=ignore
|
|
patch to skip some problematic tests:
diff -rup e2fsprogs-1.42.7.1.ddn1/e2fsck/pass1.c e2fsprogs-1.42.7.3.ddn3/e2fsck/pass1.c
--- e2fsprogs-1.42.7.1.ddn1/e2fsck/pass1.c 2013-10-14 13:19:11.000000000 -0700
+++ e2fsprogs-1.42.7.3.ddn3/e2fsck/pass1.c 2013-10-15 18:12:59.000000000 -0700
@@ -2250,6 +2250,11 @@ report_problem:
pctx->blk2 = extent.e_lblk;
pctx->num = extent.e_len;
if (fix_problem(ctx, problem, pctx)) {
+ if (ctx->invalid_bitmaps) {
+ printf("WARNING: invalid bitmaps, unable"
+ "to fix extents\n");
+ goto next;
+ }
e2fsck_read_bitmaps(ctx);
pctx->errcode =
ext2fs_extent_delete(ehandle, 0);
@@ -2489,9 +2494,14 @@ static void check_blocks(e2fsck_t ctx, s
if (extent_fs && (inode->i_flags & EXT4_EXTENTS_FL))
check_blocks_extents(ctx, pctx, &pb);
else {
+ /*
pctx->errcode = ext2fs_block_iterate3(fs, ino,
pb.is_dir ? BLOCK_FLAG_HOLE : 0,
block_buf, process_block, &pb);
+ */
+ printf("WARNING: inode %d not using extents,"
+ " skipping block check.\n", ino);
+ return;
/*
* We do not have uninitialized extents in non extent
* files.
|
|
The latest RO e2fsck: [37MB]
http://ftp.ddntsr.com/ftp/2013-10-19-lfs2_e2fsck_ro_check_2013-10-18.tgz
|
|
Andreas had mentioned on conference call that there were some OSS log messages we should watch out for. I didn't catch exactly what they were, so here are all "unusual" log messages since the last target came online.
|
|
Going through the OST15 logs, it appears that there are a whole range of inodes in the 75000-130000 range are just completely overwritten by garbage (i.e. random timestamps, block counts, sizes, feature flags, etc). There is a feature we wrote "inode badness" that should have detected this and erased those inodes completely, but it doesn't do this until pass 2 of e2fsck. I wonder if this mechanism was foiled by e2fsck being stopped early in the duplicate block pass 1b/1c before it erased the inodes? Also, in hindsight it probably makes sense to ask to clear an inode as soon as its badness exceeds the threshold, because one of the main goals of the inode badness feature is to avoid duplicate block processing on totally corrupt inodes. There may also be some benefit of saving the inode badness in the inode on disk, in case e2fsck is restarted like this.
As for your patches - the skip-invalid-bitmap patch couldn't be used as-is. At the same time, pass 1 should be modifying only the in-memory bitmaps, it isn't until pass 5 that on-disk bitmaps are updated, so it does seem that something needs to be fixed in the extent processing.
The shared=ignore patch seems reasonable. It might make sense to have this also imply E2F_SHARED_LPF, since using using those files would be dangerous. However, this would also modify the namespace in a way that would be difficult to undo later if one of the inodes was erased due to "badness" and was no longer sharing blocks.
|
|
These are messages that are to be expected in OST corruption cases like this. It means there are objects referenced by a file on the MDT, but the objects no longer exist.
This looks like an issue introduced by e2fsck linking files into lost+found or similar. OST objects should only ever have a single link. This is not in itself harmful (the object will still be deleted) and does not imply any further corruption. It may be that there is some space leaked in the filesystem that needs to be cleaned up by a later by running another e2fsck and/or deleting files from lost+found.
|
|
Hi Andreas,
I was wondering if you had thoughts on what changes to e2fsck would be most useful in resolving all the remaining corruption. It sounds like:
- move inodes to l+f on shared=ignore
- early delete bad inodes
- use other data structure to record duplicate blocks / inodes
- other?
Also we were wondering what these messages ment:
Oct 21 03:13:07 lfs-oss-2-8 kernel: LustreError: 22020:0:(client.c:841:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8117d284f800 x1449180093007020/t0 o401->@NET_0x500000ab31078_UUID:17/18 lens 512/384 e 0 to 1 dl 0 ref 1 fl Rpc:N/0/0 rc 0/0
Thanks,
Kit
|
|
I don't think "use other data structure to record duplicate blocks / inodes" was ever mentioned. The data structures themselves are fine. However, in e2fsck pass 1 there is only normally a bitmap kept of in-use blocks, and only if there are collisions in the bitmap (i.e. blocks shared by multiple users) does pass 1b/1c run to track the owning inode(s) of every block. That is done in order to reduce memory usage for block bitmap tracking significantly (by a factor of 32) during normal e2fsck runs. The one potential improvement that I mentioned was to track shared blocks in the superblock or similar (or allow it to be specified on the e2fsck command line) so that the block owners are tracked in pass 1 so that pass 1b doesn't need to scan them again. I'm not sure if that would be a significant improvement, just an idea I had.
|
|
Looking at the most recent e2fsck logs, most of the filesystems look reasonably clean (looks like LU-3542 is being hit on most of them), except for OST 38 looks like it has a lot of problems. It seems OST 38 has corruption of the OST object directory O/0 and is putting most of the objects into /lost+found. It makes sense to run ll_recover_lost_found_objs on that OST to rebuild the object directories and move the objects back out to the O/0 hierarchy.
|
|
I'd mistakenly looked at a partially-downloaded tarball, and didn't see most of the remaining problematic OSTs that are still disabled:
The corruption appears to be a result of large chunks of the inode table being overwritten by other parts of the inode table. That means there are a large number of bad inodes that are exact copies of valid inodes. This results in objects in /O/
{seq}/d*/nnnnn actually having an LMA FID or filter_fid xattr that references a different object ID than 'nnnnn'. Our plan moving forward is that I will work on enhancing ll_recover_lost_found_objs to detect and report this mismatch so that running it on the /O directory will verify the O/{seq}
/d*/nnnnn object name maps to the same FID stored in the inode xattr.
|
|
Kit, I've pushed a patch for ll_recover_lost_found_objs which should report any inconsistent objects in the O/* directory tree as discussed above. It should be run with the "-n" option on the "O" directory (instead of the "lost+found" as it usually does. This should report inodes which incorrectly report in their FID xattr that they are a different object ID. It worked OK in my simple testing here, but I would strongly recommend to run this on a test copy of the OST first. This should be best tested against a sparse copy of one of the problematic OSTs, using "e2image -r" and then mounting the raw image with "-o loop".
Please let me know how this works out.
|
|
Patch at http://review.whamcloud.com/8061
|
|
Hi Andreas,
I'm getting an error when I try to run e2image, I've done it on a couple OSTs:
[root@lfs-oss-2-4 ~]# /usr/bin/time e2image -r /dev/mapper/ost_lfs2_19 /scratch/ost_lfs2_19.img
e2image 1.42.7.wc1 (12-Apr-2013)
seek: Invalid argument
Command exited with non-zero status 1
I got an strace, would that be useful?
|
|
Actually non-sparse e2image works fine. It looks like the sparse image is having issues past 2TB. I guess the FSes on this OSS are all ext3, so that would explain it.
I'll create a dm snapshot to test the tool on.
|
|
NB - you can avoid the 2TB limit if you stripe the file more widely so that individual objects are below 2TB. If you are running an ext4-based ldiskfs (presumably yes) but on a filesystem that was formatted a while ago, you can use "tune2fs -O huge_file" to enable larger-than-2TB files also.
|
|
Sorry I was unclear which disks, I meant the system disks are formatted as ext3, where I was building the sparse file. I got a non-sparse e2image I am copying over to webspace. Nathan also set up a Lustre filesystem I will use to dump a sparse image to.
|
|
Here's the list of corruption and a general plan of attack:
ost_15 unattached inodes
ost_19 multiply-claimed block (2780 inodes), unattached inodes
ost_28 multiply-claimed blocks (52 inodes), unattached inodes
ost_32 unattached inodes - will take multiple passes
ost_34 multiply-claimed blocks (48 inodes), unattached inodes
ost_36 multiply-claimed blocks (1792 inodes), unattached inodes
ost_3 multiply-claimed blocks (376 inodes), unattached inodes
ost_45 multiply-claimed blocks (362 inodes), unattached inodes
Plan:
1) using bind mount, run new ll_recover -n on osts with multiply-claimed blocks
2) use list of duplicate files to create a debugfs script to clri and unlink each "evil twin" file
3) take downtime to execute debugfs script
4) [1 hour] run e2fsck -n on OSTs to make sure all multiply-claimed blocks are gone
a) prepare script in advance to move files to lost+found if any are left
5) [1 hour] run e2fsck -p on all OSTs, as well as ll_recover
a) it's possible that there could be unknown issues at this stage
6) [30 min] clri any multiply-claimed block files in l+f, delete all other files
a) prepare script to nuke l+f
7) [30 min] rerun e2fsck -p to verify that all OSTs are clean
a) again, it's possible that there could be unknown issues at this stage
So I am thinking 3 hours + 4 hours for unknown issues + 1 hour for startup/shutdown. What do you think of this plan/schedule?
|
|
I think it seems reasonable, so long as the new ll_recover_lost_found_objs fixes the shared block problem to a large extent.
It should be possible to do a test run a full test against a raw e2image file for rach of the OSTs. This would reduce the risk of problems during the actual repair, give some confidence that the remaining problems will be repaired, and also minimize the system downtime because the debugfs scripts can be generated while the system is still running.
Loopback mount the raw image file, run "ll_recover_lost_found_objs -n" against it and unmount. Generate and run the debugfs script against the raw image file, then run "e2fsck -fy" on the image to see what is left. If all goes well, the debugfs script can be used on the real device.
|
|
Hi Andreas,
One of the targets just went RO:
Oct 28 17:09:45 lfs-oss-2-6 kernel: LDISKFS-fs error (device dm-50): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 577corrupted: 24074 blocks free in bitmap,
24075 - in gd
It looks like I missed it when disabling targets. I had forgotten that I used the shared=ignore flag when cleaning it up, so the clean bill of health from e2fsck was an illusion.
I've marked it deactivated on the MDT. Hopefully it can hold until Wednesday.
|
|
http://review.whamcloud.com/8188 updates lustre/ChangeLog to recommend using a newer e2fsprogs for b2_1.
|
|
Hi Andreas,
I wanted to tie up the loose ends with the e2fsck patches in this thread. Is the shared=ignore patch something that could be landed? Should I create a gerrit changeset for it?
As for the skip-invalid-bitmap issue, what's the best path to resolution on that? Should I create a new Jira ticket for it or what do you suggest?
Thanks,
Kit
|
|
Any work still outstanding should be tracked under a new ticket
|
Generated at Sat Feb 10 01:39:40 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
e2fsck.patch1
lfs2_unusual_syslog_msgs_2013-10-19.txt