https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3542 Lustre <p>e2fsck doesn't actually clear deleted/unused inodes, though it claims to. I've attached a log showing what we are seeing. The customer is CalTech. </p> Centos5, e2fsprogs-1.42.7.wc1-0redhat LU-3542

deleted/unused inodes not actually cleared by e2fsck

Bug Critical Resolved Fixed Niu Yawei Kit Westneat Mon, 1 Jul 2013 14:20:16 +0000 Fri, 13 Dec 2013 20:53:20 +0000 Fri, 13 Dec 2013 20:53:20 +0000 0 10 <p>Nathaniel</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I need to increase the priority on this one. The OSTs are stopping with "ldiskfs_lookup: deleted inode referenced." Do you have any ideas for how to fix it? I assume this means that the dentries are corrupted, but it seems weird that the files don't show up when I try to ls them. Is it possible that it's something with the HTREE? There were some HTREE messages in the original e2fsck.</p> <p>Thanks.</p> <p>Raised priority to blockern and severity to 1, after Kit's last update on this problem </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> We running into a problem trying to restart Lustre after a Sev1. We have run e2fsck several times on a filesystem hit by catastrophic disk failure and cleaned up most of the corruption. However, there are a bunch of referenced deleted/cleared inodes that are not getting cleaned up. e2fsck claims to clear them, but when you rerun it, it's still there. When the OSTs hit these inodes in production, the OSTs go read-only, bringing Lustre down. So due to this, we are in Sev1 until the fs is 100% clean. I put the most recent e2fsck logs in LU-3542. Anything else I should get? </pre> </div></div> <p>ok I think I figured out how to workaround the problem. If I use debugfs, I can unlink all the troublesome files and it works ok. Is there anything I should get to try to debug the e2fsprogs issue before I unlink everything?</p> <p>Yes, I confirm that. I just tested it too and it seems to work fine!!... What puzzle me is that e2fsck does not propose/do it ...</p> <p>Would be also interesting if you can provide the 1st e2fsck log it still available ??</p> <p>Kit, can you please run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs -c -R "htree_dump O/0/d10" /dev/mapper/ost_global_7 </pre> </div></div> <p>Also if I do a clri &lt;inode&gt; with debugfs to simulate the problem, e2fsck seems to do the right thing, so I'm not sure what is weird about this filesystem.</p> <p>Very first:<br/> e2fsck -v -f -p /dev/mapper/ost_global_7MMP interval is 10 seconds and total wait time is 42 seconds. Please wait...<br/> global-OST0007: recovering journal<br/> global-OST0007: Entry '29584587' in /O/0/d11 (148471827) has deleted/unused inode 153961358. CLEARED.<br/> global-OST0007: Entry '29584575' in /O/0/d31 (148471847) has deleted/unused inode 153961348. CLEARED.<br/> global-OST0007: Entry '29584573' in /O/0/d29 (148471845) has deleted/unused inode 153961346. CLEARED.<br/> global-OST0007: Entry '29584407' in /O/0/d23 (148471839) has deleted/unused inode 153961200. CLEARED.<br/> global-OST0007: Entry '29584406' in /O/0/d22 (148471838) has deleted/unused inode 153961199. CLEARED.<br/> global-OST0007: Entry '29584589' in /O/0/d13 (148471829) has deleted/unused inode 153961359. CLEARED.<br/> global-OST0007: Directory inode 148471835, block #360, offset 0: directory corrupted</p> <p>I will try to find the e2fsck -y log. </p> <p>hey Andreas, some how I missed your comment till now, here is the htree dump</p> <p>here are the e2fsck -p outputs from the first and second runs on ost_3 (also exhibiting the same behavior) </p> <p>Kit, it isn't clear from your comment whether your use of <tt>clri &lt;inode&gt;</tt> is intended as a workaround (i.e. this allows e2fsck to correctly clean up the inode), or if you are trying (unsuccessfully) to reproduce the problem on a test filesystem to allow debugging e2fsck?</p> <p>It definitely seems possible to use debugfs to to mark the affected inodes as deleted and remove the name entries, e.g. "<tt>clri &lt;153961357&gt;</tt>" and "<tt>unlink /O/0/d10/29584586</tt>". In theory "<tt>rm /O/0/d10/29584586</tt>" should do both, but it may be there is some problem with this and maybe safer to do them separately. I'd try this first on the ost_global_7 target, since it only has a few such objects, and then run "e2fsck -fy" to see if this fixed the problem.</p> <p>You could also try running "e2fsck -fD" on ost_global_7, which should rebuild the htree directory structure on the OST, since it seems there may a problem with this as well. This isn't a requirement if it is working fine after the first e2fsck, and maybe better left to a scheduled downtime in the future.</p> <p>Hi Andreas, I was trying to use clri to simulate the failure. I tested the unlink/rm through debugfs on a snapshot and it seemed to work well. I just saw all the htree corruption and got worried about running it on the real device. </p> <p>I'll try running e2fsck -fD on the snapshot to see how it does. I have been wary of it since there used to be bugs, but it looks like all those have been fixed in this version. </p> <p>Thanks. </p> <p>Kit, another option, which <em>might</em> allow you to get the system back up and running, if e2fsck isn't fixing the problem, is to mount the OST with "-o errors=continue" which would at least avoid the OST from going read-only when it hit this error.</p> <p>Unfortunately, it seems that the "-o errors=continue" option in 2.4 is placed <em>before</em> "errors=remount-ro" in the mount options line, so it is overridden (which is itself a bug). I'm not sure if this is handled correctly in 2.1, but worthwhile to try (I don't have a 2.1 system handy to test this right now).</p> <p>The previous "e2fsck -fD" problem was only seen on MDT devices, not on OST devices. That said, it is my understanding that those problems were fixed in the version of e2fsck-1.42.7.wc1 that you are running, but I would have been leery to suggest it at this point if the issue was on an MDT device.</p> <p>If you have a snapshot, that is excellent, as it allows some margin for error if e2fsck behaves in a (more) unexpected manner.</p> <p>Hi Andreas,</p> <p>After running the e2fsck -fD, I am getting this on e2fsck -fvy:<br/> Interior extent node level 0 of inode 148471837:<br/> Logical start 980 does not match logical start 981 at next level. Fix? yes</p> <p>Inode 148471837, i_size is 2097152, should be 4022272. Fix? yes</p> <p>Inode 148471837, i_blocks is 4112, should be 2856. Fix? yes</p> <p>Inode 148471838, i_size is 2097152, should be 4005888. Fix? yes</p> <p>Inode 148471838, i_blocks is 4112, should be 2816. Fix? yes</p> <p>Inode 148471839, i_size is 2084864, should be 3952640. Fix? yes</p> <p>Inode 148471839, i_blocks is 4088, should be 2832. Fix? yes</p> <p>Inode 148471840, i_size is 2093056, should be 3948544. Fix? yes</p> <p>Inode 148471840, i_blocks is 4104, should be 2880. Fix? yes</p> <p>Inode 148471841, i_size is 2093056, should be 4001792. Fix? yes</p> <p>Inode 148471841, i_blocks is 4104, should be 2800. Fix? yes</p> <p>In your opinion, is this corruption created by the -fD or is it corruption uncovered by it?</p> <p>It looks like a bit of both. The "-fD" option re-sorts and compacts the htree directories to ensure all of the leaf blocks are valid. Normally this makes the directory smaller, which is the cause of the reduction in "i_blocks" values. Conversely, the i_size value is based on the i_blocks count, but it is fixing this before it checks the i_blocks value. That seems to be a separate bug in e2fsck.</p> <p>I don't think it will be harmful to allow these problems to be fixed, but I suspect a second e2fsck run is needed to re-fix the i_size values after i_blocks has been updated, and that should resolve the problems finally.</p> <p>These all appear to be directory inodes. Towards the end of the run, it is non-stop "Unattached inode ...".</p> <p>My snapshot ran out of space, and I was overconfident and ran it live. I'm glad the ll_recover script exists!</p> <p>ah I didn't see your response before posting. I am running this second e2fsck on a snapshot (the one with all the unattached inodes). Do you think there is any way to avoid all the Unattached inodes, or is it a necessary step at this point? Like would some combination of y/n to those i_size/i_blocks question prevent them from being lost+found?</p> <p>No, I think the unattached inodes are a consequence of the directory blocks being corrupted, and it is dumping all of the inodes from the corrupt leaf blocks into lost+found. You'll need to run ll_recover_lost_found_objs to fix them. In the not too distant future, online LFSCK in 2.5 (patch <a href="http://review.whamcloud.com/6857" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6857</a>) will be able to do this automatically at mount time, but until then it needs to be run by hand.</p> <p>Ah ok, hmm. a few questions:</p> <ul> <li>Should I run the -fD on all the OSTs then?</li> <li>I would have expected e2fsck to uncover any corrupted directories without -fD. Should I file a new bug on that?</li> <li>I was doing a read-only lfsck when the OSTs started going read-only. Is there a possibility that some of the files in the corrupt leaf nodes didn't get added to the ost DBs due to the directory corruption? Should I rerun the e2fsck object db creation step?</li> </ul> <p>Thanks for all your help!</p> <p>I'm not sure why the original e2fsck didn't show problems with the directory blocks, but the later ones do. Typically, e2fsck is very robust about fixing problems on the first pass, or restarting automatically in the rare cases it cannot.</p> <p>If the other OSTs are behaving properly, I would avoid e2fsck -fD for now. While this fixes up the htree directory structure, it also means that the directory will need to allocate new blocks as soon as new files are being allocated there (i.e. immediately for any OST).</p> <p>Kit, what is the status of this bug? Can we lower it from Sev 1?</p> <p>Hi Andreas, we are doing the final lfsck to get the list of damaged files, but we can lower the severity of this ticket. There are two e2fsck behaviors that we saw during this that seem like bugs to me:</p> <ul> <li>first is the not unlinking the deleted/cleared inodes</li> <li>second is the movement of most files to lost+found by e2fsck -D</li> </ul> <p>It might not be worth the effort to explore these at any high priority, but I think we should leave the ticket open for documentation at least.</p> <p>Thanks again for all your help and advice.</p> <p>Niu</p> <p>Can you please see what work remains on this ticket?</p> <p>THanks</p> <p>Peter</p> <p>Peter, the two questions Kit asked are probably e2fsck bugs. The remaining work is: </p> <ul class="alternate" type="square"> <li>Search to find out if the same problem was reported in Linux community before, and if there is any patch alreay. (I did an initial searching, but had no luck so far)</li> <li>Try to reproduce the probelm and trace into the e2fsck code to see if it's really some bug needs be fixed. (that requires e2fsprogs expert and could be time-consuming)</li> </ul> <p>I agree with Kit that it's not high priority job.</p> <p>Hi Niu,</p> <p>This has become a higher priority for us. The problem is that if deleted inodes are not cleared, the filesystem will go read-only when it encounters the inode. This can lead to a state where the filesystem goes read-only at a random time and only manual intervention with debugfs can bring it back to a healthy state. It has happened to us a couple of times now, so I think we need to explore problem #1 a little more closely. </p> <p>Thanks.</p> <p>Kit, I didn't know they often run into the problem of "deleted/unused inode". Which Lustre version did they use? and do you know what kind of operation could possibly caused the problem? If possible, could you collect the log on OST before the problem happen? I think it might be helpful for us to figure out how this happened.</p> <p>I'll look into the e2fsck problem at the same time. Thank you.</p> <p>Hi Niu,</p> <p>The first customer had a problem with the RAID storage which caused the ldiskfs corruption. The second customer had a power outage that we think corrupted the journal and journal replay (<a href="https://jira.whamcloud.com/browse/LU-4102" title="lots of multiply-claimed blocks in e2fsck" class="issue-link" data-issue-key="LU-4102"><del>LU-4102</del></a>). Basically when there is some kind of ldiskfs corruption, there is the possibility of getting these delete/unused inode messages, and it seems if the htrees are also corrupt, e2fsck is unable to clear them.</p> <p>Thanks,<br/> Kit</p> <p>I looked through the relevant code in pass2.c::check_dir_block():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* * Offer to clear unused inodes; <span class="code-keyword">if</span> we are going to be * restarting the scan due to bg_itable_unused being * wrong, then don't clear any inodes to avoid zapping * inodes that were skipped during pass1 due to an * incorrect bg_itable_unused; we'll get any real * problems after we restart. */ <span class="code-keyword">if</span> (!(ctx-&gt;flags &amp; E2F_FLAG_RESTART_LATER) &amp;&amp; !(ext2fs_test_inode_bitmap2(ctx-&gt;inode_used_map, dirent-&gt;inode))) problem = PR_2_UNUSED_INODE; <span class="code-keyword">if</span> (problem) { <span class="code-keyword">if</span> (fix_problem(ctx, problem, &amp;cd-&gt;pctx)) { dirent-&gt;inode = 0; dir_modified++; <span class="code-keyword">goto</span> next; </pre> </div></div> <p>It is easy to trigger the <tt>PR_2_UNUSED_INODE</tt> problem by setting nlink = 0 in the inode(s) via debugfs. However, when I run e2fsck against such a filesystem (whether with small directories or large htree directories) e2fsck fixes the problem by clearing the dirent (setting inode = 0 above, and later writing out the directory block) and a second check shows it is fixed.</p> <p>To capture a filesystem that has a persistent case of this problem (after "e2fsck -fy" didn't fix it) so that it can be debugged and fixed, please use e2image to dump the filesystem metadata. The dense image format can be efficiently compressed and transported, unlike the sparse variant of e2image:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2image -Q /dev/OSTnnnn OSTnnnn.qcow bzip2 -9 OSTnnnn.qcow </pre> </div></div> <p>Hopefully the OSTnnnn.qcow.bz2 image size is small enough for transport. It is possible to reconstitute the (uncompressed) qcow file into a raw ext4 image file that can be tested with e2fsck, debugfs, or mounted via loopback.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2image -r OSTnnnn.qcow OSTnnnn.raw </pre> </div></div> <p>I don't think any of the OSTs described in <a href="https://jira.whamcloud.com/browse/LU-4102" title="lots of multiply-claimed blocks in e2fsck" class="issue-link" data-issue-key="LU-4102"><del>LU-4102</del></a> currently has the deleted/unused inodes issue. All the ones that reported it on the r/o e2fsck had previously been clean, so I think that it's just a matter of them being in use. That being said I could get an image of the OST (ost_45) that had the error before.. Do you think that might be useful? I have the e2fsck output as well. </p> <p>Even if there isn't a 100% chance that OST has the problem, it is still worthwhile to make an image of the OST. This will first give us an idea of how long it takes to generate the image, how large it is (uncompressed and compressed), and it can also be used to test the <a href="https://jira.whamcloud.com/browse/LU-4102" title="lots of multiply-claimed blocks in e2fsck" class="issue-link" data-issue-key="LU-4102"><del>LU-4102</del></a> code. </p> <p>I got a qcow image with a file exhibiting the corruption, it's available here:<br/> <a href="http://ddntsr.com/ftp/2013-10-30-lustre-ost_lfs2_36.qcow2.bz2" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2013-10-30-lustre-ost_lfs2_36.qcow2.bz2</a> <span class="error">&#91;295M&#93;</span></p> <ol> <li>e2fsck -fp /dev/mapper/ost_lfs2_36<br/> lfs2-OST0024: Entry '62977970' in /O/0/d18 (88080410) has deleted/unused inode 1051496. CLEARED.<br/> lfs2-OST0024: 1546929/89620480 files (9.5% non-contiguous), 2367418676/5735710720 blocks</li> </ol> <ol> <li>e2fsck -fp /dev/mapper/ost_lfs2_36<br/> lfs2-OST0024: Entry '62977970' in /O/0/d18 (88080410) has deleted/unused inode 1051496. CLEARED.<br/> lfs2-OST0024: 1546929/89620480 files (9.5% non-contiguous), 2367418676/5735710720 blocks</li> </ol> <p>We ran into this problem as well. I'll attach the fsck output to this JIRA. Email me if you'd like me to send you the qcow image. </p> <p>I just uploaded my qcow image to ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-3542" title="deleted/unused inodes not actually cleared by e2fsck" class="issue-link" data-issue-key="LU-3542"><del>LU-3542</del></a>/ost000b.qcow.bz2</p> <p>The raw device of ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-3542" title="deleted/unused inodes not actually cleared by e2fsck" class="issue-link" data-issue-key="LU-3542"><del>LU-3542</del></a>/ost000b.qcow.bz2 is 16TB? It's hard for me to find a machine with that big drive to reproduce the problem, is there any smaller OST which has the same problem?</p> <p>Hi Niu, I was able to convert the qcow image to a raw (sparse) image on an XFS filesystem. It uses 3.6GB, though it reports a size of 28TB:<br/> <span class="error">&#91;root@oxonia-mds1 rcvy&#93;</span># ls -lh<br/> total 3.6G<br/> <del>rw</del>------ 1 root root 28T Nov 22 10:04 ost000b.raw</p> <blockquote> <p>Hi Niu, I was able to convert the qcow image to a raw (sparse) image on an XFS filesystem. It uses 3.6GB, though it reports a size of 28TB:<br/> <span class="error">&#91;root@oxonia-mds1 rcvy&#93;</span># ls -lh<br/> total 3.6G<br/> rw------ 1 root root 28T Nov 22 10:04 ost000b.raw</p></blockquote> <p>oh, I didn't notice it's sparse file. Then I think it can be converted on ext4 either, however, I got following error while trying to convert it on ext4 (actual size 1.6G, showed 16T):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">e2image: Invalid argument <span class="code-keyword">while</span> trying to convert qcow2 image (ost000b.qcow) into raw image </pre> </div></div> <p>If the 3.6G file you mentioned is <a href="http://ddntsr.com/ftp/2013-10-30-lustre-ost_lfs2_36.qcow2.bz2" class="external-link" target="_blank" rel="nofollow noopener">http://ddntsr.com/ftp/2013-10-30-lustre-ost_lfs2_36.qcow2.bz2</a>, could you upload it to whamcloud ftp? cause I have no permission to access the ddn ftp server.</p> <p>Hi Niu, </p> <p>I don't think ext4 supports files greater than 16TB, so you'd need to use XFS or ZFS. </p> <p>Yeah, the files on the DDN server are temporary.. I'll upload it to the Intel FTP server.</p> <p>Thanks,<br/> Kit</p> <p>It seems like this doesn't actually produce a valid raw image:<br/> e2image -r OSTnnnn.qcow OSTnnnn.raw</p> <p>I had to do:<br/> qemu-img convert -p -O raw /scratch/ost000b.qcow ost000b.raw</p> <p>to get something that worked.</p> <p>It looks like the block number is wrapping around during the io_channel write:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Entry <span class="code-quote">'3102500'</span> in /O/0/d4 (19398664) has deleted/unused inode 26072855. Clear? yes Breakpoint 1, check_dir_block (fs=&lt;value optimized out&gt;, db=0x7ffff7f14340, priv_data=0x7fffffffe180) at pass2.c:1219 1219 cd-&gt;pctx.errcode = ext2fs_write_dir_block(fs, block_nr, buf); (gdb) p block_nr $30 = 4966058525 Breakpoint 2, raw_write_blk (channel=0x647570, data=0x648670, block=671091229, count=1, bufv=0x64f060) at unix_io.c:233 (gdb) p (unsigned <span class="code-object">int</span>)4966058525 $33 = 671091229 </pre> </div></div> <p>I thought maybe it was the cache node, but that appears to use an unsigned long long to store the block.</p> <p>I'll keep looking but I thought I'd pass that info along in case it helps.</p> <p>oh I think it is the definition of ext2fs_write_dir_block:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">errcode_t ext2fs_write_dir_block(ext2_filsys fs, blk_t block, void *inbuf) typedef __u32 blk_t; typedef __u64 blk64_t; </pre> </div></div> <p>It seems like that should be blk64_t? It looks like ext2fs_write_dir_block3 uses blk64_t, but the call to ext2fs_write_dir_block already casts it down to blk_t</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Breakpoint 1, check_dir_block (fs=&lt;value optimized out&gt;, db=0x7ffff7f14a00, priv_data=0x7fffffffe180) at pass2.c:1219 1219 cd-&gt;pctx.errcode = ext2fs_write_dir_block(fs, block_nr, buf); (gdb) p block_nr $35 = 4966058603 (gdb) cont Continuing. Breakpoint 3, ext2fs_write_dir_block3 (fs=0x647420, block=671091307, inbuf=0x667270, flags=0) at dirblock.c:146 146 <span class="code-keyword">return</span> io_channel_write_blk64(fs-&gt;io, block, 1, (<span class="code-object">char</span> *) inbuf); (gdb) p (blk_t)4966058603 $36 = 671091307 </pre> </div></div> <p>Yes, I think that's probably the reason of the entries are not fixed. check_dir_block() should call ext2fs_write_dir_block3() directly.</p> <p>ok, I can get a patch for that. </p> <p>I ran gcc with -Wconversion on the source code and there are a few other cases where it converts to blk_t from blk64_t. I guess it would be good to go through them all at some point... I am not sure I know enough about ext4 to judge if the conversion is valid or not. For example, pass2.c also has a conversion on line 890:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">struct dx_dirblock_info { <span class="code-object">int</span> type; blk_t phys; <span class="code-object">int</span> flags; blk_t parent; ext2_dirhash_t min_hash; ext2_dirhash_t max_hash; ext2_dirhash_t node_min_hash; ext2_dirhash_t node_max_hash; }; ... dx_db = &amp;dx_dir-&gt;dx_block[db-&gt;blockcnt]; dx_db-&gt;type = DX_DIRBLOCK_LEAF; 890&gt;&gt; dx_db-&gt;phys = block_nr; dx_db-&gt;min_hash = ~0; dx_db-&gt;max_hash = 0; </pre> </div></div> <p>Should those be 64-bit? It seems like it, but I don't know. There are 103 cases of conversion to blk_t from blk64_t . The real number of conversions is probably higher since there are also some like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">fileio.c:164: warning: conversion to ‘blk_t’ from ‘__u64’ may alter its value res_gdt.c:140: warning: conversion to ‘blk_t’ from ‘<span class="code-object">long</span> <span class="code-object">long</span> unsigned <span class="code-object">int</span>’ may alter its value pass2.c:687: warning: conversion to ‘blk_t’ from ‘e2_blkcnt_t’ may alter its value </pre> </div></div> <p><a href="http://review.whamcloud.com/#/c/8416/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8416/</a></p> <p>This fix has landed for the next e2fsprogs release</p> Development Rank 1|hzvugv: Rank (Obsolete) 8914 Severity