https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2097 Lustre <p>Running acc-sm on a local ldiskfs test system causes e2fsck to fail when checking the MDT in sanity.sh test_17m. I'm running e2fsprogs-1.42.5.wc3-7.fc13.x86_64, which I don't <em>think</em> is the issue, but it hasn't been released yet.</p> <p>It isn't clear when the quota accounting issue is first introduced (even if it is present when the filesystem is formatted), but the problem needs to be resolved for 2.4.0 and/or the e2fsprogs-1.42.5.wc3 release.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 17m: run e2fsck against MDT which contains short/long symlink 21:55:58 (1349409358) create 512 short and long symlink files under /mnt/testfs/d0.sanity/d17m erase them recreate the 512 symlink files with a shorter string stop and checking mds1: e2fsck -fnvd /dev/vg_sookie/lvmdt1 Stopping /mnt/mds1 (opts:-f) on sookie-gig.adilger.int e2fsck 1.42.5.wc3 (15-Sep-2012) [QUOTA WARNING] Usage inconsistent for ID 0:actual (3649536, 1281) != expected (3551232, 1281) [QUOTA WARNING] Usage inconsistent for ID 0:actual (3645440, 1280) != expected (3547136, 1280) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Update quota info for quota type 0? no Update quota info for quota type 1? no testfs-MDT0000: ********** WARNING: Filesystem still has errors ********** 1293 inodes used (1.29%, out of 100000) 8 non-contiguous files (0.6%) 1 non-contiguous directory (0.1%) # of inodes with ind/dind/tind blocks: 3/0/0 17587 blocks used (35.17%, out of 50000) 0 bad blocks 1 large file 109 regular files 135 directories 0 character device files 0 block device files 0 fifos 8 links 1040 symbolic links (521 fast symbolic links) 0 sockets ------------ 1292 files Starting mds1: /dev/vg_sookie/lvmdt1 /mnt/mds1 Started testfs-MDT0000 sanity test_17m: @@@@@@ FAIL: e2fsck should not report error upon short/long symlink MDT: rc=4 </pre> </div></div> Single node test system (client, MDT, 3x OSTs on same node), x86_64<br/> Lustre master &quot;<a href="https://jira.whamcloud.com/browse/LU-1842" title="Quota enforcement landing" class="issue-link" data-issue-key="LU-1842"><strike>LU-1842</strike></a> quota: add quotactl support on qmt&quot; (commit 294aa9cb666c48e02da1057c222fe5f206ce38fc) LU-2097

sanity.sh test_17m, lfsck: e2fsck failed due to MDT quota accounting error

Bug Blocker Resolved Fixed Niu Yawei Andreas Dilger Fri, 5 Oct 2012 15:52:43 +0000 Fri, 19 Apr 2013 16:20:49 +0000 Mon, 29 Oct 2012 05:45:30 +0000 Lustre 2.4.0 Lustre 2.4.0 0 4 <p>I saw the same problem running mmp.sh test_8 on a newly-formatted MDT filesystem (this is the first test that actually runs e2fsck):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Running e2fsck on the device /dev/vg_sookie/lvmdt1 on mds1... e2fsck 1.42.5.wc3 (15-Sep-2012) [QUOTA WARNING] Usage inconsistent for ID 0:actual (1294336, 206) != expected (1273856, 206) [QUOTA WARNING] Usage inconsistent for ID 0:actual (1294336, 206) != expected (1273856, 206) MMP interval is 30 seconds and total wait time is 122 seconds. Please wait... Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Setting filetype for entry '16' in /O/1/d16 (50023) to 1. Setting filetype for entry '17' in /O/1/d17 (50024) to 1. Setting filetype for entry '18' in /O/1/d18 (50025) to 1. Setting filetype for entry '19' in /O/1/d19 (50026) to 1. Setting filetype for entry '20' in /O/1/d20 (50027) to 1. Setting filetype for entry '21' in /O/1/d21 (50028) to 1. Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information Update quota info for quota type 0? yes Update quota info for quota type 1? yes testfs-MDT0000: ***** FILE SYSTEM WAS MODIFIED ***** testfs-MDT0000: 215/100000 files (2.8% non-contiguous), 17012/50000 blocks </pre> </div></div> <p>I can imagine this happening fairly easily if we have code paths that are not updating the root quota usage properly?</p> <p>The filetype errors are in <a href="https://jira.whamcloud.com/browse/LU-1861" title="osd-ldiskfs doesn&#39;t set object file type" class="issue-link" data-issue-key="LU-1861"><del>LU-1861</del></a> and not related to this problem.</p> <p>Andreas, is this with OFD or obdfilter?</p> <p>I think I hit it with both ZFS and ldiskfs using OFD, though I don't <em>think</em> it matters, since the error is on the MDT?</p> <p>this is rather strange, because we do not modify accounting ?</p> <blockquote> <p>I think I hit it with both ZFS and ldiskfs using OFD, though I don't think it matters, since the error is on the MDT?</p></blockquote> <p>Ah, nm. I thought fsck was run against an OST.</p> <blockquote> <p>this is rather strange, because we do not modify accounting ?</p></blockquote> <p>Right, we don't, but we still bypass the vfs so there might be a code path where we forget to call ll_vfs_dq_transfer() or something similar. It might also be a bug in quota support in e2fsck or even a kernel bug (till now, nothing was checking accounting information correctness).</p> <p>Niu, could you please have a look at this one?</p> <p>I can reproduce it by "NOFORMAT=1 sh llmount.sh; ONLY=17m sh sanity.sh" or running the 17m twice "ONLY=17m sh sanity.sh; ONLY=17m sh sanity.sh", but seems OSTs doesn't have such problem, so it's unlikely a kernel or e2fsprogs bug.</p> <p>I think it because we missed ll_vfs_dq_init() for some system objects, like llog, oi, last_rcvd, etc. so the block accounting for those existing objects is missed. (if they are newly created, then there isn't any problem), I'll post a patch to fix it soon.</p> <p><a href="http://review.whamcloud.com/4220" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4220</a></p> <p>Patch is merged, just after the 2.3.53 tag. </p> <p>Was this patch actually tested to fix the problem? I still see the same issue here (running mmp.sh test 8 in this case):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>e2fsck 1.42.5.wc3 (15-Sep-2012) Starting mds1: /dev/vg_sookie/lvmdt1 /mnt/mds1 Start of /dev/vg_sookie/lvmdt1 on mds1 failed 1 [QUOTA WARNING] Usage inconsistent for ID 0:actual (1294336, 283) != expected (1253376, 279) [QUOTA WARNING] Usage inconsistent for ID 0:actual (1294336, 283) != expected (1253376, 279) : : [problems from LU-1861] : : Update quota info for quota type 0? yes Update quota info for quota type 1? yes </pre> </div></div> <p>Filesystem had previously been running other acceptance small tests, but I believe this hits early enough and consistently in sanity.sh that it is easily reproduced.</p> <p>I verified the test fixed the problem of:</p> <ul class="alternate" type="square"> <li>NOFORMAT=1 sh llmount.sh; ONLY=17m sh sanity.sh (accounting for llog objects created during mount)</li> <li>ONLY=17m sh sanity.sh; ONLY=17m sh sanity.sh (accounting for oi)</li> </ul> <p>Maybe there is till something missed, I'm checking into this. Thank you, Andreas.</p> <p>more ll_vfs_dq_init() are added: <a href="http://review.whamcloud.com/427" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/427</a></p> <p>Without above patch, I can reproduce the problem by "sh sanity.sh; ONLY=8 sh mmp.sh".</p> <p>patches landed for 2.4</p> Related LU-2663 Development Rank 1|hzv58f: Rank (Obsolete) 4381 Severity