https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1916 Lustre <p>We have a user whos quota dosn't match what du shows.</p> <p>nbp1-mds /nobackupp1/cenewma1 # lfs quota -u cenewma1 /nobackupp1<br/> Disk quotas for user cenewma1 (uid 11540):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /nobackupp1 677532808 990000000 1100000000 - 34234 75000 1000000 -<br/> nbp1-mds /nobackupp1/cenewma1 # du -sk .<br/> 492191480 .</p> <p>All her files are in one directory /nobackupp1/cenewma1</p> LU-1916

du vs lfs quota miss-match

Bug Major Resolved Cannot Reproduce Niu Yawei Mahmoud Hanafi ptr Wed, 12 Sep 2012 16:34:13 +0000 Tue, 29 Oct 2013 18:37:53 +0000 Tue, 29 Oct 2013 18:37:53 +0000 Lustre 2.1.2 0 3 <p>Niu</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>I suspect that there are some orphan objects (owned by cenewma1) on OSTs, which caused the usage of 'lfs quota' is larger than du.<br/> Could you try if rebooting OSTs can fix the problem? When OST restart, it'll replay unlink &amp; setattr log on MDS, then the orphan objects will be removed/chowned. Thanks.</p> <p>Can we determine which OST has the orphan objects?<br/> How does one get these orphan objects?<br/> Would we need to do a lfsck?</p> <p>I don't think we have any tools to determine which OST has orphan objects.</p> <p>When some client unlink files, it unlink namespace on MDS first, then issue objects destroy to OSTs, if the client crash (or some other failure) between these two steps, there will be orphan objects on OSTs.</p> <p>I don't think we need to do a lfsck, the orhpan objects will be cleared automatically once the OSTs restart.</p> <p>Could you provide the output of "lfs quota -u cenewma1 -v /nobackupp1"? lfsck can show the orphan objects and fix them, but I think orphan will be cleared once OST restart and connect to mds, so usually it's not necessary to run lfsck.</p> <p>here is the output. This filesystem has be restart. So if there were orphan object the would have been cleared.</p> <p>I calculated spaced used for users by adding up objects on each ost using a modified e2scan. That value was very close to what quota was showing. So I think that lustre is not clean up objects on the osts. There was a report of a user deleting a large file and his file count and quota size should have decreased but it did not. I been try to reproduce it but have not been able to yet. I will try to compare file counts from quota vs. real number of files.</p> <p>pfe1 ~ # lfs quota -u cenewma1 -v /nobackupp1<br/> Disk quotas for user cenewma1 (uid 11540):<br/> Filesystem kbytes quota limit grace files quota limit grace<br/> /nobackupp1 677532804 990000000 1100000000 - 34233 75000 1000000 -<br/> nbp1-MDT0000_UUID<br/> 18528 - 131072 - 34233 - 40960 -<br/> nbp1-OST0000_UUID<br/> 4627588 - 4718592 - - - - -<br/> nbp1-OST0001_UUID<br/> 5059468 - 5242880 - - - - -<br/> nbp1-OST0002_UUID<br/> 5001992 - 5111808 - - - - -<br/> nbp1-OST0003_UUID<br/> 5147528 - 5242880 - - - - -<br/> nbp1-OST0004_UUID<br/> 4397416 - 4587520 - - - - -<br/> nbp1-OST0005_UUID<br/> 5171204 - 5242880 - - - - -<br/> nbp1-OST0006_UUID<br/> 4308496 - 4456448 - - - - -<br/> nbp1-OST0007_UUID<br/> 4573088 - 4718592 - - - - -<br/> nbp1-OST0008_UUID<br/> 6079784 - 6160384 - - - - -<br/> nbp1-OST0009_UUID<br/> 5127440 - 5242880 - - - - -<br/> nbp1-OST000a_UUID<br/> 7754820 - 7864320 - - - - -<br/> nbp1-OST000b_UUID<br/> 4281648 - 4456448 - - - - -<br/> nbp1-OST000c_UUID<br/> 4527680 - 4718592 - - - - -<br/> nbp1-OST000d_UUID<br/> 4655232 - 4849664 - - - - -<br/> nbp1-OST000e_UUID<br/> 4251904 - 4325376 - - - - -<br/> nbp1-OST000f_UUID<br/> 4606736 - 4718592 - - - - -<br/> nbp1-OST0010_UUID<br/> 3764148 - 3932160 - - - - -<br/> nbp1-OST0011_UUID<br/> 4050308 - 4194304 - - - - -<br/> nbp1-OST0012_UUID<br/> 5168732 - 5242880 - - - - -<br/> nbp1-OST0013_UUID<br/> 5631040 - 5767168 - - - - -<br/> nbp1-OST0014_UUID<br/> 5040964 - 5111808 - - - - -<br/> nbp1-OST0015_UUID<br/> 3138804 - 3276800 - - - - -<br/> nbp1-OST0016_UUID<br/> 4985140 - 5111808 - - - - -<br/> nbp1-OST0017_UUID<br/> 5776324 - 5898240 - - - - -<br/> nbp1-OST0018_UUID<br/> 6531844 - 6684672 - - - - -<br/> nbp1-OST0019_UUID<br/> 4307916 - 4456448 - - - - -<br/> nbp1-OST001a_UUID<br/> 6343920 - 6422528 - - - - -<br/> nbp1-OST001b_UUID<br/> 6194384 - 6291456 - - - - -<br/> nbp1-OST001c_UUID<br/> 4251588 - 4325376 - - - - -<br/> nbp1-OST001d_UUID<br/> 4339640 - 4456448 - - - - -<br/> nbp1-OST001e_UUID<br/> 4209232 - 4325376 - - - - -<br/> nbp1-OST001f_UUID<br/> 5527532 - 5636096 - - - - -<br/> nbp1-OST0020_UUID<br/> 6629476 - 6815744 - - - - -<br/> nbp1-OST0021_UUID<br/> 3878208 - 4063232 - - - - -<br/> nbp1-OST0022_UUID<br/> 4401920 - 4587520 - - - - -<br/> nbp1-OST0023_UUID<br/> 4523984 - 4718592 - - - - -<br/> nbp1-OST0024_UUID<br/> 5144444 - 5242880 - - - - -<br/> nbp1-OST0025_UUID<br/> 4165376 - 4325376 - - - - -<br/> nbp1-OST0026_UUID<br/> 4017568 - 4194304 - - - - -<br/> nbp1-OST0027_UUID<br/> 3677424 - 3801088 - - - - -<br/> nbp1-OST0028_UUID<br/> 4681332 - 4849664 - - - - -<br/> nbp1-OST0029_UUID<br/> 6254192 - 6422528 - - - - -<br/> nbp1-OST002a_UUID<br/> 3962612 - 4063232 - - - - -<br/> nbp1-OST002b_UUID<br/> 5581092 - 5767168 - - - - -<br/> nbp1-OST002c_UUID<br/> 4681436 - 4849664 - - - - -<br/> nbp1-OST002d_UUID<br/> 3222648 - 3407872 - - - - -<br/> nbp1-OST002e_UUID<br/> 4960592 - 5111808 - - - - -<br/> nbp1-OST002f_UUID<br/> 6170920 - 6291456 - - - - -<br/> nbp1-OST0030_UUID<br/> 4947888 - 5111808 - - - - -<br/> nbp1-OST0031_UUID<br/> 4877680 - 4980736 - - - - -<br/> nbp1-OST0032_UUID<br/> 5693548 - 5767168 - - - - -<br/> nbp1-OST0033_UUID<br/> 4268804 - 4456448 - - - - -<br/> nbp1-OST0034_UUID<br/> 3973800 - 4063232 - - - - -<br/> nbp1-OST0035_UUID<br/> 5744668 - 5898240 - - - - -<br/> nbp1-OST0036_UUID<br/> 4122328 - 4194304 - - - - -<br/> nbp1-OST0037_UUID<br/> 4303528 - 4456448 - - - - -<br/> nbp1-OST0038_UUID<br/> 3404112 - 3538944 - - - - -<br/> nbp1-OST0039_UUID<br/> 8269384 - 8388608 - - - - -<br/> nbp1-OST003a_UUID<br/> 5336032 - 5505024 - - - - -<br/> nbp1-OST003b_UUID<br/> 5555316 - 5636096 - - - - -<br/> nbp1-OST003c_UUID<br/> 4060448 - 4194304 - - - - -<br/> nbp1-OST003d_UUID<br/> 6620264 - 6815744 - - - - -<br/> nbp1-OST003e_UUID<br/> 5119320 - 5242880 - - - - -<br/> nbp1-OST003f_UUID<br/> 6267080 - 6422528 - - - - -<br/> nbp1-OST0040_UUID<br/> 3356136 - 3538944 - - - - -<br/> nbp1-OST0041_UUID<br/> 5295300 - 5373952 - - - - -<br/> nbp1-OST0042_UUID<br/> 5569504 - 5636096 - - - - -<br/> nbp1-OST0043_UUID<br/> 5390596 - 5505024 - - - - -<br/> nbp1-OST0044_UUID<br/> 4988152 - 5111808 - - - - -<br/> nbp1-OST0045_UUID<br/> 3383276 - 3538944 - - - - -<br/> nbp1-OST0046_UUID<br/> 5804668 - 5898240 - - - - -<br/> nbp1-OST0047_UUID<br/> 4546556 - 4718592 - - - - -<br/> nbp1-OST0048_UUID<br/> 4378612 - 4456448 - - - - -<br/> nbp1-OST0049_UUID<br/> 5769100 - 5898240 - - - - -<br/> nbp1-OST004a_UUID<br/> 4457072 - 4587520 - - - - -<br/> nbp1-OST004b_UUID<br/> 4922704 - 5111808 - - - - -<br/> nbp1-OST004c_UUID<br/> 3756600 - 3932160 - - - - -<br/> nbp1-OST004d_UUID<br/> 5131736 - 5242880 - - - - -<br/> nbp1-OST004e_UUID<br/> 4163448 - 4325376 - - - - -<br/> nbp1-OST004f_UUID<br/> 5198396 - 5373952 - - - - -<br/> nbp1-OST0050_UUID<br/> 5358008 - 5505024 - - - - -<br/> nbp1-OST0051_UUID<br/> 5296572 - 5373952 - - - - -<br/> nbp1-OST0052_UUID<br/> 4922184 - 5111808 - - - - -<br/> nbp1-OST0053_UUID<br/> 3887868 - 4063232 - - - - -<br/> nbp1-OST0054_UUID<br/> 4526244 - 4718592 - - - - -<br/> nbp1-OST0055_UUID<br/> 5493468 - 5636096 - - - - -<br/> nbp1-OST0056_UUID<br/> 5469616 - 5636096 - - - - -<br/> nbp1-OST0057_UUID<br/> 4571472 - 4718592 - - - - -<br/> nbp1-OST0058_UUID<br/> 5106192 - 5242880 - - - - -<br/> nbp1-OST0059_UUID<br/> 4427744 - 4587520 - - - - -<br/> nbp1-OST005a_UUID<br/> 5627236 - 5767168 - - - - -<br/> nbp1-OST005b_UUID<br/> 54825564 - 54919168 - - - - -<br/> nbp1-OST005c_UUID<br/> 4643472 - 4718592 - - - - -<br/> nbp1-OST005d_UUID<br/> 4825748 - 4980736 - - - - -<br/> nbp1-OST005e_UUID<br/> 41961916 - 42074112 - - - - -<br/> nbp1-OST005f_UUID<br/> 4466272 - 4587520 - - - - -<br/> nbp1-OST0060_UUID<br/> 4388640 - 4456448 - - - - -<br/> nbp1-OST0061_UUID<br/> 4278320 - 4456448 - - - - -<br/> nbp1-OST0062_UUID<br/> 5445700 - 5636096 - - - - -<br/> nbp1-OST0063_UUID<br/> 4250208 - 4325376 - - - - -<br/> nbp1-OST0064_UUID<br/> 4386596 - 4456448 - - - - -<br/> nbp1-OST0065_UUID<br/> 4130968 - 4325376 - - - - -<br/> nbp1-OST0066_UUID<br/> 5030856 - 5111808 - - - - -<br/> nbp1-OST0067_UUID<br/> 3996656 - 4063232 - - - - -<br/> nbp1-OST0068_UUID<br/> 5600400 - 5767168 - - - - -<br/> nbp1-OST0069_UUID<br/> 4050628 - 4194304 - - - - -<br/> nbp1-OST006a_UUID<br/> 3552020 - 3670016 - - - - -<br/> nbp1-OST006b_UUID<br/> 4318852 - 4456448 - - - - -<br/> nbp1-OST006c_UUID<br/> 5103064 - 5242880 - - - - -<br/> nbp1-OST006d_UUID<br/> 4816232 - 4980736 - - - - -<br/> nbp1-OST006e_UUID<br/> 7393596 - 7471104 - - - - -<br/> nbp1-OST006f_UUID<br/> 4763896 - 4849664 - - - - -<br/> nbp1-OST0070_UUID<br/> 5462568 - 5636096 - - - - -<br/> nbp1-OST0071_UUID<br/> 4800748 - 4980736 - - - - -<br/> nbp1-OST0072_UUID<br/> 6836304 - 6946816 - - - - -<br/> nbp1-OST0073_UUID<br/> 4109016 - 4194304 - - - - -<br/> nbp1-OST0074_UUID<br/> 6586436 - 6684672 - - - - -<br/> nbp1-OST0075_UUID<br/> 4597616 - 4718592 - - - - -<br/> nbp1-OST0076_UUID<br/> 5575980 - 5767168 - - - - -<br/> nbp1-OST0077_UUID<br/> 5166580 - 5242880 - - - - -</p> <p>Looks like some objects were not destroyed eventually somehow, did the user see anything abnormal when deleting file? and any log was attached?</p> <p>If the orhpan objects were not cleared by restart, lfsck can help you to find out and fix those orphan objects.</p> <p>this can be closed</p> <p>Thanks Mahmoud</p> Development Rank 1|hzvkdz: Rank (Obsolete) 7032 Severity