|
Niu
Could you please comment on this one?
Thanks
Peter
|
|
I suspect that there are some orphan objects (owned by cenewma1) on OSTs, which caused the usage of 'lfs quota' is larger than du.
Could you try if rebooting OSTs can fix the problem? When OST restart, it'll replay unlink & setattr log on MDS, then the orphan objects will be removed/chowned. Thanks.
|
|
Can we determine which OST has the orphan objects?
How does one get these orphan objects?
Would we need to do a lfsck?
|
|
I don't think we have any tools to determine which OST has orphan objects.
When some client unlink files, it unlink namespace on MDS first, then issue objects destroy to OSTs, if the client crash (or some other failure) between these two steps, there will be orphan objects on OSTs.
I don't think we need to do a lfsck, the orhpan objects will be cleared automatically once the OSTs restart.
|
|
Could you provide the output of "lfs quota -u cenewma1 -v /nobackupp1"? lfsck can show the orphan objects and fix them, but I think orphan will be cleared once OST restart and connect to mds, so usually it's not necessary to run lfsck.
|
|
here is the output. This filesystem has be restart. So if there were orphan object the would have been cleared.
I calculated spaced used for users by adding up objects on each ost using a modified e2scan. That value was very close to what quota was showing. So I think that lustre is not clean up objects on the osts. There was a report of a user deleting a large file and his file count and quota size should have decreased but it did not. I been try to reproduce it but have not been able to yet. I will try to compare file counts from quota vs. real number of files.
pfe1 ~ # lfs quota -u cenewma1 -v /nobackupp1
Disk quotas for user cenewma1 (uid 11540):
Filesystem kbytes quota limit grace files quota limit grace
/nobackupp1 677532804 990000000 1100000000 - 34233 75000 1000000 -
nbp1-MDT0000_UUID
18528 - 131072 - 34233 - 40960 -
nbp1-OST0000_UUID
4627588 - 4718592 - - - - -
nbp1-OST0001_UUID
5059468 - 5242880 - - - - -
nbp1-OST0002_UUID
5001992 - 5111808 - - - - -
nbp1-OST0003_UUID
5147528 - 5242880 - - - - -
nbp1-OST0004_UUID
4397416 - 4587520 - - - - -
nbp1-OST0005_UUID
5171204 - 5242880 - - - - -
nbp1-OST0006_UUID
4308496 - 4456448 - - - - -
nbp1-OST0007_UUID
4573088 - 4718592 - - - - -
nbp1-OST0008_UUID
6079784 - 6160384 - - - - -
nbp1-OST0009_UUID
5127440 - 5242880 - - - - -
nbp1-OST000a_UUID
7754820 - 7864320 - - - - -
nbp1-OST000b_UUID
4281648 - 4456448 - - - - -
nbp1-OST000c_UUID
4527680 - 4718592 - - - - -
nbp1-OST000d_UUID
4655232 - 4849664 - - - - -
nbp1-OST000e_UUID
4251904 - 4325376 - - - - -
nbp1-OST000f_UUID
4606736 - 4718592 - - - - -
nbp1-OST0010_UUID
3764148 - 3932160 - - - - -
nbp1-OST0011_UUID
4050308 - 4194304 - - - - -
nbp1-OST0012_UUID
5168732 - 5242880 - - - - -
nbp1-OST0013_UUID
5631040 - 5767168 - - - - -
nbp1-OST0014_UUID
5040964 - 5111808 - - - - -
nbp1-OST0015_UUID
3138804 - 3276800 - - - - -
nbp1-OST0016_UUID
4985140 - 5111808 - - - - -
nbp1-OST0017_UUID
5776324 - 5898240 - - - - -
nbp1-OST0018_UUID
6531844 - 6684672 - - - - -
nbp1-OST0019_UUID
4307916 - 4456448 - - - - -
nbp1-OST001a_UUID
6343920 - 6422528 - - - - -
nbp1-OST001b_UUID
6194384 - 6291456 - - - - -
nbp1-OST001c_UUID
4251588 - 4325376 - - - - -
nbp1-OST001d_UUID
4339640 - 4456448 - - - - -
nbp1-OST001e_UUID
4209232 - 4325376 - - - - -
nbp1-OST001f_UUID
5527532 - 5636096 - - - - -
nbp1-OST0020_UUID
6629476 - 6815744 - - - - -
nbp1-OST0021_UUID
3878208 - 4063232 - - - - -
nbp1-OST0022_UUID
4401920 - 4587520 - - - - -
nbp1-OST0023_UUID
4523984 - 4718592 - - - - -
nbp1-OST0024_UUID
5144444 - 5242880 - - - - -
nbp1-OST0025_UUID
4165376 - 4325376 - - - - -
nbp1-OST0026_UUID
4017568 - 4194304 - - - - -
nbp1-OST0027_UUID
3677424 - 3801088 - - - - -
nbp1-OST0028_UUID
4681332 - 4849664 - - - - -
nbp1-OST0029_UUID
6254192 - 6422528 - - - - -
nbp1-OST002a_UUID
3962612 - 4063232 - - - - -
nbp1-OST002b_UUID
5581092 - 5767168 - - - - -
nbp1-OST002c_UUID
4681436 - 4849664 - - - - -
nbp1-OST002d_UUID
3222648 - 3407872 - - - - -
nbp1-OST002e_UUID
4960592 - 5111808 - - - - -
nbp1-OST002f_UUID
6170920 - 6291456 - - - - -
nbp1-OST0030_UUID
4947888 - 5111808 - - - - -
nbp1-OST0031_UUID
4877680 - 4980736 - - - - -
nbp1-OST0032_UUID
5693548 - 5767168 - - - - -
nbp1-OST0033_UUID
4268804 - 4456448 - - - - -
nbp1-OST0034_UUID
3973800 - 4063232 - - - - -
nbp1-OST0035_UUID
5744668 - 5898240 - - - - -
nbp1-OST0036_UUID
4122328 - 4194304 - - - - -
nbp1-OST0037_UUID
4303528 - 4456448 - - - - -
nbp1-OST0038_UUID
3404112 - 3538944 - - - - -
nbp1-OST0039_UUID
8269384 - 8388608 - - - - -
nbp1-OST003a_UUID
5336032 - 5505024 - - - - -
nbp1-OST003b_UUID
5555316 - 5636096 - - - - -
nbp1-OST003c_UUID
4060448 - 4194304 - - - - -
nbp1-OST003d_UUID
6620264 - 6815744 - - - - -
nbp1-OST003e_UUID
5119320 - 5242880 - - - - -
nbp1-OST003f_UUID
6267080 - 6422528 - - - - -
nbp1-OST0040_UUID
3356136 - 3538944 - - - - -
nbp1-OST0041_UUID
5295300 - 5373952 - - - - -
nbp1-OST0042_UUID
5569504 - 5636096 - - - - -
nbp1-OST0043_UUID
5390596 - 5505024 - - - - -
nbp1-OST0044_UUID
4988152 - 5111808 - - - - -
nbp1-OST0045_UUID
3383276 - 3538944 - - - - -
nbp1-OST0046_UUID
5804668 - 5898240 - - - - -
nbp1-OST0047_UUID
4546556 - 4718592 - - - - -
nbp1-OST0048_UUID
4378612 - 4456448 - - - - -
nbp1-OST0049_UUID
5769100 - 5898240 - - - - -
nbp1-OST004a_UUID
4457072 - 4587520 - - - - -
nbp1-OST004b_UUID
4922704 - 5111808 - - - - -
nbp1-OST004c_UUID
3756600 - 3932160 - - - - -
nbp1-OST004d_UUID
5131736 - 5242880 - - - - -
nbp1-OST004e_UUID
4163448 - 4325376 - - - - -
nbp1-OST004f_UUID
5198396 - 5373952 - - - - -
nbp1-OST0050_UUID
5358008 - 5505024 - - - - -
nbp1-OST0051_UUID
5296572 - 5373952 - - - - -
nbp1-OST0052_UUID
4922184 - 5111808 - - - - -
nbp1-OST0053_UUID
3887868 - 4063232 - - - - -
nbp1-OST0054_UUID
4526244 - 4718592 - - - - -
nbp1-OST0055_UUID
5493468 - 5636096 - - - - -
nbp1-OST0056_UUID
5469616 - 5636096 - - - - -
nbp1-OST0057_UUID
4571472 - 4718592 - - - - -
nbp1-OST0058_UUID
5106192 - 5242880 - - - - -
nbp1-OST0059_UUID
4427744 - 4587520 - - - - -
nbp1-OST005a_UUID
5627236 - 5767168 - - - - -
nbp1-OST005b_UUID
54825564 - 54919168 - - - - -
nbp1-OST005c_UUID
4643472 - 4718592 - - - - -
nbp1-OST005d_UUID
4825748 - 4980736 - - - - -
nbp1-OST005e_UUID
41961916 - 42074112 - - - - -
nbp1-OST005f_UUID
4466272 - 4587520 - - - - -
nbp1-OST0060_UUID
4388640 - 4456448 - - - - -
nbp1-OST0061_UUID
4278320 - 4456448 - - - - -
nbp1-OST0062_UUID
5445700 - 5636096 - - - - -
nbp1-OST0063_UUID
4250208 - 4325376 - - - - -
nbp1-OST0064_UUID
4386596 - 4456448 - - - - -
nbp1-OST0065_UUID
4130968 - 4325376 - - - - -
nbp1-OST0066_UUID
5030856 - 5111808 - - - - -
nbp1-OST0067_UUID
3996656 - 4063232 - - - - -
nbp1-OST0068_UUID
5600400 - 5767168 - - - - -
nbp1-OST0069_UUID
4050628 - 4194304 - - - - -
nbp1-OST006a_UUID
3552020 - 3670016 - - - - -
nbp1-OST006b_UUID
4318852 - 4456448 - - - - -
nbp1-OST006c_UUID
5103064 - 5242880 - - - - -
nbp1-OST006d_UUID
4816232 - 4980736 - - - - -
nbp1-OST006e_UUID
7393596 - 7471104 - - - - -
nbp1-OST006f_UUID
4763896 - 4849664 - - - - -
nbp1-OST0070_UUID
5462568 - 5636096 - - - - -
nbp1-OST0071_UUID
4800748 - 4980736 - - - - -
nbp1-OST0072_UUID
6836304 - 6946816 - - - - -
nbp1-OST0073_UUID
4109016 - 4194304 - - - - -
nbp1-OST0074_UUID
6586436 - 6684672 - - - - -
nbp1-OST0075_UUID
4597616 - 4718592 - - - - -
nbp1-OST0076_UUID
5575980 - 5767168 - - - - -
nbp1-OST0077_UUID
5166580 - 5242880 - - - - -
|
|
Looks like some objects were not destroyed eventually somehow, did the user see anything abnormal when deleting file? and any log was attached?
If the orhpan objects were not cleared by restart, lfsck can help you to find out and fix those orphan objects.
|
|
this can be closed
|
|
Thanks Mahmoud
|
Generated at Sat Feb 10 01:20:48 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.