|
Niu
Can you please help with this issue?
Thanks
Peter
|
|
What's the e2fsprogs version? Looks like it's dup of LU-3784. Could you try the following:
1. upgrade your e2fsprogs to the latest version which have the fix of LU-3784 (tune2fs: update i_size in ext2fs_file_write()) (you can download it from build.whamcloud.com)
2. disable then enable each MDT/OST device by "tune2fs -O ^quota" and "tune2fs -O quota"
|
|
The e2fsprogs RPMs installed are:
-e2fsprogs-1.42.7.wc2-7.el6.x86_64.rpm
-e2fsprogs-libs-1.42.7.wc2-7.el6.x86_64.rpm.
The procedure outlined in step #2 has been performed - we did disable/enable quota for all the mdt and ost devices with:
tune2fs -O ^quota $dev
tune2fs -O quota $dev
Same result. Any ideas on what to try next, or any debugging that can be done? Thanks.
|
|
Is this the only user who has incorrect quota usage or all other users' usage are incorrect as well? Could you try to write as the user to see if the newly written bytes can be accounted?
And I want to get confirmed that the e2fsprogs installed is the latest build from the build.whamcloud.com? I'm not sure if the older 1.42.7-wc2 include the patch.
|
|
The problem affects at least two users on one file system, and another two users on another separate lustre file system.
Today, the customer discovered the issue on another lustre fs with other users:
root@ic2n992:/pfs/data2/home# lfs quota -u ho_anfuchs .
Disk quotas for user ho_anfuchs (uid 900085):
Filesystem kbytes quota limit grace files quota limit
grace
. 1982128 0 0 - 20350 0 0
-
root@ic2n992:/pfs/data2/home# du -hs ho/ho_kim/ho_anfuchs
5.8G ho/ho_kim/ho_anfuchs
root@ic2n992:/pfs/data2/home# find ho/ho_kim/ho_anfuchs ! ( -user ho_anfuchs ) -exec ls -l {} \;
root@ic2n992:/pfs/data2/home# lfs quota -u kn_pop164377 .
Disk quotas for user kn_pop164377 (uid 900025):
Filesystem kbytes quota limit grace files quota limit
grace
. 15359272 0 0 - 167897 0
0 -
root@ic2n992:/pfs/data2/home# du -hs kn/kn_kn/kn_pop164377
71G kn/kn_kn/kn_pop164377
root@ic2n992:/pfs/data2/home# find kn/kn_kn/kn_pop164377 ! ( -user
kn_pop164377 ) -exec ls -l {} \;
The efsprogs was downloaded from whamcloud, with the following patch applied: http://git.whamcloud.com/?p=tools/e2fsprogs.git;a=commit;h=470ca046b1
I will have the customer do a write operation to see if the accounting changes.
|
|
Newly written bytes are accounted. This works for all OSTs.
|
|
I wonder whether some of the OST objects belonging to those users did not get the proper UID/GID. Could you please check on one of the OST reporting the wrong usage if all the objects have the correct UID/GID? You can do it by unmounting the OST, mounting it with -t ldiskfs and run a find command to compute usage of the user and compare it with what is reported by lfs quota. Thanks in advance.
|
|
Thanks for the response. Unfortunately, the FS is in production and unmounting anything can not be easily done. Are there any other options for gathering the information you're asking for? Please advise.
Thanks,
Oz
|
|
Did the customer ever see the problem before upgrading? We'd make sure first that these users don't have the same problem before upgranding.
|
|
Response from customer:
The file systems were going into production in September 2013 with Lustre 2.1.x. Actually, we never checked the quotas carefully before the upgrade to 2.4 (done in December 2013), i.e. it is possible that the problem existed before the upgrade. On the other hand, we also have one user which has this problem and which created all files in January (and he got access to the cluster after the 2.4 upgrade).
In addition, I also found a response to the question of Johann Lombardi from 31/Jan/14 9:41 PM without unmounting the OST by following chapter
13.14 of the Lustre manual. The file I checked had the correct UID and GID. For more details see attached file (pfs2wor2_check_quotas_bad_user_20140204.txt).
|
In addition, I also found a response to the question of Johann Lombardi from 31/Jan/14 9:41 PM without unmounting the OST by following chapter
13.14 of the Lustre manual. The file I checked had the correct UID and GID. For more details see attached file (pfs2wor2_check_quotas_bad_user_20140204.txt).
Looks you only checked one file, it's better to run a script to check all files belong to user "es_asaramet".
|
|
From customer:
pfs2wor2_check_quotas_bad_user_20140204.txt the command "lfs quota -v -u es_asaramet" shows 0 quota usage for pfs2wor2-OST0007. We have found a non empty file (refinementSurfaces.o) which is located on OST0007. Hence the 0 quota usage of lfs quota -v for this user is wrong. And since the file really belongs to the user with UID 900044 (es_asaramet) in the underlying ldiskfs, the assumption of Johann Lombardi that the UID/GID in the underlying ldiskfs might be wrong cannot be the reason for this 0 quota usage on pfs2wor2-OST0007.
More information from another ticket the customer just today opened regarding quota:
last December DDN upgraded several Lustre file systems to Lustre 2.4.
However, I now recognized that quota enforcement does not work.
For example we have:
root@ic2n993:~# lfs quota -g imk-tro /pfs/imk
Disk quotas for group imk-tro (gid 44470):
Filesystem kbytes quota limit grace files quota limit grace
/pfs/imk 3067173276* 0 1 - 11115 0 0 -
Note that the quota block limit was set to 1 before files were created.
I was just reading the Lustre manual and checked the following command:
[root@pfs2n6 ~]# lctl get_param osd-..quota_slave.info
osd-ldiskfs.pfs2wor1-MDT0000.quota_slave.info=
target name: pfs2wor1-MDT0000
pool ID: 0
type: md
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
Since "quota enabled" is set to none I assume that the following command
was not executed:
- lctl conf_param pfs2wor1.quota.ost=ug
Do you believe this assumption is correct?
BTW: I tried to check the MGS parameters according to a description of
Kit Westneat and did the following:
[root@pfs2n2 ~]# debugfs -R 'rdump CONFIGS /tmp/' /dev/mapper/vg_pfs2dat1-mgs
[root@pfs2n2 ~]# for x in /tmp/CONFIGS/*; do echo $x:; llog_reader $x | grep param ; echo; done
However, the last command hangs. The command
[root@pfs2n2 ~]# llog_reader /tmp/CONFIGS/lustre-params
permanently repeats the following line:
Bit 0 of 135136 not set
|
pfs2wor2_check_quotas_bad_user_20140204.txt the command "lfs quota -v -u es_asaramet" shows 0 quota usage for pfs2wor2-OST0007. We have found a non empty file (refinementSurfaces.o) which is located on OST0007. Hence the 0 quota usage of lfs quota -v for this user is wrong. And since the file really belongs to the user with UID 900044 (es_asaramet) in the underlying ldiskfs, the assumption of Johann Lombardi that the UID/GID in the underlying ldiskfs might be wrong cannot be the reason for this 0 quota usage on pfs2wor2-OST0007.
I see, thanks. Could you check the "/proc/fs/lustre/osd-ldiskfs/pfs2wor2-OST0007/quota_slave/acct_user" and post the result here?
I was just reading the Lustre manual and checked the following command:
[root@pfs2n6 ~]# lctl get_param osd-..quota_slave.info
osd-ldiskfs.pfs2wor1-MDT0000.quota_slave.info=
target name: pfs2wor1-MDT0000
pool ID: 0
type: md
quota enabled: none
conn to master: setup
space acct: ug
user uptodate: glb[0],slv[0],reint[0]
group uptodate: glb[0],slv[0],reint[0]
Since "quota enabled" is set to none I assume that the following command
was not executed:
lctl conf_param pfs2wor1.quota.ost=ug
Do you believe this assumption is correct?
The info you checked is for metadata, you'd check the quota_slave.info on OST to see if data quota is enabled, and enabled it by 'lctl conf_param $FSNAME.quota.ost=ug".
|
|
The customer checked it on the MDS because the example in chapter 21.2 of the Lustre Operations Manual also displays this information for the MDT.
Is this example of the manual wrong?
The OSTs also showed quota enabled: none
The customer checked and verified they have the same problem on their test system.
After "lctl conf_param pfscdat1.quota.ost=ug" on the MGS this indeed fixed the problem.
It would also be good to get a response why the llog_reader hangs permanently for some file systems while we try to check MGS parameters?
|
The customer checked it on the MDS because the example in chapter 21.2 of the Lustre Operations Manual also displays this information for the MDT.
The command should be run on MDS or OSS to check the quota status of each MDT & OST. The example in manual displayed only the output of MDS.
It would also be good to get a response why the llog_reader hangs permanently for some file systems while we try to check MGS parameters?
It looks like LU-632, is the file empty?
|
|
> The command should be run on MDS or OSS to check the quota status of each MDT & OST. The example in manual displayed only the output of MDS.
Thank you for this clarification!
> It looks like LU-632, is the file empty?
Yes, the file is indeed empty. The displayed error message was misleading.
|
|
> Niu Yawei added a comment - 13/Feb/14 8:45 AM > I see, thanks. Could you check the "/proc/fs/lustre/osd-ldiskfs/pfs2wor2-OST0007/quota_slave/acct_user" and post the result here?
A file with the requested information is attached, pfs2wor2-OST0007_acct_user_20140213.txt.
It is a bit strange that pretty few user IDs appear and that user ID
900044 is missing.
I also checked the used capacity and compared with the lfs df output and there is a difference of more than 100 GB.
|
|
Any updates?
Please let us know if you need any additional information or if there is any additional debugging we could be doing. We would like to close this one out fairly soon.
Thanks!
|
|
I cooked a debug patch for e2fsprogs: http://review.whamcloud.com/#/c/9397 and it's built on http://build.whamcloud.com/job/e2fsprogs-reviews/200/ .
Oz, could you install the e2fsprogs from http://build.whamcloud.com/job/e2fsprogs-reviews/200/ and collect the debug information while disabling/enabling quota for the OST0007 device? (by tune2fs -O ^quota & tune2fs -O quota command). Thanks a lot.
|
|
The debug e2fsprogs has been installed on the OSS, and quota has been disabled / enabled on the affected OST. The output from running the 'tune2fs O quota' can be taken from http://ddntsr.com/ftp/2014-03-12-SR28763_tunefs-O-quota.out.tgz and the new 'lctl get_param osd..quota_slave.info' output is attached.
|
|
Hi, Oz
[DEBUG] quotaio_tree.c:316:qtree_write_dquot:: writing ddquot 1: id=900044 off=0, info->dqi_entry_size=72^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=1, depth=0^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=2, depth=1^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=34, depth=2^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=36, depth=3^M
[DEBUG] quotaio_tree.c:330:qtree_write_dquot:: writing ddquot 2: id=900044 off=34168, info->dqi_entry_size=72^M
Looks the accounting information for 900044 is written back, could you verify that if the space accounting for 900044 on this OST is fixed? Thank you.
|
|
It appears the space accounting for 900044 on OST0007 is fixed. For details see attached file pfs2wor2-OST0007_acct_user_20140313.txt.
The customer performed further investigations on all 4 of the affected file systems. The details can be seen in the attached file pfs2wor2_check_quotas_bad_user_20140313.txt.
Results:
1. On the pfs2wor2 file system it seems that the quota problems of both affected users were fixed by only re-enabling quotas for OST0007.
2. For all other file systems (pfs2dat1, pfs2dat2, pfs2wor1) we still have users with quota issues. Only for the pfs2dat1 file system we found an OST which might be the reason. But please note that we would rather wait for this file system for the next maintenance with disruptive actions since it contains home directories.
Questions:
1. What might have caused these issues?
2. How can this problem be resolved and prevented from reoccurring?
NOTE: Quota was reset (disabled / enabled) on all the OSTs back in December.
|
It appears the space accounting for 900044 on OST0007 is fixed. For details see attached file pfs2wor2-OST0007_acct_user_20140313.txt.
Good news, thank you.
Questions:
1. What might have caused these issues?
2. How can this problem be resolved and prevented from reoccurring?
It could probably because the e2fsprogs on OST0007 was not uptodate, could you verify that if the e2fsprogs on OST0007(or other problematic OSTs) was same as others'?
|
|
1. The software is usually installed by pdsh, i.e. it is the same on all servers.
2. This does not explain why some OSTs on the same OSS showed no problems with their quotas.
3. Also, OST0007 did not have a quota problem for most users.
Since we see the same problem on all 4 file systems this is a general problem, i.e. not something which happened once by chance.
I just had a look at the upgrade documentation which was sent by the vendor field engineer. He wrote (translated): tunefs is problematic and does not always work. Maybe Sven can comment what exactly was meant here and if this could be the reason.
Anyway I wonder how we could clearly repair the problem for the remaining file systems during the next maintenance. I see 2 problems:
1. The vendor had written that he had done the following for one file system and this had not fixed the quota problem:
turn off quota first, turn it back on and run an e2fsck
How can we be sure to have a procedure which clearly fixes the problem?
2. The problem might move to different users after disabling and re-enabling quotas. How can we easily and quickly find out if the problem still appears?
|
|
Another interesting thing to note is both user quotas and group quotas are used, but there was not a problem with group quotas.
|
1. The software is usually installed by pdsh, i.e. it is the same on all servers.
2. This does not explain why some OSTs on the same OSS showed no problems with their quotas.
3. Also, OST0007 did not have a quota problem for most users.
Comparing the two versions of accounting information of OST0007 (before and after executing tune2fs), we can see lots of user accounting was fixed, so I think many users were having accounting problems, but not discovered. Maybe it's same to other OSTs on the same OSS?
Another possibility is that customer just missed tune2fs on OST0007?
1. The vendor had written that he had done the following for one file system and this had not fixed the quota problem:
turn off quota first, turn it back on and run an e2fsck
How can we be sure to have a procedure which clearly fixes the problem?
I think first we'd make sure we are using the correct e2fsprogs. To verify if the accounting information is fixed, you can check the "acct_user/group" in proc file.
2. The problem might move to different users after disabling and re-enabling quotas. How can we easily and quickly find out if the problem still appears?
Disable/re-enable quota is just for triggering a quotacheck, you can verify the accounting information in proc file.
Another interesting thing to note is both user quotas and group quotas are used, but there was not a problem with group quotas.
I think it probably because it was just not detected.
|
|
The customer doesn't believe that tune2fs was missed on some OSTs. Either this is a general Lustre problem or it is a problem with the vendors tunefs wrapper script.
Concerning this wrapper script, the field engineer sent the following:
> I just had a look at the upgrade documentation which was sent by the vendor: tunefs is problematic > and does not always work.
I have not completely isolated the problem yet. I think it is the EXAScaler tunefs wrapper script es_tunefs. It does a second tunefs to set the MMP timeout, which may be causing the problem.
If tunefs was not done correctly the OST do not register with the MGS correctly and the clients can not access some of the OSTs. This can be easily verified by running "lfs df" on a client. I have noticed that this behaviour is worse with Lustre 2.4.x but i have seen it with older Lustre versions as well.
Since I had noticed a strange difference between user and group quotas I wrote a perl script which checks the sum of "acct_user/group"
in proc. The perl script and a text file with the output on all file systems is attached.
Here are the results:
1. Some OSTs are affected and others are not affected, i.e.
this is an easy way to find out which OSTs are affected.
2. On the affected OSTs both inodes and kbytes are wrong.
3. We have higher group values and higher user values, i.e.
both user and group quotas are affected.
4. On the different file systems in nearly all cases either group values or user values are higher. The reason for this behaviour is not clear.
Do you have any comments or ideas about the possible reason for the problem?
|
Do you have any comments or ideas about the possible reason for the problem?
This sounds same problem as OST0007, and OST0007 can be fixed by re-run "tune2fs -O quota" (with uptodate e2fsprogs), can these problematic OSTs be fixed in same way or not?
|
|
The customer ran through the "tune2fs -O quota" procedure last week during their scheduled downtime. However, did this not resolve the problem.
.
For OST pfs2dat2-OST0000 the customer also used the patched e2fsprogs and collected all output.
The log file with the additional details can be downloaded from "http://ddntsr.com/ftp/2014-04-28-SR28763_tunefs_20140424.txt.gz" (69MB)
|
|
Oz, which uid/gid has problem on pfs2dat2-OST0000?
|
|
we do not know which uid/gid has wrong quotas on pfs2dat2-OST0000.
We used our perl script which sums up all user and group quotas of acct_user/group in proc. This should show the same results for users and groups but it does not for pfs2dat2-OST0000.
In detail, before the maintenance and after clients were unmounted the script reported this for pfs2dat2-OST0000:
Sum of inodes of users: 9353416
Sum of inodes of groups: 9447415
Sum of kbytes of users: 11926483836
Sum of kbytes of groups: 12132828844
After servers were upgraded to Lustre 2.4.3 and quotas were re-enabled (with normal e2fsprogs):
Sum of inodes of users: 9325574
Sum of inodes of groups: 9446294
Sum of kbytes of users: 11897886304
Sum of kbytes of groups: 12132673600
Note the changes although clients were not mounted in the meantime.
After just re-enabling quotas again for pfs2dat2-OST0000 (with normal e2fsprogs):
Sum of inodes of users: 9325357
Sum of inodes of groups: 9446077
Sum of kbytes of users: 11897857144
Sum of kbytes of groups: 12132644440
Note that tune2fs -O quota reported messages like these:
[ERROR] quotaio_tree.c:277:do_insert_tree:: Inserting already present quota entry (block 5).
[ERROR] quotaio_tree.c:277:do_insert_tree:: Inserting already present quota entry (block 35).
After re-enabling quotas again for pfs2dat2-OST0000 (with patched e2fsprogs):
Sum of inodes of users: 9325357
Sum of inodes of groups: 9446077
Sum of kbytes of users: 11897857144
Sum of kbytes of groups: 12132644440
It is also interesting that only one OST of the pfs2dat2 has the same value for users and groups. For the pfs2wor2 file system most OSTs show the same values. pfs2dat2 has 219 million files and stripe count 1,
pfs2wor2 has 69 million files and default stripe count 2.
Is further investigation possible with this information and with the provided tune2fs logs?
If not, the customer will develop another script to find out uids/gids with wrong quotas on pfs2dat2-OST0000. Since this makes some effort I just wanted to check if this is really needed/helpful.
|
Note the changes although clients were not mounted in the meantime.
Orphan cleanup may removed some files.
Note that tune2fs -O quota reported messages like these:
[ERROR] quotaio_tree.c:277:do_insert_tree:: Inserting already present quota entry (block 5).
[ERROR] quotaio_tree.c:277:do_insert_tree:: Inserting already present quota entry (block 35).
I noticed the UID/GID on this system is very huge, some UIDs are larger than 2G. I think there could be some defect in the e2fsprogs which handle large ID incorrectly. For example:
[DEBUG] quotaio.c:326:quota_file_create:: Creating quota ino=3, type=0^M
[DEBUG] quotaio_tree.c:316:qtree_write_dquot:: writing ddquot 1: id=2171114240 off=0, info->dqi_entry_size=72^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=1, depth=0^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=0, depth=1^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=0, depth=2^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=0, depth=3^M
e2fsprogs is writing UID 2171114240 into quota file, and later on...
[DEBUG] quotaio_tree.c:316:qtree_write_dquot:: writing ddquot 1: id=2171114240 off=0, info->dqi_entry_size=72^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=1, depth=0^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=2, depth=1^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=3, depth=2^M
[DEBUG] quotaio_tree.c:253:do_insert_tree:: inserting in tree: treeblk=4, depth=3^M
[ERROR] quotaio_tree.c:277:do_insert_tree:: Inserting already present quota entry (block 5).^M
[DEBUG] quotaio_tree.c:330:qtree_write_dquot:: writing ddquot 2: id=2171114240 off=11543712, info->dqi_entry_size=72^M
e2fsprogs tries to write some UID 2171114240 into quota file again. Looks the UID 2171114240 got duplicated in the memory dict.
I'll investigate further to see what happened when inserting large id into memory dict.
Is further investigation possible with this information and with the provided tune2fs logs?
Yes, no need to develop new script for now. I just want get confirmed from customer that they really have such large UID/GIDs.
|
|
Thanks Niu. Here is the response from the customer:
We have pretty huge UIDs/GIDs. However, they are by far not as huge as reported. The largest UID is 901987 and the largest GID is 890006.
|
|
The huge UID/GIDs may caused by a lustre defect described in LU-4345.
And looks there is a defect in e2fsprogs which could mess dict lookup when the difference of two keys greater than 2G.
static int dict_uint_cmp(const void *a, const void *b)
{
unsigned int c, d;
c = VOIDPTR_TO_UINT(a);
d = VOIDPTR_TO_UINT(b);
return c - d;
}
This function returns an unsigned int value in int type, and quota relies on this function to insert ids into dict on quotacheck. I think that's why we see dup ID on quotacheck. I'll cooke a patch to fix this soon.
|
|
http://review.whamcloud.com/10227
|
|
From the customer:
It's good news that you found possible reasons for the problem.
We will install the patches during our next maintenance which is expected to take place during the next 2 months. However, DDN will have to provide a Lustre version which includes those patches.
For the huge UID/GIDs caused by the lustre defect described in
LU-4345: Is there a way to repair the bad IDs on the OST objects?
|
For the huge UID/GIDs caused by the lustre defect described in
LU-4345: Is there a way to repair the bad IDs on the OST objects?
Fix bad IDs on existing OST objects:
- Find the objects with bad IDs first (mount the OST device with ldiskfs to check IDs of each file or use debugfs without umount)
- Get the correct ID from MDT (see Lustre manual 13.14 to identify which file the object belongs to) the set the correct IDs to these OST objects.
- Set correct IDs for the objects on OST directly.
|
|
Hello Oz,
We've recently heard from another site that Niu's fixes have resolved the quota problems they were seeing.
Has DDN installed a new version at this site, with those patches?
If so, do you have any new news on this?
Thanks,
~ jfc.
|
|
I updated the way of how to fix bad ID for OST object (see my previous comment). Thanks.
|
|
Hello Niu,
We dont think manually fixing the OST object is not a good idea. Since the filesystem have more than 100 million files.
Cu is expecting a better way to fix this issue,
Thanks,
Rajesh
|
We dont think manually fixing the OST object is not a good idea. Since the filesystem have more than 100 million files.
Cu is expecting a better way to fix this issue,
I can't think of any other good ways to fix the bad IDs. I think running a script instead of repeating the commands manually would be better?
|
|
rganesan@ddn.com, orentas, have you resolved this issue? do you need anything else from this ticket?
|
|
Yes, a long time ago. Please close. Thanks!
|
|
Thank you Sir!
|
Generated at Sat Feb 10 01:43:18 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
check_quotas_bad_user_20140123.txt