|
We are currently in a maintainance windown, and trying to fix/walk around this problem instantly. Otherwise, disabling quota feature would be the last choice.
We tried rebooting the OSTs and MDTs, runing 'tune2fs -O ^quota/quota', and doing 'lctl conf_param $FSNAME.quota.ost=none; lctl conf_param $FSNAME.quota.ost=ug;'. None of those attempts succeeded.
I am wondering whether the files under quota_slave directories are broken. Is it safe to remove all of those directories offline and restart the Lustre then? I tried that in an test environment, and nothing bad happened. But we'd like to make sure it won't cause any problem of losing data or breaking down the system. Is there any other good idea about how to fix/walk around this problem?
Thanks in advance!
|
|
Niu
Can you please advise?
Thanks
Peter
|
|
Could you please dump the limits on all the OSTs by running the following command?
And also dumps limits on the qmt so that we can compare it.
You could also try to force reintegration by running on each OSS:
As for removing the slave copies of the indexes on the OST, it was designed to work, but it does not seem to be tested in sanity-quota. Niu, could you please run additional tests to make sure it works well?
|
|
Hi Johann,
Thanks for helping. I've attached the output of 'lctl get_param osd*..quota_slave.limit'
However, nothing changes after running 'lctl set_param osd*.*.quota_slave.force_reint=1' on the first OSS. The limits are still strange.
[root@nmds06 ~]# lfs quota -u toyasuda -v /root/lustre/
Disk quotas for user toyasuda (uid 652):
Filesystem kbytes quota limit grace files quota limit grace
/root/lustre/ 46120740 0 0 - 5076 0 0 -
lustre5-MDT0000_UUID
2692 - 0 - 5076 - 0 -
lustre5-OST0000_UUID
5164 - 4194304 - - - - -
lustre5-OST0001_UUID
118936* - 4 - - - - -
lustre5-OST0002_UUID
1093400 - 3670132 - - - - -
lustre5-OST0003_UUID
26120* - 4 - - - - -
lustre5-OST0004_UUID
3204* - 4 - - - - -
lustre5-OST0005_UUID
11041916* - 4194304 - - - - -
...
|
|
Strange thing to me is why uid 3018 quota info is not in qmt_dt-0x0_glb-usr.log?
|
|
Hi Zhenyu,
The reason is because these output was captured from different file system. There are 3 file systems and all are showing the same problem.
lect02 (uid 3018) is for lustre4 file system and qmt_dt-0x0_glb-usr.log was captured from lustre5 file system. toyasuda (uid 652) is for lustre5.
|
|
While uid 652 quota info does no appear in the qmt_dt-0x0_glb-usr.log also.
|
|
I checked qmt_dt-0x0_glb-usr.log and the current quota_slave.limit on QMT for lustre5. It looks like qmt_dt-0x0_glb-usr.log is not from lustre5, sorry.
I captured quota_slave.limit again from QMT and quota_slave for file system lustre5 (lustre5_quota_slave.limit.tar.gz).
uid 652 does not appear in either log. uid 652 does not have any quota limitation and this might be a reason why it does not appear, I think.
|
|
quota_slave.limit from QMT, quota_slave for lustre5 file system
|
|
Sorry, QMT log
|
|
Sorry, many times, this one is the latest QMT log for lustre5 (captured at the same time as lustre5_quota_slave.limit.tar.gz). I uploaded different file mistakenly...
|
|
Hello, can we expect an action plan (or if we can safely remove slave copies) within a few hours? The system is currently in the maintenance window and we can do offline work, but it will over at the end of today. The customer need to determine if they continue to wait for our response or they put the system back to service without quota. This is difficult situation for them to decide. Please share us with the latest information and expectation to go. Regards,
|
|
There is some id 's granted value exceeds its hardlimit
|
|
I've tried to backup quota_slave and remove it, it does work under master branch, but not for b2_4 code.
The OI know there exists slave index file in its oi mapping file but cannot find it.
LiXi said that he can remove the quota_slave and mount successfully, I don't know how.
|
|
Hi Zhengyu,
I removed the whole directory rather than renaming it. I guess that is the difference? I got similar problem when renaming the file, so I removed it by 'rm -rf quota_slave'. BTW, I did the test on master branch of Lustre rather than 2.4.x.
|
|
I think 2.4.x does not work as master, OI code on master is more mature.
|
|
Would you mind setting the hard limit to 1MB for a UID, say 3348, then check whether its granted value falls, and set the limit back to check that space got released? If you use soft limit as well, decrease and restore it as well.
|
|
First, I disabled quota on the systems since the behavior is not stable and the customer need to bring the service back. However quota feature is required for them and I need to resolve this issue ASAP.
I tried to change the hard limit and found the following (I used admin users for the file systems).
For file system lustre4, the situation was improved in the case of UID 657, but not for UID 3018.
UID 657 did not have quota limitation, but "limit" was shown in 'lfs quota'. When I set '-B 1000000000', 'limit' was set to the same value as 'kbytes' on all the OSTs. Then when I set it back to '-B 0', 'limit' was set to '0' (lustre4_lfs.quota_UID654.657_20140626.log). However for UID 3018, 'limit' did not change even I did setquota (lustre4_lfs.quota_UID3018_20140626.log).
On file system lustre5, I did the same thing for UID 652 and mostly observed same behavior as the case of UID 657 above, but 'limit' value did not change on some of OSTs (lustre5_lfs.quota_UID652_20140626.log). Those OSTs are lustre5-OST0000-0004 and mounted on one of OSS.
I also captured debug log (with +trace, +quota) when I did setquota for UID 3018 on file system lustre4 and for UID652 on file system lustre5.
lustre4_quota.debug.UID3018_20140626.tar.gz
lustre5_quota.debug.UID652_20140626.tar.gz
|
|
Hi Zhengyu,
You are right. When I removed the quota_slave directory and tried to mount the OSD again on Lustre-2.4.2. Following LBUG happened.
LDISKFS-fs (sdb3): mounted filesystem with ordered data mode. quota=on. Opts:
LustreError: 0-0: vm14-OST0000: trigger OI scrub by RPC for [0x200000005:0x1:0x0], rc = 0 [1]
LustreError: 5147:0:(qsd_lib.c:424:qsd_qtype_init()) vm14-OST0000: can't open slave index copy [0x200000006:0x20000:0x0] -115
LustreError: 5147:0:(obd_mount_server.c:1716:server_fill_super()) Unable to start targets: -115
Lustre: Failing over vm14-OST0000
LustreError: 5172:0:(osd_internal.h:752:osd_fid2oi()) ASSERTION( !fid_is_idif(fid) ) failed: [0x100000000:0x1:0x0]
LustreError: 5172:0:(osd_internal.h:752:osd_fid2oi()) LBUG
Pid: 5172, comm: OI_scrub
Call Trace:
[<ffffffffa0358895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
[<ffffffffa0358e97>] lbug_with_loc+0x47/0xb0 [libcfs]
[<ffffffffa0b660f5>] __osd_oi_lookup+0x3a5/0x3b0 [osd_ldiskfs]
[<ffffffff8119d13d>] ? generic_drop_inode+0x1d/0x80
[<ffffffffa0b66174>] osd_oi_lookup+0x74/0x140 [osd_ldiskfs]
[<ffffffffa0b7afbf>] osd_scrub_exec+0x1af/0xf30 [osd_ldiskfs]
[<ffffffffa0b7c5f2>] ? osd_scrub_next+0x142/0x4b0 [osd_ldiskfs]
[<ffffffffa02e331c>] ? ldiskfs_read_inode_bitmap+0x5c/0x2c0 [ldiskfs]
[<ffffffffa0b76d4f>] osd_inode_iteration+0x1cf/0x570 [osd_ldiskfs]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0b7ae10>] ? osd_scrub_exec+0x0/0xf30 [osd_ldiskfs]
[<ffffffffa0b7c4b0>] ? osd_scrub_next+0x0/0x4b0 [osd_ldiskfs]
[<ffffffffa0b7932a>] osd_scrub_main+0x59a/0xd00 [osd_ldiskfs]
[<ffffffff81071a6f>] ? release_task+0x33f/0x4b0
[<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
[<ffffffffa0b78d90>] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs]
[<ffffffff8100c0ca>] child_rip+0xa/0x20
[<ffffffffa0b78d90>] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs]
[<ffffffffa0b78d90>] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Kernel panic - not syncing: LBUG
Pid: 5172, comm: OI_scrub Not tainted 2.6.32 #2
Call Trace:
[<ffffffff8150bb0c>] ? panic+0xa7/0x16f
[<ffffffffa0358eeb>] ? lbug_with_loc+0x9b/0xb0 [libcfs]
[<ffffffffa0b660f5>] ? __osd_oi_lookup+0x3a5/0x3b0 [osd_ldiskfs]
[<ffffffff8119d13d>] ? generic_drop_inode+0x1d/0x80
[<ffffffffa0b66174>] ? osd_oi_lookup+0x74/0x140 [osd_ldiskfs]
[<ffffffffa0b7afbf>] ? osd_scrub_exec+0x1af/0xf30 [osd_ldiskfs]
[<ffffffffa0b7c5f2>] ? osd_scrub_next+0x142/0x4b0 [osd_ldiskfs]
[<ffffffffa02e331c>] ? ldiskfs_read_inode_bitmap+0x5c/0x2c0 [ldiskfs]
[<ffffffffa0b76d4f>] ? osd_inode_iteration+0x1cf/0x570 [osd_ldiskfs]
[<ffffffff81051439>] ? __wake_up_common+0x59/0x90
[<ffffffffa0b7ae10>] ? osd_scrub_exec+0x0/0xf30 [osd_ldiskfs]
[<ffffffffa0b7c4b0>] ? osd_scrub_next+0x0/0x4b0 [osd_ldiskfs]
[<ffffffffa0b7932a>] ? osd_scrub_main+0x59a/0xd00 [osd_ldiskfs]
[<ffffffff81071a6f>] ? release_task+0x33f/0x4b0
[<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
[<ffffffffa0b78d90>] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs]
[<ffffffff8100c0ca>] ? child_rip+0xa/0x20
[<ffffffffa0b78d90>] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs]
[<ffffffffa0b78d90>] ? osd_scrub_main+0x0/0xd00 [osd_ldiskfs]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20
|
For file system lustre4, the situation was improved in the case of UID 657, but not for UID 3018.
UID 657 did not have quota limitation, but "limit" was shown in 'lfs quota'. When I set '-B 1000000000', 'limit' was set to the same value as 'kbytes' on all the OSTs. Then when I set it back to '-B 0', 'limit' was set to '0' (lustre4_lfs.quota_UID654.657_20140626.log). However for UID 3018, 'limit' did not change even I did setquota (lustre4_lfs.quota_UID3018_20140626.log).
When setting the limit to 0, the slaves should release all quota space unconditionally (see qsd_calc_adjust()). Could you please enable quota debug on one of the OSS that did not release space, set quota -B 0, wait for a couple of seconds and then collect logs on this OSS?
|
|
I did the same thing (setquota -B 0) and captured OSS quota debug log.
|
|
Other logs captured at the same time.
|
|
I looked at the debug logs for lustre5 and unfortunately, i have traces of neither the setquota request in the MDS logs nor the glimpse in the OSS logs  Could you please collect those logs again and use "lctl mark" to mark the beginning and end of the set quota operation?
Also, could you please make sure that all OST have successfully connected to the master and have been resynchronized? This can be done by running the following command on all the OSSs:
Thanks in advance.
|
|
I checked again quota_slave.info, but connection to master was setup on all the OSTs (lustre5_quota_slave.info.tar.gz).
I also captured the log again with the markers '########## START SETQUOTA ##########' and '########## END SETQUOTA ##########'.
I did the following in the interval.
|
|
The lustre5_quota_slave.info shows that all OSTs do not have quota enabled.
nmds05_quota_slave.info:5:quota enabled: none
noss37_quota_slave.info:5:quota enabled: none
noss37_quota_slave.info:14:quota enabled: none
noss37_quota_slave.info:23:quota enabled: none
noss37_quota_slave.info:32:quota enabled: none
noss37_quota_slave.info:41:quota enabled: none
noss38_quota_slave.info:5:quota enabled: none
noss38_quota_slave.info:14:quota enabled: none
....
You need enable quota for the test and collect the log again.
|
|
Yes, as I stated before, we disabled quota for interim remedy. We have tried changing quota setting, force_reint while quota was being enabled, but it did not work. Since we could not determine if quota exceeded on some OSTs are really effective or false positive, we ended up to disable quota and put the system back in service.
Can you please check if the quota exceeded mark "*" shown in some OSTs are really effective, or false and users can write to that OST exceeding the limit. If it is safe, we can enable quota again and capture the log.
When I changed quota setting by 'lfs setquota' while quota is being disabled, OSTs other than OST0000-OST0004 changed 'limit' value in 'lfs quota'. What is the difference between these OSTs and others where 'limit' did not changed?
Can I also ask what is the designed (expected behavior) here at the first place and what is not. When we change quota setting while quota is disabled, what value 'limit' reported by 'lfs quota' should be set to? Does the behavior change when we enable quota? How can we confirm if quota has up-to-dated correct information?
|
Yes, as I stated before, we disabled quota for interim remedy. We have tried changing quota setting, force_reint while quota was being enabled, but it did not work. Since we could not determine if quota exceeded on some OSTs are really effective or false positive, we ended up to disable quota and put the system back in service.
Understood. That said, the procedure we gave you (i.e set quota limit to 0 to force all OSTs to release space) only works if quota has been enabled. Would it be possible to enable quota for a short amount of time and rerun those commands? Once completed, you can then disable quota again if it does not work.
Can you please check if the quota exceeded mark "*" shown in some OSTs are really effective, or false and users can write to that OST exceeding the limit.
It is effective.
If it is safe, we can enable quota again and capture the log.
The thing is that setting the hard limit to 0 is expected to fix the problem (provided that quotas are on), so no log capture would be required in this case.
When I changed quota setting by 'lfs setquota' while quota is being disabled, OSTs other than OST0000-OST0004 changed 'limit' value in 'lfs quota'.
right.
What is the difference between these OSTs and others where 'limit' did not changed?
The OSTs that changed limits still hold a global quota lock, while OST0000-OST0004 do not. This quota lock isn't automatically dropped once quota is disabled and isn't enqueued if quota is disabled at mount time.
Can I also ask what is the designed (expected behavior) here at the first place and what is not. When we change quota setting while quota is disabled, what value 'limit' reported by 'lfs quota' should be set to?
The quota master should report the right limit. As for slaves, it depends on whether they still hold a global quota lock. To sum up, when quota is disabled, you should not pay attention to the hard limit on OSTs.
Does the behavior change when we enable quota?
Yes. When quota is enabled, all quota slaves have to acquire a global quota lock.
How can we confirm if quota has up-to-dated correct information?
In quota_slave.info, the number between [] is 1 when the slave is synchronized with the master and 0 otherwise.
|
|
Any update?
|
|
I talked with the customer, but unfortunately just enabling quota while the system is in service is not acceptable for the customer. To do the test safely, setting '-B 0' on all the user and then enable quota would be one option for them. Assuming we do the test, what is expected result?
Currently,
1. Sum of usage on each OST does not match the whole (reported in 3rd line of 'lfs quota')
2. 'limit' in 'lfs quota' is set to (compared to the usage);
- much greater value (e.g. usage is 0, limit is 38205292)
- a bit greater value (e.g. usage is 337530440. limit is 404591056)
- same value, zero (e.g. usage is 0, limit is also 0)
- same value, but non-zero (e.g. usage is 4, limit is also 4), quota exceeded is set.
- much smaller value (e.g. usage is 1387420, limit is 4), quota exceeded is set.
- a bit smaller value (e.g. usage is 405435848. limit is 405418036), quota exceeded is set.
3. For some users, 'limit' does not change even if we re-set the hard limit (these OSTs has 1 between [] in quota_slave.info).
4. In quota_slave.info, only OST0000-OST0004 does not have 1 between [].
What is the expected result (what it should be) for these items after setting quota limit to 0 to force all OSTs to release space?
|
I talked with the customer, but unfortunately just enabling quota while the system is in service is not acceptable for the customer. To do the test safely, setting '-B 0' on all the user and then enable quota would be one option for them. Assuming we do the test, what is expected result?
The expected result is that all OSTs will release their limit to master (which means all OSTs will have 0 limit at the end). Please collect debug log with D_QUOTA enabled when you enable quota (on both MDT and OSTs). Thank you.
|
|
We need to build a test plan for the action and so let me ask more.
What is expected after we set '-B 0' and then to the current setting again? As OSTs other than OST0000-OST0004 are synchronized with the master, can we expect the following on those synchronized OSTs?
1. same value in sum of usage on each OST and the whole (reported in 3rd line of 'lfs quota'), on file systems that do not have un-synchronized OSTs?
2. a bit greater value in 'limit'?
3. 'limit' will be updated with zero, and then to an adjusted value?
4. OST0000-OST0004 will be synchronized with the master? Or it will not even after we set '-B 0' on all OSTs?
regards,
|
We need to build a test plan for the action and so let me ask more.
sure
What is expected after we set '-B 0' and then to the current setting again?
On -B 0, all slaves should release reserved quota space.
As OSTs other than OST0000-OST0004 are synchronized with the master,
None of the OSTs are strictly synchronized with the master. If you check quota_slave.info, you will see that all report "glb[0] slv[0]". Some of them just happen to still own a glb quota lock.
can we expect the following on those synchronized OSTs?
1. same value in sum of usage on each OST and the whole (reported in 3rd line of 'lfs quota'), on file systems that do not have un-synchronized OSTs?
Usage should always be consistent, regardless of the quota enforcement status.
2. a bit greater value in 'limit'?
Those ones are expected to release all the quota space.
3. 'limit' will be updated with zero, and then to an adjusted value?
Once quota is enabled, yes.
4. OST0000-OST0004 will be synchronized with the master? Or it will not even after we set '-B 0' on all OSTs?
OST0000-OST0004 will synchronize with the master only once quota is enabled.
At this point, i would advise to proceed as follows:
1. keep quota disabled
2. dump & back up all quota (hard/soft/time) limits on the qmt. To do so, you can run the following command on the MDT and store the results:
3. set all the limits to 0 with quota disabled
4. enable quota again
5. check that all slaves are properly synchronized (should be "glb[1] slv[1]" everywhere in quota_slave.info)
6. check with lfs quota that all slaves have released quota space
7. set the user/group limit one by one and check that none of the OSTs get crazy limits again
What do you think?
|
|
Thanks Johann! I will write up a test plan and try the actions you provided. regards,
|
|
For the record, i created LU-5293 for the problem related to quota files removal on slaves.
|
|
The customer agreed with our plan and I am doing the work currently. After step 6, I set quota (-B 1000000000) on two users. The 'limit' value has now "0" for OSTs that does not have any object for the user, and 'limit' is set to the same value as usage in the case of the OSTs that has objects. For example,
On another user,
Is this correct behavior?
|
Is this correct behavior?
This is correct behavior.
|
|
I performed the action plan and confirmed quota issue has been resolved on most users, but also found a behavior that quota limit on each OST were not updated even I did 'setquota -B 0' (while quota was enabled, quota_slave was sync with the master). On the same file system, in the case of another user, limit value was updated just after I issued 'setquota -B 0'. Although I could not try on the all users, at least the behavior occurred on one user (UID: 2165). I captured debug log (lustre5_quota_UID2165_20140707.tar.gz). Even though "quota exceeded" is set on some of OSTs, I could write to that OSTs successfully. When I was doing write test on that user, finally limit value was updated just after 'setquota -B 0'. What I was doing is writing files on each OSTs, 'setquota -B 1000000000' and 'setquota -B 0' several times. Is this correct behavior?
|
|
Could you please clarify when the quota logs were collected? Just after setting the limit to 0? I only have the lfs quota output with a limit set to 10000000
|
|
In the logs, i see the glimpse related to set quota -B 1000000000:
But i don't have logs related to -B 0 where the slave is supposed to release all the quota space.
When a hard limit is enforced for a given user/group, it is actually expected to have a local slave limit higher than the actual usage as long as the sum of all slave limits (i.e. as reported by lfs quota -v) is equal to the granted field in qmt_dt-0x0_glb-usr on the quota master. When i look at this file in the tarball, i see:
It is pretty difficult for me to make any sense of those data because i have the "lfs quota -v" output for -B 10000000, debug logs for -B 1000000000 and qmt_dt-0x0_glb-usr file with -B 1000000. Could you please set the hard limit to 0, collect "lfs quota -v" as well as qmt_dt-0x0_glb-usr and then set the hard limit to 1000000000 and collect again "lfs quota -v" and qmt_dt-0x0_glb-usr ?
Thanks in advance.
|
|
Ah, i actually found logs later related to -B 0:
1. The notification related to the hard limit change is received by the slave:
2. The slave updates the lqe as well as the local copy of the global index:
3. The slave refreshes usage for this ID:
4. The slave decides to release the 4KB of quota space it owns for this user since quota isn't enforced any more for this ID:
As far as i can see, -B 0 worked as expected and all the quota space owned by the slave has been released. If you could just collect qmt_dt-0x0_glb-usr as well as "lfs quota -v" output, then we could check that limits are consistent everywhere.
|
|
When I set -B 0 for UID2165, quota limit as a whole was updated to the 0, but 'limit' on each OSTs was not updated. qmt_dt-0x0_glb-usr just after setting -B 0 was;
However, when I ran lfs quota;
Why those 'limit' were not updated to 0?
|
|
Well, i looked at logs for lustre5-OST0002 which released all the quota space (limit = 0). I am going to have a look at the logs for the other OSTs then.
|
|
Checking logs for lustre5-OST0004:
So it seems that no quota space was owned by lustre5-OST0004 and everything looks fine in the logs. Unfortunately, the logs were not collected at the same time as the lfs quota output above. Could you please:
1. set hard limit to a high value and wait for 5s
2. enable +quota and +trace in the debug mask of both MDS and OSS where lustre5-OST0004 is running.
3. run lctl mark start on MDS & OSS
4. set hard limit to 0 and wait for 5s
5. run lctl mark end on MDS & OSS
6. run lfs quota -v
7. collect lustre logs on both MDS & OSS
Thanks in advance.
|
|
Johann, I collected 'lfs quota -v' output just after 'setquota -B 0'. What I did was,
1. enable +quota and +trace in the debug mask of both MDS and OSS where lustre5-OST0004 is running.
2. run lctl mark start on MDS & OSS
3. run lfs quota -v
4. set hard limit to 1000000000
5. run lfs quota -v
6. set hard limit to 0
7. run lfs quota -v
8. wait for 5s
9. run lctl mark end on MDS & OSS
10. collect lustre logs on both MDS & OSS
The output above was captured in step 7. I also ran lfs quota -v a while (a few minutes or so) after that, and found the same output. Since this was the only user that showed this synopsis and now it was resolved somehow, I cannot capture the additional log. Is there a chance that this is a reporting issue in lfs utility?
|
|
Mitsuhiro, could you please clarify what you mean by "and now it was resolved somehow"?
|
|
After I captured lustre debug log, the customer was doing a write test using UID2165 to see how it behaves. At that time, 'limit' on each OSTs did not changed by setting hard limit to 0 or to 1000000000. They found they can write over the 'limit' shown in 'lfs quota'. While doing the test and 'lfs setquota', they also noticed 'limit' on each OSTs changed by 'lfs setquota'. Now, it can be set to "0" by 'lfs setquota -B 0' and to a value like "69652" by 'lfs setquota -B 10000000" (in this case, hard limit is set to 10GB). Thanks,
|
|
What does the behavior on UID2165 mean? Is there anything we should do to have quota behave correctly? Regards,
|
the customer was doing a write test using UID2165 to see how it behaves. At that time, 'limit' on each OSTs did not changed by setting hard limit to 0 or to 1000000000.
Do you mean that customer is writing files as UID2165 while you changing hard limit? How did you observe that limit wasn't changed by setting hard limit to 0 (could you show me the command and the output)?
They found they can write over the 'limit' shown in 'lfs quota'.
Could you explain it in detail? One possible reason is: the limit was set to 0, data could be cached on client, when user set a limit, the cache data will be flushed back anyway despite of quota limit.
While doing the test and 'lfs setquota', they also noticed 'limit' on each OSTs changed by 'lfs setquota'.
Was the limit changed as we have expected?
Now, it can be set to "0" by 'lfs setquota -B 0' and to a value like "69652" by 'lfs setquota -B 10000000" (in this case, hard limit is set to 10GB)
You mean that UID is back to normal now, right?
|
Do you mean that customer is writing files as UID2165 while you changing hard limit? How did you observe that limit wasn't changed by setting hard limit to 0 (could you show me the command and the output)?
No. I set hard limit to 0 and confirmed 'limit' did not changed. After that, the customer did write tests and found 'limit' started to change. Here is the log,
and finally it was,
Could you explain it in detail? One possible reason is: the limit was set to 0, data could be cached on client, when user set a limit, the cache data will be flushed back anyway despite of quota limit.
I don't have the log to show details unfortunately as I was not able to do a test by my own... Maybe the possibility you show is true. Thanks for pointing out.
Was the limit changed as we have expected?
Yes, as shown above, 'limit' on each OSTs started to change.
You mean that UID is back to normal now, right?
Yes, right. As far as I checked the account, I don't see a problematic behavior. However, because the customer observed the behavior above by themselves, they have a concern if quota is really behaving correctly now. I could confirm this account, but am not sure for all the account.
|
No. I set hard limit to 0 and confirmed 'limit' did not changed. After that, the customer did write tests and found 'limit' started to change. Here is the log
I didn't see global limit was set to 0 (it was from 1000000000 to 10000000) from the log.
|
I didn't see global limit was set to 0 (it was from 1000000000 to 10000000) from the log.
The log contains many meta characters and so it is a bit hard to read, but hard limit was first set to 1000000000, then to 0, finally to 10000000.
|
The log contains many meta characters and so it is a bit hard to read, but hard limit was first set to 1000000000, then to 0, finally to 10000000.
Ah, I see it. Did you wait for a while between 'lfs setquota -B 0' and 'lfs quota'?
|
|
In the log above, it was a few seconds, but this situation lasted for more than 10 minutes. I tried a couple of times during the time.
|
In the log above, it was a few seconds, but this situation lasted for more than 10 minutes. I tried a couple of times during the time.
Got it, thank you, but it's hard to tell why this happened without related logs, could you try to collect logs (in the way Johann suggested in previous comment) when you see the problem again? Thanks.
|
|
The steps Johann suggested was,
1. set hard limit to a high value and wait for 5s
2. enable +quota and +trace in the debug mask of both MDS and OSS where lustre5-OST0004 is running.
3. run lctl mark start on MDS & OSS
4. set hard limit to 0 and wait for 5s
5. run lctl mark end on MDS & OSS
6. run lfs quota -v
7. collect lustre logs on both MDS & OSS
and the log I collected was,
1. enable +quota and +trace in the debug mask of both MDS and OSS where lustre5-OST0004 is running.
2. run lctl mark start on MDS & OSS
3. run lfs quota -v
4. set hard limit to 1000000000
5. run lfs quota -v
6. set hard limit to 0
7. run lfs quota -v
8. wait for 5s
9. run lctl mark end on MDS & OSS
10. collect lustre logs on both MDS & OSS
The difference would be whether we wait for 5 seconds before 'lfs quota' (debug log was captured after the '5 seconds') or do it after waiting for 5 seconds. How does this difference in collecting the log make sense to explain the behavior?
|
The difference would be whether we wait for 5 seconds before 'lfs quota' (debug log was captured after the '5 seconds') or do it after waiting for 5 seconds. How does this difference in collecting the log make sense to explain the behavior?
If the 'lfs quota -v' is executed immediately after 'lfs setquota -B 0', the slaves might haven't release limits yet, so you'd wait for a while after setting limit, then run 'lfs quota -v' to verify limit. We checked the provided log, and the log shows that slaves released limits as expected.
|
|
As per Ihara it is ok to close this ticket. Quotas are now running ok since they were reset. It is not understood how things got into an inconsistent state but things have been running ok since the reset.
|
Generated at Sat Feb 10 01:49:48 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
lustre4_lfs.quota_UID3018_20140626.log
lustre4_quota.debug.UID3018_20140626.tar.gz
oss_quota_slave.zip