|
Here is the patch.
http://review.whamcloud.com/#/c/6901/
|
|
Description of Token Bucket Filter - http://en.wikipedia.org/wiki/Token_bucket_filter
It would also be useful to test TBF in conjunction with the ORR NRS policy, so that RPCs from clients are sorted before IO and have a better chance to have a more optimal ordering when submitted to the backing storage.
Before this can be landed, there will need to be a much better description of how this policy is used, and the performance results. As well, an update is needed for the Lustre Manual with details of how to use the policy and set limits for the NIDs.
|
|
Lai
Could you please review the supplied patch and offer advise as appropriate
Thanks
Peter
|
|
Cool!
How does the deadline scheduling interact with the tokens in the case of a conflict? I.e. if there are not enough tokens yet the deadline is imminent?
Andreas, I just realized a possible side effect of any NRS policy is that it may oddly affect adaptive timeouts by skewing the measured RPC processing time to the maximum delay induced by the policy. I suppose the worst fallout from this would be slower recovery, so maybe not so horrible.
|
|
Hi Andreas,
The policy throttles RPC based on TBF algorithm. But it schedules the handling of RPC between different NIDs according to their deadline, so it looks more like a CRR-N policy rather than ORR policy. Yes, we are trying change the policy in order to limit RPC rate of different users/groups/jobs. But I havn't got any idea about how to conjunct it with the ORR NRS policy. Any advice?
Sure, we are running benchmarks and writing description about it. The test results and documents will come along with code improvement soon.
|
|
Hi Nathan,
The current codes does not consider deadline of a RPC yet. But yeah, I think more cases should be tested to make sure it is not a big problem.
|
it schedules the handling of RPC between different NIDs according to their deadline
current codes does not consider deadline of a RPC
I am having trouble reconciling those two statements. Does the first refer to a different kind of deadline?
|
|
My preference for the long term is that we have a single "super" NRS policy that does many things at one time, or we use layers of NRS policies at the same time to achieve the optimum result.
In the case of TBF it would be possible to have it provide QOS guarantees be giving out credits to exports in an unfair manner, so clients with guaranteed bandwidth will get it, and clients with bandwidth limits will be throttled as needed.
Combining TBF and ORR seems possible, either by using TBF as a first layer filter to avoid massive client unfairness, passing the "unthrottled" requests through to the ORR filter to sort it within and between objects.
|
|
Hi Nathan,
Yes, you are right. The deadlines in the two statements mean different.
The former one refers to a time point that one of the NID's RPCs should be handled in order to achieve the deserved RPC rate. If the deadline is missed, nothing bad will happen, except the RPC rate is slower than the NID expects.
The latter one refers to traditional deadline of a RPC.
Sorry for my misusing. 
|
|
Hi Andreas,
That sounds really interesting. But in order to implement multi-layered policies, I guess the framework of NRS needs a lot of changes, right?
|
|
This is an inital benchmark resutls we did a couple of weeks ago. Tested on 16 clients with various token policy for few clients. We will test again on the latest codes and push results on here later.
|
|
This is initial version of Lustre NRS TBF design document.
|
|
The current Lustre NRS TBF codes change a lot on this basis of the last version. First, we optimize the main framework of TBF policy, which makes it easier to add new supports, e.g. Job ID support, UID/GID support. Based on that, we add the support of Job ID. Now we can utilize Job Stat mechanism to set limit to different jobs. And ofcourse, we fix some defects of TBF codes too.
|
|
Would it be possible to get the fix-version set to 2.5? We want to make sure it doesn't slip off the radar at the last minute or anything. I think that's the procedure we talked about on the CDWG call last week.
Thanks.
|
|
The NID based TBF policy works well. But we found a problem of JobID based TBF policy and have to ask for help.
The JobID based TBF policy classifies RPCs according to the Job Stat informantion of each RPC. The simplest Job Stat informantion is 'procname_uid' which can be enabled by 'lctl conf_param server1.sys.jobid_var=procname_uid'. With TBF policy, we are able to set rate limits to different kinds of RPCs. We set the RPC rate of 'dd.0' to 1 RPC/s and the RPC rate 'dd.500' to 1000 RPC/s. If TBF policy works well, when root user ran 'dd' command, an OSS service partition will never handle more than 1 RPC of it. And when user 500 ran 'dd' command, an OSS service partition will never handle more than 1000 RPC of it. Actually this works well except following condition.
When we ran 'dd' using user root and user 500 at the same time, on the same client, writing to the same OST, the performance of user 500 will decline dramatically, i.e. the performance of user 500 is highly affected by the user root.
Here is the result that we got running following command.
dd if=/dev/zero of=/mnt/lustre/fileX bs=1048576 count=XXXX
1. When user 500 ran 'dd' alone, the performance is about 80 MB/s. This is normal because the OSS's performance has an upper limit of about 80 MB/s
2. When user root ran 'dd' alone, the performance is about 2 MB/s. This is normal too, because the OSS has two partition and each has a limit of 1 RPC/s. 1 MB/RPC * 1 RPC/s * 2 = 2 MB/s
3. When user root ran 'dd', and user 500 ran 'dd' on another client, user 500 will get performance of about 80 MB/s and user root will get performance of about 2 MB/s. Please not that different processes writes to differnt files. No matter what the stripes of the files are, we get similar results. There are expected normal results.
4. When user root ran 'dd', and user 500 ran 'dd' on another client,
user 500 will get performance of about 80 MB/s and user root will get 2 MB/s. That's normal too.
5. When user root ran 'dd', and user 500 ran 'dd' on the same client, but they write to different OSTs (i.e. the stripe indexes of these files are different), user 500 will get performance of about 80 MB/s and user root will get 2 MB/s. That's normal too.
6. When user root ran 'dd', and user 500 ran 'dd' on the same client, and they write to the same OST (i.e. the stripe indexes of these files are the same), the performance of user 500 will declines to about 2 MB/s when user root is writing too. The performance of user 500 will go up immediately to 80 MB/s after user root completes its writing.
The result 6 is really strange. We think it is not likely that server side codes cause the problem since result 4 is normal. And result 5 implies that it is the OSC ranther than the OSS throttles RPC rate wrongly. Maybe when some RPCs from an OSC are appending, the OSC does not send any more RPCs? I guess maybe some mechanisms of OSC make it works like this, e.g. max RPC in flight limit? I've tried to enlarge max_rpc_in_flight argument of OSCs but got no luck.
Any suggestions you could provide to us would be greatly appreciated! Thank you in advance!
|
|
I believe your analysis of case #6 is correct - the client only has a limited number of RPCs in flight for each RPC (see also the LNET "peer credits" tunable). If the u500 IOs are blocked behind the u0 IOs, they will be limited by the slower process. This may not be strictly related to the RPCs, but rather to the higher-level RPC engine that is trying to balance IO submission between objects, and doesn't know about the NRS ordering on the server.
The first question is whether this is a use case that is important for real users? I'm not sure if there is an easy solution for how to handle this from the server.
|
|
It's been a few months since the last entry. I wanted to ask if this idea/patch is worthy of further work for inclusion in 2.7? Thanks!
|
|
Hmm.. the question from us, why not included in 2.6 or even 2.5.1 yet? The original discussion with Peter, this is not core component of Lustre, it could be landed in 2.5. or 2.4.x even. But, review didn't finish before 2.5 release.
After that, we got at least multiple inspection pass from multiple people, but rebase was needed again and again, then it needed review again.
I would request review this quickly again and we want to land this in 2.6 and 2.5.1...
|
|
No disagreement from me But I'm not a technical person - I just like the capability that TBF provides.
So we'll have to get technical people to review this.
Thanks!
|
|
OK, thanks! We hope popole can get QoS function with Lustre sooner and we want it as well!
|
|
The patch http://review.whamcloud.com/6901 was landed to master for 2.6.
This functionality also needs an update to the manual to explain what this feature does, and how to use it. Please see https://wiki.hpdd.intel.com/display/PUB/Making+changes+to+the+Lustre+Manual. Please submit an LUDOC jira ticket to track the manual update, and link it here.
|
|
No problem! We will submit a manual update soon. Thanks!
|
|
I filed LUDOC-221 to track the documentation update for the TBF feature.
|
|
Can this ticket now be closed since we have LUDOC-221 to track the manual updates?
|
|
Patch has landed for 2.6.0, using LUDOC-221 for tracking the remaining work on the manual.
|
Generated at Sat Feb 10 01:34:56 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
NRS-initial-test-result.xlsx
TBF-design-1.0.pdf