[LU-3558] NRS TBF policy for QoS purposes

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.6.0
• Labels: patch ptr
• Assignee: Lai Siyao
• Reporter: Li Xi (Inactive)

Description

NRS (Network Request Scheduler) enables the services to schedule the RPCs in different manners. And there have been a bunch of policies implemented over the main framework. Most of them are aimed at improving throughput rate or similar purposes. But we are trying to implement policies for a differnt kind of purpose, QoS.

The TBF (Token Bucket Filter) is one of the policies that we implemented for traffic control. It enforces a RPC rate limit on every client according to the NID. The handling of a RPC will be delayed until there are enough tokens for the client. Different clients are scheduled according to their deadlines, so that none of them will be starving even though the service does not have the ability to satisfy all the RPC rate requirments of clients. The RPCs from the the same clients are queued in a FIFO manner.)

Early tests show that the policy works to enforce the RPC rate limit. But more tests, bechmarks and analyses is needed for its correctness and efficiency.

Issue Links

• is blocking: LU-11431 Global QoS management based on TBF
• is related to: LU-5717 Dead lock of nrs_tbf_timer_cb
• is related to: LU-6668 Add tests for TBF
• is related to: LU-5379 Get error when has many rules in nrs tbf policy
• is related to: LU-9227 Changing rate of a TBF rule loses control in some testcases
• is related to: LU-5580 Switch between 'JOBID' and 'NID' directly in NRS TBF
• is related to: LU-5620 nrs tbf policy based on opcode
• is related to: LUDOC-221 Document Token Bucket Filter (TBF) NRS policy
• is related to: LU-4586 build failure in nrs_tbf_ctl()
• is related to: LUDOC-328 documentation updates for complex TBF policies
• is related to: LU-8008 Can't enable or add rules to TBF
• is related to: LU-5620 nrs tbf policy based on opcode
• is related to: LU-7470 Extend TBF policy with NID/JobID expressions
• is related to: LU-9228 Hard TBF Token Compensation under congestion
• is related to: LU-3266 Regression tests for NRS policies
• is related to: LU-8006 Specify ordering of TBF policy rules
• is related to: LU-8236 Wild-card in jobid TBF rule

Comments

[Li Xi (Inactive)]

Hi Andreas,

That sounds really interesting. But in order to implement multi-layered policies, I guess the framework of NRS needs a lot of changes, right?

[Li Xi (Inactive)]

Hi Nathan,

Yes, you are right. The deadlines in the two statements mean different.

The former one refers to a time point that one of the NID's RPCs should be handled in order to achieve the deserved RPC rate. If the deadline is missed, nothing bad will happen, except the RPC rate is slower than the NID expects.

The latter one refers to traditional deadline of a RPC.

Sorry for my misusing.

[Andreas Dilger]

My preference for the long term is that we have a single "super" NRS policy that does many things at one time, or we use layers of NRS policies at the same time to achieve the optimum result.

In the case of TBF it would be possible to have it provide QOS guarantees be giving out credits to exports in an unfair manner, so clients with guaranteed bandwidth will get it, and clients with bandwidth limits will be throttled as needed.

Combining TBF and ORR seems possible, either by using TBF as a first layer filter to avoid massive client unfairness, passing the "unthrottled" requests through to the ORR filter to sort it within and between objects.

[Nathan Rutman]

it schedules the handling of RPC between different NIDs according to their deadline

current codes does not consider deadline of a RPC

I am having trouble reconciling those two statements. Does the first refer to a different kind of deadline?

[Li Xi (Inactive)]

Hi Nathan,

The current codes does not consider deadline of a RPC yet. But yeah, I think more cases should be tested to make sure it is not a big problem.

[Li Xi (Inactive)]

Hi Andreas,

The policy throttles RPC based on TBF algorithm. But it schedules the handling of RPC between different NIDs according to their deadline, so it looks more like a CRR-N policy rather than ORR policy. Yes, we are trying change the policy in order to limit RPC rate of different users/groups/jobs. But I havn't got any idea about how to conjunct it with the ORR NRS policy. Any advice?

Sure, we are running benchmarks and writing description about it. The test results and documents will come along with code improvement soon.

[Nathan Rutman]

Cool!
How does the deadline scheduling interact with the tokens in the case of a conflict? I.e. if there are not enough tokens yet the deadline is imminent?

Andreas, I just realized a possible side effect of any NRS policy is that it may oddly affect adaptive timeouts by skewing the measured RPC processing time to the maximum delay induced by the policy. I suppose the worst fallout from this would be slower recovery, so maybe not so horrible.

[Peter Jones]

Lai

Could you please review the supplied patch and offer advise as appropriate

Thanks

Peter

[Andreas Dilger]

Description of Token Bucket Filter - http://en.wikipedia.org/wiki/Token_bucket_filter

It would also be useful to test TBF in conjunction with the ORR NRS policy, so that RPCs from clients are sorted before IO and have a better chance to have a more optimal ordering when submitted to the backing storage.

Before this can be landed, there will need to be a much better description of how this policy is used, and the performance results. As well, an update is needed for the Lustre Manual with details of how to use the policy and set limits for the NIDs.

[Li Xi (Inactive)]

Here is the patch.
http://review.whamcloud.com/#/c/6901/