[LU-6283] NRS Delay Policy

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.10.0
• Type: New Feature
• Priority: Minor
• Reporter: Chris Horn
• Assignee: Chris Horn
• Resolution: Fixed
• Labels: cea, patch

Description

We'd like to be able to perturb the timing of request processing at the PtlRPC layer with the goal being to simulate high server load, and find and expose timing related problems.

Our initial idea is to create an NRS policy that will delay request handling for some configurable amount of time. When the policy is started and a request arrives the policy will calculate an offset, within a defined, user-configurable range, from the request arrival time to set a request "start time". We can use the cfs_binheap implementation to store these requests and sort them based on this "start time". Request's are then removed from the binheap for handling only once we've reached/passed their start time. We could also choose to only delay some % of requests by allowing the request enqueue to fallback to FIFO (or whatever).

I have an initial implementation mostly done (just need to finish up lprocfs stuff). I appreciate any thoughts on this approach.

Issue Links

• is related to: LU-398 NRS (Network Request Scheduler )
• is related to: LUDOC-366 Document NRS Delay

Comments

[Andreas Dilger]

I think this is a very interesting way to do fault injection and load simulation.

If you are working in the NRS code, there are a number of things that could be cleaned up. LU-2667 is one example.

Also, having the ability to dynamically load NRS modules via a register and deregister function would allow the testing policy to only be loaded when needed. This was discussed several times in the context of the original NRS patches, but I'm unable to find those comments right now.

[Gerrit Updater]

Nikitas Angelinas ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14003
Subject: LU-6283 ptlrpc: re-add NRS policy registration symbol exports
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6b552ecf150edad57e14ae1795d809fe4da3fd5c

[Nikitas Angelinas]

The ptlrpc_nrs_policy_(register|unregister)() functions should allow for loading/unloading policies on demand from modules other than ptlrpc; the symbols used to be exported, but were unexported as part of LU-5829; I uploaded a short patch to re-add them at http://review.whamcloud.com/#/c/14003/.

I had tested this feature when we first landed NRS and it appeared to work fine.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/14003/
Subject: LU-6283 ptlrpc: re-add NRS policy registration symbol exports
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5167bcea2175c751c79173fd934cddcb4cd9fa7b

[Andreas Dilger]

Hi Chris, is there still a plan to land this feature for 2.8? As yet I haven't seen any signs that this is being worked on, but if 2.8 is still the target release then we need to start planning for its landing before the feature freeze.

[Gerrit Updater]

Chris Horn ([e-mail]) uploaded a new patch: http://review.whamcloud.com/14701
Subject: LU-6283 ptlrpc: Implement NRS Delay Policy
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: db28ca8c9d8008d3bc00e8c1e77b60d107cdaf1d

[Chris Horn]

Hi Andreas,
I've just pushed my code. My hope is that this is mostly complete, so I should be able to address any review feedback quickly. In any case, I'll be sure to dedicate appropriate resources so this can land for 2.8.

[Sarah Liu]

Hello Chris, if NRS Delay is targeted for 2.8, could you please upload the test plan by feature freeze? I created a ticket for tracking the test plan: https://jira.hpdd.intel.com/browse/LU-6583

Thanks!

[Cory Spitz]

Hi, Sarah. Perhaps an oversight, but LU-6583 is assigned to HPDD Triage. Should that have been assigned to Chris?

[Peter Jones]

I'm not really clear on what the intent of creating such tickets is, but JIRA tickets can only be assigned to HPDD engineers atm

[Sarah Liu]

Hello Cory,

If Chris could upload the test plan in this ticket then I can just close LU-6583. LU-6583 is for tracking the test plan.

[Cory Spitz]

We should open an LUDOC ticket to track any needed doc updates for this policy.

[Ben Evans (Inactive)]

Also need something to add the design to: http://wiki.lustre.org/Projects

[Chris Horn]

LUDOC-366 opened to track doc changes.

[James Nunez (Inactive)]

Chris - I think the only thing we are waiting on to land this feature is some kind of a feature test plan or test report. We are looking for some indication of what testing you have done to verify that this feature is working correctly and any tests added to the Lustre test suites to make sure this feature functions correctly in the future.

Please attach a test plan/report to this ticket and, I think, the feature can move ahead for Lustre 2.10.

If there's any questions about what we are looking for, you are welcome to contact me.

Thanks, James

[Chris Horn]

test_77l() is added as part of the feature implementation. It provides a good example of how other test cases might be written for other ptlrpc services. It verifies NRS delay works correctly for the ost_io service by generating a specific number of write RPCs, measuring the time it takes for the I/O to complete and comparing that with the amount of delay we configured for the ost_io service.

I did similar testing in development using a range of delay values to ensure that the random delay logic was working as expected. For example:

With a range of delay values between 1 and 10:

lctl set_param ost.OSS.ost_io.nrs_policies=delay \
				       ost.OSS.ost_io.nrs_delay_min=1 \
				       ost.OSS.ost_io.nrs_delay_max10 \
				       ost.OSS.ost_io.nrs_delay_pct=100

We'd expect a 1MB write to take at least 1 second, and not much more than 10 seconds (for an idle filesystem it should effectively be 10-11 seconds).

I performed testing in development to ensure the tunables were doing some proper sanitization of their inputs. Things like setting min > max, delay_pct < 0 and > 100, etc.

This was all pretty informal so I do not have data on the results of those testing except to say that I fixed any bugs that I found.

If the community feels that additional unit testing of the sort I've described is warranted then I can surely work to generate additional unit tests for sanityn.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/14701/
Subject: LU-6283 ptlrpc: Implement NRS Delay Policy
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 588831e9eac38b8514f2a3e71516b44fa7c4bcce

[Peter Jones]

Landed for 2.10

[Colin Faber [X] (Inactive)]

Should this be closed?

[Peter Jones]

ok I am going to stop responding to these now. Hopefully you check your email before I get too many more of these...