https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6283 Lustre <p>We'd like to be able to perturb the timing of request processing at the PtlRPC layer with the goal being to simulate high server load, and find and expose timing related problems.</p> <p>Our initial idea is to create an NRS policy that will delay request handling for some configurable amount of time. When the policy is started and a request arrives the policy will calculate an offset, within a defined, user-configurable range, from the request arrival time to set a request "start time". We can use the cfs_binheap implementation to store these requests and sort them based on this "start time". Request's are then removed from the binheap for handling only once we've reached/passed their start time. We could also choose to only delay some % of requests by allowing the request enqueue to fallback to FIFO (or whatever).</p> <p>I have an initial implementation mostly done (just need to finish up lprocfs stuff). I appreciate any thoughts on this approach.</p> LU-6283

NRS Delay Policy

New Feature Minor Resolved Fixed Chris Horn Chris Horn cea patch Wed, 25 Feb 2015 20:16:28 +0000 Wed, 29 May 2019 17:20:42 +0000 Sun, 23 Apr 2017 03:57:37 +0000 Lustre 2.10.0 0 17 <p>I think this is a very interesting way to do fault injection and load simulation.</p> <p>If you are working in the NRS code, there are a number of things that could be cleaned up. <a href="https://jira.whamcloud.com/browse/LU-2667" title="move NRS structures/definitions from lustre_net.h to new lustre_nrs.h header" class="issue-link" data-issue-key="LU-2667"><del>LU-2667</del></a> is one example.</p> <p>Also, having the ability to dynamically load NRS modules via a register and deregister function would allow the testing policy to only be loaded when needed. This was discussed several times in the context of the original NRS patches, but I'm unable to find those comments right now.</p> <p>Nikitas Angelinas (nikitas.angelinas@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/14003" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14003</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6283" title="NRS Delay Policy" class="issue-link" data-issue-key="LU-6283"><del>LU-6283</del></a> ptlrpc: re-add NRS policy registration symbol exports<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6b552ecf150edad57e14ae1795d809fe4da3fd5c</p> <p>The ptlrpc_nrs_policy_(register|unregister)() functions should allow for loading/unloading policies on demand from modules other than ptlrpc; the symbols used to be exported, but were unexported as part of <a href="https://jira.whamcloud.com/browse/LU-5829" title="too many EXPORT_SYMBOL in code" class="issue-link" data-issue-key="LU-5829"><del>LU-5829</del></a>; I uploaded a short patch to re-add them at <a href="http://review.whamcloud.com/#/c/14003/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14003/</a>.</p> <p>I had tested this feature when we first landed NRS and it appeared to work fine.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14003/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14003/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6283" title="NRS Delay Policy" class="issue-link" data-issue-key="LU-6283"><del>LU-6283</del></a> ptlrpc: re-add NRS policy registration symbol exports<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5167bcea2175c751c79173fd934cddcb4cd9fa7b</p> <p>Hi Chris, is there still a plan to land this feature for 2.8? As yet I haven't seen any signs that this is being worked on, but if 2.8 is still the target release then we need to start planning for its landing before the feature freeze.</p> <p>Chris Horn (hornc@cray.com) uploaded a new patch: <a href="http://review.whamcloud.com/14701" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14701</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6283" title="NRS Delay Policy" class="issue-link" data-issue-key="LU-6283"><del>LU-6283</del></a> ptlrpc: Implement NRS Delay Policy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: db28ca8c9d8008d3bc00e8c1e77b60d107cdaf1d</p> <p>Hi Andreas,<br/> I've just pushed my code. My hope is that this is mostly complete, so I should be able to address any review feedback quickly. In any case, I'll be sure to dedicate appropriate resources so this can land for 2.8.</p> <p>Hello Chris, if NRS Delay is targeted for 2.8, could you please upload the test plan by feature freeze? I created a ticket for tracking the test plan: <a href="https://jira.hpdd.intel.com/browse/LU-6583" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-6583</a></p> <p>Thanks!</p> <p>Hi, Sarah. Perhaps an oversight, but LU-6583 is assigned to HPDD Triage. Should that have been assigned to Chris?</p> <p>I'm not really clear on what the intent of creating such tickets is, but JIRA tickets can only be assigned to HPDD engineers atm</p> <p>Hello Cory, </p> <p>If Chris could upload the test plan in this ticket then I can just close LU-6583. LU-6583 is for tracking the test plan.</p> <p>We should open an LUDOC ticket to track any needed doc updates for this policy.</p> <p>Also need something to add the design to: <a href="http://wiki.lustre.org/Projects" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/Projects</a></p> <p><a href="https://jira.whamcloud.com/browse/LUDOC-366" title="Document NRS Delay" class="issue-link" data-issue-key="LUDOC-366"><del>LUDOC-366</del></a> opened to track doc changes.</p> <p>Chris - I think the only thing we are waiting on to land this feature is some kind of a feature test plan or test report. We are looking for some indication of what testing you have done to verify that this feature is working correctly and any tests added to the Lustre test suites to make sure this feature functions correctly in the future. </p> <p>Please attach a test plan/report to this ticket and, I think, the feature can move ahead for Lustre 2.10. </p> <p>If there's any questions about what we are looking for, you are welcome to contact me.</p> <p>Thanks, James</p> <p>test_77l() is added as part of the feature implementation. It provides a good example of how other test cases might be written for other ptlrpc services. It verifies NRS delay works correctly for the ost_io service by generating a specific number of write RPCs, measuring the time it takes for the I/O to complete and comparing that with the amount of delay we configured for the ost_io service.</p> <p>I did similar testing in development using a range of delay values to ensure that the random delay logic was working as expected. For example:</p> <p>With a range of delay values between 1 and 10:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl set_param ost.OSS.ost_io.nrs_policies=delay \ ost.OSS.ost_io.nrs_delay_min=1 \ ost.OSS.ost_io.nrs_delay_max10 \ ost.OSS.ost_io.nrs_delay_pct=100 </pre> </div></div> <p>We'd expect a 1MB write to take at least 1 second, and not much more than 10 seconds (for an idle filesystem it should effectively be 10-11 seconds).</p> <p>I performed testing in development to ensure the tunables were doing some proper sanitization of their inputs. Things like setting min &gt; max, delay_pct &lt; 0 and &gt; 100, etc.</p> <p>This was all pretty informal so I do not have data on the results of those testing except to say that I fixed any bugs that I found.</p> <p>If the community feels that additional unit testing of the sort I've described is warranted then I can surely work to generate additional unit tests for sanityn.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/14701/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/14701/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6283" title="NRS Delay Policy" class="issue-link" data-issue-key="LU-6283"><del>LU-6283</del></a> ptlrpc: Implement NRS Delay Policy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 588831e9eac38b8514f2a3e71516b44fa7c4bcce</p> <p>Landed for 2.10</p> <p>Should this be closed?</p> <p>ok I am going to stop responding to these now. Hopefully you check your email before I get too many more of these...</p> Related LU-398 LUDOC-366 Development Rank 1|hzx727: Rank (Obsolete) 17614