https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9140 Lustre <p><a href="https://testing.hpdd.intel.com/test_sets/16f600cc-f85e-11e6-b8e5-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/16f600cc-f85e-11e6-b8e5-5254006e85c2</a></p> <p>From test_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control Write speed is 4.91903 Verify the read rate is under TBF control Read speed is 19.5095 sanityn test_77j: @@@@@@ FAIL: The read rate (19.5095) exceeds 110% of preset rate (5) </pre> </div></div> <p>Info required for matching: sanityn 77j<br/> Info required for matching: sanityn 77k</p> trevis-38vm3 thru 6, Full Group test, <br/> RHEL7.3/ldiskfs, branch master, v2.9.53.19.gdad9ce4, b3534, <br/> SLES 12.2 client<br/> LU-9140

sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate

Bug Critical Resolved Fixed Qian Yingjin James Casper Tue, 21 Feb 2017 21:40:34 +0000 Wed, 19 Sep 2018 20:50:48 +0000 Thu, 19 Oct 2017 17:09:46 +0000 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.11.0 Lustre 2.10.2 0 13 <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/344078ec-0173-11e7-9255-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/344078ec-0173-11e7-9255-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/c5cefc54-08c9-11e7-8c9f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c5cefc54-08c9-11e7-8c9f-5254006e85c2</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=qian" class="user-hover" rel="qian">qian</a>, could you please have a look at this issue? test_77j was introduced by <del><a href="https://jira.whamcloud.com/browse/LU-5620" title="nrs tbf policy based on opcode" class="issue-link" data-issue-key="LU-5620"><del>LU-5620</del></a></del> and now its failure rate is about "6.00% of last 100 runs, 0 skipped (all branches) ".</p> <p>It is a very strange bug. I can not reproduce it in my test environment. I was thought it caused by the CPT &gt; 1 as the rate limit per server would reach CPT * rate. But I Checked the failure log of the testbed all with CPT = 1. Maybe the reproduce of this failure is dependent with other test cases?</p> <p>Seen again with 2.9.54 tag:</p> <p><a href="https://testing.hpdd.intel.com/test_sessions/93d0b46a-0a02-11e7-9053-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/93d0b46a-0a02-11e7-9053-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/1a6bc6e8-0a05-11e7-9053-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/1a6bc6e8-0a05-11e7-9053-5254006e85c2</a></p> <p>We run the full group on 7 configs, and this seems to move around those configs (failed 2 of 7 times on .53 and .54).</p> <p>I have ran the test more than 100 times for a Lustre configure with 7 OSTs, but it does not reproduce...</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">error() { echo <span class="code-quote">"$@"</span> exit 1 } pushd lustre-release/lustre/tests <span class="code-keyword">for</span> ((i=1; i&lt;=100; i++)) <span class="code-keyword">do</span> ONLY=<span class="code-quote">"77a 77b 77c 77d 77e 77f 77i 77j"</span> RUNAS_ID=1000 sh sanityn.sh [ $? -ne 0 ] &amp;&amp; error <span class="code-quote">"FAILED $i testing"</span> done popd </pre> </div></div> <p> </p> <p>+1 on master<br/> <a href="https://testing.hpdd.intel.com/test_sets/f9d4cb54-1897-11e7-b742-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/f9d4cb54-1897-11e7-b742-5254006e85c2</a></p> <p>Still broken <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>This is blocking patch review testing on master branch:<br/> <a href="https://testing.hpdd.intel.com/test_sets/9d3acb4c-1db4-11e7-9de9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9d3acb4c-1db4-11e7-9de9-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/864f7644-1f6c-11e7-8920-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/864f7644-1f6c-11e7-8920-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/a5d1ef7e-1f08-11e7-9073-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a5d1ef7e-1f08-11e7-9073-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/1cbc0492-1ef2-11e7-b742-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1cbc0492-1ef2-11e7-b742-5254006e85c2</a></p> <p>This is failing 5-10x per day so is causing some significant interference with new patch testing.</p> <p>I suspect that the problem may relate to running in a VM, or possibly because there are so few RPCs?</p> <p>Qian, please review one of the test results above (they should all be run with -1 debugging, e.g. <a href="https://testing.hpdd.intel.com/test_logs/8ce9c52c-1f6c-11e7-8920-5254006e85c2/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/8ce9c52c-1f6c-11e7-8920-5254006e85c2/show_text</a> ) to see why NRS is not handling this properly.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26615" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26615</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> tests: disable sanityn test_77j and test_77k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6225bfd533457ca364ff5bde613f0cf86f130f38</p> <p>Hi Andreas, I am sorry that the debugging log in NRS TBF is not enough for identifiing the problem. It needs to add new debug log to see what happend in NRS TBF.</p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/1ff0c23e-20db-11e7-b742-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1ff0c23e-20db-11e7-b742-5254006e85c2</a></p> <p>Qian, can you please submit a patch that adds the required debugging to the test and/or the kernel.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26615/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26615/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> tests: disable sanityn test_77j and test_77k<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e9b0c8b55c100e1d86093636309b385a913e927e</p> <p>Although it is still unclear to me how this problem happens, I am wondering whether we should use a special RPC type to check the rate of RPC. The software stack and also cache stack of read/write is too deap, and the speed of read/write could be affected by a lot of factors. So in order to check the speed of RPC control, a RPC type (e.g. "lfs advise") could be used.</p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/26701" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26701</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: add some debug log for NRS TBF<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ad151f5adee39af6b786f36ccc7263c43dcfbfb5</p> <p>Qian</p> <p>Do the results from the run with the denig patch give you a fuller picture as to what the problem is here?</p> <p>Peter</p> <p>Hi Peter，<br/> Yes, I think the debug patch <a href="https://review.whamcloud.com/26701" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26701</a> can help to identify the problem. I used the similar debug log to find out the failure reason of <a href="https://jira.whamcloud.com/browse/LU-9227" title="Changing rate of a TBF rule loses control in some testcases" class="issue-link" data-issue-key="LU-9227"><del>LU-9227</del></a>. Maybe the merged patch of <a href="https://jira.whamcloud.com/browse/LU-9227" title="Changing rate of a TBF rule loses control in some testcases" class="issue-link" data-issue-key="LU-9227"><del>LU-9227</del></a> have already solved the problem.</p> <p>This subtest is still failing for patches that include the <a href="https://jira.whamcloud.com/browse/LU-9227" title="Changing rate of a TBF rule loses control in some testcases" class="issue-link" data-issue-key="LU-9227"><del>LU-9227</del></a> fix:</p> <p> <a href="https://testing.hpdd.intel.com/sub_tests/a2b0f3f4-33d4-11e7-b0a8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/a2b0f3f4-33d4-11e7-b0a8-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/abff3518-3278-11e7-8847-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/abff3518-3278-11e7-8847-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/fcf81bb0-32f5-11e7-814a-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/fcf81bb0-32f5-11e7-814a-5254006e85c2</a></p> <p>We'll land the debug patch and try to catch some failures with the added debugging information.</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="https://review.whamcloud.com/26701/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26701/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: add some debug log for NRS TBF<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e0deeb237ae111e959e916ae910d15272ed4c120</p> <p>There are at least two test sessions that have the new debug information in them. I found the messages in the OST debug_logs for <br/> <a href="https://testing.hpdd.intel.com/sub_tests/9951bf06-36e4-11e7-b0a8-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/9951bf06-36e4-11e7-b0a8-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/9fd5daee-3700-11e7-8847-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/9fd5daee-3700-11e7-8847-5254006e85c2</a></p> <p>After analyzed the debug log, I found that the total time to write 100MB data is about 200s (becaused of network congestion, I think), so the measured I/O performance by dd is nearly 0.5MB/s, but the output of dd is used 'kB/s' as the unit autoatically, so the number is about 500. I think we can fix this bug in the sanityn.sh tbf_verify by adjusting the output result by dd according to its unit 'MB/s', 'GB/s' or 'kB/s'.</p> <p>Would it be better to just compute the expected runtime for I/O based on the TBF limit, and then compare the actual runtime to expected, instead of parsing the output of "dd"? </p> <p>The reason the IO is so slow is because these tests are run in a VM that may be using a shared disk also used by another VM guest. </p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/27110" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27110</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: adjust measured IOPS according to dd output<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1f9030a3624cabf633913edc84376486defa1719</p> <p>Hi Andreas,</p> <p>I just finished a patch to ajust the measured IOPS according to the unit of dd output.</p> <p>The patch above is more intuitive; while your suggestion is much simpler.</p> <p>If you think the comparson by using runtime is much better, I will push a new patch sooner. </p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27110/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27110/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: calculate the measured rate according to dd runtime<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5fb7d63e7d5ccd05f39b74b5c79d1856798477c7</p> <p>Landed for 2.10</p> <p>It looks like the issue hasn't been resolved. there is syntax error</p> <p><a href="https://testing.hpdd.intel.com/test_sets/cb4e36e6-52c9-11e7-bacf-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cb4e36e6-52c9-11e7-bacf-5254006e85c2</a><br/> tag 2.9.59</p> <p>test log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-3vm6: 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449807 s, 23.3 MB/s trevis-3vm5: 1+0 records in trevis-3vm5: 1+0 records out trevis-3vm5: 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0186742 s, 56.2 MB/s Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control (standard_in) 1: illegal character: M (standard_in) 1: syntax error Write runtime is MiB) s, speed is IOPS (standard_in) 1: syntax error /usr/lib64/lustre/tests/sanityn.sh: line 3163: [: -eq: unary operator expected sanityn test_77j: @@@@@@ FAIL: The write rate () exceeds 110% of preset rate (20) </pre> </div></div> <p>From the test log above, the format of dd output has been changed, so the parsing result is incorrect. I will make a patch sooner.</p> <p>Yingjin Qian (qian@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/27878" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27878</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: measure the runtime of dd directly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8e5988cdf728577db247afd98525a6bc1bcb7ad2</p> <p>another on b2_10:<br/> <a href="https://testing.hpdd.intel.com/test_sets/a9d32abe-9903-11e7-b778-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/a9d32abe-9903-11e7-b778-5254006e85c2</a></p> <p>another on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/bb51f0ae-991c-11e7-ba20-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bb51f0ae-991c-11e7-ba20-5254006e85c2</a></p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/bf2cc154-af15-11e7-943d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/bf2cc154-af15-11e7-943d-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/2c690a30-af38-11e7-9eeb-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2c690a30-af38-11e7-9eeb-5254006e85c2</a></p> <p>+1 on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/1da465c8-addd-11e7-bb19-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1da465c8-addd-11e7-bb19-5254006e85c2</a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/27878/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27878/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: measure the runtime of dd directly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 66abf6ffe9f032aaa7c271ee2bd1f9b569058bd5</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29686" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29686</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: measure the runtime of dd directly<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: bf27231ae6224598e2f04b2d937fd83c159dc58c</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29686/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29686/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9140" title="sanityn test_77j test_77k: The read / write rate exceeds x% of preset rate" class="issue-link" data-issue-key="LU-9140"><del>LU-9140</del></a> nrs: measure the runtime of dd directly<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 53604724c3e5dd2032303e22511f7be230d4feae</p> Duplicate LU-9308 Related LU-9227 Development Rank 1|hzz4lj: Rank (Obsolete) 9223372036854775807 Severity