https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4341 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/7756e5f2-5bb9-11e3-8d79-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/7756e5f2-5bb9-11e3-8d79-52540035b04c</a>.</p> <p>The sub-test test_170 failed with the following error:</p> <blockquote> <p>expected 31 bad lines, but got 34</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 170: test lctl df to handle corrupted log ============================================= 00:50:22 (1385974222) sanity test_170: @@@@@@ FAIL: expected 31 bad lines, but got 34 </pre> </div></div> server and client: lustre-master build # 1784 <br/> client is running SLES11 SP3 LU-4341

Failure on test suite sanity test_170: expected 31 bad lines, but got 34

Bug Blocker Resolved Cannot Reproduce Jian Yu Maloo always_except Tue, 3 Dec 2013 21:23:37 +0000 Tue, 14 Dec 2021 22:51:13 +0000 Tue, 14 Dec 2021 22:51:13 +0000 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.5.1 Lustre 2.7.0 Lustre 2.5.3 Lustre 2.8.0 0 11 <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/5/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/5/</a><br/> Distro/Arch: RHEL6.4/x86_64(server), SLES11SP3/x86_64(client)</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/0c73c65e-763c-11e3-b3c0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0c73c65e-763c-11e3-b3c0-52540035b04c</a></p> <p>More instances on Lustre b2_5 branch:<br/> <a href="https://maloo.whamcloud.com/test_sets/ba50cbe6-7ecf-11e3-925a-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ba50cbe6-7ecf-11e3-925a-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/c2ca6dea-908b-11e3-a134-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/c2ca6dea-908b-11e3-a134-52540035b04c</a></p> <p>This failure kept occurring on Lustre b2_5 branch in SLES11SP3/x86_64 client test session:<br/> <a href="https://maloo.whamcloud.com/test_sets/ee128066-990d-11e3-968c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ee128066-990d-11e3-968c-52540035b04c</a></p> <p>Lustre Build: <a href="http://build.whamcloud.com/job/lustre-b2_5/40/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-b2_5/40/</a> (2.5.1 RC2)<br/> Distro/Arch: RHEL6.5/x86_64(server), SLES11SP3/x86_64(client)</p> <p>The same failure occurred:<br/> <a href="https://maloo.whamcloud.com/test_sets/97d92b6a-a663-11e3-aac5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/97d92b6a-a663-11e3-aac5-52540035b04c</a></p> <p>another in b2_5<br/> <a href="https://maloo.whamcloud.com/test_sessions/bb218674-cc67-11e3-bda1-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/bb218674-cc67-11e3-bda1-52540035b04c</a></p> <p>Yu, Jian</p> <p>This seems to be occurring sporadically and when it does it causes review failures. Could you please look into what kind of circumstances trigger these failures?</p> <p>Thanks</p> <p>Peter</p> <p>The failure occurred on the following test sessions on Lustre b2_5 and master branches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>SLES11SP2 client + RHEL6.5 server SLES11SP3 client + RHEL6.5 server SLES11SP3 client + SLES11SP3 server (only on master branch) </pre> </div></div> <p>I'll look into the failure.</p> <p>I think this is another, but says: Error: 'expected 24 bad lines, but got 27' instead of expected 31 bad lines, but got 34,</p> <p>sles11sp3 client in b2_5:<br/> <a href="https://maloo.whamcloud.com/test_sets/63de2e74-cf07-11e3-a250-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/63de2e74-cf07-11e3-a250-52540035b04c</a></p> <p>starting to wonder if this is a high rate failure, maybe even 100%, in any sles client.</p> <p>another sles11sp3 client in master:<br/> <a href="https://maloo.whamcloud.com/test_sets/a9add412-d0ac-11e3-b9d4-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/a9add412-d0ac-11e3-b9d4-52540035b04c</a></p> <p>another sles11sp3 client in master:<br/> <a href="https://maloo.whamcloud.com/test_sets/fd386500-d167-11e3-91ff-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/fd386500-d167-11e3-91ff-52540035b04c</a></p> <p>There is a defect in sanity test_170(), and here is a patch for master branch to fix it: <a href="http://review.whamcloud.com/10296" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10296</a><br/> Since the failure cannot be reproduced by only running sanity test 170, I'm checking the previous sub-tests to see which one is the culprit.</p> <p>Finally, I found that it was sanity test 150 which caused test 170 fail on SLES11SP3 client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>run_test 150 "truncate/append tests" </pre> </div></div> <p>I've tried several ways to fix the issue but failed. Still digging.</p> <p>Just narrowed down that it was the following operation in sanity test 150 which caused test 170 fail:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>remount_client $MOUNT -&gt; zconf_mount `hostname` $1 -&gt; set_default_debug_nodes $client </pre> </div></div> <p>After commenting out "set_default_debug_nodes $client", the failure disappeared.</p> <p>It turns out that sanity test 170 is affected by the lctl debug value. With debug=-1, it passed, and with debug="rpctrace", the test failed.</p> <p>In sanity.sh, debug=-1 is set before running sub-tests, which is why only running test 170 passed. In test 150, "set_default_debug_nodes $client" made the debug value change to debug="vfstrace rpctrace dlmtrace neterror ha config ioctl super", which caused test 170 fail.</p> <p>Hi Di,</p> <p>I saw that sanity test_170() was added by you in commit d9bf86ae95a599bf10bbb05818317b48eb71db1b. Could you please give me some hints about why debug="rpctrace" affects the test results of sanity test 170 on SLES client? Thanks a lot.</p> <p>Hi, Yujian</p> <p>test_170 is supposed to verify "lctl df can identify and skip corrupted debug records", instead of abandon the whole debug log file. I guess there are some debug format problem for "rpctrace", though not sure what is the real reason here, I think you need get "$TMP/${file}_log_good" to have a look or follow the test step to repeat the test locally? Thanks.</p> <p>Thanks, Di. I can reproduce the failure every time with debug="rpctrace". I'll look into "$TMP/${tfile}_log_good".</p> <p>Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-b2_5/86/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-b2_5/86/</a> (2.5.3 RC1)</p> <p>The same failure occurred:<br/> <a href="https://testing.hpdd.intel.com/test_sets/b2a57a6e-30a3-11e4-9f57-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b2a57a6e-30a3-11e4-9f57-5254006e85c2</a></p> <p>seen in master with sles11sp3 client/server:<br/> <a href="https://testing.hpdd.intel.com/test_sets/14d6386e-8c7e-11e4-b81b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/14d6386e-8c7e-11e4-b81b-5254006e85c2</a></p> <p>hit this error in tag-2.6.94 test:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/c53f0196-b22a-11e4-af8e-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c53f0196-b22a-11e4-af8e-5254006e85c2</a></p> <p>another seen in master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/83277a58-d3cd-11e4-8c98-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/83277a58-d3cd-11e4-8c98-5254006e85c2</a></p> <p>another instance<br/> <a href="https://testing.hpdd.intel.com/test_sets/834ace12-d75c-11e4-a678-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/834ace12-d75c-11e4-a678-5254006e85c2</a></p> <p>another instance:<br/> <a href="https://testing.hpdd.intel.com/test_sets/02e36236-fe29-11e4-be9d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/02e36236-fe29-11e4-be9d-5254006e85c2</a></p> <p>Can someone please definitively understand why this test is failing for SLES, and either fix it or add it to the ALWAYS_EXCEPT list for SLES. It doesn't make sense to exclude this via <tt>envdefinitions</tt> for SLES patches, when it will still fail when someone forgets to except it.</p> <p>Bob Glossman (bob.glossman@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/16146" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16146</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4341" title="Failure on test suite sanity test_170: expected 31 bad lines, but got 34" class="issue-link" data-issue-key="LU-4341"><del>LU-4341</del></a> test: skip failing sanity test 170<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7f5668027d0b1393640c8185e8084c2957c8bdbe</p> <p>best I can do for now is push a mod to ALWAYS_EXCEPT on sles11</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16146/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16146/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-4341" title="Failure on test suite sanity test_170: expected 31 bad lines, but got 34" class="issue-link" data-issue-key="LU-4341"><del>LU-4341</del></a> test: skip failing sanity test 170<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ef63c034b437d47cd10fe7ee94ed614ac1359f44</p> <p>This appears to be failing on master on SLES11.3 tests in the past week:<br/> <a href="https://testing.hpdd.intel.com/sub_tests/05b5baa6-73f7-11e5-ada9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/05b5baa6-73f7-11e5-ada9-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/07fb912c-73df-11e5-ab44-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/07fb912c-73df-11e5-ab44-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/9c5611ca-73ea-11e5-ab44-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/9c5611ca-73ea-11e5-ab44-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/bae5eb2c-722f-11e5-b344-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/bae5eb2c-722f-11e5-b344-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/49587898-7073-11e5-b705-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/49587898-7073-11e5-b705-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/8bfb6e00-7071-11e5-b705-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/8bfb6e00-7071-11e5-b705-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/sub_tests/31313890-6f24-11e5-83a9-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/31313890-6f24-11e5-83a9-5254006e85c2</a></p> <p>The patch <a href="http://review.whamcloud.com/16146" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16146</a> has landed, but that doesn't solve the problem itself. An improved patch would only skip the "expected N bad lines, but got M" check in test_170 for SLES11.3+ and RHEL7 as well rather than the whole test_170. While in there, it should also remove the "-rf" from <tt>rm &#45;rf $DIR/$tfile</tt> since that is a file and not a directory and shouldn't fail in any case.</p> <p>One possible source of the bug is that the first <tt>cat $TMP/${tfile}&#95;log&#95;good &gt;&gt; $TMP/${tfile}&#95;logs&#95;corrupt</tt> is appending to a file (<tt>&gt;&gt;</tt>) instead of first truncating it (<tt>&gt;</tt>) so if the ${tfile}_logs_corrupt file is lingering around from a previous test run for some reason it might cause problems. It does seem like the number of bad lines is always higher than the number of expected lines, so this seems like a candidate.</p> <p>In any case, this bug cannot be closed until the actual test failure is understood and fixed.</p> Related Development Rank 1|hzwaj3: Rank (Obsolete) 11880 Severity