https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1724 Lustre <p>This issue was created by maloo for sarah &lt;sarah@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/0596b3e0-dcd3-11e1-8744-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/0596b3e0-dcd3-11e1-8744-52540035b04c</a>.</p> <p>The sub-test test_3 failed with the following error:</p> <blockquote> <p>test_3 returned 1</p></blockquote> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:24:41:Lustre: DEBUG MARKER: performance-sanity test_3: @@@@@@ FAIL: test_3 failed with 10 </pre> </div></div> LU-1724

Test failure on test suite performance-sanity, subtest test_3

Bug Blocker Resolved Duplicate Keith Mannthey Maloo Wed, 8 Aug 2012 18:41:03 +0000 Mon, 13 Aug 2012 12:20:48 +0000 Mon, 13 Aug 2012 12:20:48 +0000 Lustre 2.3.0 0 3 <p>This error may caused by the previous failure of test mds-survey</p> <p>What happened to client-27vm3 of this test run? It was the mds and it paniced or ???? </p> <p>There is a previous failure of mds-survey, it may cause the MDS abnormal.</p> <p><a href="https://maloo.whamcloud.com/test_sets/7fa0e0bc-dcd2-11e1-8744-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7fa0e0bc-dcd2-11e1-8744-52540035b04c</a></p> <p>from the MDS of the 2nd run</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00:54:47:Lustre: DEBUG MARKER: == mds-survey test 2: Metadata survey with stripe_count = 1 == 00:54:45 (1343894085) 00:54:50:Lustre: DEBUG MARKER: lctl dl 00:54:52:LustreError: 17365:0:(echo_client.c:1607:echo_md_lookup()) lookup tests: rc = -2 00:54:52:LustreError: 17365:0:(echo_client.c:1607:echo_md_lookup()) Skipped 2 previous similar messages 00:54:52:LustreError: 17365:0:(echo_client.c:1806:echo_md_destroy_internal()) Can't find child tests: rc = -2 00:54:52:LustreError: 17365:0:(echo_client.c:1806:echo_md_destroy_internal()) Skipped 2 previous similar messages 00:54:54:LustreError: 17387:0:(echo_client.c:1607:echo_md_lookup()) lookup tests1: rc = -2 00:54:54:LustreError: 17387:0:(echo_client.c:1806:echo_md_destroy_internal()) Can't find child tests1: rc = -2 01:04:40:lctl invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 </pre> </div></div> <p>The OOM killer was out running around. Once OOM is running the system behaviour becomes non-deterministic as it may choose different processes to kill. </p> <p>Why is the MDS running of of memory? </p> <p>I think that the theory is due to <a href="https://jira.whamcloud.com/browse/LU-1548" title="Test failure on test suite mds-survey, subtest test_1" class="issue-link" data-issue-key="LU-1548"><del>LU-1548</del></a>. Do you think that it is reasonable to consider this a follow-on issue, close as a duplicate and reopen if it reoccurs with the <a href="https://jira.whamcloud.com/browse/LU-1548" title="Test failure on test suite mds-survey, subtest test_1" class="issue-link" data-issue-key="LU-1548"><del>LU-1548</del></a> fix applied?</p> <p>After reviewing <a href="https://jira.whamcloud.com/browse/LU-1548" title="Test failure on test suite mds-survey, subtest test_1" class="issue-link" data-issue-key="LU-1548"><del>LU-1548</del></a> I feel these are very likely the same issue: MDS OOM under the right conditions. I think duping it is the right way to go. There are no MDS logs from the first failure but OOM can cause the network to drop and the 2nd failure was clearly OOM. </p> <p>Closing as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-1548" title="Test failure on test suite mds-survey, subtest test_1" class="issue-link" data-issue-key="LU-1548"><del>LU-1548</del></a>. Please reopen if this reoccurs with that issue fixed</p> Development Rank 1|hzvgov: Rank (Obsolete) 6353 Severity