https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4409 Lustre <p>This issue was created by maloo for wangdi &lt;di.wang@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/e08778b8-6a67-11e3-9248-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/e08778b8-6a67-11e3-9248-52540035b04c</a>.</p> <p>The sub-test test_10 failed with the following error:</p> <blockquote> <p>Lustre: DEBUG MARKER: /usr/sbin/lctl mark == insanity test 10: Tenth Failure Mode: MDT0\/OST\/MDT1 Fri Dec 20 15:09:13 PST 2013 == 15:09:13 &#40;1387580953&#41;<br/> Lustre: DEBUG MARKER: == insanity test 10: Tenth Failure Mode: MDT0/OST/MDT1 Fri Dec 20 15:09:13 PST 2013 == 15:09:13 (1387580953)<br/> Lustre: DEBUG MARKER: grep -c /mnt/mds1' ' /proc/mounts<br/> Lustre: DEBUG MARKER: umount -d /mnt/mds1<br/> Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST '<br/> Lustre: DEBUG MARKER: hostname<br/> Lustre: DEBUG MARKER: mkdir -p /mnt/mds1<br/> Lustre: DEBUG MARKER: test -b /dev/lvm-Role_MDS/P1<br/> Lustre: DEBUG MARKER: mkdir -p /mnt/mds1; mount -t lustre -o user_xattr,acl /dev/lvm-Role_MDS/P1 /mnt/mds1<br/> LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. quota=on. Opts: <br/> LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 10.10.16.180@tcp, operation ost_connect failed with -16.<br/> LustreError: Skipped 4 previous similar messages<br/> Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u<br/> LNet: 20318:0:(debug.c:218:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.<br/> LNet: 20318:0:(debug.c:218:libcfs_debug_str2mask()) Skipped 3 previous similar messages<br/> Lustre: DEBUG MARKER: e2label /dev/lvm-Role_MDS/P1 2&gt;/dev/null<br/> Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 5 clients reconnect<br/> Lustre: 7793:0:(client.c:1903:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1387581110/real 1387581110&#93;</span> req@ffff88005dd0ec00 x1454975111842816/t0(0) o38-&gt;lustre-MDT0001-osp-MDT0000@10.10.16.183@tcp:24/4 lens 400/544 e 0 to 1 dl 1387581115 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1<br/> Lustre: 7793:0:(client.c:1903:ptlrpc_expire_one_request()) Skipped 12 previous similar messages<br/> Lustre: lustre-MDT0000: Recovery over after 0:21, of 5 clients 5 recovered and 0 were evicted.<br/> LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 10.10.16.180@tcp, operation ost_connect failed with -16.<br/> LustreError: Skipped 122 previous similar messages<br/> LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 10.10.16.180@tcp, operation ost_connect failed with -16.<br/> LustreError: Skipped 120 previous similar messages<br/> LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 10.10.16.180@tcp, operation ost_connect failed with -16.<br/> LustreError: Skipped 120 previous similar messages<br/> LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 10.10.16.180@tcp, operation ost_connect failed with -16.<br/> LustreError: Skipped 120 previous similar messages<br/> LustreError: 11-0: lustre-OST0000-osc-MDT0000: Communicating with 10.10.16.180@tcp, operation ost_connect failed with -16.<br/> LustreError: Skipped 119 previous similar messages<br/> SysRq : Show State</p> </blockquote> <p>Info required for matching: insanity 10</p> LU-4409

insanity test_10 (MDT0/OST/MDT1)

Bug Blocker Resolved Fixed WC Triage Maloo dne Sat, 21 Dec 2013 18:21:09 +0000 Fri, 22 Dec 2017 11:02:23 +0000 Fri, 22 Dec 2017 11:02:23 +0000 Lustre 2.6.0 0 4 <p>Disable the test now, <a href="http://review.whamcloud.com/8650" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8650</a>, will fix this after feature freeze.</p> <p>The first failure of this test is 2013-12-07 in:<br/> <a href="https://maloo.whamcloud.com/test_sets/6eadd65c-5f5d-11e3-85c5-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6eadd65c-5f5d-11e3-85c5-52540035b04c</a><br/> which is a test run for a patch that hasn't landed yet. There is a later test run for that patch which did not hit this problem, and the patch is not yet landed, so it cannot be the cause if failures on master. </p> <p>The master test failures only started on master on 2013-12-18 and has been hit on several different patches (none of them landed), so it is likely a regression landed to master. While it failed the 3 most recent tests, there are quite a number of passing runs for insanity test_10() over the past few weeks. </p> <p>Hmm, I checked the debug log, it seems in this test, it started OST before MGS(MDT0), so OST use local config log to setup MDT, then it missed the recovery process. I do not know why it just happened recently. I think we have two options here, one is always start MDT0(MGS) during the test, or using " -o abort_recov" to start MDT/OST in this test. </p> <p>I just updated the patch 8650 to fix the problem. <a href="http://review.whamcloud.com/#/c/8650/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/8650/</a></p> <p>This subtest is currently disabled at the autotest level, so a regular test run will skip it. We need to explicitly request testing on the patch - hopefully Test-Parameters works. </p> <p>Is this ticket still an open problem? A patch for <a href="https://jira.whamcloud.com/browse/LU-2059" title="mgc to backup configuration on osd-based llogs" class="issue-link" data-issue-key="LU-2059"><del>LU-2059</del></a>, <a href="http://review.whamcloud.com/#/c/10311" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10311</a>, re-enabled insanity test 10 for all cases except when there are less than two MDTs. Yet, insanity test 10 is still skipped since it is disabled in autotest; TEI-1312. </p> <p>Insanity test 10 is passing in recent full test sessions:<br/> <a href="https://testing.hpdd.intel.com/test_sessions/d7711864-b904-11e4-a983-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/d7711864-b904-11e4-a983-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/63504c66-b8d7-11e4-a983-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/63504c66-b8d7-11e4-a983-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sessions/dba0429c-b590-11e4-9366-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/dba0429c-b590-11e4-9366-5254006e85c2</a></p> <p>I'm happy if you can renable the test in autotest. </p> <p>Test_10 has been re-enabled by <a href="http://review.whamcloud.com/10311" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10311</a>. So I guess this ticket has been fixed by <a href="https://jira.whamcloud.com/browse/LU-2059" title="mgc to backup configuration on osd-based llogs" class="issue-link" data-issue-key="LU-2059"><del>LU-2059</del></a>.</p> <p>Closer investigation will show that insanity test_10 is still being skipped for<br/> <a href="https://testing.hpdd.intel.com/sub_tests/query?utf8=%E2%9C%93&amp;test_set%5Btest_set_script_id%5D=ba1365ac-4a53-11e0-a7f6-52540025f9af&amp;sub_test%5Bsub_test_script_id%5D=bab06398-4a53-11e0-a7f6-52540025f9af&amp;sub_test%5Bstatus%5D=&amp;sub_test%5Bquery_bugs%5D=&amp;test_session%5Btest_host%5D=&amp;test_session%5Btest_group%5D=&amp;test_session%5Buser_id%5D=&amp;test_session%5Bquery_date%5D=&amp;test_session%5Bquery_recent_period%5D=86400&amp;test_node%5Bos_type_id%5D=&amp;test_node%5Bdistribution_type_id%5D=&amp;test_node%5Barchitecture_type_id%5D=&amp;test_node%5Bfile_system_type_id%5D=&amp;test_node%5Blustre_branch_id%5D=&amp;test_node_network%5Bnetwork_type_id%5D=&amp;commit=Update+results" class="external-link" target="_blank" rel="nofollow noopener">all recent lustre-review test runs</a>, with either:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>skipping ALWAYS excluded test 10 needs &gt;= 2 MDTs </pre> </div></div> <p>Since it is currently passing on full test runs, it seems safe enough to file a TEI ticket to request that test_10 be removed from the autotest exception list.</p> <p>TEI-3188 requests insanity test 10 be re-enabled in autotest.</p> Related LU-4274 Development Rank 1|hzwbrr: Rank (Obsolete) 12103 Severity