https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5706 Lustre <p>This issue was created by maloo for Bruno Faccini &lt;bfaccini62@gmail.com&gt;</p> <p>Please provide additional information about the failure here.</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/637ff760-4ac1-11e4-a839-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/637ff760-4ac1-11e4-a839-5254006e85c2</a>.</p> <p>Seems that recent master reviews builds hit this failure quite frequently. A recent master change may have introduced a regression,</p> LU-5706

conf-sanity test_57a: @@@@@@ FAIL: OST registration from failnode should fail

Bug Blocker Resolved Fixed Nathaniel Clark Maloo zfs Sat, 4 Oct 2014 14:30:51 +0000 Mon, 27 Oct 2014 18:55:41 +0000 Mon, 27 Oct 2014 18:55:41 +0000 Lustre 2.7.0 Lustre 2.5.4 Lustre 2.7.0 0 10 <p>Since 2014-10-02, the failure has been preventing Lustre b2_5 patches from passing review testing with FSTYPE=zfs:<br/> <a href="https://testing.hpdd.intel.com/test_sets/359d7a4e-4a75-11e4-95b1-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/359d7a4e-4a75-11e4-95b1-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/97cdd432-4b03-11e4-8d48-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/97cdd432-4b03-11e4-8d48-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/8709c0fc-4b0d-11e4-b999-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8709c0fc-4b0d-11e4-b999-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/40a1872c-4b98-11e4-b01b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/40a1872c-4b98-11e4-b01b-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/1636091a-4bb9-11e4-b01b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1636091a-4bb9-11e4-b01b-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/e865aef8-4bc4-11e4-b13d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e865aef8-4bc4-11e4-b13d-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/55c870b0-4be9-11e4-b01b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/55c870b0-4be9-11e4-b01b-5254006e85c2</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== conf-sanity test 57a: initial registration from failnode should fail (should return errs) == 15:21:43 (1412263303) CMD: shadow-42vm4 /usr/sbin/lctl get_param nis shadow-42vm4: error: get_param: /proc/{fs,sys}/{lnet,lustre}/nis: Found no match CMD: shadow-42vm3 grep -c /mnt/mds1' ' /proc/mounts CMD: shadow-42vm3 lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' CMD: shadow-42vm3 ! zpool list -H lustre-mdt1 &gt;/dev/null 2&gt;&amp;1 || grep -q ^lustre-mdt1/ /proc/mounts || zpool export lustre-mdt1 CMD: shadow-42vm3 tunefs.lustre --quiet --writeconf lustre-mdt1/mdt1 shadow-42vm3: shadow-42vm3: tunefs.lustre FATAL: Device lustre-mdt1/mdt1 has not been formatted with mkfs.lustre shadow-42vm3: tunefs.lustre: exiting with 19 (No such device) </pre> </div></div> <p>Another failure instance:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/c54fa3f0-4c01-11e4-bb84-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c54fa3f0-4c01-11e4-bb84-5254006e85c2</a></p> <p>Another failure on review-zfs: <a href="https://testing.hpdd.intel.com/test_sets/47c826f8-4bef-11e4-bb84-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/47c826f8-4bef-11e4-bb84-5254006e85c2</a></p> <p>Nathaniel</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>I think this definitely was introduced by <a href="https://jira.hpdd.intel.com/browse/LU-4749" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-4749</a> as there s a huge uptick in these failures since end of September in master: <a href="https://testing.hpdd.intel.com/sub_tests/query?commit=Update+results&amp;page=1&amp;sub_test[query_bugs]=&amp;sub_test[status]=FAIL&amp;sub_test[sub_test_script_id]=dc46d338-6c5a-11e0-b32b-52540025f9af&amp;test_node[architecture_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=&amp;test_node[os_type_id]=&amp;test_node_network[network_type_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=&amp;test_session[test_group]=&amp;test_session[test_host]=&amp;test_session[user_id]=&amp;test_set[test_set_script_id]=7f66aa20-3db2-11e0-80c0-52540025f9af&amp;utf8=%E2%9C%93" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/query?commit=Update+results&amp;page=1&amp;sub_test[query_bugs]=&amp;sub_test[status]=FAIL&amp;sub_test[sub_test_script_id]=dc46d338-6c5a-11e0-b32b-52540025f9af&amp;test_node[architecture_type_id]=&amp;test_node[distribution_type_id]=&amp;test_node[file_system_type_id]=&amp;test_node[lustre_branch_id]=&amp;test_node[os_type_id]=&amp;test_node_network[network_type_id]=&amp;test_session[query_date]=&amp;test_session[query_recent_period]=&amp;test_session[test_group]=&amp;test_session[test_host]=&amp;test_session[user_id]=&amp;test_set[test_set_script_id]=7f66aa20-3db2-11e0-80c0-52540025f9af&amp;utf8=%E2%9C%93</a></p> <p>page 2 does not show, but page 3 starts in 2013...</p> <p>Oleg, have you reverted <a href="http://review.whamcloud.com/11956" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11956</a> from master and <a href="http://review.whamcloud.com/12196" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12196</a> from b2_5 yet? This is causing frequent test failures on review-zfs tests.</p> <p>I don't think this was introduced by my <a href="https://jira.whamcloud.com/browse/LU-4749" title="ZFS-backed OST mkfs.lustre --servicenode does not correctly add failover_nids" class="issue-link" data-issue-key="LU-4749"><del>LU-4749</del></a> patch. As Yu Jian suggested early on, errors started to occur from the very beginning (taken from a recent <a href="https://testing.hpdd.intel.com/sub_tests/8000c0b0-4e0c-11e4-8fdd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">Maloo report</a>):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== conf-sanity test 57a: initial registration from failnode should fail (should return errs) == 09:56:58 (1412675818) CMD: shadow-42vm4 /usr/sbin/lctl get_param nis shadow-42vm4: error: get_param: /proc/{fs,sys}/{lnet,lustre}/nis: Found no match [...] </pre> </div></div> <p>This resulted in an empty string in "NID", which was then used to generate the following command line:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[...] CMD: shadow-42vm4 tunefs.lustre --failnode= lustre-ost1/ost1 checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x2 (OST ) Persistent mount opts: Parameters: mgsnode=10.1.5.248@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=10.1.5.248@tcp sys.timeout=20 failover.node=��Yi: Writing lustre-ost1/ost1 properties lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 lustre:mgsnode=10.1.5.248@tcp lustre:sys.timeout=20 lustre:failover.node=��Yi: [...] </pre> </div></div> <p>Notice that the empty "failnode" argument wasn't caught by tunefs.lustre. (Instead, convert_hostnames() returned garbage when given an empty string---a problem that needs to be fixed as well.)</p> <p>As to why "nis" did not exist, could it be that test_57a() has been (incorrectly) depending on prior tests to leave modules loaded? Consider these:</p> <ul> <li>This failure did not happen on ldiskfs where 54a to 56 were not skipped.</li> <li>This failure happened on ZFS with 54a to 56 skipped (see <a href="https://testing.hpdd.intel.com/sub_tests/8000c0b0-4e0c-11e4-8fdd-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/sub_tests/8000c0b0-4e0c-11e4-8fdd-5254006e85c2</a>).</li> <li>This failure did not happen on ZFS with only 54a to 55 (but not 56) skipped (see <a href="https://testing.hpdd.intel.com/test_sets/5f54ae08-4e07-11e4-ae94-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/5f54ae08-4e07-11e4-ae94-5254006e85c2</a>). test_56()'s reformat() call loaded the modules for test_57a() in this case.</li> </ul> <p>Now the question is if "excepting tests: 32newtarball 56 export 76a 59 64 69" started to appear around the time Oleg suspected.</p> <p>Andreas: the 12196 did not land to b2_5 yet. And I was under impression there are no b2_5 failures outside of select patches that include it, but alas, apparently there are still some failures so it must be some different recently landed thing that causes this bug.</p> <p>LiWei, test 54a is skipped for zfs since Oct 2012, so it cannot be the trigger.<br/> test 56 is not in always exclude list in b2_5 at least, so I wonder if it's also excluded manually in the testing system via a TEI ticket?</p> <p>Oleg, I wasn't implying 54a to be the problem, but one has to know which was the last not-skipped test before 57a. As to 56, I've no idea how it got into the exception list too. (We should really change exception lists only with Git commits.)</p> <p>This is causing a large number of review-zfs test failures at this point (20 yesterday), so either the original patch should be reverted, or this test disabled since we are wasting our time. This isn't the only problem with review-zfs, but only about 15% of these test sessions are passing right now.</p> <p><a href="http://review.whamcloud.com/12236" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12236</a></p> <p>The "56" in the exception list <em>was</em> indeed added by Autotest. Minh helped me remove that exception last Saturday. Now it looks this failure has gone (recent three review-zfs runs):</p> <p> <a href="https://testing.hpdd.intel.com/test_sets/aa8077d8-5224-11e4-a79f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/aa8077d8-5224-11e4-a79f-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/3d7c7ca8-5211-11e4-a79f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/3d7c7ca8-5211-11e4-a79f-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/2a42a2de-51d0-11e4-88de-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/2a42a2de-51d0-11e4-88de-5254006e85c2</a></p> <p>This buys us more time to cook <a href="http://review.whamcloud.com/12236" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12236</a>.</p> <p>patch landed to master</p> Related Development Rank 1|hzwxu7: Rank (Obsolete) 15995 Severity