https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3056 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite runs:<br/> <a href="https://maloo.whamcloud.com/test_sets/95fcddea-97b0-11e2-a652-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/95fcddea-97b0-11e2-a652-52540035b04c</a><br/> <a href="https://maloo.whamcloud.com/test_sets/810da798-9760-11e2-9ec7-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/810da798-9760-11e2-9ec7-52540035b04c</a></p> <p>The sub-test test_66 failed with the following error:</p> <blockquote> <p>replace nids failed</p></blockquote> <p>Info required for matching: conf-sanity 66</p> <p>All subsequent ZFS test suites (recovery-small, etc) fail with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Starting mds1: -o user_xattr,acl lustre-mdt1/mdt1 /mnt/mds1 CMD: wtm-16vm3 mkdir -p /mnt/mds1; mount -t lustre -o user_xattr,acl lustre-mdt1/mdt1 /mnt/mds1 wtm-16vm3: mount.lustre: according to /etc/mtab lustre-mdt1/mdt1 is already mounted on /mnt/mds1 </pre> </div></div> LU-3056

conf-sanity test_66 - replace nids failed

Bug Major Resolved Cannot Reproduce Keith Mannthey Maloo revzfs Thu, 28 Mar 2013 16:11:46 +0000 Fri, 22 Sep 2023 22:23:11 +0000 Fri, 21 Jun 2013 22:23:07 +0000 Lustre 2.4.0 0 8 <p>I asked Artem to look at this and he thinks <a href="https://jira.whamcloud.com/browse/LU-2988" title="conf-sanity 66: Modules still loaded" class="issue-link" data-issue-key="LU-2988"><del>LU-2988</del></a> might be the culprit, but is still confirming this theory.</p> <p>The error reason is because this statement doesn't return TRUE last call:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> only_mgs_is_running(struct obd_device *mgs_obd) { <span class="code-comment">/* TDB: Is global variable with devices count exists? */</span> <span class="code-object">int</span> num_devices = get_devices_count(); /* osd, MGS and MGC + self_export (wc -l /proc/fs/lustre/devices &lt;= 2) &amp;&amp; (num_exports &lt;= 2) */ <span class="code-keyword">return</span> (num_devices &lt;= 3) &amp;&amp; (mgs_obd-&gt;obd_num_exports &lt;= 2); } </pre> </div></div> <p>I am trying to figure out why this happens.</p> <p>I can't reproduce this bug locally. Statistics shows "Subtest passes: 99/100". Is it possible that something was launched in parallel with tests in that 1 failed test execution?</p> <p><a href="https://jira.whamcloud.com/browse/LU-2988" title="conf-sanity 66: Modules still loaded" class="issue-link" data-issue-key="LU-2988"><del>LU-2988</del></a> is useful for correct modules unloading.</p> <p>test_66 <br/> Error: 'replace nids failed'<br/> Failure Rate: 23.00% of last 100 executions <span class="error">&#91;all branches&#93;</span> </p> <p>23% makes me think this may need to be a blocker. </p> <p>In general there are no parallel tests. Tests are safe to assume they are the only thing running at this point in time. </p> <p>Just a quick breakdown for casual observers:</p> <p>From conf_sanity test_66 </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> echo "replace MDS nid" do_facet mgs $LCTL replace_nids $FSNAME-MDT0000 $MDS_NID || error "replace nids failed" </pre> </div></div> <p>And from that do_facet call we get:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: wtm-18vm3 /usr/sbin/lctl replace_nids lustre-MDT0000 10.10.16.188@tcp wtm-18vm3: error: replace_nids: Operation now in progress conf-sanity test_66: @@@@@@ FAIL: replace nids failed </pre> </div></div> <p>The error: replace... comes from here:<br/> In "utils/obd.c jt_replace_nids</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> rc = l2_ioctl(OBD_DEV_ID, OBD_IOC_REPLACE_NIDS, buf); if (rc &lt; 0) { fprintf(stderr, "error: %s: %s\n", jt_cmdname(argv[0]), strerror(rc = errno)); } </pre> </div></div> <p>This is seen on the MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 28590:0:(mgs_llog.c:1286:mgs_replace_nids()) Only MGS is allowed to be started </pre> </div></div> <p>And that leads to the above only_mgs_is_running and mgs_replace_nids in lustre/mgs/mgs_llog.c</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> /* We can not change nids if not only MGS is started */ if (!only_mgs_is_running(mgs_obd)) { CERROR("Only MGS is allowed to be started\n"); GOTO(out, rc = -EINPROGRESS); } </pre> </div></div> <p>Also </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> (wc -l /proc/fs/lustre/devices &lt;= 2) &amp;&amp; (num_exports &lt;= 2) */ return (num_devices &lt;= 3) &amp;&amp; (mgs_obd-&gt;obd_num_exports &lt;= 2); </pre> </div></div> <p>Should num_devices be 2 or 3? </p> <p>The untested debug patch can be found here: <a href="http://review.whamcloud.com/5940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5940</a> <br/> It is a rare path so it should not hurt.</p> <p>If we don't want to land the patch I will have it run conf_sanity alot. </p> <p>I haven't seen this conf-sanity/66 fail for any patch based past <a href="https://jira.whamcloud.com/browse/LU-2988" title="conf-sanity 66: Modules still loaded" class="issue-link" data-issue-key="LU-2988"><del>LU-2988</del></a>. Granted, at this point the sample set is pretty small at this point.</p> <p>Ahh thanks I didn't have lu-2988 on my radar. </p> <p>I checked as well and I don't see any errors in the past two days. </p> <p>Dup of <a href="https://jira.whamcloud.com/browse/LU-2988" title="conf-sanity 66: Modules still loaded" class="issue-link" data-issue-key="LU-2988"><del>LU-2988</del></a>. </p> <p>This issue is believed to be fixed. A patch was landed April 1st or so and no errors have been seen since. </p> <p>Please reopen if the errors continue. </p> <p>Failure on current master with fix for <a href="https://jira.whamcloud.com/browse/LU-2988" title="conf-sanity 66: Modules still loaded" class="issue-link" data-issue-key="LU-2988"><del>LU-2988</del></a>:<br/> <a href="https://maloo.whamcloud.com/test_sets/ae3e5ec6-a104-11e2-b1c3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/ae3e5ec6-a104-11e2-b1c3-52540035b04c</a></p> <p>this message in log </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">20000000:00020000:0.0:1365501872.365629:0:20267:0:(mgs_llog.c:1286:mgs_replace_nids()) Only MGS is allowed to be started</pre> </div></div> <p>Do anybody know what changed in code tree, so this become wrong?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-comment">/*(wc -l /proc/fs/lustre/devices &lt;= 3) &amp;&amp; (num_exports &lt;= 2) */</span> </pre> </div></div> <p>&gt;The untested debug patch can be found here: <a href="http://review.whamcloud.com/5940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5940</a> </p> <p>&gt;If we don't want to land the patch I will have it run conf_sanity alot.</p> <p>Keith, do you have output with patch applied and test failed?</p> <p>There is no debug output yet as the problem was thought fixed. I will revisit the patch. </p> <p><a href="http://review.whamcloud.com/6005" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6005</a> is a conf-sanity run with the debug patch. </p> <p><a href="http://review.whamcloud.com/5940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5940</a> has been rebased for possible inclusion.</p> <p>I definitely don't think this needs to be a 2.4.0 blocker, since replace_nids is a very rarely used code path. The only potential reason for increased priority might be the frequency to other patches failing due to this bug, but I don't see very many failures due to this specific bug (several other conf-sanity failures are increasing the test failure rates).</p> <p><a href="http://review.whamcloud.com/5940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5940</a> has been resubmitted for testing in the effort to land it after the 2.4 split. We still see the issue on Master a few times a week and it will be good to know more about out what is causing the issue.</p> <p>Quick update:</p> <p>conf_sanity test_66 has not failed in a few weeks. The "replace nids failed" error really dropped off after 2013-04-29. Perhaps some code path has changed. </p> <p>Still no sign of the "replace nids failed" errors. </p> <p>We no longer see this issue. Please reopen if this starts to trigger again. There is on sense landing a debug patch for a problem that does not happen. </p> Related LU-11990 LU-3793 LU-5137 Development Rank 1|hzvmkn: Rank (Obsolete) 7450 Severity