https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4200 Lustre <p>This issue was created by maloo for Nathaniel Clark &lt;nathaniel.l.clark@intel.com&gt;</p> <p>This issue relates to the following test suite run:<br/> <a href="http://maloo.whamcloud.com/test_sets/d95b0f48-264f-11e3-8f1c-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/d95b0f48-264f-11e3-8f1c-52540035b04c</a><br/> <a href="http://maloo.whamcloud.com/test_sets/2d26cba2-266f-11e3-b741-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/2d26cba2-266f-11e3-b741-52540035b04c</a></p> <p>The sub-test test_66 failed with the following error:</p> <blockquote> <p>replace nids failed</p></blockquote> <p>Info required for matching: conf-sanity 66</p> LU-4200

Test failure on test suite conf-sanity test_66: replace nids failed

Bug Major Resolved Fixed Nathaniel Clark Maloo Mon, 4 Nov 2013 14:07:59 +0000 Fri, 22 Sep 2023 22:26:33 +0000 Mon, 24 Mar 2014 17:15:39 +0000 Lustre 2.6.0 0 6 <p>another<br/> <a href="https://maloo.whamcloud.com/test_sets/9612221a-462c-11e3-b5e8-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/9612221a-462c-11e3-b5e8-52540035b04c</a></p> <p>again:<br/> <a href="https://maloo.whamcloud.com/test_sets/6d46bfb4-518d-11e3-9ca9-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6d46bfb4-518d-11e3-9ca9-52540035b04c</a></p> <p>It looks like this is failing at least twice a day in recent weeks.</p> <p>MDT/MGS debug log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>20000000:00000001:1.0:1386348576.067947:0:21057:0:(mgs_llog.c:1214:mgs_replace_nids()) Process entered 20000000:00020000:1.0:1386348576.068074:0:21057:0:(mgs_llog.c:1224:mgs_replace_nids()) Only MGS is allowed to be started 20000000:00000001:1.0:1386348576.068947:0:21057:0:(mgs_llog.c:1225:mgs_replace_nids()) Process leaving via out (rc=18446744073709551501 : -115 : 0xffffffffffffff8d) 20000000:00000001:1.0:1386348576.068947:0:21057:0:(mgs_llog.c:1265:mgs_replace_nids()) Process leaving (rc=18446744073709551501 : -115 : ffffffffffffff8d) </pre> </div></div> <p>Another failure instance:</p> <p><a href="https://maloo.whamcloud.com/test_sets/5f4a3ae6-8e8f-11e3-8d06-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/5f4a3ae6-8e8f-11e3-8d06-52540035b04c</a></p> <p>Debugging patch:<br/> <a href="http://review.whamcloud.com/9160" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9160</a></p> <p>I think this is caused by only_mgs_is_running() being confused by what it thinks the right value for obd_num_exports is. I think that obd_num_exports is being incremented because of LWP connections from the OSTs.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> only_mgs_is_running(struct obd_device *mgs_obd) { <span class="code-comment">/* TDB: Is global variable with devices count exists? */</span> <span class="code-object">int</span> num_devices = get_devices_count(); /* osd, MGS and MGC + self_export (wc -l /proc/fs/lustre/devices &lt;= 2) &amp;&amp; (num_exports &lt;= 2) */ <span class="code-keyword">return</span> (num_devices &lt;= 3) &amp;&amp; (mgs_obd-&gt;obd_num_exports &lt;= 2); } </pre> </div></div> <p>One option is to change only_mgs_is_running() to iterate over the MGS exports and return an error if any of them are not self or LWP connections.</p> <p><a href="http://review.whamcloud.com/9650" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9650</a></p> <p>Patch landed to master</p> <p>After this patch landed the test is failed in environment there mgs is dedicated to separate node. Connections from OSTs and MDSs are ignored because have OBD_CONNECT_MDS_MDS flag set. I am surprised why connection to MGS have this flag set, probably because actually OBD_CONNECT_MNE_SWAB is set and it have the same value.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define OBD_CONNECT_MNE_SWAB OBD_CONNECT_MDS_MDS </pre> </div></div> <p>Probably we could exclude OBD_CONNECT_MDS_MDS from exceptions, because if LWP connections exists, the we can't assume "MGS is alone". </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lustre/mgs/mgs_llog.c b/lustre/mgs/mgs_llog.c index e48f888..ef42e34 100644 --- a/lustre/mgs/mgs_llog.c +++ b/lustre/mgs/mgs_llog.c @@ -1165,8 +1165,6 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> only_mgs_is_running(struct obd_device *mgs_obd) <span class="code-comment">/* skip self export */</span> <span class="code-keyword">if</span> (exp == mgs_obd-&gt;obd_self_export) <span class="code-keyword">continue</span>; - <span class="code-keyword">if</span> (exp_connect_flags(exp) &amp; OBD_CONNECT_MDS_MDS) - <span class="code-keyword">continue</span>; ++num_exports; </pre> </div></div> <p> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a>, why have you ignore exports with OBD_CONNECT_MDS_MDS flag set? </p> <p>Thanks.</p> Related LU-11990 Development Rank 1|hzw7uv: Rank (Obsolete) 11419 Severity