https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8397 Lustre <p>mgsnode's list of nids is allowed to use comma as separator (see 36.15.3 of Lustre Operations Manual), however lustre_start_mgc() assumes that only colon can separate nids. Due to that nids after comma do not get into list of import's connections and those nids do not get involved.</p> LU-8397

take comma as separator of mgsnode's list

Bug Minor Resolved Fixed Emoly Liu Vladimir Saveliev Thu, 14 Jul 2016 12:55:14 +0000 Wed, 25 Oct 2017 18:35:30 +0000 Wed, 26 Apr 2017 07:21:59 +0000 Lustre 2.10.0 0 10 <p>Vladimir Saveliev (vladimir_saveliev@xyratex.com) uploaded a new patch: <a href="http://review.whamcloud.com/21308" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21308</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8397" title="take comma as separator of mgsnode&#39;s list" class="issue-link" data-issue-key="LU-8397"><del>LU-8397</del></a> mgc: take comma as separator of mgsnode's list<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d967d2d09409aaaf036abe3001457260aff15292</p> <p>Emoly</p> <p>Could you please take care of reviews etc for this patch?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Bobi,</p> <p>I found the following codes in <tt>lustre_start_mgc()</tt> introduced by patch <a href="https://review.whamcloud.com/7509" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/7509</a> for <del><a href="https://jira.whamcloud.com/browse/LU-3829" title="MDT mount fails if mkfs.lustre is run with multiple mgsnode arguments on MDSs where MGS is not running" class="issue-link" data-issue-key="LU-3829"><del>LU-3829</del></a></del> might have some issue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* * LU-3829. * Here we only take the first mgsnid as its primary * serving mgs node, the <span class="code-keyword">rest</span> mgsnid will be taken as * failover mgs node, otherwise they would be takens * as multiple nids of a single mgs node. */ <span class="code-keyword">while</span> (class_parse_nid(ptr, &amp;nid, &amp;ptr) == 0) { rc = do_lcfg(mgcname, nid, LCFG_ADD_UUID, niduuid, NULL, NULL, NULL); <span class="code-keyword">if</span> (rc == 0) { i = 1; <span class="code-keyword">break</span>; } } </pre> </div></div> <p>For multiple NIDs separated by <b>commas</b> on one MGS node, the above codes would only add the first NID and then break the while loop. After <tt>lustre_start_simple()</tt> is performed, the remaining <b>comma-separated</b> MGS NID(s) would not be added because the following condition doesn't work:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* Add any failover MGS nids */</span> i = 1; <span class="code-keyword">while</span> (ptr &amp;&amp; ((*ptr == <span class="code-quote">':'</span> || class_find_param(ptr, PARAM_MGSNODE, &amp;ptr) == 0))) { </pre> </div></div> <p><tt>ptr</tt> is previously assigned with <tt>ptr = lsi-&gt;lsi_lmd-&gt;lmd_mgs</tt>, and for comma-separated MGS NIDs, <tt>ptr</tt> now points to a ','. There is also no <tt>mgsnode=</tt> in the string, so the above condition does not work. All of the NIDs after the first comma would not be added.</p> <p>It seems this is the root cause of the latest issue reported by Darby in <del><a href="https://jira.whamcloud.com/browse/LU-8311" title="Target does not mount with the new mgsnode parameter format in case of multirail configuration" class="issue-link" data-issue-key="LU-8311"><del>LU-8311</del></a></del>. The <tt>ptr</tt> has a value of <tt>192.52.98.30@tcp,10.148.0.30@o2ib:192.52.98.31@tcp,10.148.0.31@o2ib</tt>, but from the debug log in debug.log.ib_and_eth, I can only see the first NID <tt>192.52.98.30@tcp</tt> was added.</p> <p>This evening I compiled the 2.9.0 release with patch 7905 reverted.  I can confirm that this did allow me to failover our OST's properly.  Once the OST were mounted with the NIDS shown in the comment above we see both TCP NIDS in /proc/fs/lustre/mgc/MGC192.52.98.30@tcp/import and the MGC connects with the secondary MDS/MGS when we failover.  </p> <p>Thank you very much Darby for the confirmation.</p> <p>I'm sorry - it turns out I was wrong when I said reverting this patch fixed our situation.  I think what happened is that I forgot to stop lnet before I uninstalled stock 2.9.0.  I then installed and tested the patched version of 2.9.0 - but the lnet kernel modules from stock 2.9.0 stayed loaded.  So I was using some weird combination of a patched and stock version of 2.9.0.  I've repeated the tests using a version of 2.9.0 with patch 7905 reverted and the situation is actually worse - with both single rail (tcp only) or dual rail (tcp and ib) I only see one entry for failover_nids in /proc/fs/lustre/mgc/MGC192.52.98.30@tcp/import.  I can repeat the tests and dump debug logs if that would help.  </p> <p>Hi Darby,<br/> I just updated patch <a href="https://review.whamcloud.com/21308" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/21308</a> on master branch to fix the issue in my previous comment. The patch can be applied to Lustre 2.9.0 release directly. Could you please try it? Thank you.</p> <p>I just tested 2.9.0 with this patch and it does indeed fix our issues.  I brought up our filesystem while configured on both TCP and IB, mounted both TCP and IB clients and failed over the MDS/MGS to the secondary.  The OST's and the clients all pick up the secondary MGS now.  Info from one of our OSS's is below - this is after the failover.  Thank you very much!  </p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@hpfs-fsl-oss01 ~]# cat /proc/fs/lustre/mgc/MGC192.52.98.30@tcp/import import: name: MGC192.52.98.30@tcp target: MGS state: FULL connect_flags: [ version, adaptive_timeouts, mds_mds_connection, full20, imp_recov, bulk_mbits ] connect_data: flags: 0x2000011005000020 instance: 0 target_version: 2.9.0.0 import_flags: [ pingable, connect_tried ] connection: failover_nids: [ 192.52.98.31@tcp, 192.52.98.30@tcp ] current_connection: 192.52.98.31@tcp connection_attempts: 3 generation: 2 in-progress_invalidations: 0 [root@hpfs-fsl-oss01 ~]# zfs get all | grep mgs oss01-0/ost-fsl lustre:mgsnode 192.52.98.30@tcp,10.148.0.30@o2ib:192.52.98.31@tcp,10.148.0.31@o2ib local [root@hpfs-fsl-oss01 ~]# </pre> </div></div> <p>Thank you very much, Darby. I'll push the patch to be ready to land.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/21308/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/21308/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8397" title="take comma as separator of mgsnode&#39;s list" class="issue-link" data-issue-key="LU-8397"><del>LU-8397</del></a> mgc: add comma-separated nids for primary MGS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 672e182049e1a636d8591748764950f99c4b71d9</p> <p>Patch landed to master branch for Lustre 2.10.0.</p> Related LU-3829 LU-10162 Development Rank 1|hzyhg7: Rank (Obsolete) 9223372036854775807 Severity