https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10124 Lustre <p>When importing a yaml config file for peers the import does not correctly set the Multi-Rail property when it is false.</p> <p>An example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">peer: - primary nid: 10.112.1.60@o2ib8 Multi-Rail: False peer ni: - nid: 10.112.1.60@o2ib8 state: up </pre> </div></div> <p>When imported it results in a running config of:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">peer: - primary nid: 10.112.1.60@o2ib8 Multi-Rail: True peer ni: - nid: 10.112.1.60@o2ib8 state: up </pre> </div></div> <p>For our config this isn't an issue yet, but as we will have a mix of multi-rail and non-multi-rail nodes this could be an issue moving forward.</p> Centos 7.4 LU-10124

lnetctl: lnetctl import --add not importing peers correctly

Bug Minor Resolved Fixed Sonia Sharma Malcolm Haak - NCI lnet lnetctl Mon, 16 Oct 2017 23:42:56 +0000 Tue, 9 Jun 2020 14:50:55 +0000 Fri, 21 Sep 2018 15:13:56 +0000 Lustre 2.10.0 Lustre 2.10.1 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.10.7 0 6 <p>We have encountered a second issue when importing peers.</p> <p>Two peers with a single primary nid were merged during import into one peer with two peer ni's</p> <p>The YAML file being used was one that was exported from the lnet router it was imported on.</p> <p>Not sure if it is a race condition when importing 4000+ peers or some other issue.</p> <p>Sonia, <br/> Would you please look into this issue?</p> <p>Thank you.</p> <p>Sonia Sharma (sonia.sharma@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31138" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31138</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10124" title="lnetctl: lnetctl import --add not importing peers correctly" class="issue-link" data-issue-key="LU-10124"><del>LU-10124</del></a> lnet: Correctly add peer MR value while importing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ddd0aedbe7a26053ae9c3da2c82c9e7a3e5a7c76</p> <p>This patch should fix up that one issue,</p> <p>But the second issue of peer merging does not appear to be solved.</p> <p>Did you want a second ticket for this issue?</p> <p>I will get you some log lines from the affected nodes.</p> <p>log line on lnet router is:</p> <p>LNetError: 8507:0:(peer.c:806:lnet_add_peer_ni_to_prim_lpni()) Cannot add NID 10.9.60.1@o2ib3 owned by peer 10.9.60.1@o2ib3 to peer 10.9.12.38@o2ib3</p> <p>The two 10.9.x.x addresses mentioned correspond to adjacent entries in the config file.</p> <p>Is the issue happening even after applying the patch?<br/> When the MR value is correctly imported, it would know that the peer is MR and thus another NID should be merged to the same peer.</p> <p>I think you misunderstand. It's merging peers that AREN'T supposed to be merged.</p> <p>Say peer A is in the file with a nid of 10.9.12.38@o2ib3:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">peer: - primary nid: 10.9.12.38@o2ib3 Multi-Rail: True peer ni: - nid: 10.9.12.28@o2ib3 state: up </pre> </div></div> <p>and peer B is next in the YAML file with a nid of 10.9.60.1@o2ib3</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">peer: - primary nid: 10.9.60.1@o2ib3 Multi-Rail: False peer ni: - nid: 10.9.60.1@o2ib3 state: up </pre> </div></div> <p>So the resulting peer config YAML file should look like</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">peer: - primary nid: 10.9.12.38@o2ib3 Multi-Rail: False peer ni: - nid: 10.9.12.38@o2ib3 state: up - primary nid: 10.9.60.1@o2ib3 Multi-Rail: False peer ni: - nid: 10.9.60.1@o2ib3 state: up </pre> </div></div> <p>It's trying to add 10.9.60.1@o2ib3 as an extra peer ni to 10.9.12.38@o2ib3. </p> <p>This is wrong. They are separate peers. I've checked the YAML file and they are both described in YAML correctly. There is something wrong with the YAML parser that it causing it to not parse correctly. </p> <p>Oh okay. So I noticed that I never updated the patch which had issue. Just did that.</p> <p>And now with the patch, I just tried it on my system and I could not replicate the issue.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@lutfRtr1-linux ~]# lnetctl ping 10.211.55.9@tcp ping:     - primary nid: 10.211.55.9@tcp       Multi-Rail: False       peer ni:         - nid: 10.211.55.9@tcp [root@lutfRtr1-linux lustre-release]# lnetctl peer add --prim_nid 10.9.60.24@tcp [root@lutfRtr1-linux lustre-release]# lnetctl peer show peer:     - primary nid: 10.211.55.9@tcp       Multi-Rail: False       peer ni:         - nid: 10.211.55.9@tcp           state: NA     - primary nid: 10.9.60.24@tcp       Multi-Rail: True       peer ni:         - nid: 10.9.60.24@tcp           state: NA [root@lutfRtr1-linux lustre-release]# lnetctl export &gt; out.yaml [root@lutfRtr1-linux lustre-release]# lnetctl peer show peer: [root@lutfRtr1-linux lustre-release]# lnetctl <span class="code-keyword">import</span> &lt; out.yaml [root@lutfRtr1-linux lustre-release]# lnetctl peer show peer:     - primary nid: 10.211.55.9@tcp       Multi-Rail: False       peer ni:         - nid: 10.211.55.9@tcp           state: NA     - primary nid: 10.9.60.24@tcp       Multi-Rail: True       peer ni:         - nid: 10.9.60.24@tcp           state: NA</pre> </div></div> <p>How are you adding peers? Can you list the commands you are running to add peers.</p> <p>Though I tried both ways - using "lnetctl" command and running traffic and was able to import peers correctly.</p> <p>I think its a race condition. It happens because we are importing ~4000 nodes while in production (so along side normal discovery)</p> <p>I doubt you will trigger it with two.</p> <p>Hi Malcolm,</p> <p>Can you please attach here the YAML file you are using for configuration. We can try reproducing the issue using that YAML file.</p> <p>Thanks</p> <p>Sonia Sharma (sonia.sharma@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32255" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32255</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10124" title="lnetctl: lnetctl import --add not importing peers correctly" class="issue-link" data-issue-key="LU-10124"><del>LU-10124</del></a> lnet: Correctly add peer MR value while importing<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: c6fcf5a01fa4da0b026498b16927fa6c86cc1918</p> <p>Just pushed the back-ported patch for b2_10 to make it easy for you to apply the patch and test.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/31138/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31138/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10124" title="lnetctl: lnetctl import --add not importing peers correctly" class="issue-link" data-issue-key="LU-10124"><del>LU-10124</del></a> lnet: Correctly add peer MR value while importing<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 99494a28e6afde7c59e7f03045e63028ece1064d</p> <p>Landed for 2.12</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32255/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32255/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10124" title="lnetctl: lnetctl import --add not importing peers correctly" class="issue-link" data-issue-key="LU-10124"><del>LU-10124</del></a> lnet: Correctly add peer MR value while importing<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 8103e94c1bd3000bc25da0d05f0ef3cafa1f91fd</p> Related Development Epic/Theme lnet lustre-2.10.1 Rank 1|hzzlyf: Rank (Obsolete) 9223372036854775807 Severity