https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9990 Lustre <p>Recently I started to run into issues with the MDT failing to mount randomly. Now with the latest master the MDT fails to mount every single time. Looking at the debug log I noticed the following error on the MDT:</p> <p>(client.c:96:ptlrpc_uuid_to_connection()) cannot find peer MGC10.37.248.196@o2ib1_0!</p> Latest lustre 2.10.5X running on RHEL7.4 with default OFED. Using IB for LND. LU-9990

MDS fails to mount due to (client.c:96:ptlrpc_uuid_to_connection()) cannot find peer MGC10.37.248.196@o2ib1 _0!

Bug Blocker Resolved Fixed Amir Shehata James A Simmons Thu, 14 Sep 2017 14:37:13 +0000 Mon, 6 Nov 2017 15:57:24 +0000 Tue, 24 Oct 2017 12:51:42 +0000 Lustre 2.11.0 Lustre 2.11.0 0 12 <p>I attached full debug logs from the MDS/MGS.</p> <p>Dropping severity because I take it that this is not a production service interruption</p> <p>Just our test bed is busted <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>James</p> <p>Could you please elaborate as to the exact commit you are seeing this with and the last commit that you did not see this problem?</p> <p>Peter</p> <p>James,</p> <p>Could you be more specific about the Lustre version here? (Saying 'latest' does not age well.) Also can you give a recent Lustre version where you didn't see this issue?</p> <p>Dose this still happen if you set lnet_peer_discovery_disabled=1 in the lnet module parameters?</p> <p>James, can you also try: <a href="https://review.whamcloud.com/#/c/29007/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29007/</a></p> <p>I did and it still fail to bring up the file system <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> John was right. When I set lnet_peer_discovery_disabled=1 everything worked as usual.</p> <p>in the dump-mds.log I see:<br/> (lib-move.c:3199:LNetGet()) Error sending GET to 12345-10.37.202.59@o2ib1: -113</p> <p>Is that issue resolved?</p> <p>Can you also turn on net and neterror when you're mounting and failing, and attach the output.</p> <p>That is due to patch <a href="https://jira.whamcloud.com/browse/LU-9810" title="Melanox OFED 4.1 support" class="issue-link" data-issue-key="LU-9810"><del>LU-9810</del></a> lnet: prefer Fast Reg. I had to revert it locally to make things work better. I will post new logs shortly.</p> <p>With the revert of <a href="https://jira.whamcloud.com/browse/LU-9810" title="Melanox OFED 4.1 support" class="issue-link" data-issue-key="LU-9810"><del>LU-9810</del></a> prefer Fast Reg and the fix from <a href="https://jira.whamcloud.com/browse/LU-9933" title="Hitting ASSERTION in lnet_peer_add_nid()" class="issue-link" data-issue-key="LU-9933"><del>LU-9933</del></a> + <a href="https://jira.whamcloud.com/browse/LU-9992" title="Multi-Rail: use lolnd when sending locally" class="issue-link" data-issue-key="LU-9992"><del>LU-9992</del></a> everything seems to work now.</p> <p>Ah, I found what caused this issue. I have a reproducer. Its very simple, add routes that are all down to your yaml config file and you will not be able to mount your file system. We have Cray routers and they have been done recently. So add to your yaml config file something like this:</p> <p>route:</p> <ul class="alternate" type="square"> <li>net: gni1<br/> gateway: 10.37.202.60@o2ib1<br/> hop: 1<br/> priority: 0<br/> state: down</li> <li>net: gni1<br/> gateway: 10.37.202.59@o2ib1<br/> hop: 1<br/> priority: 0<br/> state: down</li> <li>net: gni1<br/> gateway: 10.37.202.61@o2ib1<br/> hop: 1<br/> priority: 0<br/> state: down</li> </ul> <p>And you will see problems.</p> <p>I couldn't reproduce this on my local setup. I'm assuming that the downed gateways are not needed for the FS to be mounted, correct? IE do you need those routes for communication? If that's not the case, can you please enable net and neterror and dump the logs after the problem happens to so I can take a look.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl set_param debug=+net lctl set_param debug=+neterror lctl dk &gt; log </pre> </div></div> <p>James, </p> <p>can you confirm - <a href="https://jira.whamcloud.com/browse/LU-9180" title="Upstream ko2iblnd does not work with map_on_demand &lt;256" class="issue-link" data-issue-key="LU-9180"><del>LU-9180</del></a> isn't caused it bug? if no, can you test with patches from <a href="https://jira.whamcloud.com/browse/LU-9810" title="Melanox OFED 4.1 support" class="issue-link" data-issue-key="LU-9810"><del>LU-9810</del></a> ?<br/> thanks.</p> <p>I have been testing without <a href="https://jira.whamcloud.com/browse/LU-9810" title="Melanox OFED 4.1 support" class="issue-link" data-issue-key="LU-9810"><del>LU-9810</del></a> patches. This bug is still their. Its just when I run with <a href="https://jira.whamcloud.com/browse/LU-9810" title="Melanox OFED 4.1 support" class="issue-link" data-issue-key="LU-9810"><del>LU-9810</del></a> patches new bugs show up <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Amir I will get new debug logs for you Friday.</p> <p>James,</p> <p>what HW you use for testing? if it MLX5, these patches do nothing for you. MLX5 uses a FastReg only model, while MLX4 support a both Fast and FMR.</p> <p>I don't have access to HW IB for now, so may check only with VM's + M-OFED 4.1+ Soft IB</p> <p>Sorry I was having a hard time reproducing this problem. Its not the route configuration that breaks lnet but the numa node setting in my lnet.conf that did. I ended up removing the numa stuff from my config file. If you add</p> <p>numa:<br/> range: 0</p> <p>to your lnet yaml config file you will see this breakage. </p> <p>I believe the numa range defaults to 0. When you remove it and you do "lnetctl numa show" do you see a different value for the range?</p> <p><span class="error">&#91;root@ninja34 ~&#93;</span># lnetctl global show<br/> global:<br/> numa_range: 0<br/> max_intf: 200<br/> discovery: 1</p> <p>Does the numa_range have to be "under" global: in the YAML config file?</p> <p>so the output is a little strange. It looks like you're using the latest master. But the configuration you're feeding in seems to have been generated from 2.10. numa should be fed in under the global, as shown in the global show output you pasted above. Now thinking about it, this is a backwards compatibility issue since 2.10 is already out. I'll have to make lnetctl handle the older configuration as well for master. </p> <p>Do you see "call back for 'numa' not found" error when you configure with the numa block?</p> <p>I think I know what the problem is. When the parser encounters a problem in the YAML file, it'll quit and simply stop configuring the rest of the items. So it could be that when it hits this error it doesn't finish the configuration leading to the problem you're seeing.</p> <p>Are you calling "lnetctl import" from a script? If so, I think you should be checking if the command succeeds or fails. If it fails you should assume that the node is not configured properly.</p> <p>Can you verify if my theory is correct?</p> <p>Yes I do see a "call back from 'num' not found error when I start up. Also I was using lnet.conf from my 2.10 setup. I'm running the lnetctl import from the command line not script.</p> <p>ok. I'll make a change to handle "numa" entry in YAML file so I can make master backwards compatible with 2.10. In the meantime, if there is an error configuring, you should assume that the configuration is not complete, and the node is not really usable.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29333" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29333</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9990" title="MDS fails to mount due to (client.c:96:ptlrpc_uuid_to_connection()) cannot find peer MGC10.37.248.196@o2ib1 _0!" class="issue-link" data-issue-key="LU-9990"><del>LU-9990</del></a> lnet: add backwards compatibility for YAML config<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e72cdc1373dfb930eccbc5d9afba215d8368b331</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29333/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29333/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9990" title="MDS fails to mount due to (client.c:96:ptlrpc_uuid_to_connection()) cannot find peer MGC10.37.248.196@o2ib1 _0!" class="issue-link" data-issue-key="LU-9990"><del>LU-9990</del></a> lnet: add backwards compatibility for YAML config<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3187d551d538bd8203c7156daaa617620c6569ab</p> <p>Landed for 2.11</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ashehata" class="user-hover" rel="ashehata">ashehata</a> said we don't need this for LTS</p> Related LU-9933 LU-9992 LU-9810 Development Rank 1|hzzk6n: Rank (Obsolete) 9223372036854775807 Severity