https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11840 Lustre <p>In recent Lustre releases, some specific filesystem could not be mounted due to a communication error between clients and servers, depending on the LNET configuration.</p> <p>If we have a filesystem running on a host with 2 interfaces, let say tcp0 and tcp1 and the devices are setup to reply on both interfaces (formatted with --servicenode IP1@tcp0,IP2@tcp1).</p> <p>If a client is connected only to tcp0 and try to mount this filesystem, it fails with an I/O error because it is trying to connect using tcp1 interface.</p> <p>Mount failed:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mount -t lustre x.y.z.a@tcp:/lustre /mnt/lustre mount.lustre: mount x.y.z.a@tcp:/lustre at /mnt/client failed: Input/output error Is the MGS running? </pre> </div></div> <p>dmesg shows that communication fails using the wrong IP</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[422880.743179] LNetError: 19787:0:(lib-move.c:1714:lnet_select_pathway()) no route to a.b.c.d@tcp1</pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lnetctl peer show peer: - primary nid: a.b.c.d@tcp1 Multi-Rail: False peer ni: - nid: x.y.z.a@tcp state: NA - nid: 0@&lt;0:0&gt; state:</pre> </div></div> <p>Ping is OK though:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl ping x.y.z.a@tcp 12345-0@lo 12345-a.b.c.d@tcp1 12345-x.y.z.a@tcp</pre> </div></div> <p> </p> <p>This was tested with 2.10.5 and 2.12 as server versions and 2.10, 2.11 and 2.12 as client.</p> <p>Only 2.10 client is able to mount the filesystem properly with this configuration</p> <p> </p> <p>I git-bisected the regression down to <tt>0f1aaad <a href="https://jira.whamcloud.com/browse/LU-9480" title="LNet Dynamic Discovery" class="issue-link" data-issue-key="LU-9480"><del>LU-9480</del></a> lnet: implement Peer Discovery</tt></p> <p>Looking at debug log, the client:</p> <ul> <li>setups the peer with the proper NI</li> <li>the pings the peer</li> <li>updates the local peer info with the wrong NI coming from the ping reply</li> </ul> <p>data in the reply seems to announce the tcp1 IP as the primary nid.</p> <p>The client will then use this NI to contact the server even if it has no direct connection to it (tcp1) and has a correct one for the same peer (tcp0).</p> LU-11840

Multi rail dynamic discovery prevent mounting filesystem when some NIC is unreachable

Bug Minor Open Unresolved Amir Shehata Aurelien Degremont Tue, 8 Jan 2019 13:50:33 +0000 Wed, 25 Nov 2020 15:03:48 +0000 Lustre 2.11.0 Lustre 2.12.0 1 17 <p>I made more tests.</p> <p>2.12 client and 2.12 server: OK</p> <p>2.10 client and 2.10 client: OK</p> <p> </p> <p>2.12 client with 2.10.5 server: broken</p> <p> </p> <p>Looks like it is related to MR capable/Discovery capable feature.</p> <p> </p> <p>Aurelien, do you see this with 2.11-&gt; 2.10.5?</p> <p>I'm suspecting a timeout length issue. But if you could verify the above, it'll prove it to me.</p> <p>2.11 client and 2.10.5 server: broken</p> <p> </p> <p>Moreover, I can workaround the problem if I add the peer first:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetctl peer add --prim_nid x.y.z.a@tcp mount -t lustre x.y.z.a@tcp:/fsx /mnt/client/</pre> </div></div> <p>Timeout is a good lead as it seems to be what I see in the logs.</p> <p> </p> <p>Looks like if the server is 2.12, all clients (2.10, 2.11 and 2.12) successfully mount the FS.</p> <p>If you have a potential understanding of the problem and you have trails/ideas I can follow or dig in, let me know.</p> <p>It seems like there is a misunderstanding around the primary_nid. So when you add the peer explicitly it works, but if you don't, then discovery depends on the list of NIDs which come back in the ping response. The first NID in that list is considered the primary_nid. Based on this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lctl ping x.y.z.a@tcp 12345-0@lo 12345-a.b.c.d@tcp1 12345-x.y.z.a@tcp</pre> </div></div> <p>The tcp1 interface is first. So LNet thinks it's the primary NID of the node and tries to send messages to it, but there are no routes.</p> <p>If you set up the server in such a way that the tcp interface is first, do you resolve the problem?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lctl ping x.y.z.a@tcp 12345-0@lo 12345-x.y.z.a@tcp 12345-a.b.c.d@tcp1 </pre> </div></div> <p>Wooh! It works! Thanks a lot! I will work on a workaround based on that.</p> <p> </p> <p>But, I thought, especially since MR appeared that it would select the best available route, and in this case, even if the primary nid is not reachable, there is one which is. Did I misunderstand the MR features?</p> <p>LNeth Health, which is in 2.12, should be able to do that. You'll need to enable it.</p> <p><a href="https://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.mrhealth" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.mrhealth</a></p> <p>Although, I'm a bit confused by the test matrix you outlined. When you tested 2.12-&gt;2.12 it should still be the same problem. But you're saying it works. When you have 2.12 on the servers, was the order of the NIDs still the same? Or in these test runs the order of the NIDs was:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lctl ping x.y.z.a@tcp 12345-0@lo 12345-x.y.z.a@tcp 12345-a.b.c.d@tcp1 </pre> </div></div> <p>I confirm that with a 2.12 server, where tcp1 is declared first, 2.12 client mount is still ok.</p> <p> </p> <p>That's also why I'm thinking there is some MR magic in action here. Looks like the peer state in 2.12 client is not the same if the server supports 'push' or not. It seems you're right that in the simple case, it will just use the first NID that is returned when being pinged.</p> <p> </p> <p>The problem is that there is still a regression between 2.10 and 2.12 in my opinion. A setup that did not look unsupported was working with a 2.10 client and is no more with a 2.12.</p> <p>I think the 2.12 client rely on a server side feature that was introduced after 2.10 to properly setup its peer state and use the correct one when mounting. If the server is a 2.10 one, the push/discovery feature did not exist and the client does not try to do something smart with all the available IPs. It will just try the first one and fails.</p> <p> </p> <p>More test results, with a 2.12 client and 2.10 server, and NIDs being set in non-optimal order (ping returns tcp1 as first nid):</p> <ul> <li>Add peer before before mount: <tt>lnetctl peer add --prim_nid x.y.z.a@tcp</tt>: Mount is OK</li> <li>Add peer, declared as non_mr, before mount: <tt>lnetctl peer add --prim_nid x.y.z.a@tcp --non_mr:</tt> Error</li> </ul> <p> </p> <ul> <li>Add peer, bad nid as prim_nid, 2 nids declared: <tt>lnetctl peer add --prim_nid a.b.c.d@tcp1 --nid x.y.z.a@tcp,a.b.c.d@tcp1</tt>: OK</li> <li>Add peer, bad nid as prim_nid, 2 nids declared as non_MR: <tt>lnetctl peer add --prim_nid a.b.c.d@tcp1 --nid x.y.z.a@tcp,a.b.c.d@tcp1 --non_mr</tt>: Error</li> </ul> <p> </p> <p>So that means that to have it working, we need either:</p> <ul> <li>tcp0 as the first NID returned by server</li> <li>the peer being explicitly declared as MultiRail capable, whatever the prim_nid is.</li> </ul> <p>Hope that helps you understand the problem.</p> <p>As discussed today, the work around where you configure the tcp NID to be primary on the server will work in your case.</p> <p>In the mean time I've been looking at a way to resolve the incompatibility between discovery enabled node and a non-discovery capable node (IE 2.10.x) and I have hit a snag.</p> <p>I'm testing two different scenarios</p> <ol> <li>OST(2.12) MDT(2.10.x) Client(2.12)</li> <li>OST(2.10.x) MDT(2.10.x) Client (2.12)</li> </ol> <p>Unfortunately, lustre does its own NID lookup without using LNet to pull the NID information in both scenarios, particularly, here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 779 /** 780 * Retrieve MDT nids from the client log, then start the lwp device. 781 * there are only two scenarios which would include mdt nid. 782 * 1. 783 * marker 5 (flags=0x01, v2.1.54.0) lustre-MDTyyyy <span class="code-quote">'add mdc'</span> xxx- 784 * add_uuid nid=192.168.122.162@tcp(0x20000c0a87aa2) 0: 1:192.168.122.162@tcp 785 * attach 0:lustre-MDTyyyy-mdc 1:mdc 2:lustre-clilmv_UUID 786 * setup 0:lustre-MDTyyyy-mdc 1:lustre-MDTyyyy_UUID 2:192.168.122.162@tcp 787 * add_uuid nid=192.168.172.1@tcp(0x20000c0a8ac01) 0: 1:192.168.172.1@tcp 788 * add_conn 0:lustre-MDTyyyy-mdc 1:192.168.172.1@tcp 789 * modify_mdc_tgts add 0:lustre-clilmv 1:lustre-MDTyyyy_UUID xxxx 790 * marker 5 (flags=0x02, v2.1.54.0) lustre-MDTyyyy <span class="code-quote">'add mdc'</span> xxxx- 791 * 2. 792 * marker 7 (flags=0x01, v2.1.54.0) lustre-MDTyyyy <span class="code-quote">'add failnid'</span> xxxx- 793 * add_uuid nid=192.168.122.2@tcp(0x20000c0a87a02) 0: 1:192.168.122.2@tcp 794 * add_conn 0:lustre-MDTyyyy-mdc 1:192.168.122.2@tcp 795 * marker 7 (flags=0x02, v2.1.54.0) lustre-MDTyyyy <span class="code-quote">'add failnid'</span> xxxx- 796 **/ 797 <span class="code-keyword">static</span> <span class="code-object">int</span> client_lwp_config_process(<span class="code-keyword">const</span> struct lu_env *env, 798 »·······»·······»·······»······· struct llog_handle *handle, 799 »·······»·······»·······»······· struct llog_rec_hdr *rec, void *data) </pre> </div></div> <p>Lustre tries to retrieve the MDT nids from the client log and it looks at the first NID in the list. In both cases the OST is unable to mount the MGS, because it's using the tcp1 NID to get the peer and ends in this error:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (events.c:543:ptlrpc_uuid_to_peer()) 192.168.122.117@tcp1-&gt;12345-&lt;?&gt; (client.c:97:ptlrpc_uuid_to_connection()) cannot find peer 192.168.122.117@tcp1! </pre> </div></div> <p>This error is independent from the backwards compatibility issue. My config looks like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> OST: ---- net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.122.114@tcp status: up interfaces: 0: eth0 - nid: 192.168.122.115@tcp status: up interfaces: 0: eth1 MDT: ---- net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp1 local NI(s): - nid: 192.168.122.117@tcp1 status: up interfaces: 0: eth0 - net type: tcp local NI(s): - nid: 192.168.122.118@tcp status: up interfaces: 0: eth1 </pre> </div></div> <p>I'm curious how  you setup your OSTs so you don't run into the problem above?</p> <p>My LNET setup looks like the MDT one. There is 2 LND, tcp0 and tcp1 and only one interface for each of them.</p> <p>We did the test together on a simple system where both MDT and OST where on the same server, but I do not think this makes a difference here.</p> <p>Looking at my MGS client llog, it looks rather like the case #1.</p> <p>Devices were formatted specifying a simple service node option (see ticket description).</p> <p>@ashehata Did you make any progress on this topic?</p> <p>I'm facing a similar issue with a pure 2.10.5 configuration.</p> <p>Lustre servers have both tcp0 and tcp1 NID. MDT/OSTs are setup to use both of them. But Lustre servers will try to communicate only using the first configured interface. If it fails (timeout), they will never try the second one.</p> <p>Do you have any clue?</p> <p>Yes. I believe that's the issue I pointed to here: <a href="https://jira.whamcloud.com/browse/LU-11840?focusedCommentId=240077&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-240077" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11840?focusedCommentId=240077&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-240077</a></p> <p>Lustre (not LNet) does its own NID lookup based on logs. The assumption inherit in the code is that there is only one NID per node. Which is not right.</p> <p>I'm working on a solution. Will update the ticket when I have a patch to test.</p> <p>Thanks a lot! Do you have a rough idea if this is days or weeks of work?</p> <p>I don't think that it's a huge amount of work but I am focused on 2.13 feature work ATM so have not looked at it in much detail yet</p> <p>OK, understood.</p> <p>A simple question based on your config output. Should we declare 0@lo in a lnet.conf file? Used with lnetctl import.</p> <p>I could not find a clear statement on that looking at different places.</p> <p>No. 0@lo will always get ignored because it's created implicitly. So you don't have to have it in the lnet.conf file.</p> <p>There is actually a patch</p> <p><a href="https://jira.whamcloud.com/browse/LU-10452" title="lnetctl export/import suggested improvements" class="issue-link" data-issue-key="LU-10452"><del>LU-10452</del></a> lnet: cleanup YAML output</p> <p>which allows you to use a "–backup" option to print a YAML block with only the elements needed to reconfigure a system.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lnetctl net show --backup #also when you export that backup feature is automatically set lnetctl export &gt; lnet.conf</pre> </div></div> <p>Really helpful. Thank you!</p> <p>For the records, disabling LNET discovery seems to workaround the issue</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnetctl set discovery 0</pre> </div></div> <p>before mounting the Lustre client.</p> <p>I realized I could not reproduced this bug with latest master branch code.</p> <p>I tracked down this behavior change between 2.12.54 and 2.12.55, so very likely related to MR Routing feature landing. I did not track to which patch precisely.</p> <p> </p> <p>Lustre 2.12 is still impacted though.</p> <p>Hi, I have got the similar  problem when client/ server / router are all "lustre2.12"</p> <p>The Lustre client  try to communicate only using the first configured interface of server , they will never try the second one.</p> <p>clientA:<br/> options lnet networks="tcp(eno1)" routes="o2ib 172.26.1.222@tcp"</p> <p>router:<br/> options lnet networks="tcp(eno1),o2ib(ib0)" "forwarding=enabled"</p> <p>server:<br/> options lnet networks="tcp2(eno1),o2ib0(ib0)" routes="tcp 172.20.0.222@o2ib" <br/> =&gt; clientA mount server o2ib fail<br/> options lnet networks="o2ib0(ib0),tcp2(eno1)" routes="tcp 172.20.0.222@o2ib" <br/> =&gt; clientA mount server o2ib ok</p> <p>AFAICS I also have this issue:</p> <ul> <li>server (lustre-2.10.4-1.el7.x86_64): <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> options lnet networks=tcp0(en0),o2ib0(in0)</pre> </div></div></li> </ul> <ul> <li>client (lustre-2.12.5-RC1-0.el7.x86_64): <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> options lnet networks=<span class="code-quote">"o2ib(ib0)"</span></pre> </div></div></li> </ul> <p>These two workarounds seem to work (only <b>very</b> limited testing so far):</p> <ol> <li>Configuring LNET tcp on the client (although I actually only want to use IB): <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> options lnet networks=<span class="code-quote">"o2ib(ib0),tcp(enp3s0f0)"</span> </pre> </div></div></li> </ol> <ol> <li>Executing this before the actual Lustre mount: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lnetctl set discovery 0</pre> </div></div></li> </ol> <p>Krasten,</p> <p>There is a known incompatibility between discovery enabled lustre and older lustre versions, IE 2.10.</p> <p>We have a ticket open for it and we're looking at how we can resolve this:</p> <p><a href="https://jira.whamcloud.com/browse/LU-13548" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-13548</a></p> <p>Currently disabling discovery on 2.12 is the best workaround.</p> Related LU-13548 Development Rank 1|i008zz: Rank (Obsolete) 9223372036854775807 Severity