https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13566 Lustre <p>In a Multi-Rail setup using ethernet interfaces, it appears like there is a wrong mapping between the LNet level NID and the ethernet interfaces.<br/> When sending traffic, LNet reports all NIDs are being used for traffic, but when we use <tt>netstat -i</tt> to monitor LNet traffic, we only see traffic on a subset of the interfaces.</p> <p>When we restrict traffic from LNet on a subset of the NIDs, even for that subset, the interfaces don't match. For example <tt>netstat -i</tt> can show traffic on eth0 and eth2. But LNet shows that it's using eth1 and eth2.</p> <p>However, when using iperf, all ethernet interfaces are used according to <tt>netstat -i</tt></p> <p>This behavior is easily reproducible on a simple 2 VM MR setup.</p> LU-13566

socklnd: wrong NID to interface mapping

Bug Minor Resolved Fixed Amir Shehata Amir Shehata Fri, 15 May 2020 15:53:20 +0000 Wed, 29 Jul 2020 19:07:29 +0000 Sun, 28 Jun 2020 15:02:54 +0000 Lustre 2.14.0 0 5 <p>Is this happening with multiple Ethernet interfaces on the same subnet? I recall ages ago that there was a problem with "source routing" for ethernet, in that the kernel would select whatever interface it wanted on that subnet, even if LNet is trying to use a specific interface for outgoing packets.</p> <p>This <em>might</em> be helped by patch <a href="https://review.whamcloud.com/37702" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37702</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-10391" title="LNET: Support IPv6" class="issue-link" data-issue-key="LU-10391">LU-10391</a> socklnd: use interface index to track local addr</tt>" to ensure that the specific interface is used rather than trying to use the address to guide interface selection.</p> <p>I actually found a problem with that patch. It breaks binding a socket to the correct interface. As a result we keep binding to the same interface.</p> <p>However, even when I fixed this issue netstat -i still shows all traffic going over only one of the interfaces. I'm continuing my investigation.</p> <p>*Correction the same problem was there from the beginning in socklnd. It was not introduced by the patch.</p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/38743" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38743</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13566" title="socklnd: wrong NID to interface mapping" class="issue-link" data-issue-key="LU-13566"><del>LU-13566</del></a> socklnd: fix local interface binding<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6ba5f24c9f3845dc991bb45d344172af0c9e6d90</p> <p>For tcp workloads it's important to properly set the ARP, reverse path filtering and routing config, to make sure packets egress over the intended interfaces in a multi-rail setup.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38743/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38743/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13566" title="socklnd: wrong NID to interface mapping" class="issue-link" data-issue-key="LU-13566"><del>LU-13566</del></a> socklnd: fix local interface binding<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a7c9aba5eb96dd1e53899108a65af381b49e657b</p> <p>Landed for 2.14</p> Related LU-13790 Development Rank 1|i010hj: Rank (Obsolete) 9223372036854775807 Severity