https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10825 Lustre <p>Recently, we prepare to deployment a lustre with multi-rail, but i don't known how to enable dynamic discovery.</p> <p>We use lustre-2.10.3, it seems dynamic discovery is implementd in version 2.11.</p> <p>We have about 2 mgs/mds, 6 oss and 512 client nodes, how to configure static multi-rail with a large number of nodes ?</p> LU-10825

Configuring multi-rail with a large number of nodes

Improvement Minor Resolved Fixed Amir Shehata Taizeng Wu Mon, 19 Mar 2018 13:01:48 +0000 Mon, 20 Aug 2018 15:40:37 +0000 Wed, 2 May 2018 04:06:25 +0000 Lustre 2.10.3 Lustre 2.12.0 Lustre 2.10.5 0 6 <p>Can you explain to me your MR deployment? Do all nodes have multiple interfaces? or the clients only? servers only?</p> <p>All nodes have two interfaces (1 Mellanox FDR card with two port).</p> <p>I am trying to configure remote peer to include oss and clients on the mds/mgs nodes, remote peer to include mds/mgs and client on the oss nodes, remote peer to include mds/mgs and oss on the client nodes. Is this configuration correct ?</p> <p>Then i mkfs or mount lustre to use MR's primary nid.</p> <p>When i turn down a interface on server node, i found lustre filesystem hung sometimes. </p> <p>This issue may be caused by ARP ( <a href="http://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node" class="external-link" target="_blank" rel="nofollow noopener">http://wiki.lustre.org/LNet_Router_Config_Guide#ARP_flux_issue_for_MR_node</a>).</p> <p>I follow the ARP flux guide to configure, but lustre filesystem still hung sometimes when turn down a interface (dmesg report "Request set has failed due to network error" about node which turn down a interface).</p> <p>—</p> <p>Servers OS Version: RHEL 7.4</p> <p>Clients OS Version: RHEL 6.8</p> <p> </p> <p>I'm currently working on a patch to make it easier to configure large systems without Dynamic Discovery.</p> <p>But for now you'll need to configure the servers to know about the client's interfaces and you'll need to configure the clients to know about the server's interfaces. And since the OSS/MGS communicate you'll need to configure these to know about each other's interfaces.</p> <p>MR doesn't handle interface down cases. If you intentionally (or unintentionally) bring down an interface, it will interfere with the file system operations as you've seen. We're currently working on a feature, LNet Health, which will be able to handle this particular interface failures.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31785" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31785</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> libcfs: generate ip addresses<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eb88bfc56451ad790445daf1d5be303915b596b9</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31786" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31786</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> lnet: add ip2nets syntax handling for peer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3edcd387af4028e93f5d4df9caed9a36539ccbf6</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31785/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31785/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> libcfs: generate ip addresses<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4c5f788397213aa41356df1f96f7ade58653973a</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31786/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31786/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> lnet: add ip2nets syntax handling for peer<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 70c95457f6836a9c0a9e95ae0c4bdd20f99a8747</p> <p>Landed for 2.12</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32249" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32249</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> libcfs: generate ip addresses<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: a22011c6d2cd804413b5f7e8353b687fb742a495</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/32250" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32250</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> lnet: add ip2nets syntax handling for peer<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: fc78b0ee95a2ee85121e84e5d104f5c268aae26f</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32249/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32249/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> libcfs: generate ip addresses<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: a48dc3fd0f738b545571a6d2cfdeb337f2d3243b</p> <p>John L. Hammond (jhammond@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/32250/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32250/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10825" title="Configuring multi-rail with a large number of nodes" class="issue-link" data-issue-key="LU-10825"><del>LU-10825</del></a> lnet: add ip2nets syntax handling for peer<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: e124f39b6b4dd56780ba4490b81dca32ab08575c</p> Development Rank 1|hzzuj3: Rank (Obsolete) 9223372036854775807