https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1071 Lustre <p>We need to get upwards of 10k of ip2nets and 10k of route data loaded into LNet at module startup time.</p> LU-1071

able to handle many lnet routes

Improvement Minor Resolved Duplicate Amir Shehata Alexander Boyko bgti patch Mon, 6 Feb 2012 06:57:25 +0000 Sun, 31 Jan 2021 19:51:12 +0000 Mon, 29 May 2017 03:17:07 +0000 0 13 <p>patch is submitted to gerrit <a href="http://review.whamcloud.com/2102" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2102</a> </p> <p>What about this patch? Should I discard request? </p> <p>Cray is using this patch and would prefer that it gets landed to master.</p> <p>Any objection to landing the proposed patch?</p> <p>The proposed Dynamic LNet Config project will be providing a better way of doing this. However, as that project is waiting for OpenSFS feedback, it won't be happening soon.</p> <p>I have no objections to landing this patch if this functionality is needed now. Just be aware a better/preferred way will be landing in a future release.</p> <p>Is there any update about the Dynamic LNet Config project? Do we still care about this patch? </p> <p>Dynamic LNet Config is still on the OpenSFS funding list. Hopefully we will know soon if it is a green light or not.</p> <p>Ok, I'd prefer to land this patch with the understanding that an LNet configuration rework is coming down the line.</p> <p>Patch was rebased against master. A bird in the hand is worth two in the bush.</p> <p>As Issac Huang comment in the patch review:</p> <p>Is there any reason a properly timed 'lctl add_route' command script can't address this issue? </p> <p>"Is there any reason a properly timed 'lctl add_route' command script can't address this issue?" Functionally, no. But it is easier to manage and more reliable with the proposed enhancement, imo.</p> <p>In general the kernel reading in a file for configuration is not something I would want to do if there is another way. Is 'lctl add_route' unreliable when you dump info into in mass? </p> <p>We (Cray &amp; Xyratex) didn't even attempt to create scripts to drop in so many entries so I can't tell you.</p> <p>The proposed approach for Dynamic LNet Config is very similar to using "lctl add_route". The focus is to move the complexity of dealing with large volumes of routes from kernel space to user space. Such an approach is more flexible as scripts can be created in user space to do pretty much whatever your specific needs are without having to change the LNet codebase.</p> <p>ip2nets: scripting here is not always possible, and frankly requires the duplication of code that is already in lnet.ko in the kernel. </p> <p>routes: we need the full set of routes available at mount time, for proper mount-on-boot for client nodes and automatic failover/reboot on server side. To prevent wrapping or rewriting mount.lustre, this needs to be handled in the module load chain. We had tried using modprobe install hooks for this, but the end result was racy and error prone. You can't hook into lnet.ko, as that requires the loading of all the LNDs before lctl --net XYX will work, etc. </p> <p>Given that we have a fairly low-impact patch that is already done and ready to go in, why would we hold it off for some unknown feature like Dynamic LNet Config ? If DLC fixes this in the future, we could replace it - but right now only one is coded and viable. </p> <p>Since Isaac and Doug both referenced "Dynamic LNet Config" - can you share the design? It is very hard to contribute code when we don't know the plans for the future. </p> <p>I think it is agreed-ed we are not going to block this patch for the sake of "Dynamic LNet Config", you know as much about it as I do. It sounds like just using 'lctl add_route' it is not possible to accomplish what is needed since modprobe install hooks don't work. </p> <p>Sorry this patch is not ready to go in as is. The file handling is very best case scenario: at a minimum we need to be sure the the file is not larger than the fixed size, introducing code that might only read SOME of the data not good. I am sure this patch works in the specific usage it was designed for but I don't think it is asking a lot to make it a little more generally safe as it could be in the code base for awhile. </p> <p> I would also hope that we all want to understand what is/will be possible, ie what are the new maximum values. How many routes/ ip2nets can be added with this patch in place? So far 10k/10k has been stated but no indication has been given about the filesize. </p> <p>Xyratex <a href="http://jira-nss.xy01.xyratex.com:8080/browse/MRP-342" class="external-link" target="_blank" rel="nofollow noopener">MRP-342</a></p> <p>Is there any update for dynamic-lnet-config or this patch? I would hope we can land something to fill this gap.</p> <p>Now that OpenSFS seems to be moving the RFP process forward, there is a hope that the Dynamic LNet Config work can make the 2.4 release of Lustre.</p> <p>Would that be soon enough or will this patch be needed in an earlier release?</p> <p>This patch has not been resubmitted and there are outstanding issues covered in review. There is no solution proposed at this point. </p> <p>Without any update I will close this LU at the end of the month. </p> <p>It seems Dynamic Lnet Config is moving. The main LU can be see at <a href="http://jira.whamcloud.com/browse/LU-2456" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-2456</a>. </p> <p>Yes, the Dynamic LNet Config feature is landing in 2.4 and brings with it the ability to configure an almost unlimited number of routes. It does this by letting you dynamically configure one route at a time. That means you will still need a script in user space to take a list of routes and to invoke the "lent" utility for each route entry to get it configured.</p> <p>Ticket <a href="https://jira.whamcloud.com/browse/LU-2950" title="Port LNET routes_config change done by Fujitsu in latest lustre" class="issue-link" data-issue-key="LU-2950"><del>LU-2950</del></a> has implement a way from user space to configure many LNet routes which will mirror what Dynamic Lnet Config will be doing. <a href="https://jira.whamcloud.com/browse/LU-2950" title="Port LNET routes_config change done by Fujitsu in latest lustre" class="issue-link" data-issue-key="LU-2950"><del>LU-2950</del></a> will be landing in 2.5 and is ready to cherry pick now.</p> <p>Note: this addresses routes, not ip2nets entries. Those have to wait for the full Dynamic LNet Config.</p> Duplicate LU-2456 LU-14391 Bugzilla ID 18766.0 Development Rank 1|hzvn5j: Rank (Obsolete) 7561