[LU-1071] able to handle many lnet routes Created: 06/Feb/12 Updated: 31/Jan/21 Resolved: 29/May/17 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor |
| Reporter: | Alexander Boyko | Assignee: | Amir Shehata (Inactive) |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | bgti, patch | ||
| Issue Links: |
|
||||||||||||
| Bugzilla ID: | 18,766 | ||||||||||||
| Rank (Obsolete): | 7561 | ||||||||||||
| Description |
|
We need to get upwards of 10k of ip2nets and 10k of route data loaded into LNet at module startup time. |
| Comments |
| Comment by Alexander Boyko [ 06/Feb/12 ] |
|
patch is submitted to gerrit http://review.whamcloud.com/2102 |
| Comment by Alexander Boyko [ 21/Mar/12 ] |
|
What about this patch? Should I discard request? |
| Comment by Cory Spitz [ 21/Mar/12 ] |
|
Cray is using this patch and would prefer that it gets landed to master. |
| Comment by Cory Spitz [ 28/Jun/12 ] |
|
Any objection to landing the proposed patch? |
| Comment by Doug Oucharek (Inactive) [ 28/Jun/12 ] |
|
The proposed Dynamic LNet Config project will be providing a better way of doing this. However, as that project is waiting for OpenSFS feedback, it won't be happening soon. I have no objections to landing this patch if this functionality is needed now. Just be aware a better/preferred way will be landing in a future release. |
| Comment by Keith Mannthey (Inactive) [ 13/Sep/12 ] |
|
Is there any update about the Dynamic LNet Config project? Do we still care about this patch? |
| Comment by Doug Oucharek (Inactive) [ 13/Sep/12 ] |
|
Dynamic LNet Config is still on the OpenSFS funding list. Hopefully we will know soon if it is a green light or not. |
| Comment by Cory Spitz [ 17/Sep/12 ] |
|
Ok, I'd prefer to land this patch with the understanding that an LNet configuration rework is coming down the line. |
| Comment by Alexander Boyko [ 19/Sep/12 ] |
|
Patch was rebased against master. A bird in the hand is worth two in the bush. |
| Comment by Keith Mannthey (Inactive) [ 25/Sep/12 ] |
|
As Issac Huang comment in the patch review: Is there any reason a properly timed 'lctl add_route' command script can't address this issue? |
| Comment by Cory Spitz [ 03/Oct/12 ] |
|
"Is there any reason a properly timed 'lctl add_route' command script can't address this issue?" Functionally, no. But it is easier to manage and more reliable with the proposed enhancement, imo. |
| Comment by Keith Mannthey (Inactive) [ 03/Oct/12 ] |
|
In general the kernel reading in a file for configuration is not something I would want to do if there is another way. Is 'lctl add_route' unreliable when you dump info into in mass? |
| Comment by Cory Spitz [ 03/Oct/12 ] |
|
We (Cray & Xyratex) didn't even attempt to create scripts to drop in so many entries so I can't tell you. |
| Comment by Doug Oucharek (Inactive) [ 03/Oct/12 ] |
|
The proposed approach for Dynamic LNet Config is very similar to using "lctl add_route". The focus is to move the complexity of dealing with large volumes of routes from kernel space to user space. Such an approach is more flexible as scripts can be created in user space to do pretty much whatever your specific needs are without having to change the LNet codebase. |
| Comment by Nic Henke (Inactive) [ 04/Oct/12 ] |
|
ip2nets: scripting here is not always possible, and frankly requires the duplication of code that is already in lnet.ko in the kernel. routes: we need the full set of routes available at mount time, for proper mount-on-boot for client nodes and automatic failover/reboot on server side. To prevent wrapping or rewriting mount.lustre, this needs to be handled in the module load chain. We had tried using modprobe install hooks for this, but the end result was racy and error prone. You can't hook into lnet.ko, as that requires the loading of all the LNDs before lctl --net XYX will work, etc. Given that we have a fairly low-impact patch that is already done and ready to go in, why would we hold it off for some unknown feature like Dynamic LNet Config ? If DLC fixes this in the future, we could replace it - but right now only one is coded and viable. Since Isaac and Doug both referenced "Dynamic LNet Config" - can you share the design? It is very hard to contribute code when we don't know the plans for the future. |
| Comment by Keith Mannthey (Inactive) [ 04/Oct/12 ] |
|
I think it is agreed-ed we are not going to block this patch for the sake of "Dynamic LNet Config", you know as much about it as I do. It sounds like just using 'lctl add_route' it is not possible to accomplish what is needed since modprobe install hooks don't work. Sorry this patch is not ready to go in as is. The file handling is very best case scenario: at a minimum we need to be sure the the file is not larger than the fixed size, introducing code that might only read SOME of the data not good. I am sure this patch works in the specific usage it was designed for but I don't think it is asking a lot to make it a little more generally safe as it could be in the code base for awhile. I would also hope that we all want to understand what is/will be possible, ie what are the new maximum values. How many routes/ ip2nets can be added with this patch in place? So far 10k/10k has been stated but no indication has been given about the filesize. |
| Comment by Nathan Rutman [ 21/Nov/12 ] |
|
Xyratex MRP-342 |
| Comment by Keith Mannthey (Inactive) [ 27/Nov/12 ] |
|
Is there any update for dynamic-lnet-config or this patch? I would hope we can land something to fill this gap. |
| Comment by Doug Oucharek (Inactive) [ 28/Nov/12 ] |
|
Now that OpenSFS seems to be moving the RFP process forward, there is a hope that the Dynamic LNet Config work can make the 2.4 release of Lustre. Would that be soon enough or will this patch be needed in an earlier release? |
| Comment by Keith Mannthey (Inactive) [ 28/Nov/12 ] |
|
This patch has not been resubmitted and there are outstanding issues covered in review. There is no solution proposed at this point. |
| Comment by Keith Mannthey (Inactive) [ 04/Jan/13 ] |
|
Without any update I will close this LU at the end of the month. |
| Comment by Keith Mannthey (Inactive) [ 11/Jan/13 ] |
|
It seems Dynamic Lnet Config is moving. The main LU can be see at http://jira.whamcloud.com/browse/LU-2456. |
| Comment by Doug Oucharek (Inactive) [ 14/Jan/13 ] |
|
Yes, the Dynamic LNet Config feature is landing in 2.4 and brings with it the ability to configure an almost unlimited number of routes. It does this by letting you dynamically configure one route at a time. That means you will still need a script in user space to take a list of routes and to invoke the "lent" utility for each route entry to get it configured. |
| Comment by Doug Oucharek (Inactive) [ 05/Apr/13 ] |
|
Ticket Note: this addresses routes, not ip2nets entries. Those have to wait for the full Dynamic LNet Config. |