https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4147 Lustre <p>Attached you'll find a spreadsheet of our thoughts on setting ko2iblnd parameters and kgnilnd parameters for our center. I apologize that I've dropped the ball on this; we're planning on implementing these settings (in the agreed upon column) on Tuesday 10/29. There are some points where we need some guidance on the recommended values. They are noted. </p> <p>If there are any questions please do not hesitate to ask. Likely there will be more comments from ORNL folks as I publicize this jira ticket.</p> LU-4147

LNET parameter recommendations

Bug Minor Resolved Fixed James Nunez Jason Hill Fri, 25 Oct 2013 19:10:45 +0000 Mon, 18 Nov 2013 15:46:19 +0000 Mon, 18 Nov 2013 15:46:19 +0000 Lustre 2.4.1 0 9 <p>Liang, </p> <p>Would you please comment on the proposed lnet/ko2iblnd/kgnilnd values for ORNL's systems?</p> <p>Thank you,<br/> James</p> <p>Sorry I just come back from vacation, maybe it's too late to add comment, but…</p> <ul> <li>server <ul> <li>o2iblnd::fmr/pmr_pool_size and fmr_flush_trigger, they are unused unless "map_on_demand" is set or it's running on some of Chelsio infiniband</li> <li>o2iblnd::ntx, I think 4096 is a good value in this case.</li> <li>o2iblnd::keepalive and timeout, because timeout is 100 in your setting (timeout value of a TX), it's probably meaningful to have keepalive &gt;= timeout</li> <li>o2iblnd::peer_credits, any reason that you want to increase it to 63? It's ok if it's a empirical value, otherwise I think 16 can saturate most links (I remember some users report that 16 peercredits can saturate FDR infiniband)</li> <li>o2iblnd::peertimeout, this is only checked on router</li> </ul> </li> <li>RTR <ul> <li>o2iblnd::ntx, double of credits could be fine, it wouldn't consume too much memory.</li> <li>o2iblnd::peer_credits, same as comment for server</li> <li>o2iblnd::peer_buffer_credits, this means LNet will ignore lent::peer_buffer_credits, but lnet::peer_buffer_credits is also set in the proposal. Also, is current value (128) an empirical value?</li> </ul> </li> <li>o2iblnd client <ul> <li>same comment as server</li> </ul> </li> </ul> <p>When would you consider using map_on_demand?</p> <p>I would say, unless there is performance issue while benchmarking with lnet_selftest bulk, or memory allocation failure in o2iblnd while running w/o map_on_demand (they can happen on some HCAs), otherwise please just keep it turned off.</p> <p>Jason or James, </p> <p>Are there any more LNET parameter questions/issues that we need to address or should we close this ticket?</p> <p>Thanks, <br/> James</p> <p>Go ahead and close this. We've implemented the changes already. Some of the notes from Liang were too late to influence what we put in production but it has not seemed to be to any detriment. </p> <p>Request completed.</p> Development Rank 1|hzw6wf: Rank (Obsolete) 11262 Severity