https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16823 Lustre <p>When nodes are connecting to peers the sender should set a bit in the connection that it supports IPv6 (large) NIDs. This would inform LNet discovery whether it is safe to reply with large NIDs during connection or ping replies.</p> <p>At the Lustre level new clients should set an <tt>OBD_CONNECT2_LARGE_NID=0x100000000ULL</tt> flag in <tt>obd_connect_data</tt> so that the MGS knows whether it can safely reply with large NIDs in <tt>mgs_nidtbl_entry</tt>. That avoids the need to backport a patch (ala <a href="https://jira.whamcloud.com/browse/LU-13306" title="allow clients to accept mgs_nidtbl_entry with IPv6 NIDs" class="issue-link" data-issue-key="LU-13306"><del>LU-13306</del></a>) to allow old clients to mount a server with both IPv4 and IPv6 NIDs configured. </p> LU-16823

add LNet and OBD connect flags for IPv6 peers

Improvement Minor Resolved Fixed James A Simmons Andreas Dilger IPv6 Thu, 11 May 2023 19:26:49 +0000 Sun, 7 Jan 2024 18:14:34 +0000 Wed, 3 Jan 2024 14:23:39 +0000 Lustre 2.16.0 0 4 <p>I haven't looked into all of the details here, but this would potentially allow a better alternative to patching old clients to ignore large NIDs.</p> <p><a href="https://review.whamcloud.com/#/c/fs/lustre-release/+/51108" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/fs/lustre-release/+/51108</a> covers the goal of this ticket.</p> <p>While 51108 landed we do need one more patch here to communicate that the MGS supports large NIDs.</p> <p>James, adding the handling for the <tt>OBD_CONNECT2_LARGE_NID</tt> feature is relatively straight forward:</p> <ul class="alternate" type="square"> <li>the 51108 patch has already handled the mechanics of adding a new connect flag (definition, wiretest/wirecheck, <tt>obd_connect_names[]</tt>, etc.)</li> </ul> <p>When the client and MGS support for handling large NIDs is finished, a patch should be pushed that:</p> <ul class="alternate" type="square"> <li>adds this flag to <tt>data-&gt;ocd_connect_flags2</tt> on the client when they are connecting to the MGS (and possibly MDS and OSS, not sure)</li> <li>adds this flag to <tt>MGS_CONNECT_SUPPORTED2</tt> (and <tt>MDS_CONNECT_SUPPORTED2</tt> and <tt>OSS_CONNECT_SUPPORTED2</tt> if needed)</li> <li>the old MGS will mask off <tt>OBD_CONNECT2_LARGE_NID</tt> in the reply to new clients, because it is not in the old <tt>MGS_CONNECT_SUPPORTED2</tt></li> <li>the new MGS will reply with <tt>OBD_CONNECT2_LARGE_NID</tt> to clients that send it</li> <li>the MGS can check this on client exports to determine if they support large NIDs, as needed</li> <li>clients can check this to determine if the MGS supports large NIDs, as needed</li> </ul> <p>We don't want to land the "supported" patch until the code is (substantially) working, otherwise a client/server might advertise their support for this feature, but not actually work yet.</p> <p>I was looking at adding the final touches for this work and I found one place its not a easy replacement. For lmv_setup() we call LNetGetId() which only gets small size NIDs. I don't see an easy way to get the connect flag here.  Any suggestions? Should we move to another setup function that has an export as a parameter?</p> <p>James, are you referring to this hunk of code to initialize the <tt>lmv_qos_rr_index</tt> starting value:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* * initialize rr_index to lower 32bit of netid, so that client * can distribute subdirs evenly from the beginning. */ <span class="code-keyword">while</span> (LNetGetId(i++, &amp;lnet_id, <span class="code-keyword">false</span>) != -ENOENT) { <span class="code-keyword">if</span> (!nid_is_lo0(&amp;lnet_id.nid)) { lmv-&gt;lmv_qos_rr_index = ntohl(lnet_id.nid.nid_addr[0]); <span class="code-keyword">break</span>; } } </pre> </div></div> <p>That code doesn't really need to have the full NID. The main goal is that each client is initialized in some way to a well-balanced starting value (instead of 0) so that clients don't all start creating subdirectories on MDT0000 and go in lockstep across MDTs. Using the NID is deterministic and likely gives us a more uniform distribution compared to a purely random number, because it normally only changes the low bits among nearby clients in a single job. It could use any other kind of value that is slowly changing for each client, but it would need to be available early during the mount process.</p> <p>I don't know if IPv6 NIDs have the same property or not, or if they have too much "random" stuff in them that they may be wildly imbalanced?</p> <p>We could potentially have the MDS assign each client a sequential "client number" for this purpose in the mount reply, but that might be imbalanced for clients that are allocated into the same job because it would only "globally" be uniform (though still better than a purely random number).</p> <p>In any case, the <tt>lmv_qos_rr_index</tt> doesn't have to be perfect, as it drifts over time anyway, but it avoids the "thundering herd" problem at initial mount time.</p> <p>A similar solution is needed for <tt>lmv_select_statfs_mdt()</tt> to select an MDT to send <tt>MDS_STATFS</tt> RPCs to:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* choose initial MDT <span class="code-keyword">for</span> <span class="code-keyword">this</span> client */</span> <span class="code-keyword">for</span> (i = 0;; i++) { struct lnet_processid lnet_id; <span class="code-keyword">if</span> (LNetGetId(i, &amp;lnet_id, <span class="code-keyword">false</span>) == -ENOENT) <span class="code-keyword">break</span>; <span class="code-keyword">if</span> (!nid_is_lo0(&amp;lnet_id.nid)) { /* We dont need a full 64-bit modulus, just enough * to distribute the requests across MDTs evenly. */ lmv-&gt;lmv_statfs_start = nidhash(&amp;lnet_id.nid) % lmv-&gt;lmv_mdt_count; <span class="code-keyword">break</span>; } } </pre> </div></div> <p>and it probably makes sense that these both use the same mechanism instead of fetching the NIDs each time.</p> <p>Then tendency is for IPv6 to be very random in an network. Need to think about a solution for this. Looking at lmv_setup() its processing an lcfg that contains a lmv_desc. Perhaps we can in mgs_llog.c create a record for lmv_desc with new info for the index to be used?</p> <p>"James Simmons &lt;jsimmons@infradead.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53398" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53398</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16823" title="add LNet and OBD connect flags for IPv6 peers" class="issue-link" data-issue-key="LU-16823"><del>LU-16823</del></a> lustre: test if large nid is support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 64bc4ffbff2c8a6927f8c9474887a32ec528e1b9</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53398/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53398/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16823" title="add LNet and OBD connect flags for IPv6 peers" class="issue-link" data-issue-key="LU-16823"><del>LU-16823</del></a> lustre: test if large nid is support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 165cf78ab54e6e8d172f999940c62afabc043cd5</p> <p>Work is complete</p> Related LU-10391 LU-13306 Development Rank 1|i03l7b: Rank (Obsolete) 9223372036854775807 Severity