https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13307 Lustre <p>Add <tt>LCFG_NODEMAP_ADD_RANGEv6</tt> and <tt>LCFG_NODEMAP_DEL_RANGEv6</tt> records to hold IPv6 addresses. There may need to be some better semantics for what constitutes a "range" for IPv6 addresses, as we definitely do not want enumeration of addresses involved.</p> LU-13307

add LCFG_NODEMAP_ADD_RANGEv6 records for IPv6

Improvement Minor Resolved Fixed James A Simmons Andreas Dilger IPv6 Fri, 28 Feb 2020 12:43:16 +0000 Sat, 20 Jan 2024 14:29:38 +0000 Sat, 20 Jan 2024 14:29:38 +0000 Lustre 2.16.0 Lustre 2.16.0 0 5 <p>After recently digging into the nodemap code, I'm trying to figure out how to configure nodemaps with IPv6 addresses.</p> <p>My understanding of IPv6 is superficial, so it isn't clear to me how a HPC cluster of nodes would be addressed? In some documentation, the interface identifier is randomly assigned (formerly based on the Ethernet MAC address), so no "NID range" is possible, and the nodemap might need thousands of individual records to describe the nodes, yuck. It also seems possible that the interface identifier part of the IPv6 address would be sequentially assigned to the client nodes, which would map will to the existing nodemap functionality.</p> <p>For IPv4 addresses, nodemap stores the <tt>nrr_start_nid</tt> and <tt>nrr_end_nid</tt> range in a 32-byte <tt>struct nodemap_range_rec</tt> using the <tt>NODEMAP_RANGE_IDX</tt> keys. The nodemap index contains only 32-byte records, so it would be possible to store one, but not two, 20-byte IPv6 NIDs in a single record. </p> <p>To distinguish IPv6 NIDs in the nodemap config index, a new <tt>NODEMAP_RANGE6_IDX</tt> with <tt>nodemap_range6_rec</tt> records could be used:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> struct nodemap_range6_rec { __u64 nr6_count; struct nr6_start_nid; __u32 nr6_padding; }; </pre> </div></div> <p>Since the nodemap range does not need to have a different <tt>nud_size</tt>, <tt>nid_type</tt>, <tt>nid_num</tt> or IPv6 subnet, it would be possible to store the 20-byte <tt>struct lnet_nid</tt> plus a <tt>_<em>u64</tt> count of NIDs that are in this range. The <tt></em>_u64 nr6_count</tt> corresponds to the <a href="https://en.m.wikipedia.org/wiki/IPv6_address" class="external-link" target="_blank" rel="nofollow noopener">interface identifier</a> part of the IPv6 address, so is large enough to contain all NIDs in a given subnet, if necessary. </p> <p>This works for <b>inclusive</b> ranges (ie. nodes X-Y belong to a nodemap), but not very good to specify <b>exclusive</b> ranges (ie. "all other NIDs belong to this nodemap"). Any thoughts? </p> <p>James, do you have any clusters at ORNL that are configured with IPv6 addresses? It would be useful to see what the address assignments look like in that case. </p> <p>It is important to be aware that a network interface can (and usually does) have multiple IPv6 addresses.</p> <p>There is a link-local address which cannot be routed, there maybe a an address based on the hardware MAC address, there might be 1 or more temporary addresses that are assigned randomly and used as the source of outgoing connections and are discarded after a time when not in use.  There might be an address assigned by dhcp6.  And there could be addresses that are manually assigned.</p> <p>Servers typically have a manually assigned address which is published in the DNS.   Hosts that only act as a client could only have randomly assigned addresses - one permanent for the rare case when an incoming connection is needed, one or two which are temporary and recycles, and used for all outgoing connections.</p> <p>NFS and Lustre both  set IPV6_PREFER_SRC_PUBLIC so when connecting to a server the client will advertise a stable IPv6 address that will not expire.  However if there are multiple possible public source addresses for the chosen interface, there is not currently any code to choose between them.</p> <p>With this context: What are nodemaps used for?  grouping servers? grouping clients?  Both?</p> <p>Can we just assume that people who want to use nodemaps will configure appropriate IPv6 addresses and NIDs to make the mapping convenient?</p> <p>I would MUCH rather limit support to net-masks, avoiding any suggestion of numeric ranges which are not power-of-2 in size and alignment.</p> <p>Neil, nodemaps are used for managing client-specific configuration.</p> <p>Originally nodemaps were developed for UID/GID mapping for clients connecting from different "workgroup" clusters that are/were not managed with the same central user administration. They are also used to direct clients to mount only a subdirectory of the filesystem, and to limit root access/admin permissions to the rest of the filesystem (root squash, and more).</p> <p>Because nodemaps can control admin access on clients, the clients can also use strong authentication (shared secret key or Kerberos) in addition to the IP address to ensure that clients are not spoofing their address. </p> <p>Definitely the clients will need to use consistent source addresses, and/or advertise all of their addresses at connect time to the servers, so that the server knows that later RPCs are from the same client, or they will be rejected. </p> <p>It would seem reasonable to change <tt>nodemap_idx_type</tt> similar to my previous suggestion, but instead of (or in addition to) <tt>nr6_count</tt> to add a <tt>nr6_netmask_bits</tt> so that it can encode e.g. <tt>192.168.10.1/24</tt> or <tt>::1/52</tt> (or whatever is used for IPv6 addresses):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">enum</span> nodemap_idx_type { : NODEMAP_RANGE6_IDX = 6 }; struct nodemap_range6_rec { __u64 nr6_count; struct nr6_start_nid; __u32 nr6_netmask_bits; }; </pre> </div></div> <p>"James Simmons &lt;jsimmons@infradead.org&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53135" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53135</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13307" title="add LCFG_NODEMAP_ADD_RANGEv6 records for IPv6" class="issue-link" data-issue-key="LU-13307"><del>LU-13307</del></a> nodemap: have nodemap_add_member support large NIDs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4aa8457f95ee21d72a35d596082f34f6fbee1ace</p> <p>This patch allows a single nodemap entry to be specified but not a range of NIDs. I don't think this ticket should be closed when 53135 is landed, since we really need to have a solution for specifying multiple IPv6 NIDs into a range efficiently instead of listing 5000 NIDs individually for multiple nodemaps. </p> <p>Currently working on <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53135" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53135.</a></p> <p>Works for IPv4 and it mostly works for IPv6. Still some issues for final deleting of the nodemap due to the records support missing currently. Also for this patch only one large NID is stored for each nodemap.</p> <p> </p> <p>Got  <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53135" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53135 </a>fully working for IPv6 and IPv4. Its ready for review. Once this lands we can put lustre into a feature freeze.</p> <p>Per comments in the patch, it would be much better if IPv6 netmask could handle more than just a single NID per entry. Otherwise, I'm worried at the prospect of each connect walking a linked list of 5000+ clients in a nodemap on every connection to find if the NID belongs there or not. I don't think this is <b>fatal</b> as it would only apply to sites that actually do this, but it would be much better to fix it if possible before the release.</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53135/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53135/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13307" title="add LCFG_NODEMAP_ADD_RANGEv6 records for IPv6" class="issue-link" data-issue-key="LU-13307"><del>LU-13307</del></a> nodemap: have nodemap_add_member support large NIDs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0ea23e019454e03810d532758afa52aa054995d8</p> <p>Merged for 2.16</p> Related LU-10391 LU-3527 LU-14288 Development Rank 1|i00uiv: Rank (Obsolete) 9223372036854775807