https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5092 Lustre <p>This ticket relates to transfer of idmap information between the server nodes. The goal is that the MGS stores the UID/GID maps in an index (similar to how quota is storing the master quota files) and these are transferred to the other servers to keep the mappings consistent.</p> LU-5092

nodemap: transfer idmaps between MGS, MDT, OST

Technical task LU-3291 Minor Resolved Fixed Kit Westneat Kit Westneat patch Tue, 20 May 2014 19:14:57 +0000 Thu, 23 Mar 2017 19:33:42 +0000 Thu, 11 Aug 2016 12:00:37 +0000 Lustre 2.7.0 Lustre 2.9.0 0 12 <p>Kit had a question about API usage for creating the mapping indices as named objects on the MGS and transferring these to the servers. There may be an arbitrary number of maps for different remote systems, so a single reserved FID is not going to be sufficient. It probably makes sense to reserve a FID sequence for nodemap to hold all of the idmaps, and the OIDs in this sequence are used as needed for different idmaps.</p> <p>Should these idmaps be created under CONFIGS, or in their own top-level directory? It probably makes sense to put it under CONFIGS (maybe as a subdirectory), since AFAIK the MGS OSDs namespace is only under CONFIGS, and we don't want to mess with files under / and confuse a shared MDS OSD?</p> <p>Is there any documentation on how to create these objects via OSD API, or examples for Kit to follow?</p> <p>Thanks for creating this ticket for me Andreas. Here's a quick recap of what I've tried and the results I've gotten. </p> <p>I was planning on just using one index file and holding all the maps within it, at least to start with. So a sequence might not be necessary, I'm not sure. I added an OID to enum local_oid, and called:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lu_local_obj_fid(&amp;fid, NODEMAP_OID); </pre> </div></div> <p>and added this to the osd_lf_maps:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-keyword">const</span> struct osd_lf_map osd_lf_maps[] <span class="code-comment">/* nodemap */</span> { <span class="code-quote">"nodemap"</span>, { FID_SEQ_LOCAL_FILE, NODEMAP_OID, 0 }, OLF_SHOW_NAME, NULL, NULL }, </pre> </div></div> <p>I then tried to create the file with local_index_find_or_create_with_fid, but the dt_devices did not like that:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> nodemap_idx = local_index_find_or_create_with_fid(env, dev, &amp;fid, parent, nodemap_idx_filename, mode, &amp;dt_nodemap_features); </pre> </div></div> <p>But that got me a couple different LBUGs. Using mgs_bottom:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">&lt;0&gt;LustreError: 1332:0:(mdd_object.c:107:mdd_env_info()) ASSERTION( info != ((void *)0) ) failed: &lt;0&gt;LustreError: 1332:0:(mdd_object.c:107:mdd_env_info()) LBUG &lt;4&gt;Pid: 1332, comm: lctl &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0312895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa0312e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0d2f5e1&gt;] mdd_env_info+0x61/0x70 [mdd] &lt;4&gt; [&lt;ffffffffa0d2ffbd&gt;] mdd_object_start+0x4d/0x100 [mdd] &lt;4&gt; [&lt;ffffffffa047e42a&gt;] lu_object_alloc+0x12a/0x320 [obdclass] &lt;4&gt; [&lt;ffffffffa047ef84&gt;] lu_object_find_at+0x204/0x350 [obdclass] &lt;4&gt; [&lt;ffffffffa047e94d&gt;] ? lu_object_put+0xad/0x330 [obdclass] &lt;4&gt; [&lt;ffffffffa048158c&gt;] dt_locate_at+0x1c/0xa0 [obdclass] &lt;4&gt; [&lt;ffffffffa0461a36&gt;] local_index_find_or_create_with_fid+0x196/0x220 [obdclass] </pre> </div></div> <p>Using mgs_dt_dev:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">&lt;0&gt;LustreError: 4598:0:(mgs_handler.c:1311:mgs_object_alloc()) ASSERTION( hdr == ((void *)0) ) failed: &lt;0&gt;LustreError: 4598:0:(mgs_handler.c:1311:mgs_object_alloc()) LBUG &lt;4&gt;Pid: 4598, comm: lctl &lt;4&gt; &lt;4&gt;Call Trace: &lt;4&gt; [&lt;ffffffffa0312895&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] &lt;4&gt; [&lt;ffffffffa0312e97&gt;] lbug_with_loc+0x47/0xb0 [libcfs] &lt;4&gt; [&lt;ffffffffa0b91d98&gt;] mgs_object_alloc+0x148/0x220 [mgs] &lt;4&gt; [&lt;ffffffffa045f5e8&gt;] ls_object_init+0x38/0x140 [obdclass] &lt;4&gt; [&lt;ffffffffa047e3d8&gt;] lu_object_alloc+0xd8/0x320 [obdclass] &lt;4&gt; [&lt;ffffffffa047eea9&gt;] lu_object_find_at+0x129/0x350 [obdclass] &lt;4&gt; [&lt;ffffffffa048158c&gt;] dt_locate_at+0x1c/0xa0 [obdclass] &lt;4&gt; [&lt;ffffffffa0460d12&gt;] __local_file_create+0x72/0x8c0 [obdclass] &lt;4&gt; [&lt;ffffffffa04619ed&gt;] local_index_find_or_create_with_fid+0x14d/0x220 [obdclass] </pre> </div></div> <p>Adding dt_nodemap_features in case that's useful to see:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-comment">/* nodemap files */</span> <span class="code-keyword">const</span> struct dt_index_features dt_nodemap_features = { .dif_flags = DT_IND_UPDATE, .dif_keysize_min = sizeof(__u64), <span class="code-comment">/* 64-bit nodemap id/record id */</span> .dif_keysize_max = sizeof(__u64), <span class="code-comment">/* 64-bit nodemap id/record id */</span> .dif_recsize_min = sizeof(struct nodemap_rec), <span class="code-comment">/* 32 bytes */</span> .dif_recsize_max = sizeof(struct nodemap_rec), <span class="code-comment">/* 32 bytes */</span> .dif_ptrsize = 4 }; EXPORT_SYMBOL(dt_nodemap_features); </pre> </div></div> <p>Good to see you still around Kit! I have updated the reporter field to show you as the originator of this ticket</p> <p>Thanks Peter! I was wondering when someone might be able to take a look at this. I'm a bit reticent to invest a ton more time on this work until I know that I am headed in the right direction. </p> <p>Kit</p> <p>Most likely not until 2.6 goes GA and work on 2.7 starts.</p> <p>Peter</p> <p>Peter, Kit is mostly just looking for some advice from Alex, Niu, and Johann about the right way to implement the nodemap transfer between the MDT and OST. He is planning on using the quota index transfer that Johann implemented in 2.4 as the model for the UID/GID mapping transfer. It makes sense to provide feedback on whether this is the right approach to begin with, and guidance on the right implementation <em>before</em> it is done rather than afterward.</p> <p>Thanks Andreas. Yes, I understood that Kit wanted guidance up front before starting in earnest on this approach. I was just trying to give him a rough estimate as to when I expected people to be free to do so.</p> <p>I've been chipping away at this some. Specifically I've been looking at using the mgs_config_read RPC and returning all the index entries in the response body. The issue I'm running into is that mgs_config_res response only contains an offset, while the indexes use offset and version. Because llogs entries are immutable with monotonically increasing IDs, the offset and version are essentially the same, so the client can just ask for records older than its last record. The index offsets however are based on the hashes of the indices, and so the index file has a separate version variable. By examining the version variable, the client can see if the index file has changed while it was reading (between RPCs). The lack of version in the mgs_config_read RPC response makes it hard to send the index file. </p> <p>Here are some of my thoughts:</p> <ul class="alternate" type="square"> <li>A straight forward solution would be to add the version to the response or request. I'm not sure how feasible that is though. There is also a logname field in the request which maybe could be repurposed?</li> </ul> <ul class="alternate" type="square"> <li>Converting the on-disk format from indexes to llogs might make it easier to pass by RPC. The concern with llogs is that they have a fixed number of change entries before they need to be recreated, and the nodemaps might exceed that. One llog per nodemap might be feasible, but if someone makes a lot of changes to a nodemap, it's likely going to eventually fail. On the other hand, it's fairly easy to delete and readd a nodemap, which could potentially recreate the llog file.</li> </ul> <ul class="alternate" type="square"> <li>Another idea would be to store the last requested index version number with the MGC export. Then when the MGC requests the index with an offset, the MGC could return an error if the index version has changed since the offset 0 request.</li> </ul> <p>What do you all think is the best way to go about this? </p> <p>Hi, Kit. OBD_IDX_READ RPC was designed for transferring index files, I think you can use it to transfer UID/GID mapping files. (needs adding such PRC handler for MGS).</p> <p>patch to create and save nodemaps:<br/> <a href="http://review.whamcloud.com/11813" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11813</a></p> <p>patch to transfer nodemaps:<br/> <a href="http://review.whamcloud.com/11830" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11830</a></p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/14254" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14254</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: add structure to hold nodemap config<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fe0fde70326348ad17ac25490eac946c438e4eb9</p> <p>Hi Kit. I update all your patches related to this ticket. Also added in node map data being stored for ZFS besides ldiskfs support. The only reservation I have for 14254 is that nm_id is to small. I think using unsigned int is too small. I could easily see our center wide file system exhausted id namespace quickly.</p> <p>James, I'm not sure why you think a 32-bit nm_id would run out? The nm_id is for the number of different remote clusters with different UID spaces. It doesn't relate to the number of remote nodes in a given mapping or the range of UID or GID values that can be mapped.</p> <p>Ah I see. I was under the impression it was related to the number of nodes plus the UID space in the the cluster. I was thinking of the different use cases if you had 25K clients and thousands of users total like we do and in that case it would be possible to exhaust the space. Since this is not the case then that size should be large enough (famous last words).</p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/14885" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14885</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: convert member hash to a list<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: eb202c118887f73c17dfe47cca4823f643088499</p> <p>Sorry Kit but my cfs_hash cleanups broke your patches. I updated the first two in the series. Having trouble updating the 3rd one so far. Hope that helps you out.</p> <p>Thanks James, I put a comment on the first one. I'll update the third. Just to confirm, you mean change 14254?</p> <p>Yes change 14254. I seen your comment. I missed that change. Thanks for looking over the update.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14885/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14885/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: convert member hash to a list<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1c7f9caa99df3082a3e506673721d359147843d3</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/14254/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14254/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: add structure to hold nodemap config<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2e0d4e010eb4c2e28507f55c1b649d70bccd06a0</p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/16941" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16941</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: handle config changes while mid-flight<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 77833fc234a487ad7434a7d369a832cabd5c1aa2</p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/17503" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17503</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: save nodemaps to targets for caching<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fdc3bb0e8999181266b9f5f1b661d9cef25d5a8e</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/11813/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11813/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: save id maps to targets in new index file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 928714dddabb2dbc4fc93101f23eaa671099dbea</p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/18554" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18554</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: remove nodemap_idx_action, only act on MGS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 262ae321da1e2acfb13f42344016e21e3ff95cf7</p> <p>Kit Westneat (kit.westneat@gmail.com) uploaded a new patch: <a href="http://review.whamcloud.com/19674" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19674</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: users of ted_nodemap should take ref<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 08a81b6025bad4ed446088a4d4753eab34121d47</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19674/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19674/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: users of ted_nodemap should take ref<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 34ba26ea9884df0758f8ec8dd1047b4cf474af09</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/11830/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11830/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: transfer nodemaps between MGS, MDTs, and OSTs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 685b0943eb4fe55720458326a5d6c4949457bf7a</p> <p>Kit was is the dependency chain now? Currently it is confusing to know what order is important.</p> <p>Change 16941 allows for config files &gt; the size of 1 RPC</p> <p>Change 18554 sets up 17503, which caches the config on targets so they don't need the MGS to startup with nodemap.</p> <p>So change 16941 is stand alone, but 17503 requires that 18554 be landed first.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16941/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16941/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: handle config changes while mid-flight<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d3ca7a90b4a22908e212ef008fa78932541ef017</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18554/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18554/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: remove nodemap_idx_action, only act on MGS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: ffe234a077576f0f66f6fe9c165be180133be89a</p> <p>Kit</p> <p>It looks like there is still one patch - <a href="http://review.whamcloud.com/#/c/17503/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17503/</a> - being tracked under this ticket. Does that need to land for 2.9?</p> <p>Peter</p> <p>Hi Peter,</p> <p>It would be best if it did, but it's not mandatory. Without that patch, if the MGS goes away, the OSSes and MDSes will not be able to load the nodemap configuration if they get restarted.</p> <p>Thanks,<br/> Kit</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17503/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17503/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5092" title="nodemap: transfer idmaps between MGS, MDT, OST" class="issue-link" data-issue-key="LU-5092"><del>LU-5092</del></a> nodemap: save nodemaps to targets for caching<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: be7efb4ce06996444c08914305a73833a7123eeb</p> <p>Landed for 2.9</p> Related LU-3291 LU-6409 Development Rank 1|hzwmtj: Rank (Obsolete) 14037