https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5783 Lustre <p>The OFED has support of new memory registration mechanisms that the o2iblnd currently don't use.</p> <ul class="alternate" type="square"> <li>IB_WR_FAST_REG_MR: work request based asynchronous memory registration. It removes memory registration out of the critical path and has the potential of improving throughput.</li> <li>IB_WR_REG_INDIR_MR: allows registration of non-page aligned memory fragments. See:<br/> <a href="https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg21430.html" class="external-link" target="_blank" rel="nofollow noopener">https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg21430.html</a></li> </ul> <p>The 1st one may improve throughput and the 2nd one may solve the long-standing issue of "too many RDMA frags" failures we had (forgot which ticket, Liang?).</p> <p>We should do some investigation and decide whether to add support of these OFED features to the o2iblnd.</p> LU-5783

o2iblnd: investigate new memory registration mechanisms

Improvement Minor Resolved Fixed Dmitry Eremin Isaac Huang Tue, 21 Oct 2014 21:00:38 +0000 Sat, 4 Jun 2016 13:49:13 +0000 Fri, 13 May 2016 13:01:50 +0000 Lustre 2.9.0 0 18 <p>I believe this is the replacement for FMR in the mlx5 driver</p> <p>Just was told by a coworker that the above apis are being thrown out and replaced by a new RDMA api. When they API hits FMR will be removed from all drivers. You see the details in this thread:</p> <p><a href="http://article.gmane.org/gmane.linux.drivers.rdma/29040" class="external-link" target="_blank" rel="nofollow noopener">http://article.gmane.org/gmane.linux.drivers.rdma/29040</a></p> <p>Started work on a patch to support Fast Registration API.</p> <p>Dmitry Eremin (dmitry.eremin@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17606" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17606</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5783" title="o2iblnd: investigate new memory registration mechanisms" class="issue-link" data-issue-key="LU-5783"><del>LU-5783</del></a> o2iblnd: Add Fast Reg memory registration support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 833db95ca7ada950c06de0e5bbdd1ca3da83c3db</p> <p>I'm sorry, but i not able to post comments into gerrit directly.<br/> But patch is completely wrong. <br/> First of all - you don't need an alloc any pages via ib_alloc_fast_reg_page_list, but you need an just map pages from MD to dma addresses. <br/> second - you must change a QP attributes. you don't need so large WR / SGE limits if fast registration enabled - typically you needs an 7 WR * num_parallel sends.<br/> third - you must INVALIDATE a memory region after use, it's possible via READ_INV command, but for write you need additional WR with invalidate operation.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Consumer interface details: - A new device capability flag IB_DEVICE_MEM_MGT_EXTENSIONS is added to indicate device support for these features. - New send work request opcodes IB_WR_FAST_REG_MR, IB_WR_LOCAL_INV, IB_WR_RDMA_READ_WITH_INV are added. - A new consumer API function, ib_alloc_mr() is added to allocate fast register memory regions. - New consumer API functions, ib_alloc_fast_reg_page_list() and ib_free_fast_reg_page_list() are added to allocate and free device-specific memory for fast registration page lists. - A new consumer API function, ib_update_fast_reg_key(), is added to allow the key portion of the R_Key and L_Key of a fast registration MR to be updated. Consumers call this if desired before posting a IB_WR_FAST_REG_MR work request. Consumers can use this as follows: - MR is allocated with ib_alloc_mr(). - Page list memory is allocated with ib_alloc_fast_reg_page_list(). - MR R_Key/L_Key "key" field is updated with ib_update_fast_reg_key(). - MR made VALID and bound to a specific page list via ib_post_send(IB_WR_FAST_REG_MR) - MR made INVALID via ib_post_send(IB_WR_LOCAL_INV), ib_post_send(IB_WR_RDMA_READ_WITH_INV) or an incoming send with invalidate operation. - MR is deallocated with ib_dereg_mr() - page lists dealloced via ib_free_fast_reg_page_list(). </pre> </div></div> <p>This work might have flaws but it works like a charm on my mlx5 system.</p> <p>James,</p> <p>mlx5 system may work in older style. I have run o2iblnd without modification on own mlx5 system. why not ?<br/> FRM codepath is inactive for common use, it's activated just with map_on_demand option. In common case o2iblnd uses an global MR mapping </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (hdev-&gt;ibh_mr_shift == 64) { LIBCFS_ALLOC(hdev-&gt;ibh_mrs, 1 * sizeof(*hdev-&gt;ibh_mrs)); <span class="code-keyword">if</span> (hdev-&gt;ibh_mrs == NULL) { CERROR(<span class="code-quote">"Failed to allocate MRs table\n"</span>); <span class="code-keyword">return</span> -ENOMEM; } hdev-&gt;ibh_mrs[0] = NULL; hdev-&gt;ibh_nmrs = 1; mr = ib_get_dma_mr(hdev-&gt;ibh_pd, acflags); <span class="code-keyword">if</span> (IS_ERR(mr)) { CERROR(<span class="code-quote">"Failed ib_get_dma_mr : %ld\n"</span>, PTR_ERR(mr)); kiblnd_hdev_cleanup_mrs(hdev); <span class="code-keyword">return</span> PTR_ERR(mr); } hdev-&gt;ibh_mrs[0] = mr; <span class="code-keyword">goto</span> out; } </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">struct ib_mr * kiblnd_find_rd_dma_mr(kib_hca_dev_t *hdev, kib_rdma_desc_t *rd) { struct ib_mr *prev_mr; struct ib_mr *mr; <span class="code-object">int</span> i; LASSERT (hdev-&gt;ibh_mrs[0] != NULL); <span class="code-keyword">if</span> (*kiblnd_tunables.kib_map_on_demand &gt; 0 &amp;&amp; *kiblnd_tunables.kib_map_on_demand &lt;= rd-&gt;rd_nfrags) <span class="code-keyword">return</span> NULL; <span class="code-keyword">if</span> (hdev-&gt;ibh_nmrs == 1) </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-comment">/* looking <span class="code-keyword">for</span> pre-mapping MR */</span> mr = kiblnd_find_rd_dma_mr(hdev, rd); <span class="code-keyword">if</span> (mr != NULL) { <span class="code-comment">/* found pre-mapping MR */</span> rd-&gt;rd_key = (rd != tx-&gt;tx_rd) ? mr-&gt;rkey : mr-&gt;lkey; <span class="code-keyword">return</span> 0; } <span class="code-keyword">if</span> (net-&gt;ibn_fmr_ps != NULL) <span class="code-keyword">return</span> kiblnd_fmr_map_tx(net, tx, rd, nob); <span class="code-keyword">else</span> <span class="code-keyword">if</span> (net-&gt;ibn_pmr_ps != NULL) <span class="code-keyword">return</span> kiblnd_pmr_map_tx(net, tx, rd, nob); </pre> </div></div> <p>So just enough to send a local dma address for mapped page as part of IB_WR_RDMA_WRITE / IB_WR_RDMA_READ request.<br/> all pages sends in WR chain.</p> <p>simple check. if fast reg used - you may safety drop number WR per QP from 256 * num_parallel_send to 5*parallel_sends.<br/> explanation - with older code - you need a fill remote address + R_key for each IB_WR_RDMA_WRITE / IB_WR_RDMA_READ request. But with IB memory extension you able to have a single WR request to point whole transfer. Page list to fill an tranfer will take from fast reg operation and addressed by special L_Key.</p> <p>It's way to solve problems with OOM on nodes reported may times in past.</p> <p>Actually for my mlx5 stack testing I am enabling map_on_demand and it is working.</p> <p>well, with enabling map_on_depand it may work, but anyway.<br/> 1) IB_WR_RDMA_WRITE_IMM, IB_WR_RDMA_READ_INV should be reused to avoid extra WR.<br/> 2) total number WR per QP need adjusted to reflect extra WR used for mapping purpose.<br/> 3) don't prefer a FMR if exist. Fast Reg supported for most cards now, and it's better choose.<br/> 4) don't use a "invalidate-&gt;register" WR links it's directly affects (1).</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ <span class="code-keyword">if</span> (!frd-&gt;frd_valid) { + wrq = &amp;frd-&gt;frd_inv_wr; + wrq-&gt;next = &amp;frd-&gt;frd_fastreg_wr; + } <span class="code-keyword">else</span> { + wrq = &amp;frd-&gt;frd_fastreg_wr; + } </pre> </div></div> <p>It may a simplify a code by avoid calling a invalidate on error case, but it block IB card optimization. per IB spec WR workflow it should be in order<br/> 1) register<br/> 2) write/read<br/> 3) invalidate</p> <p>Oh I agree. Because something works doesn't mean it is done right <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> I agree with you that perhaps we should just remove FMR support at this since it will be going away.</p> <p>well, i know were you take an original code <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> but one note.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ <span class="code-comment">/* Prepare FASTREG WR */</span> + memset(&amp;fastreg_wr, 0, sizeof(fastreg_wr)); + fastreg_wr.opcode = IB_WR_FAST_REG_MR; + fastreg_wr.send_flags = IB_SEND_SIGNALED; + fastreg_wr.wr.fast_reg.iova_start = desc-&gt;data_frpl-&gt;page_list[0] + offset; + fastreg_wr.wr.fast_reg.page_list = desc-&gt;data_frpl; + fastreg_wr.wr.fast_reg.page_list_len = page_list_len; + fastreg_wr.wr.fast_reg.page_shift = SHIFT_4K; + fastreg_wr.wr.fast_reg.length = data_size; + fastreg_wr.wr.fast_reg.rkey = desc-&gt;data_mr-&gt;rkey; + fastreg_wr.wr.fast_reg.access_flags = (IB_ACCESS_LOCAL_WRITE | + IB_ACCESS_REMOTE_WRITE | + IB_ACCESS_REMOTE_READ); + </pre> </div></div> <p>so iser don't have two page lists as in patch. It uses directly an preallocated DMA region pages. Probably our's code will have two more memory usage if we don't use same pages to transfer and mapping.</p> <p>&gt; I agree with you that perhaps we should just remove FMR support at this since it will be going away.<br/> not at all. Yes, Fast reg introduced in 2008, but not at all mlx4 cards may support it (if you have a good melanox contact please ask him).</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> ((dev-&gt;dev-&gt;caps.bmme_flags &amp; MLX4_BMME_FLAG_LOCAL_INV) &amp;&amp; (dev-&gt;dev-&gt;caps.bmme_flags &amp; MLX4_BMME_FLAG_REMOTE_INV) &amp;&amp; (dev-&gt;dev-&gt;caps.bmme_flags &amp; MLX4_BMME_FLAG_FAST_REG_WR)) props-&gt;device_cap_flags |= IB_DEVICE_MEM_MGT_EXTENSIONS; </pre> </div></div> <p>so probably better to just lower priority for FMR.</p> <p>one more note - don't use a flag, but have two pointers to function to map and to unmap region, as code completely different. </p> <p>I know we have mthca cards in use which use FMR and I don't believe support fast reg so I'm fine with lowering the priority but not removing the FMR code altogether. I'm having some issues with the patch but there is already some good feedback here towards a second revision so I'll wait until then before I dig deeper.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17606/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17606/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5783" title="o2iblnd: investigate new memory registration mechanisms" class="issue-link" data-issue-key="LU-5783"><del>LU-5783</del></a> o2iblnd: Add Fast Reg memory registration support<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: f2600449ced3a2f78c4b3e650ff29bb88d1e7e45</p> <p>I like to see, Intel don't work in cooperations with other developers, as it land a patch without resolving comments in ticket.</p> <p>You don't review the patch in gerrit nor do you push a patch somewhere. I understand you don't want to use the gerrit system without gmail support but if you want to improve this work I suggest you create a github branch with your fix. I would gladly push the fix to gerrit for you. You need to figure a way to work with us.</p> <p>Intel don't fix a gerrit to ability to review. When you have a fix own gerrit to work with google ?<br/> I was asked for two years, new gerrit versions work fine with google, but Intel stick on older and buggy version.</p> <p>Alexey, we discussed your issue at the OpenSFS LWG and we decided to stick with the status quo.<br/> James, it was very gracious and helpful of you to offer to push a fix to gerri on behalf of Alexey. Alexey, I agree with James, you need to find a way to work with us. Please take him up on his offer, but I suggest you find a way to work with the community more generally. If you'd like the LWG to revisit this topic, please send mail to the LWG list.</p> <p>As far as aI can see this has been landed for 2.9</p> Related LU-6850 LU-3322 LU-7124 Development Rank 1|hzwz5z: Rank (Obsolete) 16229