Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-6850] Remove use of ib_reg_phys_mr() from o2iblnd https://jira.whamcloud.com/browse/LU-6850 Lustre <p>I was informed by Chuck Lever that the kernel guys consider ib_reg_phys_mr() obsolete and will be removing it from upstream OFED. We need to remove use of this routine from o2iblnd or change to a more appropriate API.</p> LU-6850

Remove use of ib_reg_phys_mr() from o2iblnd

Bug Critical Resolved Fixed Amir Shehata Doug Oucharek Tue, 14 Jul 2015 21:10:39 +0000 Mon, 30 Jan 2017 21:55:21 +0000 Tue, 6 Oct 2015 14:19:16 +0000 Lustre 2.8.0 0 11 <p>Is it going to be replaced by something else or simply removed?</p> <p>This thread seems to explain what is going to happen: <a href="http://www.spinics.net/lists/linux-nfs/msg52108.html" class="external-link" target="_blank" rel="nofollow noopener">http://www.spinics.net/lists/linux-nfs/msg52108.html</a></p> <p>The ko2iblnd driver needs to change to using FRMR asap.</p> <p>Amir</p> <p>Could you please look into what is going to be required here?</p> <p>Thanks</p> <p>Peter</p> <p>Looks like ib_reg_phys_mr() was used as a fallback for ib_get_dma_mr(), which is guaranteed to work except perhaps for some weird Chelsio HCA. I remembered Liang said he'd want to remove the hacks for some Chelsio support, so looks like ib_reg_phys_mr() call can be removed together with the Chelsio hacks Liang wanted to remove.</p> <p>Liang, can you please confirm?</p> <p>Looks like the patch to remove this call has been submitted already to the kernel. Looks like we will need to fix this very soon.<br/> Here is the patch on the RDMA discussion group:</p> <p>----<del>Original Message</del>----<br/> From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-<br/> owner@vger.kernel.org] On Behalf Of Chuck Lever<br/> Sent: Monday, July 20, 2015 12:05 PM<br/> To: linux-rdma@vger.kernel.org; linux-nfs@vger.kernel.org<br/> Subject: <span class="error">[PATCH v3 15/15]</span> core: Remove the ib_reg_phys_mr() and<br/> ib_rereg_phys_mr() verbs</p> <p>The verbs are obsolete. The ib_rereg_phys_mr() verb is not used by kernel<br/> ULPs, and the last ib_reg_phys_mr() call site in the kernel tree has now been<br/> removed.</p> <p>Two staging tree call sites remain in the Lustre client. The Lustre team has<br/> been notified of the deprecation of reg_phys_mr.</p> <p>Signed-off-by: Chuck Lever <chuck.lever@oracle.com><br/> Acked-by: Doug Ledford <dledford@redhat.com><br/> —<br/> drivers/infiniband/core/verbs.c | 67 ---------------------------------------<br/> include/rdma/ib_verbs.h | 46 ---------------------------<br/> 2 files changed, 113 deletions<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/forbidden.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c<br/> index bac3fb4..30eb245 100644<br/> — a/drivers/infiniband/core/verbs.c<br/> +++ b/drivers/infiniband/core/verbs.c<br/> @@ -1144,73 +1144,6 @@ struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int<br/> mr_access_flags) } EXPORT_SYMBOL(ib_get_dma_mr);</p> <p>-struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd,</p> <ul class="alternate" type="square"> <li>struct ib_phys_buf *phys_buf_array,</li> <li>int num_phys_buf,</li> <li>int mr_access_flags,</li> <li>u64 *iova_start)<br/> -{</li> <li>struct ib_mr *mr;</li> <li>int err;<br/> -</li> <li>err = ib_check_mr_access(mr_access_flags);</li> <li>if (err)</li> <li>return ERR_PTR(err);<br/> -</li> <li>if (!pd->device->reg_phys_mr)</li> <li>return ERR_PTR(-ENOSYS);<br/> -</li> <li>mr = pd->device->reg_phys_mr(pd, phys_buf_array, num_phys_buf,</li> <li>mr_access_flags, iova_start);<br/> -</li> <li>if (!IS_ERR(mr)) { - mr->device = pd->device; - mr->pd = pd; - mr->uobject = NULL; - atomic_inc(&pd->usecnt); - atomic_set(&mr->usecnt, 0); - } <p>-</p></li> <li>return mr;<br/> -}<br/> -EXPORT_SYMBOL(ib_reg_phys_mr);<br/> -<br/> -int ib_rereg_phys_mr(struct ib_mr *mr,</li> <li>int mr_rereg_mask,</li> <li>struct ib_pd *pd,</li> <li>struct ib_phys_buf *phys_buf_array,</li> <li>int num_phys_buf,</li> <li>int mr_access_flags,</li> <li>u64 *iova_start)<br/> -{</li> <li>struct ib_pd *old_pd;</li> <li>int ret;<br/> -</li> <li>ret = ib_check_mr_access(mr_access_flags);</li> <li>if (ret)</li> <li>return ret;<br/> -</li> <li>if (!mr->device->rereg_phys_mr)</li> <li>return -ENOSYS;<br/> -</li> <li>if (atomic_read(&mr->usecnt))</li> <li>return -EBUSY;<br/> -</li> <li>old_pd = mr->pd;<br/> -</li> <li>ret = mr->device->rereg_phys_mr(mr, mr_rereg_mask, pd,</li> <li>phys_buf_array, num_phys_buf,</li> <li>mr_access_flags, iova_start);<br/> -</li> <li>if (!ret && (mr_rereg_mask & IB_MR_REREG_PD)) { - atomic_dec(&old_pd->usecnt); - atomic_inc(&pd->usecnt); - } <p>-</p></li> <li>return ret;<br/> -}<br/> -EXPORT_SYMBOL(ib_rereg_phys_mr);<br/> -<br/> int ib_query_mr(struct ib_mr *mr, struct ib_mr_attr *mr_attr) { return mr->device->query_mr ? diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index b0f898e..43c1cf0 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -2760,52 +2760,6 @@ static inline void ib_dma_free_coherent(struct ib_device *dev, }</li> </ul> <p>/**</p> <ul class="alternate" type="square"> <li>* ib_reg_phys_mr - Prepares a virtually addressed memory region for use</li> <li>* by an HCA.</li> <li>* @pd: The protection domain associated assigned to the registered region.</li> <li>* @phys_buf_array: Specifies a list of physical buffers to use in the</li> <li>* memory region.</li> <li>* @num_phys_buf: Specifies the size of the phys_buf_array.</li> <li>* @mr_access_flags: Specifies the memory access rights.</li> <li>* @iova_start: The offset of the region's starting I/O virtual address.</li> <li>*/<br/> -struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd,</li> <li>struct ib_phys_buf *phys_buf_array,</li> <li>int num_phys_buf,</li> <li>int mr_access_flags,</li> <li>u64 *iova_start);<br/> -<br/> -/**</li> <li>* ib_rereg_phys_mr - Modifies the attributes of an existing memory region.</li> <li>* Conceptually, this call performs the functions deregister memory region</li> <li>* followed by register physical memory region. Where possible,</li> <li>* resources are reused instead of deallocated and reallocated.</li> <li>* @mr: The memory region to modify.</li> <li>* @mr_rereg_mask: A bit-mask used to indicate which of the following</li> <li>* properties of the memory region are being modified.</li> <li>* @pd: If %IB_MR_REREG_PD is set in mr_rereg_mask, this field specifies</li> <li>* the new protection domain to associated with the memory region,</li> <li>* otherwise, this parameter is ignored.</li> <li>* @phys_buf_array: If %IB_MR_REREG_TRANS is set in mr_rereg_mask, this</li> <li>* field specifies a list of physical buffers to use in the new</li> <li>* translation, otherwise, this parameter is ignored.</li> <li>* @num_phys_buf: If %IB_MR_REREG_TRANS is set in mr_rereg_mask, this</li> <li>* field specifies the size of the phys_buf_array, otherwise, this</li> <li>* parameter is ignored.</li> <li>* @mr_access_flags: If %IB_MR_REREG_ACCESS is set in mr_rereg_mask,<br/> this</li> <li>* field specifies the new memory access rights, otherwise, this</li> <li>* parameter is ignored.</li> <li>* @iova_start: The offset of the region's starting I/O virtual address.</li> <li>*/<br/> -int ib_rereg_phys_mr(struct ib_mr *mr,</li> <li>int mr_rereg_mask,</li> <li>struct ib_pd *pd,</li> <li>struct ib_phys_buf *phys_buf_array,</li> <li>int num_phys_buf,</li> <li>int mr_access_flags,</li> <li>u64 *iova_start);<br/> -<br/> -/**</li> </ul> <ul> <li>ib_query_mr - Retrieves information about a specific memory region.</li> <li>@mr: The memory region to retrieve information about.</li> <li>@mr_attr: The attributes of the specified memory region.</li> </ul> <p>–<br/> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body<br/> of a message to majordomo@vger.kernel.org More majordomo info at<br/> <a href="http://vger.kernel.org/majordomo-info.html" class="external-link" target="_blank" rel="nofollow noopener">http://vger.kernel.org/majordomo-info.html</a></p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/15788" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15788</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6850" title="Remove use of ib_reg_phys_mr() from o2iblnd " class="issue-link" data-issue-key="LU-6850"><del>LU-6850</del></a> lnet: remove references to ib_reg_phsy_mr()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 86cf812d2f8167aa6447b35a2d92f7dcdc3f9e59</p> <p>I just tried this with our Mellanox 2.4 stack using the mlx5 driver and it totally breaks ko2iblnd. If this patch is merged users stuck with hardware needing to use the mlx5 driver will no function. Before we delete this patch we need to support FRWR for the mlx5 driver. </p> <p>James: Do you have a link to documentation on FRWR? Is this something unique to MLX or can we switch to it for all IB access?</p> <p>It is unique to mlx5 driver. I will ask tomorrow one of my co-workers where I can get docs on this.</p> <p>Here is a link to docs : <a href="http://www.rdmaconsortium.org/home/RNIC_Verbs_Overview2.pdf" class="external-link" target="_blank" rel="nofollow noopener">http://www.rdmaconsortium.org/home/RNIC_Verbs_Overview2.pdf</a></p> <p>Also review the kernel thread called "IB/core: Introduce new fast registration API" and looked for comments by Jason Gunthorpe about how the API should be used</p> <p>Lastly the thread "Kernel fast memory registration API proposal <span class="error">[RFC]</span>" in the rdma list is very good.</p> <p>Another update on the latest infiniband API. It appears yet another new RDMA API has been purposed. If that lands FMR will be removed as well. See ticket <a href="https://jira.whamcloud.com/browse/LU-5783" title="o2iblnd: investigate new memory registration mechanisms" class="issue-link" data-issue-key="LU-5783"><del>LU-5783</del></a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15788/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15788/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6850" title="Remove use of ib_reg_phys_mr() from o2iblnd " class="issue-link" data-issue-key="LU-6850"><del>LU-6850</del></a> lnet: remove references to ib_reg_phys_mr()<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0f34ed0c1caf5cbe8965af730466b11760de8782</p> <p>Landed for 2.8</p> Related LU-5783 Development Rank 1|hzxi6n: Rank (Obsolete) 9223372036854775807 Severity