Is it going to be replaced by something else or simply removed?
This thread seems to explain what is going to happen: http://www.spinics.net/lists/linux-nfs/msg52108.html
The ko2iblnd driver needs to change to using FRMR asap.
Amir
Could you please look into what is going to be required here?
Thanks
Peter
Looks like ib_reg_phys_mr() was used as a fallback for ib_get_dma_mr(), which is guaranteed to work except perhaps for some weird Chelsio HCA. I remembered Liang said he'd want to remove the hacks for some Chelsio support, so looks like ib_reg_phys_mr() call can be removed together with the Chelsio hacks Liang wanted to remove.
Liang, can you please confirm?
Looks like the patch to remove this call has been submitted already to the kernel. Looks like we will need to fix this very soon.
Here is the patch on the RDMA discussion group:
----Original Message----
From: linux-rdma-owner@vger.kernel.org [mailto:linux-rdma-
owner@vger.kernel.org] On Behalf Of Chuck Lever
Sent: Monday, July 20, 2015 12:05 PM
To: linux-rdma@vger.kernel.org; linux-nfs@vger.kernel.org
Subject: [PATCH v3 15/15] core: Remove the ib_reg_phys_mr() and
ib_rereg_phys_mr() verbs
The verbs are obsolete. The ib_rereg_phys_mr() verb is not used by kernel
ULPs, and the last ib_reg_phys_mr() call site in the kernel tree has now been
removed.
Two staging tree call sites remain in the Lustre client. The Lustre team has
been notified of the deprecation of reg_phys_mr.
Signed-off-by: Chuck Lever <chuck.lever@oracle.com>
Acked-by: Doug Ledford <dledford@redhat.com>
—
drivers/infiniband/core/verbs.c | 67 ---------------------------------------
include/rdma/ib_verbs.h | 46 ---------------------------
2 files changed, 113 deletions
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index bac3fb4..30eb245 100644
— a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1144,73 +1144,6 @@ struct ib_mr *ib_get_dma_mr(struct ib_pd *pd, int
mr_access_flags) } EXPORT_SYMBOL(ib_get_dma_mr);
-struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd,
- struct ib_phys_buf *phys_buf_array,
- int num_phys_buf,
- int mr_access_flags,
- u64 *iova_start)
-{ - struct ib_mr *mr;
- int err;
- - err = ib_check_mr_access(mr_access_flags);
- if (err)
- return ERR_PTR(err);
- - if (!pd->device->reg_phys_mr)
- return ERR_PTR(-ENOSYS);
- - mr = pd->device->reg_phys_mr(pd, phys_buf_array, num_phys_buf,
- mr_access_flags, iova_start);
- - if (!IS_ERR(mr))
{
- mr->device = pd->device;
- mr->pd = pd;
- mr->uobject = NULL;
- atomic_inc(&pd->usecnt);
- atomic_set(&mr->usecnt, 0);
- }
-
- return mr;
-}
-EXPORT_SYMBOL(ib_reg_phys_mr);
-
-int ib_rereg_phys_mr(struct ib_mr *mr, - int mr_rereg_mask,
- struct ib_pd *pd,
- struct ib_phys_buf *phys_buf_array,
- int num_phys_buf,
- int mr_access_flags,
- u64 *iova_start)
-{ - struct ib_pd *old_pd;
- int ret;
- - ret = ib_check_mr_access(mr_access_flags);
- if (ret)
- return ret;
- - if (!mr->device->rereg_phys_mr)
- return -ENOSYS;
- - if (atomic_read(&mr->usecnt))
- return -EBUSY;
- - old_pd = mr->pd;
- - ret = mr->device->rereg_phys_mr(mr, mr_rereg_mask, pd,
- phys_buf_array, num_phys_buf,
- mr_access_flags, iova_start);
- - if (!ret && (mr_rereg_mask & IB_MR_REREG_PD))
{
- atomic_dec(&old_pd->usecnt);
- atomic_inc(&pd->usecnt);
- }
-
- return ret;
-}
-EXPORT_SYMBOL(ib_rereg_phys_mr);
-
int ib_query_mr(struct ib_mr *mr, struct ib_mr_attr *mr_attr) { return mr->device->query_mr ? diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index b0f898e..43c1cf0 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -2760,52 +2760,6 @@ static inline void ib_dma_free_coherent(struct ib_device *dev, }
/**
- * ib_reg_phys_mr - Prepares a virtually addressed memory region for use
- * by an HCA.
- * @pd: The protection domain associated assigned to the registered region.
- * @phys_buf_array: Specifies a list of physical buffers to use in the
- * memory region.
- * @num_phys_buf: Specifies the size of the phys_buf_array.
- * @mr_access_flags: Specifies the memory access rights.
- * @iova_start: The offset of the region's starting I/O virtual address.
- */
-struct ib_mr *ib_reg_phys_mr(struct ib_pd *pd, - struct ib_phys_buf *phys_buf_array,
- int num_phys_buf,
- int mr_access_flags,
- u64 *iova_start);
-
-/** - * ib_rereg_phys_mr - Modifies the attributes of an existing memory region.
- * Conceptually, this call performs the functions deregister memory region
- * followed by register physical memory region. Where possible,
- * resources are reused instead of deallocated and reallocated.
- * @mr: The memory region to modify.
- * @mr_rereg_mask: A bit-mask used to indicate which of the following
- * properties of the memory region are being modified.
- * @pd: If %IB_MR_REREG_PD is set in mr_rereg_mask, this field specifies
- * the new protection domain to associated with the memory region,
- * otherwise, this parameter is ignored.
- * @phys_buf_array: If %IB_MR_REREG_TRANS is set in mr_rereg_mask, this
- * field specifies a list of physical buffers to use in the new
- * translation, otherwise, this parameter is ignored.
- * @num_phys_buf: If %IB_MR_REREG_TRANS is set in mr_rereg_mask, this
- * field specifies the size of the phys_buf_array, otherwise, this
- * parameter is ignored.
- * @mr_access_flags: If %IB_MR_REREG_ACCESS is set in mr_rereg_mask,
this - * field specifies the new memory access rights, otherwise, this
- * parameter is ignored.
- * @iova_start: The offset of the region's starting I/O virtual address.
- */
-int ib_rereg_phys_mr(struct ib_mr *mr, - int mr_rereg_mask,
- struct ib_pd *pd,
- struct ib_phys_buf *phys_buf_array,
- int num_phys_buf,
- int mr_access_flags,
- u64 *iova_start);
-
-/**
- ib_query_mr - Retrieves information about a specific memory region.
- @mr: The memory region to retrieve information about.
- @mr_attr: The attributes of the specified memory region.
–
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body
of a message to majordomo@vger.kernel.org More majordomo info at
http://vger.kernel.org/majordomo-info.html
Amir Shehata (amir.shehata@intel.com) uploaded a new patch: http://review.whamcloud.com/15788
Subject: LU-6850 lnet: remove references to ib_reg_phsy_mr()
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 86cf812d2f8167aa6447b35a2d92f7dcdc3f9e59
I just tried this with our Mellanox 2.4 stack using the mlx5 driver and it totally breaks ko2iblnd. If this patch is merged users stuck with hardware needing to use the mlx5 driver will no function. Before we delete this patch we need to support FRWR for the mlx5 driver.
James: Do you have a link to documentation on FRWR? Is this something unique to MLX or can we switch to it for all IB access?
It is unique to mlx5 driver. I will ask tomorrow one of my co-workers where I can get docs on this.
Here is a link to docs : http://www.rdmaconsortium.org/home/RNIC_Verbs_Overview2.pdf
Also review the kernel thread called "IB/core: Introduce new fast registration API" and looked for comments by Jason Gunthorpe about how the API should be used
Lastly the thread "Kernel fast memory registration API proposal [RFC]" in the rdma list is very good.
Another update on the latest infiniband API. It appears yet another new RDMA API has been purposed. If that lands FMR will be removed as well. See ticket LU-5783
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/15788/
Subject: LU-6850 lnet: remove references to ib_reg_phys_mr()
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 0f34ed0c1caf5cbe8965af730466b11760de8782
Landed for 2.8