https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9461 Lustre <p>The lustre client mount fail after update IB driver and Lustre client patch(<a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a>).<br/> Should I apply any other patch for new IB driver?</p> <p>mount fail error mesage<br/> [ 5713.280039] LNet: Using FastReg for registration<br/> [ 5713.370689] LNet: Added LNI 192.168.2.220@o2ib0 <span class="error">&#91;8/256/0/180&#93;</span><br/> [ 5736.543149] LNetError: 0:0:(o2iblnd_cb.c:3436:kiblnd_qp_event()) 192.168.2.201@o2ib0: Async QP event type 3<br/> [ 5743.539710] Lustre: 15524:0:(client.c:2111:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: <span class="error">&#91;sent 1493897510/real 1493897510&#93;</span> req@ffff881011ef0300 x1566465051328608/t0(0) o503-&gt;MGC192.168.2.201@o2ib0@192.168.2.201@o2ib0:26/25 lens 272/8416 e 0 to 1 dl 1493897517 ref 2 fl Rpc:X/0/ffffffff rc 0/-1<br/> [ 5743.539728] LustreError: 166-1: MGC192.168.2.201@o2ib0: Connection to MGS (at 192.168.2.201@o2ib0) was lost; in progress operations using this service will fail<br/> [ 5743.539899] LustreError: 15c-8: MGC192.168.2.201@o2ib0: The configuration from log 'hpcfs-client' failed (-5). This may be the result of communication errors between this node and the MGS, a bad configuration, or other errors. See the syslog for more information.<br/> [ 5743.540282] Lustre: Unmounted hpcfs-client<br/> [ 5743.544658] LustreError: 15524:0:(obd_mount.c:1449:lustre_fill_super()) Unable to mount (-5)</p> CentOS7.3<br/> Lustre 2.9.0 + cherry-picked as e4297ef38561f1e788ba73ca0c8078a09dc8c303<br/> MLNX_OFED_LINUX-4.0-2.0.0.1-rhel7.3<br/> IB: Mellanox ConnectX-4 adapter EDR LU-9461

lustre client mount fail after update IB driver and Lustre patch.

Bug Minor Resolved Duplicate Amir Shehata sebg-crd-pm llnl Mon, 8 May 2017 01:46:45 +0000 Mon, 18 Sep 2017 21:29:56 +0000 Wed, 30 Aug 2017 12:59:52 +0000 1 8 <p>Hi there</p> <p>The <a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a> patch will allow building with MOFED 4.0 but there are still known issues attempting to run.</p> <p>Doug</p> <p>Do you have any suggestions here</p> <p>Peter</p> <p>The "Async QP event type 3" is a IB_EVENT_QP_ACCESS_ERR.  This error will stop the connection from continuing and explains all the errors which follow in your logs.</p> <p>This error happens when a call to ib_create_qp() fails.  It could fail if the version of the parameters being passed in is wrong (i.e. the size of the parameter structure is incorrect).  This could be related to the other issue I am currently working on that involves MOFED 4.  If MOFED 4 has changed the structure we use as a parameter to this call and we have not adapted to that change, we could see an error like this.</p> <p>Does this error happen on each mount attempt or was this a one off?</p> <p>This error happen on each mount attempt.(the test lustre filesystem servers OFED is 3.4)<br/> And "Async QP event type 3" also happened when lustre servers mount mgs/mds/....with OFED4. </p> <p>I ran into this "Async" error at the same time as the issues I talk about in <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>.  They are related.  When I have a fix for <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>, this issue will be addressed as well.</p> <p>I think it's my fault.<br/> Could you try this patch on top of <a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a> and <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">diff --git a/lnet/klnds/o2iblnd/o2iblnd.c b/lnet/klnds/o2iblnd/o2iblnd.c index 047fe3c..ba7829b 100644 --- a/lnet/klnds/o2iblnd/o2iblnd.c +++ b/lnet/klnds/o2iblnd/o2iblnd.c @@ -1900,8 +1900,6 @@ again: <span class="code-keyword">return</span> n &lt; 0 ? n : -EINVAL; } - mr-&gt;iova = iov; - wr = &amp;frd-&gt;frd_fastreg_wr; memset(wr, 0, sizeof(*wr)); </pre> </div></div> <p>I'm curious as to why you would not want to set the mr-&gt;iova value?  Is this an unneeded step?</p> <p>Hi Doug, </p> <p>I believe the issue only applies to mlx5 cards using MOFED4.</p> <p>in MOFED4, mr-&gt;iova is set by ib_map_mr_sg()-&gt;ib_sg_to_pages()</p> <p>It doesn't make sense to reset mr-&gt;iova after calling ib_map_mr_sg().</p> <p>That line of code was introduced to address an similar issue, see the comments on</p> <p><a href="https://review.whamcloud.com/#/c/19168/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/19168/</a></p> <p>I've done some testing using MOFED4 + lustre-release with mlx4 cards forcing fast reg as well, so far I've seen no problems.</p> <p>I can mount lustre ok ( 2.9.57_69_g0bc1964 + <a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a> / <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> and this patch (- mr-&gt;iova = iov<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> )</p> <p>Is it ok to apply only <a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a> + <a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a> / <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> and (- mr-&gt;iova = iov<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> patchs to 2.9.0?<br/> Sould I apply any other patchs to Lustre2.9.0 for mlx5 cards using MOFED4 ? Thanks for your suggestion</p> <p>Li: Thank you for the information.  I checked and you are right, ib_map_mr_sg() does set mr-&gt;iova so that line is not needed (could cause a problem).  I will update <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> with a removal of that line.</p> <p><a href="https://jira.hpdd.intel.com/secure/ViewProfile.jspa?name=sebg-crd-pm" class="external-link" target="_blank" rel="nofollow noopener"> sebg-crd-pm</a>: Correct, you only need --<del><a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a></del>-- + <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> (with the removal of setting mr-&gt;iova) + <a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a>.</p> <p>We have just encountered this issue as well. Is it possible to have <a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a>, <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>, and <a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a> backported to the lustre 2.5 and 2.8 branches?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=dinatale2" class="user-hover" rel="dinatale2">dinatale2</a> can you please open a new ticket to track this request?</p> <p>Hi, </p> <p>I got create striped directory error in Lustre 2.10 with <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> patch (<a href="https://review.whamcloud.com/#/c/28237/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28237/</a>) for OFED4.0</p> <p><span class="error">&#91;root@hsm client&#93;</span># lfs mkdir -c 2 dir1<br/> error on LL_IOC_LMV_SETSTRIPE 'dir1' (3): Input/output error<br/> error: mkdir: create stripe dir 'dir1' failed</p> <p>Should I apply any other patch for this issue? Thanks</p> <p>This issue is being tracked under <a href="https://jira.whamcloud.com/browse/LU-9958" title="Create striped directory fail in 2.10(with LU-9500 patch) " class="issue-link" data-issue-key="LU-9958"><del>LU-9958</del></a></p> Related LU-9500 Development Rank 1|hzzc2f: Rank (Obsolete) 9223372036854775807 Severity