https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9958 Lustre <p>Hi,<br/> I try to use OFED4.0 driver in Lustre 2.10 with <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> patch (<a href="https://review.whamcloud.com/#/c/28237/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28237/</a>) but got create stripe directory error. <br/> In <a href="https://jira.whamcloud.com/browse/LU-9461" title="lustre client mount fail after update IB driver and Lustre patch." class="issue-link" data-issue-key="LU-9461"><del>LU-9461</del></a>, I got infomation Lustre 2.9 have to apply <a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a>/<a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a>/ <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>.<br/> Then test it OFED4.0 in Lustre2.10 + <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> patch( <a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a>/<a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a> in Lustre 2.10 )<br/> Should I appy any other patch for this issue? Thanks.</p> <p>//two mdts must in different servers<br/> <span class="error">&#91;root@hsm client&#93;</span># lfs mkdir -c 2 dir1<br/> error on LL_IOC_LMV_SETSTRIPE 'dir1' (3): Input/output error<br/> error: mkdir: create stripe dir 'dir1' failed</p> Lustre 2.10 (lustre-release-58fd06e) + <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><strike>LU-9500</strike></a> patch for OFED4.0<br/> Melanox IB EDR + MLNX_OFED_LINUX-4.0-2.0.0.1-rhel7.3-x86_64.tgz<br/> Test with 2 MDS servers (1 MDT/Server) LU-9958

Create striped directory fail in 2.10(with LU-9500 patch)

Bug Critical Resolved Cannot Reproduce Lai Siyao sebg-crd-pm Fri, 8 Sep 2017 11:37:59 +0000 Sat, 14 Oct 2017 00:20:10 +0000 Thu, 5 Oct 2017 05:34:00 +0000 0 5 <p>Hello,</p> <p>Please attach the entire log for us to review.</p> <p>Thanks,</p> <p>Brad</p> <p>FYI</p> <p>lfs mkdir -c 2 /mnt/client/dir3<br/> error on LL_IOC_LMV_SETSTRIPE 'dir3' (3): Input/output error<br/> error: mkdir: create stripe dir 'dir3' failed</p> <p>see attached file <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/28256/28256_mdt1.log" title="mdt1.log attached to LU-9958">mdt1.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/28257/28257_mdt0.log" title="mdt0.log attached to LU-9958">mdt0.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/28258/28258_client.log" title="client.log attached to LU-9958">client.log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>Hi Brad,</p> <p>Do you have any update after reviewing logs?</p> <p>Thanks!</p> <p>Lai</p> <p>Can you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>in mdt1.log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00010000:00000001:6.0:1505117481.059504:0:5182:0:(ldlm_lib.c:3268:target_bulk_io()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) 00000020:00000001:6.0:1505117481.059508:0:5182:0:(out_handler.c:982:out_handle()) Process leaving via out_free (rc=18446744073709547410 : -4206 : 0xffffffffffffef92) </pre> </div></div> <p>which caused mdt0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:9.0:1505117529.647439:0:3485:0:(osp_trans.c:1204:osp_send_update_req()) Process leaving (rc=18446744073709551506 : -110 : ffffffffffffff92) ... 00000020:00000001:1.0:1505117529.647589:0:3442:0:(update_trans.c:1091:top_trans_stop()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) ... 00000004:00000001:1.0:1505117529.647779:0:3442:0:(mdt_reint.c:526:mdt_create()) Process leaving via put_child (rc=18446744073709551611 : -5 : 0xfffffffffffffffb) </pre> </div></div> <p>It looks to be network IO on mdt1 timed out, could you verify the network on mdt1 is working correctly?</p> <p>I test this bug again in 2.10.1-RC1(no add any patch).</p> <p>It looks to be network IO on mdt1 timed out, could you verify the network on mdt1 is working correctly?<br/> =&gt;all osp state looks like normal<br/> <span class="error">&#91;mdt1 server&#93;</span><br/> ./osp/jlustre-MDT0000-osp-MDT0001/state:current_state: FULL<br/> ./osp/jlustre-MDT0000-osp-MDT0001/import: state: FULL<br/> ./osp/jlustre-OST0000-osc-MDT0001/state:current_state: FULL<br/> ./osp/jlustre-OST0000-osc-MDT0001/import: state: FULL<br/> <span class="error">&#91;mdt0 server&#93;</span><br/> ./osp/jlustre-MDT0001-osp-MDT0000/state:current_state: FULL<br/> ./osp/jlustre-MDT0001-osp-MDT0000/import: state: FULL<br/> ./osp/jlustre-OST0000-osc-MDT0000/state:current_state: FULL<br/> ./osp/jlustre-OST0000-osc-MDT0000/import: state: FULL</p> <p><span class="error">&#91;/var/log/message in mdt0 server&#93;</span><br/> Sep 19 02:50:14 ossb2 kernel: LNetError: 21764:0:(o2iblnd.c:1940:kiblnd_fmr_pool_map()) Failed to map mr 10/11 elements<br/> Sep 19 02:50:14 ossb2 kernel: LNetError: 21764:0:(o2iblnd_cb.c:560:kiblnd_fmr_map_tx()) Can't map 41033 pages: -22<br/> Sep 19 02:50:14 ossb2 kernel: LNetError: 21764:0:(o2iblnd_cb.c:1554:kiblnd_send()) Can't setup GET sink for 172.20.110.209@o2ib: -22<br/> Sep 19 02:50:14 ossb2 kernel: LustreError: 21764:0:(events.c:449:server_bulk_callback()) event type 5, status -5, desc ffff88086ea2e400<br/> Sep 19 02:51:54 ossb2 kernel: LustreError: 21764:0:(ldlm_lib.c:3237:target_bulk_io()) @@@ timeout on bulk WRITE after 100+0s req@ffff880457208c50 x1578948605516272/t0(0) o1000-&gt;jlustre-MDT0001-mdtlov_UUID@172.20.110.209@o2ib:210/0 lens 376/0 e 4 to 0 dl 1505803920 ref 1 fl Interpret:/0/ffffffff rc 0/-1</p> <p><span class="error">&#91;/var/log/messages in mdt1 server&#93;</span><br/> Sep 19 14:51:22 ossb1 kernel: LustreError: 11-0: jlustre-MDT0000-osp-MDT0001: operation out_update to node 172.20.110.210@o2ib failed: rc = -110<br/> Sep 19 14:51:22 ossb1 kernel: LustreError: 31069:0:(layout.c:2085:__req_capsule_get()) @@@ Wrong buffer for field `object_update_reply' (1 of 1) in format `OUT_UPDATE': 0 vs. 4096 (server)#012 req@ffff8807d3aa7800 x1578948605516272/t0(0) o1000-&gt;jlustre-MDT0000-osp-MDT0001@172.20.110.210@o2ib:24/4 lens 376/192 e 4 to 0 dl 1505803889 ref 2 fl Interpret:ReM/0/0 rc -110/-110<br/> Sep 19 14:51:24 ossb1 kernel: LustreError: 30780:0:(llog_cat.c:773:llog_cat_cancel_records()) jlustre-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog</p> <p>It looks to be network IO on mdt1 timed out, could you verify the network on mdt1 is working correctly?<br/> =&gt;how to verify network on mdt1 is working correctly? Could you give me anu comment? Thanks.</p> <p>Hi Lai,</p> <p> Do you need more detail log? or you have already reproduce it in your site. Thanks.</p> <p>can you test 'lfs mkdir -i 1 dir1' to create a remote directory?</p> <p>create a remote directory =&gt;fail</p> <p><span class="error">&#91;root@robin client&#93;</span># lfs mkdir -i 0 dir0<br/> <span class="error">&#91;root@robin client&#93;</span># lfs mkdir -i 1 dir1<br/> error on LL_IOC_LMV_SETSTRIPE 'dir1' (3): Input/output error<br/> error: mkdir: create stripe dir 'dir1' failed<br/> <span class="error">&#91;root@robin client&#93;</span># lfs mkdir -c 2 dir2<br/> error on LL_IOC_LMV_SETSTRIPE 'dir2' (3): Input/output error<br/> error: mkdir: create stripe dir 'dir2' failed</p> <p> I have also test create striped directory successed when the two mdts in the same server.(transfer message by loopback device) <br/> So I guess there is something wrong between MDT transfer message by IB .</p> <p>Any update ? Thanks.</p> <p>This bug can not be reproduced in release 2.10.1 </p> <p>Good news - thanks</p> Duplicate LU-10010 Development Rank 1|hzzjtb: Rank (Obsolete) 9223372036854775807 Severity