https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10089 Lustre <p>The following group of messages appear in the console logs of MDTs.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-10-04 08:11:00 [407096.858161] LNetError: 174158:0:(o2iblnd.c:1893:kiblnd_fmr_pool_map()) Failed to map mr 10/11 elements 2017-10-04 08:11:00 [407096.869697] LNetError: 174158:0:(o2iblnd_cb.c:590:kiblnd_fmr_map_tx()) Can't map 41033 pages: -22 2017-10-04 08:11:00 [407096.880686] LNetError: 174158:0:(o2iblnd_cb.c:1582:kiblnd_send()) Can't setup GET sink for 172.19.1.112@o2ib100: -22 2017-10-04 08:11:00 [407096.893504] LustreError: 174158:0:(events.c:449:server_bulk_callback()) event type 5, status -5, desc ffff883ebaa9bb00 2017-10-04 08:12:40 [407196.901157] LustreError: 174158:0:(ldlm_lib.c:3186:target_bulk_io()) @@@ timeout on bulk WRITE after 100+0s req@ffff883f27232850 x1579913603429696/t0(0) o1000-&gt;lquake-MDT0001-mdtlov_UUID@172.19.1.112@o2ib100:-1/-1 lens 352/0 e 0 to 0 dl 1507130003 ref 1 fl Interpret:/0/ffffffff rc 0/-1 </pre> </div></div> <p>The nodes have Mellanox ConnectX-4 IB adapters.</p> Build Version: 2.8.0_12.chaos<br/> See <a href="ssh://review.whamcloud.com/fs/lustre-release-fe-llnl">ssh://review.whamcloud.com/fs/lustre-release-fe-llnl</a><br/> RHEL 7.4 kernel<br/> LU-10089

kiblnd_fmr_pool_map() Failed to map mr 10/11 elements

Bug Critical Resolved Fixed Amir Shehata Olaf Faaland llnl Thu, 5 Oct 2017 21:56:40 +0000 Fri, 2 Feb 2018 16:54:30 +0000 Tue, 24 Oct 2017 13:06:28 +0000 Lustre 2.11.0 0 9 <p>The file system has 16 MDTs, each on a separate MDS. This log snippet is from the server hosting MGS and MDT0, NID 172.19.1.111@o2ib100. The nodes referenced by NIDs in the range172.19.1.112@o2ib100 to 172.19.1.126@o2ib100 are the other MDSs.</p> <p>The console log for the MGS/MDS0000 node, which the "Failed to map mr" messages appear in, is attached as console.log.jet1.lu-10089.txt</p> <p>I spot checked a few examples. There appears to be a corresponding error message on the node referred to by the "timeout on bulk WRITE" message. For example:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jet1: 2017-10-05 06:55:32 [488964.187806] LNetError: 174158:0:(o2iblnd.c:1893:kiblnd_fmr_pool_map()) Failed to map mr 10/11 elements 2017-10-05 06:55:32 [488964.199374] LNetError: 174158:0:(o2iblnd_cb.c:590:kiblnd_fmr_map_tx()) Can't map 41033 pages: -22 2017-10-05 06:55:32 [488964.210361] LNetError: 174158:0:(o2iblnd_cb.c:1582:kiblnd_send()) Can't setup GET sink for 172.19.1.112@o2ib100: -22 2017-10-05 06:55:32 [488964.223175] LustreError: 174158:0:(events.c:449:server_bulk_callback()) event type 5, status -5, desc ffff883ebaa9bf00 2017-10-05 06:57:12 [489064.231425] LustreError: 174158:0:(ldlm_lib.c:3186:target_bulk_io()) @@@ timeout on bulk WRITE after 100+0s req@ffff883f27239c50 x1579913603914692/t0(0) o1000-&gt;lquake-MDT0001-mdtlov_UUID@172.19.1.112@o2ib100:-1/-1 lens 352/0 e 2 to 0 dl 1507211843 ref 1 fl Interpret:/0/ffffffff rc 0/-1 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>jet2: 2017-10-05 06:57:12 [489015.158245] LustreError: 11-0: lquake-MDT0000-osp-MDT0001: operation out_update to node 172.19.1.111@o2ib100 failed: rc = -110 2017-10-05 06:57:12 [489015.173453] LustreError: 16866:0:(layout.c:2025:__req_capsule_get()) @@@ Wrong buffer for field `object_update_reply' (1 of 1) in format `OUT_UPDATE': 0 vs. 4096 (server) 2017-10-05 06:57:12 [489015.173453] req@ffff882b342cdd00 x1579913603914692/t0(0) o1000-&gt;lquake-MDT0000-osp-MDT0001@172.19.1.111@o2ib100:24/4 lens 352/192 e 2 to 0 dl 1507211888 ref 2 fl Interpret:ReM/0/0 rc -110/-110 2017-10-05 06:57:16 [489019.757615] LustreError: 17382:0:(llog_cat.c:744:llog_cat_cancel_records()) lquake-MDT0000-osp-MDT0001: fail to cancel 1 of 1 llog-records: rc = -116 </pre> </div></div> <p>The nodes use ntp to sync and all report they are within the same 1/20th of a second or so.</p> <p>Can you try: <a href="https://review.whamcloud.com/29290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290</a></p> <blockquote><p>Can you try: <a href="https://review.whamcloud.com/29290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290</a></p></blockquote> <p>Yes, I will.</p> <p>Creating remote directories seems to hang, then fail, and trigger these messages. Is there likely a separate problem, or does it make sense that these two symptoms would be connected?</p> <p>I was able to reproduce this issue on mlx5 as well.</p> <p>It looks to be due to:<br/> <a href="https://jira.whamcloud.com/browse/LU-6215" title="Sync Lustre external tree with lustre linux kernel client" class="issue-link" data-issue-key="LU-6215"><del>LU-6215</del></a> o2iblnd: port to new fast reg API introduced in 4.4</p> <p>Will need to investigate this.</p> <p>Amir,</p> <p>OK. Please check our patch stack to see what we've got. Pointer is under "Environment" above.</p> <p>With patch 29290, I see the following in the console logs of the MDS's, on a quiet filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2017-10-05 18:19:01 [ 1342.476344] LustreError: 15171:0:(events.c:449:server_bulk_callback()) event type 5, status -5, desc ffff887f17347e00 2017-10-05 18:19:01 [ 1342.481443] mlx5_0:dump_cqe:262:(pid 15170): dump error cqe 2017-10-05 18:19:01 [ 1342.481444] 00000000 00000000 00000000 00000000 2017-10-05 18:19:01 [ 1342.481445] 00000000 00000000 00000000 00000000 2017-10-05 18:19:01 [ 1342.481445] 00000000 00000000 00000000 00000000 2017-10-05 18:19:01 [ 1342.481445] 00000000 9d005304 0800180b 000ff6d2 2017-10-05 18:19:01 [ 1342.481630] LustreError: 15170:0:(events.c:449:server_bulk_callback()) event type 5, status -5, desc ffff887f1cb76800 2017-10-05 18:19:01 [ 1342.486614] mlx5_0:dump_cqe:262:(pid 15170): dump error cqe 2017-10-05 18:19:01 [ 1342.486617] 00000000 00000000 00000000 00000000 2017-10-05 18:19:01 [ 1342.486618] 00000000 00000000 00000000 00000000 2017-10-05 18:19:01 [ 1342.486619] 00000000 00000000 00000000 00000000 2017-10-05 18:19:01 [ 1342.486620] 00000000 9d005304 0800180c 000ef0d2 2017-10-05 18:19:01 [ 1342.486810] LustreError: 15170:0:(events.c:449:server_bulk_callback()) event type 5, status -5, desc ffff887f17347a00 </pre> </div></div> <p>Olaf are you carrying the <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> patch (<a href="https://review.whamcloud.com/27149" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/27149</a>)</p> <p>James:<br/> Yes. We are carrying backports of patches from <a href="https://jira.whamcloud.com/browse/LU-9026" title="Adapt to the removal of ib_get_dma_mr()" class="issue-link" data-issue-key="LU-9026"><del>LU-9026</del></a>, <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>, and <a href="https://jira.whamcloud.com/browse/LU-9472" title="FastReg (MLX5) support breaks when map_on_demand &gt; 0" class="issue-link" data-issue-key="LU-9472"><del>LU-9472</del></a>. I don't have the gerritt links handy but can get them.</p> <p>Amir,<br/> Have you learned anything more? Thanks.</p> <p>Olaf, yes. we fixed this issue for MLX-5 but the same fix doesn't work on MLX-4. I can upload a test patch for MLX-5 for now since I think this is what you're using, correct?</p> <p>Amir, thanks.</p> <p>We have both MLX-4 and MLX-5 hardware in our center, so we need both to work. </p> <p>You're correct, the symptoms reported in this issue appear on servers with MLX-5. The filesystem is mounted via routers with MLX-5 and OmniPath and clients using OmniPath (based on the loaded drivers, mlx5_core/ib).</p> <p>I can give your existing patch a try if it would be helpful.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29551" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29551</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10089" title="kiblnd_fmr_pool_map() Failed to map mr 10/11 elements" class="issue-link" data-issue-key="LU-10089"><del>LU-10089</del></a> o2iblnd: use IB_MR_TYPE_SG_GAPS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c0527177a8e5690c7e9418f4da3070fecf5ef5a4</p> <p>Ok, so I think we have it resolved. There are three different fixes:<br/> 1. <a href="https://review.whamcloud.com/29551" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29551</a> - this is required for MLX-5, and it's particular for fastreg<br/> 2. <a href="https://review.whamcloud.com/29290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290</a> - this is required for MLX-4 and MLX-5, since FMR and fastreg have different requirements for the RDMA descriptor<br/> 3. <a href="https://review.whamcloud.com/29341" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29341</a> - this is a revert for <a href="https://jira.whamcloud.com/browse/LU-9810" title="Melanox OFED 4.1 support" class="issue-link" data-issue-key="LU-9810"><del>LU-9810</del></a> which is required for MLX-4 and OPA, since both seem to support fastreg, but fastreg doesn't work for either of them. So we need to continue preferring FMR. I'm investigating why MLX-4 and OPA both don't support fastreg properly. Since fastreg is a MLX feature it makes sense that OPA doesn't support it, but not sure why MLX-4 chokes on it. </p> <p>If you could try these three patches and see if they resolve your issue.</p> <p>I ran on an MLX-5 machine with good success. I'll test on MLX-4 and OPA this morning.</p> <p>With brief testing, I see no errors on MLX-4 and OPA machines. What is the next step?</p> <p>I believe that I'll push for landing these three patches as they stabilize master as well.</p> <p>I'm not sure how you guys will pick up the patches. Do you maintain your own tree? or do would you need this patches backported?</p> <p>Backport them to 2.8 fe, please. They will likely apply cleanly.</p> <p>The relationship between our stack and 2.8fe is complicated but they are not that different, and we will be switching to 2.8fe + a small stack of commits very soon.</p> <p>Note that we cannot land these patches to our production tree until they are through your review and testing process, and are merged to master at a minimum.  Let me know if there's anything I can do to help that along.</p> <p>Some notes:<br/> 1. Fastreg with MLX4 + OFED + ib_alloc_mr(.., IB_MR_TYPE_MEM_REG): gets BIND_ERR <br/> 2. Fastreg with MLX4 + MOFED 4.1 + ib_alloc_mr(.., IB_MR_TYPE_MEM_REG): works<br/> 3. IB_MR_TYPE_SG_GAPS is not supported for MLX4 on either OFED or MOFED<br/> 4. MLX4 FMR mapping works differently on MOFED vs OFED. </p> <ul class="alternate" type="square"> <li><a href="https://review.whamcloud.com/29290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290</a> is needed for OFED</li> <li>but not needed for MOFED (although it doesn't seem to hurt)</li> </ul> <p>So the best solution is to:<br/> 1. Always use FMR for MLX4<br/> 2. Always use FMR for OPA<br/> 3. FMR is not available for MLX5 so fastreg will be used</p> <p>The three patches I described earlier seems like the ideal solution for now.</p> <p>This appears to be working well in my tests.</p> <p>Hi Amir,</p> <p>I see <a href="https://review.whamcloud.com/#/c/29551/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29551/</a> has status "Ready to land", but hasn't been landed.  Is there further work needed, or is it just waiting for the next time a set of patches get merged?  Thanks.</p> <p>Their is a question about querying the IB device to see if it supports IB_MR_TYPE_SG_GAPS instead of assuming that IB_MR_TYPE_SG_GAPS is always the case.</p> <p>I updated <a href="https://review.whamcloud.com/#/c/29551/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/29551/</a> to address the comments.</p> <p>One thing to note, if you're using OPA you should use map-on-demand set to 256. I'm still analyzing this issue and hopefully will have a patch soon. This issue is tracked under <a href="https://jira.whamcloud.com/browse/LU-10129" title="map-on-demand set to 32 doesn&#39;t work on OPA" class="issue-link" data-issue-key="LU-10129"><del>LU-10129</del></a></p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29551/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29551/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10089" title="kiblnd_fmr_pool_map() Failed to map mr 10/11 elements" class="issue-link" data-issue-key="LU-10089"><del>LU-10089</del></a> o2iblnd: use IB_MR_TYPE_SG_GAPS<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1b609396e468949f2420f14fed5ebfc999366b62</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29771" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29771</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10089" title="kiblnd_fmr_pool_map() Failed to map mr 10/11 elements" class="issue-link" data-issue-key="LU-10089"><del>LU-10089</del></a> o2iblnd: use IB_MR_TYPE_SG_GAPS<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: aacf8a650f495f50faf0135c6201ad0e446faf74</p> <p>we don't need this in 2.10</p> Blocker LU-10115 Related LU-9983 LU-10091 Development Rank 1|hzzlfr: Rank (Obsolete) 9223372036854775807 Severity