https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10775 Lustre <p>running IOR on locally built lustre branch b2_10 at commit 0f6c448, a couple of initial data transfers work but quickly start to fail, with server side messages like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(sec.c:2363:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(4194304)  req@ffff880f052d8050 x1593867370500512/t0(0) o4-&gt;d0c9fb64-cf93-52c4-8daf-a80ac8484f6b@194.1.0.2@o2ib4:76/0 lens 608/448 e 0 to 0 dl 1520037046 ref 1 fl Interpret:H/2/0 rc 0/0 </pre> </div></div> <p> </p> <p>config arg: --disable-gss </p> <p>module opts all defaults on both sides, perhaps something needs changed for ARM client?</p> <p>server has mdt + 3 osts on one node for testing, no lnet routers</p> <p>IB mlx5 connections.</p> RHEL 7.4 ARM client vs x86 server LU-10775

(sec.c:2363:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(2097152)

Bug Minor Resolved Fixed James A Simmons Ruth Klundt Mon, 5 Mar 2018 19:05:53 +0000 Wed, 19 Dec 2018 20:53:59 +0000 Mon, 16 Apr 2018 13:56:49 +0000 Lustre 2.10.3 0 7 <p>It appears that the page size is 64k on the ARM client. so the workaround of reducing max_pages_per_rpc to 16 works to get rid of this problem. </p> <p>I see the same exact errors. I thought this issue was something else. I have a early patch to resolve this but its not complete. Let me finish up another thing I'm working on and I will look into it.</p> <p>Give patch <a href="https://review.whamcloud.com/#/c/31559" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31559</a> a try</p> <p>Thanks James, will do.</p> <p>sorry for the delay, the cluster has moved ahead on mofed version (MLNX_OFED_LINUX-4.2-1.4.6.0) and build/insmod of ko2iblnd is problematic now. Not sure if my build is wrong, pointing away from /usr/src/ofa-kernel to the actual kernel rpmbuild directory let's lustre config and build. But lots of these on insmod:</p> <p><span class="error">&#91;170344.847903&#93;</span> ko2iblnd: disagrees about version of symbol ib_create_cq<br/> <span class="error">&#91;170344.847905&#93;</span> ko2iblnd: Unknown symbol ib_create_cq (err -22)</p> <p> </p> <p>restoring config default for o2ib to /usr/src/ofa-kernel, 'make' works. The resulting ko2iblnd.ko loads. lctl ping generates a server side error:</p> <p>LNet: 14703:0:(o2iblnd_cb.c:2355:kiblnd_passive_connect()) Can't accept conn from 194.1.0.2@o2ib4 (version 12): max_frags 16 incompatible without FMR pool (256 wanted)</p> <p>The server is running a 2.10-ish commit 2f379be, without your patch. guess I should have patched both sides..</p> <p> </p> <p>Sadly Lustre 2.10 is missing patches to make it properly work. A bunch of fixes when into 2.11 to make lustre work with newer OFED stacks or new kernel IB stacks. I have been testing with 2.11 with my one additonal patch.</p> <p>I was at 2.11 RC1 with the patch on the client side, 2.10 without the patch on the server. </p> <p>After removing the patch the client connects fine. I'm defaulting any module options. </p> <p>This seems like an interop issue to me. </p> <p> </p> <p>That is unexpected considering x86 sends 1MB packets with and without the patch. Its ARM/Power8 that is sending 16 MB packets. I can tell you that the patch on x86 platforms will work with x86 systems without the patch. I have run the upstream client which lacks the patch against patched servers. So we have:</p> <p>patched x86 &lt;-&gt; patched x86 works</p> <p>unpatch x86 &lt;-&gt; unpatch x86 works</p> <p>unpatch x86 &lt;-&gt; patched x86 works</p> <p>patched x86 &lt;-&gt; unpatched x86 ??? should work</p> <p> </p> <p>patched ARM &lt;-&gt; patched x86 works</p> <p>unpatch ARM &lt;-&gt; unpatch x86  fails</p> <p>unpatch ARM &lt;-&gt; pacthed x86 ?? should fail since ARM is not addressed</p> <p>patched ARM &lt;-&gt; unpatch x86 fails</p> <p>Did you trying the server side with the patch to see if it works?</p> <p>Actually I realized I have been testing with an unpatched 2.11 server and it does work. The problem is lustre 2.10 is missing a bunch of fixes to properly support newer MOFED stack. Things like queue pair manage and map_on_demand have changed dramatically. Amir can you put together a list of missing patches for 2.10 to make this work?</p> <p>whoa, thanks but no need to patch 2.10 to make this work, I'm fine with moving the server to 2.11, it's a tiny toy fs, no routers.</p> <p>I'm far from grasping the whole map_on_demand ish, but maybe I just needed to set it to 256, don't think I did that.</p> <p>ps the test cluster is under work again...so next try will be in a while.</p> <p> </p> <p>Ruth, is the server side running RHEL 7.2 or earlier?</p> <p>Looking through the code the reason you'd get:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LNet: 14703:0:(o2iblnd_cb.c:2355:kiblnd_passive_connect()) Can't accept conn from 194.1.0.2@o2ib4 (version 12): max_frags 16 incompatible without FMR pool (256 wanted) </pre> </div></div> <p>is because you're not using FMR. This would occur if HAVE_IB_GET_DMA_MR is defined. I believe this is defined for RHEL 7.2 and earlier.</p> <p>you would be able to avoid this issue by setting map-on-demand to 16 on the server side as well.</p> <p>Can you try that and see if it resolves the issue?</p> <p>James, I consider the map-on-demand changes to be mini-feature. Not sure if it's the best decision to backport that to 2.10.</p> <p>However, we might consider porting the below patch to 2.10, because it fixes a bug</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LU-10213 lnd: calculate qp max_send_wrs properly </pre> </div></div> <p>setting map-on_demand to 16 is not going to help. I have tried it before. We are going to need the map-on-demand changes for b2_10. As he pointed out using the 64K page patch that is slated for  2.12 will break interop with x86 2.10 server when using ARM clients since it lacks the all the changes to make it possible. So we have a choice here, state that in order to user ARM clients you must use at least a 2.11 server, or back port a bunch of o2iblnd patches to make it possible. Also many of the changes missing from 2.10 make using newer MOFED possible.  Do we say you have to stay on a MOFED 3.X version for 2.10?</p> <p>Amir, the server side is RHEL 7.4, I built the 2.10 at 0f6c448. The ofed is MLNX_OFED_LINUX-4.2-1.0.0.0.</p> <p>configure reports yes to checking if 'ib_get_dma_mr' exists, but also:</p> <p>WARNING: "ib_get_dma_mr" <span class="error">&#91;/build_area/lustre-release/build/conftest.ko&#93;</span> undefined!</p> <p>&gt; nm /lib/modules/3.10.0-693.el7.x86_64/extra/mlnx-ofa_kernel/drivers/infiniband/core/ib_core.ko | grep ib_get_dma</p> <p>0000000000006e50 T ib_get_dma_mr</p> <p>Setting map_on_demand=16 on the server works, traffic is moving, Thanks. (I guess that would not work if there were other clients mounting, with a different setting though.)</p> <p>The client side is now:</p> <p>2.11.0_RC2 + '<a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: make LNET_MAX_IOV dependent on page size'</p> <p>                    + <a href="https://jira.whamcloud.com/browse/LU-10560" title="Fixes for 4.14 kernel" class="issue-link" data-issue-key="LU-10560"><del>LU-10560</del></a> libcfs: Use kernel_write when appropriate</p> <p>rhel7.5, kernel 4.14.0-49.el7a.aarch64 and MLNX_OFED_LINUX-4.3-1.0.1.0 </p> <p> </p> <p>Ruth can you join the LWG call today?</p> <p>yes I'll be on</p> <p> </p> <p>Now that <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> landed this ticket can be closed. As a note we need to document on the wiki for ARM/Power8 systems that you need to set map_on_demand to 16 on the back end x86 servers for lustre 2.10 version.</p> <p> </p> Duplicate LU-10157 Development Rank 1|hzztvz: Rank (Obsolete) 9223372036854775807 Severity