https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10394 Lustre <p>mlx5 performance is down 2+ GB/s when using IB_MR_TYPE_SG_GAPS as compared to IB_MR_TYPE_MEM_REG.</p> <p> </p> <p>mlx5 with SG GAPS</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>---------------------------------------------------------- Running test: lst add_test --batch rperf --concurrency 32 --distribute 1:1 --from clients --to servers brw read size=1M Client Read RPC/s: 17426.3333333333 Client Write RPC/s: 8713.77777777778 Client Read MiB/s: 8713.46111111111 Client Write MiB/s: 1.33 ---------------------------------------------------------- Running test: lst add_test --batch rperf --concurrency 64 --distribute 1:1 --from clients --to servers brw read size=1M Client Read RPC/s: 17408.3333333333 Client Write RPC/s: 8705.22222222222 Client Read MiB/s: 8704.06666666667 Client Write MiB/s: 1.33 ---------------------------------------------------------- Running test: lst add_test --batch rperf --concurrency 128 --distribute 1:1 --from clients --to servers brw read size=1M Client Read RPC/s: 17388.4444444444 Client Write RPC/s: 8697 Client Read MiB/s: 8695.54666666667 Client Write MiB/s: 1.32777777777778 ---------------------------------------------------------- Running test: lst add_test --batch wperf --concurrency 32 --distribute 1:1 --from clients --to servers brw write size=1M Client Read RPC/s: 17712.1111111111 Client Write RPC/s: 8856.55555555555 Client Read MiB/s: 1.35 Client Write MiB/s: 8855.53111111111 ---------------------------------------------------------- Running test: lst add_test --batch wperf --concurrency 64 --distribute 1:1 --from clients --to servers brw write size=1M Client Read RPC/s: 17705.7777777778 Client Write RPC/s: 8853.66666666667 Client Read MiB/s: 1.35 Client Write MiB/s: 8853.18555555556 ---------------------------------------------------------- Running test: lst add_test --batch wperf --concurrency 128 --distribute 1:1 --from clients --to servers brw write size=1M Client Read RPC/s: 17697.3333333333 Client Write RPC/s: 8854.44444444445 Client Read MiB/s: 1.34888888888889 Client Write MiB/s: 8850.95777777778 </pre> </div></div> <p>mlx5 without SG GAPS</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>---------------------------------------------------------- Running test: lst add_test --batch rperf --concurrency 32 --distribute 1:1 --from clients --to servers brw read size=1M Client Read RPC/s: 22449.5555555556 Client Write RPC/s: 11227 Client Read MiB/s: 11224.5033333333 Client Write MiB/s: 1.71222222222222 ---------------------------------------------------------- Running test: lst add_test --batch rperf --concurrency 64 --distribute 1:1 --from clients --to servers brw read size=1M Client Read RPC/s: 22308.6666666667 Client Write RPC/s: 11154.3333333333 Client Read MiB/s: 11155.7288888889 Client Write MiB/s: 1.7 ---------------------------------------------------------- Running test: lst add_test --batch rperf --concurrency 128 --distribute 1:1 --from clients --to servers brw read size=1M Client Read RPC/s: 21549.1 Client Write RPC/s: 10737.4 Client Read MiB/s: 11135.278 Client Write MiB/s: 1.638 ---------------------------------------------------------- Running test: lst add_test --batch wperf --concurrency 32 --distribute 1:1 --from clients --to servers brw write size=1M Client Read RPC/s: 22178.3333333333 Client Write RPC/s: 11090.8888888889 Client Read MiB/s: 1.69 Client Write MiB/s: 11088.7822222222 ---------------------------------------------------------- Running test: lst add_test --batch wperf --concurrency 64 --distribute 1:1 --from clients --to servers brw write size=1M Client Read RPC/s: 22198.6666666667 Client Write RPC/s: 11099.8888888889 Client Read MiB/s: 1.69111111111111 Client Write MiB/s: 11100.1666666667 ---------------------------------------------------------- Running test: lst add_test --batch wperf --concurrency 128 --distribute 1:1 --from clients --to servers brw write size=1M Client Read RPC/s: 22162.6666666667 Client Write RPC/s: 11085.7777777778 Client Read MiB/s: 1.68777777777778 Client Write MiB/s: 11083.5477777778 </pre> </div></div> <p>o2iblnd parameters:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options ko2iblnd timeout=10 options ko2iblnd peer_timeout=0 options ko2iblnd keepalive=30 options ko2iblnd credits=2048 options ko2iblnd ntx=2048 options ko2iblnd peer_credits=16 options ko2iblnd concurrent_sends=16 </pre> </div></div> CentOS 7.4<br/> 2.10.56_1_g11aae87-1.el7.centos.x86_64 LU-10394

IB_MR_TYPE_SG_GAPS mlx5 LNet performance drop

Bug Critical Resolved Fixed Amir Shehata Ian Ziemba Thu, 14 Dec 2017 23:30:35 +0000 Mon, 7 Jan 2019 19:08:16 +0000 Fri, 9 Feb 2018 13:02:24 +0000 Lustre 2.11.0 Lustre 2.11.0 0 16 <p>Amir</p> <p>Could you please advise?</p> <p>Peter</p> <p>Here is my proposed solution. Introduce a tunable to enable GAP support. it defaults to 0 in order to use MEM_REG. if the tunable is set to 1 and the mlx card has gap support then use GAP support and add a warning that performance degradation is expected with this configuration.</p> <p>This is safe because</p> <p> <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> osp: align the OSP request size by 4k</p> <p>closes any scenario where we could introduce discontigous fragments.</p> <p>The tunable can be turned on if in the future a layer uses LNet with discontigous gaps.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30749" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30749</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10394" title="IB_MR_TYPE_SG_GAPS mlx5 LNet performance drop" class="issue-link" data-issue-key="LU-10394"><del>LU-10394</del></a> lnd: default to using MEM_REG<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 74dced457067c9135365ea040e7071c87cd75375</p> <p>Hi Amir,</p> <p>I tried running twice with this patch, and both times, my system has panic with the following message.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 872.303780] LNetError: 2362:0:(o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed: [ 872.304325] LNetError: 2362:0:(o2iblnd_cb.c:991:kiblnd_check_sends_locked()) LBUG [ 872.304699] Pid: 2362, comm: kiblnd_sd_00_01 [ 872.304704] Call Trace: [ 872.304805] [&lt;ffffffffc0a1e7ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] [ 872.304901] [&lt;ffffffffc0a1e83c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] [ 872.304967] [&lt;ffffffffc0ab352b&gt;] kiblnd_check_sends_locked+0xd8b/0xd90 [ko2iblnd] [ 872.305008] [&lt;ffffffffc0524d53&gt;] ? mlx5_ib_post_recv+0x1f3/0x240 [mlx5_ib] [ 872.305074] [&lt;ffffffffc0ab4e06&gt;] kiblnd_post_rx+0x156/0x4e0 [ko2iblnd] [ 872.305138] [&lt;ffffffffc0ab536a&gt;] kiblnd_recv+0x1da/0x7b0 [ko2iblnd] [ 872.305266] [&lt;ffffffffc08db1fc&gt;] ? lnet_mt_match_md+0x8c/0x1b0 [lnet] [ 872.305392] [&lt;ffffffffc08e3573&gt;] lnet_ni_recv+0xc3/0x320 [lnet] [ 872.305523] [&lt;ffffffffc08e3cc1&gt;] lnet_recv_put+0x81/0xb0 [lnet] [ 872.305641] [&lt;ffffffffc08e5ee6&gt;] lnet_parse_local+0x5a6/0xd40 [lnet] [ 872.305762] [&lt;ffffffffc08e6f4a&gt;] lnet_parse+0x8ca/0xfc0 [lnet] [ 872.305825] [&lt;ffffffffc0ab3035&gt;] ? kiblnd_check_sends_locked+0x895/0xd90 [ko2iblnd] [ 872.305860] [&lt;ffffffffc051c608&gt;] ? mlx5_ib_poll_cq+0x418/0xf10 [mlx5_ib] [ 872.305925] [&lt;ffffffffc0ab5ce3&gt;] kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd] [ 872.305990] [&lt;ffffffffc0abc90f&gt;] kiblnd_scheduler+0xf0f/0x1150 [ko2iblnd] [ 872.306004] [&lt;ffffffff810c93f5&gt;] ? sched_clock_cpu+0x85/0xc0 [ 872.306016] [&lt;ffffffff8102954d&gt;] ? __switch_to+0xcd/0x500 [ 872.306031] [&lt;ffffffff810c6440&gt;] ? default_wake_function+0x0/0x20 [ 872.306096] [&lt;ffffffffc0abba00&gt;] ? kiblnd_scheduler+0x0/0x1150 [ko2iblnd] [ 872.306109] [&lt;ffffffff810b252f&gt;] kthread+0xcf/0xe0 [ 872.306122] [&lt;ffffffff810b2460&gt;] ? kthread+0x0/0xe0 [ 872.306135] [&lt;ffffffff816b8798&gt;] ret_from_fork+0x58/0x90 [ 872.306147] [&lt;ffffffff810b2460&gt;] ? kthread+0x0/0xe0 </pre> </div></div> <p>I think this panic is unrelated to your patch. Do you want me to open another ticket?</p> <p>Thanks</p> <p>This has already been fixed already. Please checkout <a href="https://jira.whamcloud.com/browse/LU-10459" title="LBUG o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:" class="issue-link" data-issue-key="LU-10459"><del>LU-10459</del></a></p> <p>Thanks. The use_fastreg_gaps parameter works as expected. When set to 0, 1M performance is 11 GBps. When set to 1, 1M performance is 8.9 GBps.</p> <p>Ok thanks. good to know that SG_GAPS drops performance. I'll make sure to record that on the LNet Wiki</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30749/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30749/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10394" title="IB_MR_TYPE_SG_GAPS mlx5 LNet performance drop" class="issue-link" data-issue-key="LU-10394"><del>LU-10394</del></a> lnd: default to using MEM_REG<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 27116ee18c04057936837f6f0aef3f4c09c21d78</p> <p>Landed for 2.11</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33149" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33149</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10394" title="IB_MR_TYPE_SG_GAPS mlx5 LNet performance drop" class="issue-link" data-issue-key="LU-10394"><del>LU-10394</del></a> lnd: default to using MEM_REG<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: ed0687e222c2714b4a1d6d12ff2bce6612c8bb82</p> Duplicate Related LU-10373 LU-11105 Development Rank 1|hzzpiv: Rank (Obsolete) 9223372036854775807 Severity