QDR <-> QDR Test 1: LTO - Qlogic Compute node LFROM - Qlogic Lnet router HCA cmdline# TM=300 LTO=192.168.55.78@o2ib LFROM=192.168.55.231@o2ib /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 1831.82 MiB/s Min: 1831.82 MiB/s Max: 1831.82 MiB/s [W] Avg: 0.28 MiB/s Min: 0.28 MiB/s Max: 0.28 MiB/s [LNet Rates of lto] [R] Avg: 1832 RPC/s Min: 1832 RPC/s Max: 1832 RPC/s [W] Avg: 3664 RPC/s Min: 3664 RPC/s Max: 3664 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.28 MiB/s Min: 0.28 MiB/s Max: 0.28 MiB/s [W] Avg: 1831.45 MiB/s Min: 1831.45 MiB/s Max: 1831.45 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [lnet01]root: QDR <-> QDR Test 2: LTO - Qlogic Lnet router HCA LFROM - Qlogic Compute node QDR Test 2: cmdline# TM=300 LTO=192.168.55.231@o2ib LFROM=192.168.55.78@o2ib /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 1092.56 MiB/s Min: 1092.56 MiB/s Max: 1092.56 MiB/s [W] Avg: 0.17 MiB/s Min: 0.17 MiB/s Max: 0.17 MiB/s [LNet Rates of lto] [R] Avg: 1095 RPC/s Min: 1095 RPC/s Max: 1095 RPC/s [W] Avg: 2188 RPC/s Min: 2188 RPC/s Max: 2188 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.17 MiB/s Min: 0.17 MiB/s Max: 0.17 MiB/s [W] Avg: 1092.82 MiB/s Min: 1092.82 MiB/s Max: 1092.82 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [lnet01]root: OPA <-> OPA Test 1: LTO - OPA Compute node LFROM - OPA Lnet router HCA cmdline# TM=300 LTO=192.168.44.199@o2ib44 LFROM=192.168.44.15@o2ib44 /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 11504.75 MiB/s Min: 11504.75 MiB/s Max: 11504.75 MiB/s [W] Avg: 1.76 MiB/s Min: 1.76 MiB/s Max: 1.76 MiB/s [LNet Rates of lto] [R] Avg: 11504 RPC/s Min: 11504 RPC/s Max: 11504 RPC/s [W] Avg: 23007 RPC/s Min: 23007 RPC/s Max: 23007 RPC/s [LNet Bandwidth of lto] [R] Avg: 1.76 MiB/s Min: 1.76 MiB/s Max: 1.76 MiB/s [W] Avg: 11504.75 MiB/s Min: 11504.75 MiB/s Max: 11504.75 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [lnet01]root: OPA <-> OPA Test 2: LTO - OPA Lnet router HCA LFROM - OPA Compute node cmdline# TM=300 LTO=192.168.44.15@o2ib44 LFROM=192.168.44.199@o2ib44 /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 11838.27 MiB/s Min: 11838.27 MiB/s Max: 11838.27 MiB/s [W] Avg: 1.81 MiB/s Min: 1.81 MiB/s Max: 1.81 MiB/s [LNet Rates of lto] [R] Avg: 11841 RPC/s Min: 11841 RPC/s Max: 11841 RPC/s [W] Avg: 23677 RPC/s Min: 23677 RPC/s Max: 23677 RPC/s [LNet Bandwidth of lto] [R] Avg: 1.81 MiB/s Min: 1.81 MiB/s Max: 1.81 MiB/s [W] Avg: 11838.27 MiB/s Min: 11838.27 MiB/s Max: 11838.27 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [lnet01]root: Ethernet <-> Ethernet Test 1: LTO - VM with Mellanox 100G NIC LFROM - Lnet router with Mellanox 100G NIC cmdline# TM=300 LTO=10.8.49.155@tcp201 LFROM=10.8.49.15@tcp201 /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 1774.43 MiB/s Min: 1774.43 MiB/s Max: 1774.43 MiB/s [W] Avg: 0.27 MiB/s Min: 0.27 MiB/s Max: 0.27 MiB/s [LNet Rates of lto] [R] Avg: 1775 RPC/s Min: 1775 RPC/s Max: 1775 RPC/s [W] Avg: 3549 RPC/s Min: 3549 RPC/s Max: 3549 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.27 MiB/s Min: 0.27 MiB/s Max: 0.27 MiB/s [W] Avg: 1774.43 MiB/s Min: 1774.43 MiB/s Max: 1774.43 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [lnet01]root: Ethernet <-> Ethernet Test 2: LTO - Lnet router with Mellanox 100G NIC LFROM - VM with Mellanox 100G NIC cmdline# TM=300 LTO=10.8.49.15@tcp201 LFROM=10.8.49.155@tcp201 /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 3323.19 MiB/s Min: 3323.19 MiB/s Max: 3323.19 MiB/s [W] Avg: 0.51 MiB/s Min: 0.51 MiB/s Max: 0.51 MiB/s [LNet Rates of lto] [R] Avg: 3326 RPC/s Min: 3326 RPC/s Max: 3326 RPC/s [W] Avg: 6650 RPC/s Min: 6650 RPC/s Max: 6650 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.51 MiB/s Min: 0.51 MiB/s Max: 0.51 MiB/s [W] Avg: 3322.99 MiB/s Min: 3322.99 MiB/s Max: 3322.99 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [lnet01]root: QDR <-> OPA Test 1: LTO - Qlogic Compute node LFROM - OPA Compute node cmdline# TM=300 LTO=192.168.55.78@o2ib LFROM=192.168.44.199@o2ib44 /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 2801.27 MiB/s Min: 2801.27 MiB/s Max: 2801.27 MiB/s [W] Avg: 0.43 MiB/s Min: 0.43 MiB/s Max: 0.43 MiB/s [LNet Rates of lto] [R] Avg: 2799 RPC/s Min: 2799 RPC/s Max: 2799 RPC/s [W] Avg: 5600 RPC/s Min: 5600 RPC/s Max: 5600 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.43 MiB/s Min: 0.43 MiB/s Max: 0.43 MiB/s [W] Avg: 2800.71 MiB/s Min: 2800.71 MiB/s Max: 2800.71 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [root@john99 ~]# QDR <-> OPA Test 2: LTO - OPA Compute node LFROM - Qlogic Compute node cmdline# TM=300 LTO=192.168.44.199@o2ib44 LFROM=192.168.55.78@o2ib /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 0.01 MiB/s Min: 0.01 MiB/s Max: 0.01 MiB/s [W] Avg: 0.01 MiB/s Min: 0.01 MiB/s Max: 0.01 MiB/s [LNet Rates of lto] [R] Avg: 2 RPC/s Min: 2 RPC/s Max: 2 RPC/s [W] Avg: 2 RPC/s Min: 2 RPC/s Max: 2 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s [W] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s lfrom: 12345-192.168.55.78@o2ib: [Session 32 brw errors, 0 ping errors] [RPC: 1 errors, 0 dropped, 31 expired] Total 1 error nodes in lfrom lto: Total 0 error nodes in lto Batch is stopped session is ended [root@john99 ~]# LFROM node dmesg: LustreError: 1512:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-192.168.44.199@o2ib44 failed with -103 LNet: 1514:0:(rpc.c:1069:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-192.168.44.199@o2ib44, timeout 64. LustreError: 1509:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-192.168.44.199@o2ib44 failed with -110 LustreError: 1510:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-192.168.44.199@o2ib44 failed with -110 LustreError: 1509:0:(brw_test.c:344:brw_client_done_rpc()) Skipped 29 previous similar messages Eth <-> OPA Test 1: LTO - VM with Mellanox 100G NIC LFROM - OPA Compute node cmdline# TM=300 LTO=10.8.49.155@tcp201 LFROM=192.168.44.199@o2ib44 /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 1807.70 MiB/s Min: 1807.70 MiB/s Max: 1807.70 MiB/s [W] Avg: 0.28 MiB/s Min: 0.28 MiB/s Max: 0.28 MiB/s [LNet Rates of lto] [R] Avg: 1807 RPC/s Min: 1807 RPC/s Max: 1807 RPC/s [W] Avg: 3614 RPC/s Min: 3614 RPC/s Max: 3614 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.28 MiB/s Min: 0.28 MiB/s Max: 0.28 MiB/s [W] Avg: 1807.50 MiB/s Min: 1807.50 MiB/s Max: 1807.50 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [root@john99 ~]# Eth <-> OPA Test 2: LTO - VM with Mellanox 100G NIC LFROM - OPA Compute node cmdline# TM=300 LTO=192.168.44.199@o2ib44 LFROM=10.8.49.155@tcp201 /opt/lustre/bin/lst-bench.sh [LNet Rates of lfrom] [R] Avg: 6171 RPC/s Min: 6171 RPC/s Max: 6171 RPC/s [W] Avg: 3085 RPC/s Min: 3085 RPC/s Max: 3085 RPC/s [LNet Bandwidth of lfrom] [R] Avg: 3085.75 MiB/s Min: 3085.75 MiB/s Max: 3085.75 MiB/s [W] Avg: 0.47 MiB/s Min: 0.47 MiB/s Max: 0.47 MiB/s [LNet Rates of lto] [R] Avg: 3088 RPC/s Min: 3088 RPC/s Max: 3088 RPC/s [W] Avg: 6174 RPC/s Min: 6174 RPC/s Max: 6174 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.47 MiB/s Min: 0.47 MiB/s Max: 0.47 MiB/s [W] Avg: 3086.36 MiB/s Min: 3086.36 MiB/s Max: 3086.36 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended [root@john99 ~]# Eth <-> Qlogic Test 1: LTO - VM with Mellanox 100G NIC LFROM - Qlogic Compute node cmdline# TM=300 LTO=10.8.49.155@tcp201 LFROM=192.168.55.78@o2ib /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 0.01 MiB/s Min: 0.01 MiB/s Max: 0.01 MiB/s [W] Avg: 0.01 MiB/s Min: 0.01 MiB/s Max: 0.01 MiB/s [LNet Rates of lto] [R] Avg: 0 RPC/s Min: 0 RPC/s Max: 0 RPC/s [W] Avg: 0 RPC/s Min: 0 RPC/s Max: 0 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s [W] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s lfrom: 12345-192.168.55.78@o2ib: [Session 32 brw errors, 0 ping errors] [RPC: 1 errors, 0 dropped, 63 expired] Total 1 error nodes in lfrom lto: Total 0 error nodes in lto Batch is stopped session is ended [root@john99 ~]# LFROM node dmesg: LustreError: 1512:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-192.168.44.199@o2ib44 failed with -103 LNet: 1514:0:(rpc.c:1069:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-192.168.44.199@o2ib44, timeout 64. LustreError: 1509:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-192.168.44.199@o2ib44 failed with -110 LustreError: 1510:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-192.168.44.199@o2ib44 failed with -110 LustreError: 1509:0:(brw_test.c:344:brw_client_done_rpc()) Skipped 29 previous similar messages LNet: 1514:0:(rpc.c:1069:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.8.49.155@tcp201, timeout 64. LNet: 1514:0:(rpc.c:1069:srpc_client_rpc_expired()) Skipped 30 previous similar messages LustreError: 1511:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-10.8.49.155@tcp201 failed with -110 LustreError: 1511:0:(brw_test.c:344:brw_client_done_rpc()) Skipped 31 previous similar messages Eth <-> Qlogic Test 2: LTO - Qlogic Compute node LFROM - VM with Mellanox 100G NIC cmdline# TM=300 LTO=192.168.55.78@o2ib LFROM=10.8.49.155@tcp201 /opt/lustre/bin/lst-bench.sh [LNet Bandwidth of lfrom] [R] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s [W] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s [LNet Rates of lto] [R] Avg: 0 RPC/s Min: 0 RPC/s Max: 0 RPC/s [W] Avg: 0 RPC/s Min: 0 RPC/s Max: 0 RPC/s [LNet Bandwidth of lto] [R] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s [W] Avg: 0.00 MiB/s Min: 0.00 MiB/s Max: 0.00 MiB/s lfrom: 12345-10.8.49.155@tcp201: [Session 32 brw errors, 0 ping errors] [RPC: 0 errors, 0 dropped, 64 expired] Total 1 error nodes in lfrom lto: 12345-192.168.55.78@o2ib: [Session 0 brw errors, 0 ping errors] [RPC: 1 errors, 0 dropped, 63 expired] Total 1 error nodes in lto Batch is stopped session is ended [root@john99 ~]# LTO node dmesg: [Tue Aug 21 22:26:11 2018] LNet: 25532:0:(rpc.c:1069:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-192.168.55.78@o2ib, timeout 64. [Tue Aug 21 22:26:11 2018] LNet: 25532:0:(rpc.c:1069:srpc_client_rpc_expired()) Skipped 31 previous similar messages [Tue Aug 21 22:26:11 2018] LustreError: 25512:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-192.168.55.78@o2ib failed with -110 [Tue Aug 21 22:26:11 2018] LustreError: 25512:0:(brw_test.c:344:brw_client_done_rpc()) Skipped 31 previous similar messages