Run from sprig5 (client evicted): # each OSS node root@sprig5:/tmp:$ lctl ping 11.3.0.14@o2ib 12345-0@lo 12345-11.3.0.14@o2ib root@sprig5:/tmp:$ lctl ping 11.3.0.15@o2ib 12345-0@lo 12345-11.3.0.15@o2ib # each MDS node root@sprig5:/tmp:$ lctl ping 11.3.0.12@o2ib 12345-0@lo 12345-11.3.0.12@o2ib root@sprig5:/tmp:$ lctl ping 11.3.0.13@o2ib 12345-0@lo 12345-11.3.0.13@o2ib # # lnet_selftest using all MDS and OSS nodes and sprig3 (a working client) and sprig5 (an evicted client): # [root@sprig-oss1 ~]# lnet_selftest.sh -c "11.3.0.49@o2ib 11.3.0.47@o2ib" -s "11.3.0.14@o2ib 11.3.0.15@o2ib 11.3.0.12@o2ib 11.3.0.13@o2ib" You need to manually load lnet_selftest on all nodes modprobe lnet_selftest LST_SESSION=20147 SESSION: read/write FEATURES: 0 TIMEOUT: 300 FORCE: No 11.3.0.14@o2ib are added to session 11.3.0.15@o2ib are added to session 11.3.0.12@o2ib are added to session 11.3.0.13@o2ib are added to session 11.3.0.49@o2ib are added to session 11.3.0.47@o2ib are added to session Test was added successfully Test was added successfully batch is running now [LNet Rates of servers] [R] Avg: 5266 RPC/s Min: 3 RPC/s Max: 11160 RPC/s [W] Avg: 6921 RPC/s Min: 3 RPC/s Max: 14619 RPC/s [LNet Bandwidth of servers] [R] Avg: 1804.94 MB/s Min: 0.00 MB/s Max: 3850.15 MB/s [W] Avg: 1654.46 MB/s Min: 0.00 MB/s Max: 3460.02 MB/s [LNet Rates of servers] [R] Avg: 5264 RPC/s Min: 3 RPC/s Max: 11136 RPC/s [W] Avg: 6917 RPC/s Min: 3 RPC/s Max: 14590 RPC/s [LNet Bandwidth of servers] [R] Avg: 1803.77 MB/s Min: 0.00 MB/s Max: 3839.35 MB/s [W] Avg: 1653.75 MB/s Min: 0.00 MB/s Max: 3455.62 MB/s [LNet Rates of servers] [R] Avg: 5262 RPC/s Min: 5 RPC/s Max: 11100 RPC/s [W] Avg: 6912 RPC/s Min: 5 RPC/s Max: 14538 RPC/s [LNet Bandwidth of servers] [R] Avg: 1799.52 MB/s Min: 0.00 MB/s Max: 3821.75 MB/s [W] Avg: 1651.05 MB/s Min: 0.00 MB/s Max: 3440.42 MB/s [LNet Rates of servers] [R] Avg: 5280 RPC/s Min: 3 RPC/s Max: 11198 RPC/s [W] Avg: 6937 RPC/s Min: 3 RPC/s Max: 14668 RPC/s [LNet Bandwidth of servers] [R] Avg: 1809.96 MB/s Min: 0.00 MB/s Max: 3861.92 MB/s [W] Avg: 1657.22 MB/s Min: 0.00 MB/s Max: 3471.32 MB/s [LNet Rates of servers] [R] Avg: 5278 RPC/s Min: 4 RPC/s Max: 11191 RPC/s [W] Avg: 6935 RPC/s Min: 4 RPC/s Max: 14662 RPC/s [LNet Bandwidth of servers] [R] Avg: 1807.84 MB/s Min: 0.00 MB/s Max: 3857.95 MB/s [W] Avg: 1657.56 MB/s Min: 0.00 MB/s Max: 3472.03 MB/s [LNet Rates of servers] [R] Avg: 5270 RPC/s Min: 4 RPC/s Max: 11155 RPC/s [W] Avg: 6925 RPC/s Min: 4 RPC/s Max: 14614 RPC/s [LNet Bandwidth of servers] [R] Avg: 1805.32 MB/s Min: 0.00 MB/s Max: 3845.95 MB/s [W] Avg: 1656.25 MB/s Min: 0.00 MB/s Max: 3461.22 MB/s [LNet Rates of servers] [R] Avg: 5265 RPC/s Min: 3 RPC/s Max: 11136 RPC/s [W] Avg: 6919 RPC/s Min: 3 RPC/s Max: 14593 RPC/s [LNet Bandwidth of servers] [R] Avg: 1804.71 MB/s Min: 0.00 MB/s Max: 3839.31 MB/s [W] Avg: 1654.07 MB/s Min: 0.00 MB/s Max: 3458.31 MB/s [LNet Rates of servers] [R] Avg: 5261 RPC/s Min: 8 RPC/s Max: 11149 RPC/s [W] Avg: 6909 RPC/s Min: 8 RPC/s Max: 14601 RPC/s [LNet Bandwidth of servers] [R] Avg: 1799.04 MB/s Min: 0.00 MB/s Max: 3838.75 MB/s [W] Avg: 1648.71 MB/s Min: 0.00 MB/s Max: 3452.03 MB/s [LNet Rates of servers] [R] Avg: 5259 RPC/s Min: 3 RPC/s Max: 11136 RPC/s [W] Avg: 6912 RPC/s Min: 3 RPC/s Max: 14589 RPC/s [LNet Bandwidth of servers] [R] Avg: 1802.17 MB/s Min: 0.00 MB/s Max: 3839.95 MB/s [W] Avg: 1652.95 MB/s Min: 0.00 MB/s Max: 3455.42 MB/s [LNet Rates of servers] [R] Avg: 5247 RPC/s Min: 4 RPC/s Max: 11086 RPC/s [W] Avg: 6895 RPC/s Min: 4 RPC/s Max: 14525 RPC/s [LNet Bandwidth of servers] [R] Avg: 1796.76 MB/s Min: 0.00 MB/s Max: 3821.31 MB/s [W] Avg: 1648.52 MB/s Min: 0.00 MB/s Max: 3439.91 MB/s [LNet Rates of servers] [R] Avg: 5288 RPC/s Min: 3 RPC/s Max: 11208 RPC/s [W] Avg: 6948 RPC/s Min: 3 RPC/s Max: 14682 RPC/s [LNet Bandwidth of servers] [R] Avg: 1812.24 MB/s Min: 0.00 MB/s Max: 3865.15 MB/s [W] Avg: 1660.81 MB/s Min: 0.00 MB/s Max: 3475.83 MB/s [LNet Rates of servers] [R] Avg: 5275 RPC/s Min: 3 RPC/s Max: 11144 RPC/s [W] Avg: 6932 RPC/s Min: 3 RPC/s Max: 14601 RPC/s [LNet Bandwidth of servers] [R] Avg: 1807.52 MB/s Min: 0.00 MB/s Max: 3842.75 MB/s [W] Avg: 1657.85 MB/s Min: 0.00 MB/s Max: 3458.22 MB/s [LNet Rates of servers] [R] Avg: 5257 RPC/s Min: 5 RPC/s Max: 11085 RPC/s [W] Avg: 6906 RPC/s Min: 5 RPC/s Max: 14518 RPC/s [LNet Bandwidth of servers] [R] Avg: 1798.81 MB/s Min: 0.00 MB/s Max: 3818.91 MB/s [W] Avg: 1648.82 MB/s Min: 0.00 MB/s Max: 3433.11 MB/s [LNet Rates of servers] [R] Avg: 5271 RPC/s Min: 3 RPC/s Max: 11140 RPC/s [W] Avg: 6928 RPC/s Min: 3 RPC/s Max: 14596 RPC/s [LNet Bandwidth of servers] [R] Avg: 1805.39 MB/s Min: 0.00 MB/s Max: 3839.15 MB/s [W] Avg: 1657.06 MB/s Min: 0.00 MB/s Max: 3458.22 MB/s [LNet Rates of servers] [R] Avg: 5261 RPC/s Min: 4 RPC/s Max: 11102 RPC/s [W] Avg: 6911 RPC/s Min: 4 RPC/s Max: 14544 RPC/s [LNet Bandwidth of servers] [R] Avg: 1802.52 MB/s Min: 0.00 MB/s Max: 3827.95 MB/s [W] Avg: 1650.65 MB/s Min: 0.00 MB/s Max: 3443.42 MB/s [LNet Rates of servers] [R] Avg: 5294 RPC/s Min: 4 RPC/s Max: 11190 RPC/s [W] Avg: 6956 RPC/s Min: 4 RPC/s Max: 14661 RPC/s [LNet Bandwidth of servers] [R] Avg: 1814.41 MB/s Min: 0.00 MB/s Max: 3859.12 MB/s [W] Avg: 1662.57 MB/s Min: 0.00 MB/s Max: 3471.52 MB/s [LNet Rates of servers] [R] Avg: 5264 RPC/s Min: 3 RPC/s Max: 11155 RPC/s [W] Avg: 6919 RPC/s Min: 3 RPC/s Max: 14617 RPC/s [LNet Bandwidth of servers] [R] Avg: 1803.89 MB/s Min: 0.00 MB/s Max: 3845.95 MB/s [W] Avg: 1654.81 MB/s Min: 0.00 MB/s Max: 3463.22 MB/s [LNet Rates of servers] [R] Avg: 5268 RPC/s Min: 6 RPC/s Max: 11110 RPC/s [W] Avg: 6919 RPC/s Min: 6 RPC/s Max: 14551 RPC/s [LNet Bandwidth of servers] [R] Avg: 1801.57 MB/s Min: 0.00 MB/s Max: 3824.95 MB/s [W] Avg: 1651.65 MB/s Min: 0.00 MB/s Max: 3442.42 MB/s [LNet Rates of servers] [R] Avg: 5278 RPC/s Min: 3 RPC/s Max: 11155 RPC/s [W] Avg: 6935 RPC/s Min: 3 RPC/s Max: 14614 RPC/s [LNet Bandwidth of servers] [R] Avg: 1809.46 MB/s Min: 0.00 MB/s Max: 3846.72 MB/s [W] Avg: 1657.22 MB/s Min: 0.00 MB/s Max: 3460.52 MB/s # # lnet_selftest using all MDS and OSS nodes and only sprig5 (an evicted client): # [root@sprig-oss1 ~]# lnet_selftest.sh -c "11.3.0.49@o2ib" -s "11.3.0.14@o2ib 11.3.0.15@o2ib 11.3.0.12@o2ib 11.3.0.13@o2ib" You need to manually load lnet_selftest on all nodes modprobe lnet_selftest LST_SESSION=20599 SESSION: read/write FEATURES: 0 TIMEOUT: 300 FORCE: No 11.3.0.14@o2ib are added to session 11.3.0.15@o2ib are added to session 11.3.0.12@o2ib are added to session 11.3.0.13@o2ib are added to session 11.3.0.49@o2ib are added to session Test was added successfully Test was added successfully batch is running now [LNet Rates of servers] [R] Avg: 2496 RPC/s Min: 3 RPC/s Max: 9973 RPC/s [W] Avg: 3290 RPC/s Min: 3 RPC/s Max: 13148 RPC/s [LNet Bandwidth of servers] [R] Avg: 849.60 MB/s Min: 0.00 MB/s Max: 3398.40 MB/s [W] Avg: 793.85 MB/s Min: 0.00 MB/s Max: 3175.40 MB/s [LNet Rates of servers] [R] Avg: 2495 RPC/s Min: 6 RPC/s Max: 9947 RPC/s [W] Avg: 3286 RPC/s Min: 6 RPC/s Max: 13110 RPC/s [LNet Bandwidth of servers] [R] Avg: 845.78 MB/s Min: 0.00 MB/s Max: 3383.13 MB/s [W] Avg: 791.14 MB/s Min: 0.00 MB/s Max: 3164.57 MB/s [LNet Rates of servers] [R] Avg: 2485 RPC/s Min: 3 RPC/s Max: 9930 RPC/s [W] Avg: 3275 RPC/s Min: 3 RPC/s Max: 13092 RPC/s [LNet Bandwidth of servers] [R] Avg: 845.63 MB/s Min: 0.00 MB/s Max: 3382.52 MB/s [W] Avg: 790.74 MB/s Min: 0.00 MB/s Max: 3162.97 MB/s [LNet Rates of servers] [R] Avg: 2480 RPC/s Min: 4 RPC/s Max: 9903 RPC/s [W] Avg: 3269 RPC/s Min: 4 RPC/s Max: 13061 RPC/s [LNet Bandwidth of servers] [R] Avg: 842.45 MB/s Min: 0.00 MB/s Max: 3369.80 MB/s [W] Avg: 789.50 MB/s Min: 0.00 MB/s Max: 3158.00 MB/s [LNet Rates of servers] [R] Avg: 2493 RPC/s Min: 3 RPC/s Max: 9957 RPC/s [W] Avg: 3285 RPC/s Min: 3 RPC/s Max: 13125 RPC/s [LNet Bandwidth of servers] [R] Avg: 847.83 MB/s Min: 0.00 MB/s Max: 3391.32 MB/s [W] Avg: 792.44 MB/s Min: 0.00 MB/s Max: 3169.77 MB/s [LNet Rates of servers] [R] Avg: 2479 RPC/s Min: 3 RPC/s Max: 9908 RPC/s [W] Avg: 3269 RPC/s Min: 3 RPC/s Max: 13069 RPC/s [LNet Bandwidth of servers] [R] Avg: 843.48 MB/s Min: 0.00 MB/s Max: 3373.92 MB/s [W] Avg: 790.29 MB/s Min: 0.00 MB/s Max: 3161.17 MB/s [LNet Rates of servers] [R] Avg: 2490 RPC/s Min: 7 RPC/s Max: 9924 RPC/s [W] Avg: 3280 RPC/s Min: 7 RPC/s Max: 13083 RPC/s [LNet Bandwidth of servers] [R] Avg: 842.95 MB/s Min: 0.00 MB/s Max: 3371.80 MB/s [W] Avg: 790.10 MB/s Min: 0.00 MB/s Max: 3160.40 MB/s [LNet Rates of servers] [R] Avg: 2487 RPC/s Min: 3 RPC/s Max: 9939 RPC/s [W] Avg: 3278 RPC/s Min: 3 RPC/s Max: 13104 RPC/s [LNet Bandwidth of servers] [R] Avg: 846.63 MB/s Min: 0.00 MB/s Max: 3386.52 MB/s [W] Avg: 791.59 MB/s Min: 0.00 MB/s Max: 3166.37 MB/s