fir-io1-s1: -- Logs begin at Wed 2019-10-16 11:09:56 PDT. -- fir-io1-s1: Oct 19 10:57:36 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-io1-s1: Oct 19 10:57:36 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s1: Oct 19 11:07:46 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-io1-s1: Oct 19 11:07:46 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s1: Oct 19 11:18:01 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-io1-s1: Oct 19 11:18:01 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s1: Oct 19 11:28:11 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-io1-s1: Oct 19 11:28:11 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s1: Oct 19 11:38:21 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-io1-s1: Oct 19 11:38:21 fir-io1-s1 kernel: LNetError: 60979:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s2: -- Logs begin at Wed 2019-10-16 11:10:15 PDT. -- fir-io2-s2: Oct 19 10:57:01 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s2: Oct 19 10:57:01 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s2: Oct 19 11:07:11 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s2: Oct 19 11:07:11 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s2: Oct 19 11:17:21 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s2: Oct 19 11:17:21 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s2: Oct 19 11:27:31 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s2: Oct 19 11:27:31 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s2: Oct 19 11:37:41 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s2: Oct 19 11:37:41 fir-io2-s2 kernel: LNetError: 57813:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s1: -- Logs begin at Wed 2019-10-16 11:09:56 PDT. -- fir-io3-s1: Oct 19 10:57:58 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io3-s1: Oct 19 10:57:58 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s1: Oct 19 11:08:07 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io3-s1: Oct 19 11:08:07 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s1: Oct 19 11:18:17 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io3-s1: Oct 19 11:18:17 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s1: Oct 19 11:28:27 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io3-s1: Oct 19 11:28:27 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s1: Oct 19 11:38:37 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io3-s1: Oct 19 11:38:37 fir-io3-s1 kernel: LNetError: 59554:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s2: -- Logs begin at Wed 2019-10-16 11:36:03 PDT. -- fir-io1-s2: Oct 19 10:58:02 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io1-s2: Oct 19 10:58:02 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s2: Oct 19 11:08:12 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io1-s2: Oct 19 11:08:12 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s2: Oct 19 11:18:22 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io1-s2: Oct 19 11:18:22 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s2: Oct 19 11:28:32 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io1-s2: Oct 19 11:28:32 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io1-s2: Oct 19 11:38:43 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io1-s2: Oct 19 11:38:43 fir-io1-s2 kernel: LNetError: 20957:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s2: -- Logs begin at Wed 2019-10-16 11:27:14 PDT. -- fir-io4-s2: Oct 19 10:59:08 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s2: Oct 19 10:59:08 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s2: Oct 19 11:09:18 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s2: Oct 19 11:09:18 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s2: Oct 19 11:19:28 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s2: Oct 19 11:19:28 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s2: Oct 19 11:29:38 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s2: Oct 19 11:29:38 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s2: Oct 19 11:39:48 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s2: Oct 19 11:39:48 fir-io4-s2 kernel: LNetError: 45387:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s2: -- Logs begin at Wed 2019-10-16 11:10:30 PDT. -- fir-md1-s2: Oct 19 11:13:55 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s2: Oct 19 11:13:55 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s2: Oct 19 11:24:05 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s2: Oct 19 11:24:05 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s2: Oct 19 11:34:15 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s2: Oct 19 11:34:15 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s1: -- Logs begin at Wed 2019-10-16 11:10:19 PDT. -- fir-io2-s1: Oct 19 10:58:02 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s1: Oct 19 10:58:02 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s1: Oct 19 11:08:12 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s1: Oct 19 11:08:12 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s1: Oct 19 11:18:22 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s1: Oct 19 11:18:22 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s1: Oct 19 11:28:32 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s1: Oct 19 11:28:32 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io2-s1: Oct 19 11:38:42 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-io2-s1: Oct 19 11:38:42 fir-io2-s1 kernel: LNetError: 24865:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s3: -- Logs begin at Wed 2019-10-16 11:27:21 PDT. -- fir-md1-s3: Oct 19 10:52:35 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s3: Oct 19 10:52:35 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s3: Oct 19 11:02:45 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s3: Oct 19 11:02:45 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s3: Oct 19 11:12:55 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s3: Oct 19 11:12:55 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s3: Oct 19 11:23:05 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s3: Oct 19 11:23:05 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s3: Oct 19 11:33:10 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s3: Oct 19 11:33:10 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s4: -- Logs begin at Wed 2019-10-16 11:27:21 PDT. -- fir-md1-s4: Oct 19 10:52:52 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-md1-s4: Oct 19 10:52:52 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s4: Oct 19 11:03:02 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-md1-s4: Oct 19 11:03:02 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s4: Oct 19 11:13:12 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-md1-s4: Oct 19 11:13:12 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s4: Oct 19 11:23:23 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-md1-s4: Oct 19 11:23:23 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s4: Oct 19 11:33:32 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-md1-s4: Oct 19 11:33:32 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s1: -- Logs begin at Wed 2019-10-16 11:10:02 PDT. -- fir-md1-s1: Oct 19 11:22:20 fir-md1-s1 kernel: LNetError: 35332:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-md1-s1: Oct 19 11:22:20 fir-md1-s1 kernel: LNetError: 35332:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s1: Oct 19 11:32:30 fir-md1-s1 kernel: LNetError: 35332:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-md1-s1: Oct 19 11:32:30 fir-md1-s1 kernel: LNetError: 35332:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s2: -- Logs begin at Wed 2019-10-16 11:09:56 PDT. -- fir-io3-s2: Oct 19 10:57:37 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-io3-s2: Oct 19 10:57:37 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s2: Oct 19 11:07:51 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-io3-s2: Oct 19 11:07:51 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s2: Oct 19 11:18:01 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-io3-s2: Oct 19 11:18:01 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s2: Oct 19 11:28:06 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-io3-s2: Oct 19 11:28:06 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io3-s2: Oct 19 11:38:16 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-io3-s2: Oct 19 11:38:16 fir-io3-s2 kernel: LNetError: 61521:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s1: -- Logs begin at Wed 2019-10-16 11:27:16 PDT. -- fir-io4-s1: Oct 19 10:59:08 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s1: Oct 19 10:59:08 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s1: Oct 19 11:09:18 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s1: Oct 19 11:09:18 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s1: Oct 19 11:19:28 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s1: Oct 19 11:19:28 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s1: Oct 19 11:29:38 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s1: Oct 19 11:29:38 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-io4-s1: Oct 19 11:39:48 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.201@o2ib7 added to recovery queue. Health = 900 fir-io4-s1: Oct 19 11:39:48 fir-io4-s1 kernel: LNetError: 49537:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s1: Oct 19 11:42:40 fir-md1-s1 kernel: LNetError: 35332:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.210@o2ib7 added to recovery queue. Health = 900 fir-md1-s1: Oct 19 11:42:40 fir-md1-s1 kernel: LNetError: 35332:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s3: Oct 19 11:43:20 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s3: Oct 19 11:43:20 fir-md1-s3 kernel: LNetError: 18464:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s4: Oct 19 11:43:42 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.209@o2ib7 added to recovery queue. Health = 900 fir-md1-s4: Oct 19 11:43:42 fir-md1-s4 kernel: LNetError: 18507:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages fir-md1-s2: Oct 19 11:44:25 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) lpni 10.0.10.202@o2ib7 added to recovery queue. Health = 900 fir-md1-s2: Oct 19 11:44:25 fir-md1-s2 kernel: LNetError: 35672:0:(peer.c:3451:lnet_peer_ni_add_to_recoveryq_locked()) Skipped 79 previous similar messages