Feb 13 09:48:37 oak-md1-s1 kernel: LustreError: 137-5: oak-MDT0003_UUID: not available for connect from 10.51.16.8@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
Feb 13 09:48:37 oak-md1-s1 kernel: LustreError: Skipped 15 previous similar messages
Feb 13 09:48:42 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-osp-MDT0002: operation mds_statfs to node 10.0.2.52@o2ib5 failed: rc = -107
Feb 13 09:48:42 oak-md1-s1 kernel: Lustre: oak-MDT0000-osp-MDT0002: Connection to oak-MDT0000 (at 10.0.2.52@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Feb 13 09:48:42 oak-md1-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 13 09:48:45 oak-md1-s1 kernel: LustreError: 137-5: oak-MDT0003_UUID: not available for connect from 10.50.8.16@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
Feb 13 09:48:45 oak-md1-s1 kernel: LustreError: Skipped 2470 previous similar messages
Feb 13 09:49:01 oak-md1-s1 kernel: LustreError: 137-5: oak-MDT0003_UUID: not available for connect from 10.50.13.4@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
Feb 13 09:49:01 oak-md1-s1 kernel: LustreError: Skipped 11190 previous similar messages
Feb 13 09:49:18 oak-md1-s1 kernel: Lustre: oak-MDT0001: Connection restored to b6558a5c-1a3d-4260-b9fe-bea2d1135fe6 (at 0@lo)
Feb 13 09:49:18 oak-md1-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 13 09:49:21 oak-md1-s1 kernel: Lustre: oak-MDT0003: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
Feb 13 09:49:22 oak-md1-s1 kernel: LustreError: 3948:0:(mgs_handler.c:282:mgs_revoke_lock()) MGS: can't take cfg lock for 0x6b616f/0x2 : rc = -11
Feb 13 09:49:22 oak-md1-s1 kernel: Lustre: oak-MDD0003: changelog on
Feb 13 09:49:22 oak-md1-s1 kernel: Lustre: oak-MDT0003: in recovery but waiting for the first client to connect
Feb 13 09:49:22 oak-md1-s1 kernel: Lustre: oak-MDT0003: Will be in recovery for at least 2:30, or until 1837 clients reconnect
Feb 13 09:49:22 oak-md1-s1 kernel: Lustre: oak-MDT0003: Connection restored to 930fa9cb-a616-c251-f068-fc37f44d4ee3 (at 10.0.3.36@o2ib5)
Feb 13 09:49:22 oak-md1-s1 kernel: Lustre: Skipped 1 previous similar message
Feb 13 09:49:23 oak-md1-s1 kernel: Lustre: oak-MDT0003: Connection restored to 91b0cc4c-9474-b486-9164-0941db7abe41 (at 10.0.3.57@o2ib5)
Feb 13 09:49:23 oak-md1-s1 kernel: Lustre: Skipped 189 previous similar messages
Feb 13 09:49:26 oak-md1-s1 kernel: Lustre: oak-MDT0003: Connection restored to bc804fc6-1115-4f44-a41f-e00c1f81a31d (at 10.0.3.18@o2ib5)
Feb 13 09:49:26 oak-md1-s1 kernel: Lustre: Skipped 166 previous similar messages
Feb 13 09:49:26 oak-md1-s1 kernel: Lustre: oak-MDT0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
Feb 13 09:49:27 oak-md1-s1 kernel: Lustre: oak-MDD0000: changelog on
Feb 13 09:49:27 oak-md1-s1 kernel: Lustre: oak-MDT0000: in recovery but waiting for the first client to connect
Feb 13 09:49:27 oak-md1-s1 kernel: Lustre: oak-MDT0000: Will be in recovery for at least 2:30, or until 1837 clients reconnect
Feb 13 09:49:27 oak-md1-s1 kernel: Lustre: oak-MDT0000: nosquash_nids set to 10.49.0.[11-12]@o2ib1 10.50.0.1@o2ib2 10.50.0.[11-12]@o2ib2 10.51.0.[1-2]@o2ib3 10.51.0.[11-18]@o2ib3 10.0.2.[1-3]@o2ib5 10.0.2.6@o2ib5 10.0.2.46@o2ib5 10.0.2.[51-58]@o2ib5 10.0.2.[66-77]@o2ib5 10.0.2.79@o2ib5 10.0.2.[82-92]@o2ib5 10.0.2.[101-120]@o2ib5 10.0.2.[221-223]@o2ib5 10.0.2.[226-227]@o2ib5 10.0.2.[228-229]@o2ib5 10.0.2.[232-236]@o2ib5 10.0.2.[240-252]@o2ib5 10.0.2.[113]@o2ib5 10.0.3.[1-254]@o2ib5 10.210.12.6@tcp1 10.210.15.[1-2,253-254]@tcp1 10.210.14.[34-35]@tcp1 10.210.13.31@tcp1
Feb 13 09:49:27 oak-md1-s1 kernel: Lustre: oak-MDT0000: root_squash is set to 99:99
Feb 13 09:49:27 oak-md1-s1 kernel: Lustre: Skipped 4 previous similar messages
Feb 13 09:49:27 oak-md1-s1 kernel: LustreError: 119931:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa1390fb7b850 x1738420438157184/t0(0) o601->oak-MDT0000-lwp-OST0149_UUID@10.0.2.104@o2ib5:20/0 lens 336/0 e 0 to 0 dl 1676310665 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:49:27 oak-md1-s1 kernel: LustreError: 119931:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 3 previous similar messages
Feb 13 09:49:28 oak-md1-s1 kernel: LustreError: 60177:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa11d7bc6cc80 x1738420438234368/t0(0) o601->oak-MDT0000-lwp-OST0145_UUID@10.0.2.104@o2ib5:21/0 lens 336/0 e 0 to 0 dl 1676310666 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:49:28 oak-md1-s1 kernel: LustreError: 60177:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 239 previous similar messages
Feb 13 09:49:29 oak-md1-s1 kernel: LustreError: 73081:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa12d8ed35580 x1748920082410880/t0(0) o601->oak-MDT0000-lwp-OST01f0_UUID@10.0.2.101@o2ib5:22/0 lens 336/0 e 0 to 0 dl 1676310667 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:49:29 oak-md1-s1 kernel: LustreError: 73081:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 2440 previous similar messages
Feb 13 09:49:31 oak-md1-s1 kernel: LustreError: 60279:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa129b84e5100 x1738527497864320/t0(0) o601->oak-MDT0000-lwp-OST018d_UUID@10.0.2.112@o2ib5:24/0 lens 336/0 e 0 to 0 dl 1676310669 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:49:31 oak-md1-s1 kernel: LustreError: 60279:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 11691 previous similar messages
Feb 13 09:49:31 oak-md1-s1 kernel: Lustre: oak-MDT0003: Connection restored to ab7ba83c-2cca-97ea-50ec-1944bdf54bea (at 10.210.15.134@tcp1)
Feb 13 09:49:31 oak-md1-s1 kernel: Lustre: Skipped 354 previous similar messages
Feb 13 09:49:35 oak-md1-s1 kernel: LustreError: 117785:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa11f48d13a80 x1738527498180928/t0(0) o601->oak-MDT0000-lwp-OST0173_UUID@10.0.2.112@o2ib5:60/0 lens 336/0 e 0 to 0 dl 1676310705 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:49:35 oak-md1-s1 kernel: LustreError: 117785:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 7589 previous similar messages
Feb 13 09:49:43 oak-md1-s1 kernel: LustreError: 67627:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa11dbf430900 x1738527498371712/t0(0) o601->oak-MDT0000-lwp-OST0173_UUID@10.0.2.112@o2ib5:68/0 lens 336/0 e 0 to 0 dl 1676310713 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:49:43 oak-md1-s1 kernel: LustreError: 67627:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 3559 previous similar messages
Feb 13 09:49:51 oak-md1-s1 kernel: Lustre: oak-MDT0000: Connection restored to b6558a5c-1a3d-4260-b9fe-bea2d1135fe6 (at 0@lo)
Feb 13 09:49:51 oak-md1-s1 kernel: Lustre: Skipped 3 previous similar messages
Feb 13 09:49:59 oak-md1-s1 kernel: LustreError: 14308:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa132568c9680 x1738527498406400/t0(0) o601->oak-MDT0000-lwp-OST016f_UUID@10.0.2.112@o2ib5:84/0 lens 336/0 e 0 to 0 dl 1676310729 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:49:59 oak-md1-s1 kernel: LustreError: 14308:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 5215 previous similar messages
Feb 13 09:50:08 oak-md1-s1 kernel: Lustre: oak-MDT0003: Connection restored to 919d6b9e-d487-da25-18aa-a2206d91b08a (at 10.51.16.21@o2ib3)
Feb 13 09:50:08 oak-md1-s1 kernel: Lustre: Skipped 1832 previous similar messages
Feb 13 09:50:27 oak-md1-s1 kernel: LustreError: 167-0: oak-MDT0000-lwp-MDT0002: This client was evicted by oak-MDT0000; in progress operations using this service will fail.
Feb 13 09:50:27 oak-md1-s1 kernel: LustreError: Skipped 1 previous similar message
Feb 13 09:50:31 oak-md1-s1 kernel: LustreError: 85594:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa12426a91850 x1748909586522816/t0(0) o601->oak-MDT0000-lwp-OST01d3_UUID@10.0.2.102@o2ib5:116/0 lens 336/0 e 0 to 0 dl 1676310761 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:50:31 oak-md1-s1 kernel: LustreError: 85594:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 24147 previous similar messages
Feb 13 09:51:35 oak-md1-s1 kernel: LustreError: 60357:0:(tgt_handler.c:526:tgt_filter_recovery_request()) @@@ not permitted during recovery  req@ffffa13372640480 x1738551880507456/t0(0) o601->oak-MDT0000-lwp-OST0166_UUID@10.0.2.111@o2ib5:180/0 lens 336/0 e 0 to 0 dl 1676310825 ref 1 fl Interpret:/0/ffffffff rc 0/-1
Feb 13 09:51:35 oak-md1-s1 kernel: LustreError: 60357:0:(tgt_handler.c:526:tgt_filter_recovery_request()) Skipped 55842 previous similar messages
Feb 13 09:51:38 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-MDT0002: operation quota_acquire to node 0@lo failed: rc = -11
Feb 13 09:51:38 oak-md1-s1 kernel: LustreError: Skipped 20 previous similar messages
Feb 13 09:51:42 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11
Feb 13 09:51:42 oak-md1-s1 kernel: LustreError: Skipped 18 previous similar messages
Feb 13 09:51:44 oak-md1-s1 kernel: Lustre: oak-MDT0003-osp-MDT0000: Connection restored to 10.0.2.51@o2ib5 (at 0@lo)
Feb 13 09:51:44 oak-md1-s1 kernel: Lustre: Skipped 2033 previous similar messages
Feb 13 09:51:44 oak-md1-s1 kernel: Lustre: oak-MDT0003: Recovery over after 2:22, of 1837 clients 1837 recovered and 0 were evicted.
Feb 13 09:51:44 oak-md1-s1 kernel: LustreError: 188898:0:(llog_cat.c:533:llog_cat_current_log()) oak-MDD0003: next log does not exist!
Feb 13 09:51:54 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-MDT0002: operation quota_acquire to node 0@lo failed: rc = -11
Feb 13 09:51:54 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-MDT0002: operation quota_acquire to node 0@lo failed: rc = -11
Feb 13 09:51:54 oak-md1-s1 kernel: LustreError: Skipped 21 previous similar messages
Feb 13 09:51:54 oak-md1-s1 kernel: LustreError: Skipped 25 previous similar messages
Feb 13 09:52:22 oak-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1676310742.6098
Feb 13 09:52:39 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-MDT0002: operation quota_acquire to node 0@lo failed: rc = -11
Feb 13 09:52:39 oak-md1-s1 kernel: LustreError: Skipped 41 previous similar messages
Feb 13 09:52:57 oak-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1676310777.1090
Feb 13 09:52:59 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-MDT0001: operation quota_acquire to node 0@lo failed: rc = -11
Feb 13 09:52:59 oak-md1-s1 kernel: LustreError: Skipped 78 previous similar messages
Feb 13 09:53:03 oak-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1676310783.188891
Feb 13 09:53:08 oak-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1676310788.188918
Feb 13 09:53:17 oak-md1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1676310797.188904
Feb 13 09:53:39 oak-md1-s1 kernel: LustreError: 11-0: oak-MDT0000-lwp-MDT0002: operation quota_acquire to node 0@lo failed: rc = -11
Feb 13 09:53:39 oak-md1-s1 kernel: LustreError: Skipped 25 previous similar messages
Feb 13 09:53:39 oak-md1-s1 kernel: Lustre: oak-MDT0000-osp-MDT0003: Connection restored to 10.0.2.51@o2ib5 (at 0@lo)
Feb 13 09:53:39 oak-md1-s1 kernel: LustreError: 86451:0:(llog_cat.c:533:llog_cat_current_log()) oak-MDD0000: next log does not exist!
Feb 13 09:53:39 oak-md1-s1 kernel: Lustre: Skipped 2 previous similar messages
Feb 13 09:53:39 oak-md1-s1 kernel: Lustre: oak-MDT0000: Recovery over after 4:12, of 1837 clients 1837 recovered and 0 were evicted.