Mar 17 08:59:18 sh-101-09 kernel: LustreError: 11-0: fir-MDT0001-mdc-ffff9dc7b03c0000: operation mds_close to node 10.0.10.52@o2ib7 failed: rc = -19
Mar 17 08:59:18 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 17 08:59:26 sh-101-09 kernel: Lustre: fir-MDT0002-mdc-ffff9dc7b03c0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 17 08:59:26 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 17 09:01:14 sh-101-09 kernel: Lustre: 93483:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552838467/real 1552838467]  req@ffff9db0778c5700 x1626173969598592/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1552838474 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
Mar 17 09:01:14 sh-101-09 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail
Mar 17 09:01:59 sh-101-09 kernel: LustreError: 96368:0:(lmv_obd.c:1412:lmv_statfs()) can't stat MDS #0 (fir-MDT0001-mdc-ffff9dc7b03c0000), error -19
Mar 17 09:01:59 sh-101-09 kernel: LustreError: 96368:0:(llite_lib.c:1807:ll_statfs_internal()) md_statfs fails: rc = -19
Mar 17 09:02:03 sh-101-09 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x974d7e52603ff1 to 0xe9a38a3206368234
Mar 17 09:02:03 sh-101-09 kernel: Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7)
Mar 17 09:02:20 sh-101-09 kernel: Lustre: fir-MDT0000-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7)
Mar 19 18:57:33 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047032/real 1553047032]  req@ffff9dc71334a100 x1626174016953328/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553047053 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 19 18:57:33 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 18:57:33 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 18:57:33 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 19 18:57:54 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047053/real 1553047053]  req@ffff9dc71334a100 x1626174016953328/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553047074 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 19 18:57:54 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 18:57:54 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 18:58:15 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047074/real 1553047074]  req@ffff9dc71334a100 x1626174016953328/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553047095 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 19 18:58:15 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 18:58:15 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 18:58:36 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047095/real 1553047095]  req@ffff9dc71334a100 x1626174016953328/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553047116 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 19 18:58:36 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 18:58:36 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 18:58:57 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047116/real 1553047116]  req@ffff9dc71334a100 x1626174016953328/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553047137 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 19 18:58:57 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 18:58:57 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 18:59:18 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047137/real 1553047137]  req@ffff9dc71334a100 x1626174016953328/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553047158 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 19 18:59:18 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 18:59:18 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 18:59:39 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047158/real 1553047158]  req@ffff9dc71334a100 x1626174016953328/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553047179 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 19 18:59:39 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 18:59:39 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 19:00:12 sh-101-09 kernel: Lustre: 107262:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047181/real 1553047181]  req@ffff9dc747a68900 x1626174016972992/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 568/3376 e 0 to 1 dl 1553047212 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 19 19:00:12 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 19:00:12 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 19:00:12 sh-101-09 kernel: Lustre: 107262:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Mar 19 19:01:45 sh-101-09 kernel: Lustre: 107255:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047274/real 1553047274]  req@ffff9db794e78f00 x1626174016973120/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553047305 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Mar 19 19:01:45 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 19:01:45 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 19 19:01:45 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 19:01:45 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 19 19:01:45 sh-101-09 kernel: Lustre: 107255:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Mar 19 19:04:15 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553047424/real 1553047424]  req@ffff9db9595d1500 x1626174016990432/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553047455 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 19 19:04:15 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 19 19:04:15 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 19 19:04:15 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 19 19:04:15 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 19 19:04:15 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
Mar 20 00:21:43 sh-101-09 kernel: LustreError: 11-0: fir-MDT0001-mdc-ffff9dc7b03c0000: operation ldlm_enqueue to node 10.0.10.52@o2ib7 failed: rc = -107
Mar 20 00:21:43 sh-101-09 kernel: LustreError: Skipped 1 previous similar message
Mar 20 00:21:43 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 00:21:43 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 20 00:21:43 sh-101-09 kernel: LustreError: 167-0: fir-MDT0001-mdc-ffff9dc7b03c0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail.
Mar 20 00:21:43 sh-101-09 kernel: LustreError: 107259:0:(file.c:4393:ll_inode_revalidate_fini()) fir: revalidate FID [0x24000e4ad:0xc014:0x0] error: rc = -5
Mar 20 00:21:43 sh-101-09 kernel: LustreError: 107261:0:(llite_lib.c:1551:ll_md_setattr()) md_setattr fails: rc = -108
Mar 20 00:21:43 sh-101-09 kernel: LustreError: 78081:0:(vvp_io.c:1495:vvp_io_init()) fir: refresh file layout [0x24000e4ad:0xc014:0x0] error -108.
Mar 20 00:21:43 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 00:21:43 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 20 00:22:14 sh-101-09 kernel: Lustre: 78086:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553066503/real 1553066503]  req@ffff9db4463ee300 x1626174027355200/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553066534 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 20 00:22:14 sh-101-09 kernel: Lustre: 78086:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Mar 20 00:22:45 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 00:22:45 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 00:22:45 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 00:22:45 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 00:23:16 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553066565/real 1553066565]  req@ffff9dbee8f91500 x1626174027355344/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 0 to 1 dl 1553066596 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 00:23:16 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Mar 20 00:24:44 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553066653/real 1553066653]  req@ffff9dbe48c4b000 x1626174027370320/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 0 to 1 dl 1553066684 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 20 00:24:44 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 00:24:44 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 20 00:24:44 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 00:24:44 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 20 00:24:44 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Mar 20 00:26:48 sh-101-09 kernel: LustreError: 167-0: fir-MDT0001-mdc-ffff9dc7b03c0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail.
Mar 20 00:26:48 sh-101-09 kernel: LustreError: 78088:0:(vvp_io.c:1495:vvp_io_init()) fir: refresh file layout [0x24000e4ad:0xc014:0x0] error -108.
Mar 20 00:27:19 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553066808/real 1553066808]  req@ffff9db961740900 x1626174027384640/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 0 to 1 dl 1553066839 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 20 00:27:19 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
Mar 20 00:27:19 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 00:27:19 sh-101-09 kernel: Lustre: Skipped 4 previous similar messages
Mar 20 00:27:19 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 00:27:19 sh-101-09 kernel: Lustre: Skipped 4 previous similar messages
Mar 20 00:31:47 sh-101-09 kernel: Lustre: 107256:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553067076/real 1553067076]  req@ffff9dbbb446bf00 x1626174027400848/t0(0) o55->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 472/224 e 0 to 1 dl 1553067107 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 00:31:47 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 00:31:47 sh-101-09 kernel: Lustre: Skipped 7 previous similar messages
Mar 20 00:31:47 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 00:31:47 sh-101-09 kernel: Lustre: Skipped 7 previous similar messages
Mar 20 00:31:47 sh-101-09 kernel: Lustre: 107256:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Mar 20 02:20:11 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553073580/real 1553073580]  req@ffff9db9595d7800 x1626174038218000/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 576/2088 e 0 to 1 dl 1553073611 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 20 02:20:11 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 02:20:11 sh-101-09 kernel: Lustre: Skipped 15 previous similar messages
Mar 20 02:20:11 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 02:20:11 sh-101-09 kernel: Lustre: Skipped 15 previous similar messages
Mar 20 02:20:11 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
Mar 20 02:21:44 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553073673/real 1553073673]  req@ffff9db9595d7800 x1626174038218000/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 576/2088 e 0 to 1 dl 1553073704 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 02:21:44 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 02:21:44 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 20 02:21:44 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 02:21:44 sh-101-09 kernel: Lustre: Skipped 2 previous similar messages
Mar 20 02:21:44 sh-101-09 kernel: Lustre: 78079:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Mar 20 02:22:15 sh-101-09 kernel: LustreError: 167-0: fir-MDT0001-mdc-ffff9dc7b03c0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail.
Mar 20 02:22:15 sh-101-09 kernel: LustreError: 78084:0:(llite_lib.c:1551:ll_md_setattr()) md_setattr fails: rc = -5
Mar 20 02:22:15 sh-101-09 kernel: LustreError: 78079:0:(file.c:4393:ll_inode_revalidate_fini()) fir: revalidate FID [0x240005ab2:0x1e49e:0x0] error: rc = -5
Mar 20 02:22:15 sh-101-09 kernel: Lustre: 93503:0:(llite_lib.c:2733:ll_dirty_page_discard_warn()) fir: dirty page discard: 10.0.10.51@o2ib7:10.0.10.52@o2ib7:/fir/fid: [0x24000ed00:0x10:0x0]// may get corrupted (rc -108)
Mar 20 02:22:15 sh-101-09 kernel: LustreError: 107259:0:(vvp_io.c:1495:vvp_io_init()) fir: refresh file layout [0x24000ed00:0x10:0x0] error -108.
Mar 20 02:22:20 sh-101-09 kernel: julia[117508]: segfault at 18 ip 00007fc5da76f3d2 sp 00007fff3acf86e0 error 4 in libhdf5.so.100.0.1[7fc5da68f000+37c000]
Mar 20 03:34:10 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553078028/real 1553078028]  req@ffff9dba188c5400 x1626174045654272/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 1 to 1 dl 1553078049 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 20 03:34:10 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Mar 20 03:34:10 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 03:34:10 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 03:34:10 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 03:34:10 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 03:34:31 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553078050/real 1553078050]  req@ffff9dba188c5400 x1626174045654272/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 1 to 1 dl 1553078071 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Mar 20 03:34:31 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 03:34:31 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 03:35:13 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553078092/real 1553078092]  req@ffff9dba188c5400 x1626174045654272/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 1 to 1 dl 1553078113 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 03:35:13 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Mar 20 03:35:13 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 03:35:13 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 03:35:13 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 03:35:13 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 03:36:37 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553078176/real 1553078176]  req@ffff9dba188c5400 x1626174045654272/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 1 to 1 dl 1553078197 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 03:36:37 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages
Mar 20 03:36:37 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 03:36:37 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 20 03:36:37 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 03:36:37 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 20 03:39:04 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553078323/real 1553078323]  req@ffff9dba188c5400 x1626174045654272/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 1 to 1 dl 1553078344 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 03:39:04 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Mar 20 03:39:04 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 03:39:04 sh-101-09 kernel: Lustre: Skipped 6 previous similar messages
Mar 20 03:39:04 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 03:39:04 sh-101-09 kernel: Lustre: Skipped 6 previous similar messages
Mar 20 03:43:20 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553078569/real 1553078569]  req@ffff9db452ddf800 x1626174045729520/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 568/3376 e 0 to 1 dl 1553078600 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Mar 20 03:43:20 sh-101-09 kernel: Lustre: 107254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
Mar 20 03:43:20 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 03:43:20 sh-101-09 kernel: Lustre: Skipped 10 previous similar messages
Mar 20 03:43:20 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 03:43:20 sh-101-09 kernel: Lustre: Skipped 10 previous similar messages
Mar 20 03:44:53 sh-101-09 kernel: LustreError: 167-0: fir-MDT0001-mdc-ffff9dc7b03c0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail.
Mar 20 07:30:13 sh-101-09 kernel: Lustre: 107261:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553092192/real 1553092192]  req@ffff9db8bda08c00 x1626174119377488/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 1 to 1 dl 1553092213 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 20 07:30:13 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 07:30:13 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 20 07:30:13 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 07:30:13 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 20 07:30:13 sh-101-09 kernel: Lustre: 107261:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Mar 20 07:31:24 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553092253/real 1553092253]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553092284 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Mar 20 07:31:24 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message
Mar 20 07:31:24 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 07:31:24 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 07:31:24 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 07:31:24 sh-101-09 kernel: Lustre: Skipped 1 previous similar message
Mar 20 07:33:59 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553092408/real 1553092408]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553092439 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 07:33:59 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
Mar 20 07:33:59 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 07:33:59 sh-101-09 kernel: Lustre: Skipped 4 previous similar messages
Mar 20 07:33:59 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 07:33:59 sh-101-09 kernel: Lustre: Skipped 4 previous similar messages
Mar 20 07:38:38 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553092687/real 1553092687]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553092718 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 07:38:38 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
Mar 20 07:38:38 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 07:38:38 sh-101-09 kernel: Lustre: Skipped 8 previous similar messages
Mar 20 07:38:38 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 07:38:38 sh-101-09 kernel: Lustre: Skipped 8 previous similar messages
Mar 20 07:47:25 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553093214/real 1553093214]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553093245 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 07:47:25 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
Mar 20 07:47:25 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 07:47:25 sh-101-09 kernel: Lustre: Skipped 16 previous similar messages
Mar 20 07:47:25 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 07:47:25 sh-101-09 kernel: Lustre: Skipped 16 previous similar messages
Mar 20 07:57:46 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553093834/real 1553093834]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553093865 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 07:57:46 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
Mar 20 07:57:46 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 07:57:46 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 07:57:46 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 07:57:46 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 08:08:06 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553094455/real 1553094455]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553094486 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 08:08:06 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
Mar 20 08:08:06 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 08:08:06 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 08:08:06 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 08:08:06 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 08:18:26 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553095075/real 1553095075]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553095106 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 08:18:26 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
Mar 20 08:18:26 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 08:18:26 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 08:18:26 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 08:18:26 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 08:28:46 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553095695/real 1553095695]  req@ffff9dc66f0d0300 x1626174119380752/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 0 to 1 dl 1553095726 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 08:28:46 sh-101-09 kernel: Lustre: 107258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages
Mar 20 08:28:46 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 08:28:46 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 08:28:46 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 08:28:46 sh-101-09 kernel: Lustre: Skipped 19 previous similar messages
Mar 20 08:36:23 sh-101-09 kernel: LustreError: 11-0: fir-MDT0001-mdc-ffff9dc7b03c0000: operation ldlm_enqueue to node 10.0.10.52@o2ib7 failed: rc = -107
Mar 20 08:36:23 sh-101-09 kernel: LustreError: 167-0: fir-MDT0001-mdc-ffff9dc7b03c0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail.
Mar 20 08:55:34 sh-101-09 kernel: Lustre: 78087:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553097313/real 1553097313]  req@ffff9dbd48ac2d00 x1626174131593696/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 1 to 1 dl 1553097334 ref 2 fl Rpc:X/0/ffffffff rc 0/-1
Mar 20 08:55:34 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 08:55:34 sh-101-09 kernel: Lustre: Skipped 13 previous similar messages
Mar 20 08:55:34 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 08:55:34 sh-101-09 kernel: Lustre: Skipped 13 previous similar messages
Mar 20 08:55:34 sh-101-09 kernel: Lustre: 78087:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
Mar 20 08:56:58 sh-101-09 kernel: Lustre: 78082:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553097397/real 1553097397]  req@ffff9dbd48ac4500 x1626174131594016/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 568/3376 e 1 to 1 dl 1553097418 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 08:56:58 sh-101-09 kernel: Lustre: 107256:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553097397/real 1553097397]  req@ffff9da78e541800 x1626174131593872/t0(0) o55->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 472/224 e 1 to 1 dl 1553097418 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Mar 20 08:56:58 sh-101-09 kernel: Lustre: 78086:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553097397/real 1553097397]  req@ffff9dbd48ac2a00 x1626174131593856/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 480/568 e 1 to 1 dl 1553097418 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Mar 20 08:56:58 sh-101-09 kernel: Lustre: 106303:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553097397/real 1553097397]  req@ffff9da955ca6000 x1626174131594032/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 576/2088 e 1 to 1 dl 1553097418 ref 2 fl Rpc:X/2/ffffffff rc -11/-1
Mar 20 08:56:58 sh-101-09 kernel: Lustre: 107256:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
Mar 20 08:56:58 sh-101-09 kernel: Lustre: 78086:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
Mar 20 08:56:58 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 08:56:58 sh-101-09 kernel: Lustre: 106303:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
Mar 20 08:56:58 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 20 08:56:58 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 08:56:58 sh-101-09 kernel: Lustre: Skipped 3 previous similar messages
Mar 20 08:59:46 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete
Mar 20 08:59:46 sh-101-09 kernel: Lustre: Skipped 7 previous similar messages
Mar 20 08:59:46 sh-101-09 kernel: Lustre: fir-MDT0001-mdc-ffff9dc7b03c0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7)
Mar 20 08:59:46 sh-101-09 kernel: Lustre: Skipped 7 previous similar messages
Mar 20 09:00:28 sh-101-09 kernel: LustreError: 167-0: fir-MDT0001-mdc-ffff9dc7b03c0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail.
Mar 20 09:00:28 sh-101-09 kernel: LustreError: 106303:0:(file.c:4393:ll_inode_revalidate_fini()) fir: revalidate FID [0x24000ed7c:0xdd:0x0] error: rc = -5
Mar 20 09:00:28 sh-101-09 kernel: LustreError: 78082:0:(llite_lib.c:1551:ll_md_setattr()) md_setattr fails: rc = -108
Mar 20 09:00:28 sh-101-09 kernel: LustreError: 106303:0:(file.c:4393:ll_inode_revalidate_fini()) Skipped 2 previous similar messages
Mar 20 09:02:01 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553097690/real 1553097690]  req@ffff9da91507ce00 x1626174131669920/t0(0) o101->fir-MDT0001-mdc-ffff9dc7b03c0000@10.0.10.52@o2ib7:12/10 lens 600/2088 e 0 to 1 dl 1553097721 ref 2 fl Rpc:X/2/ffffffff rc 0/-1
Mar 20 09:02:01 sh-101-09 kernel: Lustre: 194029:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 39 previous similar messages
Mar 20 09:44:20 sh-101-09 kernel: Lustre: DEBUG MARKER: Wed Mar 20 09:44:20 2019