Feb 08 11:24:24 fir-io1-s1 kernel: LNet: HW NUMA nodes: 4, HW CPU cores: 48, npartitions: 4 Feb 08 11:24:24 fir-io1-s1 kernel: alg: No test for adler32 (adler32-zlib) Feb 08 11:24:24 fir-io1-s1 kernel: Lustre: Lustre: Build Version: 2.12.0 Feb 08 11:24:25 fir-io1-s1 kernel: LNet: Using FastReg for registration Feb 08 11:24:25 fir-io1-s1 kernel: LNet: Added LNI 10.0.10.101@o2ib7 [8/256/0/180] Feb 08 11:24:33 fir-io1-s1 kernel: md: md0 stopped. Feb 08 11:24:33 fir-io1-s1 kernel: async_tx: api initialized (async) Feb 08 11:24:33 fir-io1-s1 kernel: xor: automatically using best checksumming function: Feb 08 11:24:33 fir-io1-s1 kernel: avx : 9636.000 MB/sec Feb 08 11:24:33 fir-io1-s1 kernel: raid6: sse2x1 gen() 6097 MB/s Feb 08 11:24:33 fir-io1-s1 kernel: raid6: sse2x2 gen() 11339 MB/s Feb 08 11:24:33 fir-io1-s1 kernel: raid6: sse2x4 gen() 12957 MB/s Feb 08 11:24:33 fir-io1-s1 kernel: raid6: avx2x1 gen() 14257 MB/s Feb 08 11:24:33 fir-io1-s1 kernel: raid6: avx2x2 gen() 18871 MB/s Feb 08 11:24:33 fir-io1-s1 kernel: raid6: avx2x4 gen() 18851 MB/s Feb 08 11:24:33 fir-io1-s1 kernel: raid6: using algorithm avx2x2 gen() (18871 MB/s) Feb 08 11:24:33 fir-io1-s1 kernel: raid6: using avx2x2 recovery algorithm Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-20 operational as raid disk 0 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-23 operational as raid disk 9 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-55 operational as raid disk 8 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-42 operational as raid disk 7 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-47 operational as raid disk 6 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-93 operational as raid disk 5 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-108 operational as raid disk 4 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-18 operational as raid disk 3 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-90 operational as raid disk 2 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: device dm-56 operational as raid disk 1 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md0: raid level 6 active with 10 out of 10 devices, algorithm 2 Feb 08 11:24:34 fir-io1-s1 kernel: md0: detected capacity change from 0 to 64011422924800 Feb 08 11:24:34 fir-io1-s1 kernel: md: md10 stopped. Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: not clean -- starting background reconstruction Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-11 operational as raid disk 0 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-6 operational as raid disk 9 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-5 operational as raid disk 8 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-66 operational as raid disk 7 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-75 operational as raid disk 6 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-57 operational as raid disk 5 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-62 operational as raid disk 4 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-95 operational as raid disk 3 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-104 operational as raid disk 2 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: device dm-12 operational as raid disk 1 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md10: raid level 6 active with 10 out of 10 devices, algorithm 2 Feb 08 11:24:34 fir-io1-s1 kernel: md10: detected capacity change from 0 to 64011422924800 Feb 08 11:24:34 fir-io1-s1 kernel: md: resync of RAID array md10 Feb 08 11:24:34 fir-io1-s1 kernel: md: md6 stopped. Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-96 operational as raid disk 0 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-70 operational as raid disk 9 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-79 operational as raid disk 8 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-114 operational as raid disk 7 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-110 operational as raid disk 6 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-107 operational as raid disk 5 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-88 operational as raid disk 4 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-86 operational as raid disk 3 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-65 operational as raid disk 2 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: device dm-99 operational as raid disk 1 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md6: raid level 6 active with 10 out of 10 devices, algorithm 2 Feb 08 11:24:34 fir-io1-s1 kernel: md6: detected capacity change from 0 to 64011422924800 Feb 08 11:24:34 fir-io1-s1 kernel: md: md2 stopped. Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: not clean -- starting background reconstruction Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-54 operational as raid disk 0 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-35 operational as raid disk 9 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-26 operational as raid disk 8 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-30 operational as raid disk 7 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-116 operational as raid disk 6 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-37 operational as raid disk 5 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-52 operational as raid disk 4 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-14 operational as raid disk 3 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-44 operational as raid disk 2 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: device dm-46 operational as raid disk 1 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md2: raid level 6 active with 10 out of 10 devices, algorithm 2 Feb 08 11:24:34 fir-io1-s1 kernel: md2: detected capacity change from 0 to 64011422924800 Feb 08 11:24:34 fir-io1-s1 kernel: md: resync of RAID array md2 Feb 08 11:24:34 fir-io1-s1 kernel: md: md4 stopped. Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: not clean -- starting background reconstruction Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-27 operational as raid disk 0 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-50 operational as raid disk 9 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-34 operational as raid disk 8 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-105 operational as raid disk 7 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-33 operational as raid disk 6 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-19 operational as raid disk 5 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-40 operational as raid disk 4 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-78 operational as raid disk 3 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-22 operational as raid disk 2 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: device dm-53 operational as raid disk 1 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md4: raid level 6 active with 10 out of 10 devices, algorithm 2 Feb 08 11:24:34 fir-io1-s1 kernel: md4: detected capacity change from 0 to 64011422924800 Feb 08 11:24:34 fir-io1-s1 kernel: md: resync of RAID array md4 Feb 08 11:24:34 fir-io1-s1 kernel: md: md8 stopped. Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: not clean -- starting background reconstruction Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-74 operational as raid disk 0 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-2 operational as raid disk 9 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-1 operational as raid disk 8 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-69 operational as raid disk 7 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-71 operational as raid disk 6 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-112 operational as raid disk 5 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-82 operational as raid disk 4 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-92 operational as raid disk 3 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-91 operational as raid disk 2 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: device dm-80 operational as raid disk 1 Feb 08 11:24:34 fir-io1-s1 kernel: md/raid:md8: raid level 6 active with 10 out of 10 devices, algorithm 2 Feb 08 11:24:34 fir-io1-s1 kernel: md8: detected capacity change from 0 to 64011422924800 Feb 08 11:24:34 fir-io1-s1 kernel: md: resync of RAID array md8 Feb 08 11:24:35 fir-io1-s1 kernel: LDISKFS-fs warning (device md0): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Feb 08 11:24:35 fir-io1-s1 kernel: LDISKFS-fs warning (device md10): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Feb 08 11:24:35 fir-io1-s1 kernel: LDISKFS-fs warning (device md6): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Feb 08 11:24:35 fir-io1-s1 kernel: LDISKFS-fs warning (device md2): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Feb 08 11:24:35 fir-io1-s1 kernel: LDISKFS-fs warning (device md4): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Feb 08 11:24:35 fir-io1-s1 kernel: LDISKFS-fs warning (device md8): ldiskfs_multi_mount_protect:321: MMP interval 42 higher than expected, please wait. Feb 08 11:25:00 fir-io1-s1 kernel: md: md8: resync done. Feb 08 11:25:17 fir-io1-s1 kernel: LDISKFS-fs (md0): file extents enabled, maximum tree depth=5 Feb 08 11:25:17 fir-io1-s1 kernel: LDISKFS-fs (md10): file extents enabled, maximum tree depth=5 Feb 08 11:25:17 fir-io1-s1 kernel: LDISKFS-fs (md6): file extents enabled, maximum tree depth=5 Feb 08 11:25:17 fir-io1-s1 kernel: LDISKFS-fs (md2): file extents enabled, maximum tree depth=5 Feb 08 11:25:17 fir-io1-s1 kernel: LDISKFS-fs (md4): file extents enabled, maximum tree depth=5 Feb 08 11:25:18 fir-io1-s1 kernel: LDISKFS-fs (md8): file extents enabled, maximum tree depth=5 Feb 08 11:25:19 fir-io1-s1 kernel: LDISKFS-fs (md10): recovery complete Feb 08 11:25:19 fir-io1-s1 kernel: LDISKFS-fs (md10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Feb 08 11:25:19 fir-io1-s1 kernel: LDISKFS-fs (md0): recovery complete Feb 08 11:25:19 fir-io1-s1 kernel: LDISKFS-fs (md0): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Feb 08 11:25:20 fir-io1-s1 kernel: LDISKFS-fs (md6): recovery complete Feb 08 11:25:20 fir-io1-s1 kernel: LDISKFS-fs (md2): recovery complete Feb 08 11:25:20 fir-io1-s1 kernel: LDISKFS-fs (md6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Feb 08 11:25:20 fir-io1-s1 kernel: LDISKFS-fs (md2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Feb 08 11:25:20 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0000_UUID: not available for connect from 10.8.18.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 08 11:25:20 fir-io1-s1 kernel: LDISKFS-fs (md8): recovery complete Feb 08 11:25:20 fir-io1-s1 kernel: LDISKFS-fs (md8): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Feb 08 11:25:20 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000a_UUID: not available for connect from 10.8.4.32@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 08 11:25:20 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Feb 08 11:25:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Not available for connect from 10.9.104.4@o2ib4 (not set up) Feb 08 11:25:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Feb 08 11:25:21 fir-io1-s1 kernel: Lustre: fir-OST0000: in recovery but waiting for the first client to connect Feb 08 11:25:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Will be in recovery for at least 2:30, or until 548 clients reconnect Feb 08 11:25:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4376dd10-0b2f-912c-8c73-35fe5309aa36 (at 10.8.22.25@o2ib6) Feb 08 11:25:21 fir-io1-s1 kernel: LDISKFS-fs (md4): recovery complete Feb 08 11:25:21 fir-io1-s1 kernel: LDISKFS-fs (md4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Feb 08 11:25:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Denying connection for new client 43b10f1e-70d4-fb89-bd84-bb2b7f7a2036(at 10.8.2.22@o2ib6), waiting for 548 known clients (4 recovered, 2 in progress, and 0 evicted) already passed deadline 2:30 Feb 08 11:25:21 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0004_UUID: not available for connect from 10.8.30.26@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 08 11:25:21 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e6f15ae7-1884-d676-5174-cffd0284b92d (at 10.8.1.30@o2ib6) Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: Skipped 36 previous similar messages Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: fir-OST000a: Denying connection for new client 9ea86c7e-c197-f9f9-5450-9cd734e89e2d(at 10.8.1.9@o2ib6), waiting for 524 known clients (13 recovered, 6 in progress, and 0 evicted) already passed deadline 2:30 Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: fir-OST0006: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: fir-OST0006: in recovery but waiting for the first client to connect Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: fir-OST0006: Will be in recovery for at least 2:30, or until 556 clients reconnect Feb 08 11:25:22 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a8fc24fc-68d9-a5c4-e6f4-94cbb871c8dc (at 10.9.114.8@o2ib4) Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: Skipped 228 previous similar messages Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: fir-OST0006: Denying connection for new client 96fb107e-6354-4a71-2925-e1f8a9a58d15(at 10.9.103.21@o2ib4), waiting for 556 known clients (49 recovered, 19 in progress, and 0 evicted) already passed deadline 2:31 Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: fir-OST0004: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900 Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: fir-OST0004: in recovery but waiting for the first client to connect Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: fir-OST0004: Will be in recovery for at least 2:30, or until 533 clients reconnect Feb 08 11:25:23 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 11:25:25 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to bd4172ee-b21a-171f-590d-07ab0a2614bb (at 10.8.12.25@o2ib6) Feb 08 11:25:25 fir-io1-s1 kernel: Lustre: Skipped 906 previous similar messages Feb 08 11:25:25 fir-io1-s1 kernel: Lustre: fir-OST0006: Denying connection for new client bde58792-3602-962e-df58-c34b9dbd9136(at 10.9.101.64@o2ib4), waiting for 556 known clients (186 recovered, 62 in progress, and 0 evicted) already passed deadline 2:33 Feb 08 11:25:25 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Feb 08 11:25:29 fir-io1-s1 kernel: md: md10: resync done. Feb 08 11:25:29 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6f29d7ca-d9bc-eef1-1913-bbb7c0bca1a0 (at 10.8.1.5@o2ib6) Feb 08 11:25:29 fir-io1-s1 kernel: Lustre: Skipped 790 previous similar messages Feb 08 11:25:29 fir-io1-s1 kernel: Lustre: fir-OST000a: Denying connection for new client ba7fdf11-52aa-7842-bf5d-1305febabf57(at 10.8.1.11@o2ib6), waiting for 524 known clients (281 recovered, 74 in progress, and 0 evicted) already passed deadline 2:37 Feb 08 11:25:29 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 08 11:25:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to d457dd0b-bcd3-cc31-d257-0e8a7a8274a4 (at 10.8.21.10@o2ib6) Feb 08 11:25:37 fir-io1-s1 kernel: Lustre: Skipped 845 previous similar messages Feb 08 11:25:37 fir-io1-s1 kernel: Lustre: fir-OST0006: Denying connection for new client f74f6780-500b-a99c-769a-05932d2be074(at 10.9.102.10@o2ib4), waiting for 556 known clients (381 recovered, 102 in progress, and 0 evicted) already passed deadline 2:45 Feb 08 11:25:37 fir-io1-s1 kernel: Lustre: Skipped 25 previous similar messages Feb 08 11:25:53 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to dfe698bc-81af-2a92-fd5f-f1174b08b3ad (at 10.9.107.7@o2ib4) Feb 08 11:25:53 fir-io1-s1 kernel: Lustre: Skipped 334 previous similar messages Feb 08 11:25:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Denying connection for new client 801024c6-19a5-7a6f-eec8-46f11d47fa3f(at 10.9.105.19@o2ib4), waiting for 548 known clients (427 recovered, 115 in progress, and 0 evicted) already passed deadline 3:02 Feb 08 11:25:53 fir-io1-s1 kernel: Lustre: Skipped 38 previous similar messages Feb 08 11:25:54 fir-io1-s1 kernel: md: md4: resync done. Feb 08 11:26:13 fir-io1-s1 kernel: md: md2: resync done. Feb 08 11:26:25 fir-io1-s1 kernel: Lustre: fir-OST0006: Denying connection for new client bde58792-3602-962e-df58-c34b9dbd9136(at 10.9.101.64@o2ib4), waiting for 556 known clients (428 recovered, 123 in progress, and 0 evicted) already passed deadline 3:33 Feb 08 11:26:25 fir-io1-s1 kernel: Lustre: Skipped 150 previous similar messages Feb 08 11:27:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Denying connection for new client dffc9b35-6604-58e1-2def-b2b41c77d1e7(at 10.9.101.44@o2ib4), waiting for 555 known clients (434 recovered, 118 in progress, and 0 evicted) already passed deadline 4:37 Feb 08 11:27:29 fir-io1-s1 kernel: Lustre: Skipped 274 previous similar messages Feb 08 11:27:51 fir-io1-s1 kernel: Lustre: fir-OST0000: recovery is timed out, evict stale exports Feb 08 11:27:51 fir-io1-s1 kernel: Lustre: fir-OST0000: disconnecting 2 stale clients Feb 08 11:27:52 fir-io1-s1 kernel: LustreError: 168-f: fir-OST0000: BAD WRITE CHECKSUM: from 12345-10.8.6.22@o2ib6 via 10.0.10.203@o2ib7 inode [0x2c000164b:0xea36:0x0] object 0x6c0000400:526318 extent [269819904-272842751]: client csum 6d0b976, server csum 665e94ba Feb 08 11:27:52 fir-io1-s1 kernel: Lustre: fir-OST000a: recovery is timed out, evict stale exports Feb 08 11:27:52 fir-io1-s1 kernel: Lustre: fir-OST000a: disconnecting 3 stale clients Feb 08 11:27:52 fir-io1-s1 kernel: LustreError: 168-f: fir-OST000a: BAD WRITE CHECKSUM: from 12345-10.9.104.46@o2ib4 via 10.0.10.212@o2ib7 inode [0x2c00016fc:0x5d:0x0] object 0x580000400:512960 extent [268435456-272629759]: client csum a8e7fafd, server csum 7932fd8f Feb 08 11:27:52 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Recovery over after 2:31, of 548 clients 546 recovered and 2 were evicted. Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:180420 to 0x6c0000402:180449 Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:528144 to 0x6c0000400:528321 Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:164455 to 0x6c0000401:164481 Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:704410 to 0x0:704449 Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST000a: Recovery already passed deadline 4:00, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 7d64396e-7a86-2e01-38b5-8f4fd2cfeb04 (at 10.8.19.4@o2ib6) Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: Skipped 36 previous similar messages Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:528571 to 0x580000400:528673 Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:164809 to 0x580000401:164833 Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:181413 to 0x580000402:181441 Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:704154 to 0x0:704193 Feb 08 11:27:53 fir-io1-s1 kernel: LustreError: 168-f: fir-OST0002: BAD WRITE CHECKSUM: from 12345-10.9.104.46@o2ib4 via 10.0.10.211@o2ib7 inode [0x2c0003bc0:0x6:0x0] object 0x5c0000400:512702 extent [10981376-292892671]: client csum 8c7122b6, server csum 80114b3 Feb 08 11:27:53 fir-io1-s1 kernel: LustreError: Skipped 7 previous similar messages Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST0004: recovery is timed out, evict stale exports Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: fir-OST0004: disconnecting 3 stale clients Feb 08 11:27:53 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Recovery already passed deadline 4:01, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:527896 to 0x8c0000402:527969 Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:180294 to 0x8c0000401:180321 Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Recovery over after 2:31, of 533 clients 530 recovered and 3 were evicted. Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:704289 to 0x0:704321 Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:164457 to 0x8c0000400:164481 Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:528307 to 0x5c0000400:528321 Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:181566 to 0x5c0000402:181601 Feb 08 11:27:54 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:704151 to 0x0:704193 Feb 08 11:27:56 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:164773 to 0x5c0000401:164801 Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Recovery already passed deadline 4:05, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:528028 to 0xc40000402:528065 Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:704915 to 0x0:704929 Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:180262 to 0xc40000401:180289 Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:164474 to 0xc40000400:164513 Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Recovery over after 2:36, of 556 clients 551 recovered and 5 were evicted. Feb 08 11:27:58 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 11:28:05 fir-io1-s1 kernel: Lustre: fir-OST0008: recovery is timed out, evict stale exports Feb 08 11:28:05 fir-io1-s1 kernel: Lustre: fir-OST0008: disconnecting 2 stale clients Feb 08 11:28:05 fir-io1-s1 kernel: LustreError: 168-f: fir-OST0008: BAD WRITE CHECKSUM: from 12345-10.8.2.17@o2ib6 via 10.0.10.203@o2ib7 inode [0x2c0003bc9:0x3:0x0] object 0xc80000402:526349 extent [348127232-352321535]: client csum fbeab40f, server csum 2e8098bb Feb 08 11:28:05 fir-io1-s1 kernel: LustreError: Skipped 7 previous similar messages Feb 08 11:28:07 fir-io1-s1 kernel: Lustre: fir-OST0008: Recovery already passed deadline 4:02, It is most likely due to DNE recovery is failed or stuck, please wait a few more minutes or abort the recovery. Feb 08 11:28:07 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:704944 to 0x0:704961 Feb 08 11:28:07 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:528189 to 0xc80000402:528225 Feb 08 11:28:07 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:180299 to 0xc80000401:180321 Feb 08 11:28:07 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:164568 to 0xc80000400:164577 Feb 08 11:28:07 fir-io1-s1 kernel: Lustre: fir-OST0008: Recovery over after 2:44, of 536 clients 534 recovered and 2 were evicted. Feb 08 11:28:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 804b7596-eb7f-ca8b-a586-917ca597cb67 (at 10.8.3.21@o2ib6) Feb 08 11:28:58 fir-io1-s1 kernel: Lustre: Skipped 1104 previous similar messages Feb 08 11:31:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 40a4c4f2-a940-462d-8df0-96fdeeb554f6 (at 10.9.101.33@o2ib4) Feb 08 11:31:15 fir-io1-s1 kernel: Lustre: Skipped 237 previous similar messages Feb 08 11:35:36 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9633991e-ce4f-d92c-b6aa-ec983a0f2b80 (at 10.8.8.23@o2ib6) Feb 08 11:35:36 fir-io1-s1 kernel: Lustre: Skipped 343 previous similar messages Feb 08 11:44:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to cc4ba4bd-a4c2-aaab-d4c9-cbd3275e0887 (at 10.8.20.13@o2ib6) Feb 08 11:44:08 fir-io1-s1 kernel: Lustre: Skipped 905 previous similar messages Feb 08 11:51:47 fir-io1-s1 kernel: Lustre: 94238:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655496/real 1549655496] req@ffff98576ed57800 x1624929852313648/t0(0) o106->fir-OST0004@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655507 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 08 11:51:47 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655496/real 1549655496] req@ffff985761d34b00 x1624929852313632/t0(0) o106->fir-OST0000@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655507 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 08 11:51:47 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 08 11:51:58 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655507/real 1549655507] req@ffff985761d34b00 x1624929852313632/t0(0) o106->fir-OST0000@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655518 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:51:58 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 08 11:52:09 fir-io1-s1 kernel: Lustre: 96785:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655518/real 1549655518] req@ffff98385abd7b00 x1624929852313680/t0(0) o106->fir-OST0008@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655529 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:52:09 fir-io1-s1 kernel: Lustre: 96785:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 08 11:52:20 fir-io1-s1 kernel: Lustre: 96770:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655529/real 1549655529] req@ffff984835846600 x1624929852313664/t0(0) o106->fir-OST0006@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655540 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:52:20 fir-io1-s1 kernel: Lustre: 96770:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 08 11:52:31 fir-io1-s1 kernel: Lustre: 96770:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655540/real 1549655540] req@ffff984835846600 x1624929852313664/t0(0) o106->fir-OST0006@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655551 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:52:31 fir-io1-s1 kernel: Lustre: 96770:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 08 11:52:42 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655551/real 1549655551] req@ffff985761d34b00 x1624929852313632/t0(0) o106->fir-OST0000@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655562 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:52:42 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 08 11:53:04 fir-io1-s1 kernel: Lustre: 96785:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655573/real 1549655573] req@ffff98385abd7b00 x1624929852313680/t0(0) o106->fir-OST0008@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655584 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:53:04 fir-io1-s1 kernel: Lustre: 96785:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 08 11:53:37 fir-io1-s1 kernel: Lustre: 94238:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655606/real 1549655606] req@ffff98576ed57800 x1624929852313648/t0(0) o106->fir-OST0004@10.8.10.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655617 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:53:37 fir-io1-s1 kernel: Lustre: 94238:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 08 11:54:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 73392b4c-c1a6-8e85-a092-5760cb397f19 (at 10.8.18.30@o2ib6) Feb 08 11:54:13 fir-io1-s1 kernel: Lustre: Skipped 1148 previous similar messages Feb 08 11:54:30 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c379ecdc-6a44-758c-7e7d-c6e0c5b472e3 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575de9f000, cur 1549655670 expire 1549655520 last 1549655443 Feb 08 11:54:44 fir-io1-s1 kernel: Lustre: 96377:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549655677/real 1549655677] req@ffff986780829e00 x1624929852530656/t0(0) o106->fir-OST0002@10.8.13.14@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549655684 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 11:54:44 fir-io1-s1 kernel: Lustre: 96377:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Feb 08 11:55:19 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client baf8fd69-3c06-48ef-44c1-991a98b1784d (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986782578400, cur 1549655719 expire 1549655569 last 1549655492 Feb 08 11:55:19 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 08 12:04:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d165ae17-8944-f365-7713-d429fbe1daab (at 10.8.1.23@o2ib6) Feb 08 12:04:13 fir-io1-s1 kernel: Lustre: Skipped 1103 previous similar messages Feb 08 12:14:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 75c103c5-8c22-70ed-cfb0-bd07e014990e (at 10.8.11.36@o2ib6) Feb 08 12:14:13 fir-io1-s1 kernel: Lustre: Skipped 1394 previous similar messages Feb 08 12:14:58 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 843044c5-b529-765e-1bbc-6729c498b3a5 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985838d89000, cur 1549656898 expire 1549656748 last 1549656671 Feb 08 12:14:58 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 12:21:18 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 647d49b6-6175-c5f2-9b13-ecaf32f220d3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834434c00, cur 1549657278 expire 1549657128 last 1549657051 Feb 08 12:21:18 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 12:24:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 91c3019e-2880-7f92-0b14-6eb0f2cbe1dd (at 10.9.101.65@o2ib4) Feb 08 12:24:17 fir-io1-s1 kernel: Lustre: Skipped 1350 previous similar messages Feb 08 12:31:29 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8bc0b59d-4176-29ca-8993-ccbed2d8cd47 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98783b87ac00, cur 1549657889 expire 1549657739 last 1549657662 Feb 08 12:31:29 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 12:32:25 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 43706dc7-f5fd-aac3-6e24-87e00964e876 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867832d1400, cur 1549657945 expire 1549657795 last 1549657718 Feb 08 12:32:25 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 08 12:34:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 1e63f594-4caa-53ef-0c47-c05ca8852eb7 (at 10.8.13.25@o2ib6) Feb 08 12:34:23 fir-io1-s1 kernel: Lustre: Skipped 926 previous similar messages Feb 08 12:36:42 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b846950b-5bea-339d-e553-c5a9af56a183 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762a5d400, cur 1549658202 expire 1549658052 last 1549657975 Feb 08 12:36:42 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 12:44:24 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ce2a5a1a-545c-760b-44a7-8c19aadb7a36 (at 10.9.107.71@o2ib4) Feb 08 12:44:24 fir-io1-s1 kernel: Lustre: Skipped 475 previous similar messages Feb 08 12:46:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5d8ca009-b48c-98af-0dc9-d8b96f873f78 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a285c00, cur 1549658814 expire 1549658664 last 1549658587 Feb 08 12:46:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 12:48:10 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2bc95971-57af-d09b-f861-ded15d800fe7 (at 10.8.18.35@o2ib6) in 185 seconds. I think it's dead, and I am evicting it. exp ffff986785d7bc00, cur 1549658890 expire 1549658740 last 1549658705 Feb 08 12:48:10 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 12:48:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2bc95971-57af-d09b-f861-ded15d800fe7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767576000, cur 1549658932 expire 1549658782 last 1549658705 Feb 08 12:50:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 438aa5ac-b1bd-ac26-4f1d-9c6d84ed5c1c (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bd77c00, cur 1549659057 expire 1549658907 last 1549658830 Feb 08 12:50:57 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 12:53:21 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c9b98366-8eaa-5511-9f67-d357f3125295 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1551000, cur 1549659201 expire 1549659051 last 1549658974 Feb 08 12:53:21 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 08 12:54:31 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 795ed85c-3d14-47f4-c094-a6509583ab56 (at 10.8.11.24@o2ib6) Feb 08 12:54:31 fir-io1-s1 kernel: Lustre: Skipped 667 previous similar messages Feb 08 12:58:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c862f72b-deb0-77c5-1dfa-a5f9f53b6901 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575cd50400, cur 1549659502 expire 1549659352 last 1549659275 Feb 08 12:58:22 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 13:04:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fd01c7fd-7bec-8a77-ad04-40f1cfa2b200 (at 10.8.25.10@o2ib6) Feb 08 13:04:31 fir-io1-s1 kernel: Lustre: Skipped 477 previous similar messages Feb 08 13:10:34 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4a38089b-7869-6534-2324-e00dc2c27342 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a6a400, cur 1549660234 expire 1549660084 last 1549660007 Feb 08 13:10:34 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 08 13:14:36 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 08 13:14:36 fir-io1-s1 kernel: Lustre: Skipped 680 previous similar messages Feb 08 13:21:17 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5c7ff6bb-6ea2-5c1f-a993-db1bea2df0d0 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847ffc3c000, cur 1549660877 expire 1549660727 last 1549660650 Feb 08 13:21:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 13:24:37 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9334fe18-06e4-7654-e3bd-8d4e4ade184d (at 10.8.8.8@o2ib6) Feb 08 13:24:37 fir-io1-s1 kernel: Lustre: Skipped 745 previous similar messages Feb 08 13:34:39 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 028d6433-9e7d-1b84-c8b7-1bb2a8570ec4 (at 10.8.1.4@o2ib6) Feb 08 13:34:39 fir-io1-s1 kernel: Lustre: Skipped 1059 previous similar messages Feb 08 13:43:14 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a6007f90-c23e-97b9-ab81-12d11027971d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483548dc00, cur 1549662194 expire 1549662044 last 1549661967 Feb 08 13:43:14 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 08 13:44:39 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 793135a4-9596-5826-6c94-8f13559030ac (at 10.8.25.6@o2ib6) Feb 08 13:44:39 fir-io1-s1 kernel: Lustre: Skipped 1125 previous similar messages Feb 08 13:54:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1815ed5c-ff36-8b6d-f1d6-dad784199dec (at 10.9.107.72@o2ib4) Feb 08 13:54:39 fir-io1-s1 kernel: Lustre: Skipped 1209 previous similar messages Feb 08 14:04:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ffcc7510-f875-7549-61d4-9f6248a33eef (at 10.9.105.7@o2ib4) Feb 08 14:04:41 fir-io1-s1 kernel: Lustre: Skipped 951 previous similar messages Feb 08 14:06:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e8e6de7b-513a-3c56-a9e0-d3dc547c4d21 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c6d9400, cur 1549663595 expire 1549663445 last 1549663368 Feb 08 14:06:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 14:09:58 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ffe9b055-91d6-08c3-4a31-409cd95c94d0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838aec00, cur 1549663798 expire 1549663648 last 1549663571 Feb 08 14:09:58 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 14:14:41 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1ba5aef6-2be1-c2db-6ba0-6e3a31e32627 (at 10.9.107.41@o2ib4) Feb 08 14:14:41 fir-io1-s1 kernel: Lustre: Skipped 954 previous similar messages Feb 08 14:22:04 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 66c8983d-6fa8-e6fd-bb13-0d30be6003a5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780863c00, cur 1549664524 expire 1549664374 last 1549664297 Feb 08 14:22:04 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 08 14:24:43 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 08 14:24:43 fir-io1-s1 kernel: Lustre: Skipped 1032 previous similar messages Feb 08 14:34:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b41950de-0614-6c1a-0d53-d43c60fe0f33 (at 10.9.102.1@o2ib4) Feb 08 14:34:43 fir-io1-s1 kernel: Lustre: Skipped 642 previous similar messages Feb 08 14:44:45 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.8.2.22@o2ib6) Feb 08 14:44:45 fir-io1-s1 kernel: Lustre: Skipped 786 previous similar messages Feb 08 14:54:47 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 031d42d0-894f-b37f-ebbd-45aa637dedc9 (at 10.9.107.33@o2ib4) Feb 08 14:54:47 fir-io1-s1 kernel: Lustre: Skipped 745 previous similar messages Feb 08 15:04:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9d602dc8-8051-8982-51eb-1bb4250b93cd (at 10.8.22.18@o2ib6) Feb 08 15:04:52 fir-io1-s1 kernel: Lustre: Skipped 594 previous similar messages Feb 08 15:14:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.106.66@o2ib4) Feb 08 15:14:53 fir-io1-s1 kernel: Lustre: Skipped 456 previous similar messages Feb 08 15:16:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5bfe4986-235b-4715-c985-36823d3f937e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c85b400, cur 1549667778 expire 1549667628 last 1549667551 Feb 08 15:16:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 15:24:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6f43d80a-d4ed-b27e-b29c-0d22cee6d831 (at 10.9.105.8@o2ib4) Feb 08 15:24:55 fir-io1-s1 kernel: Lustre: Skipped 524 previous similar messages Feb 08 15:34:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4e48592f-b97d-5c93-9da4-86c872d7a486 (at 10.9.107.43@o2ib4) Feb 08 15:34:58 fir-io1-s1 kernel: Lustre: Skipped 980 previous similar messages Feb 08 15:36:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fc0522a2-e86f-7812-92f9-18c8c5b33bdc (at 10.9.105.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987832657c00, cur 1549668974 expire 1549668824 last 1549668747 Feb 08 15:36:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 15:36:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client fc0522a2-e86f-7812-92f9-18c8c5b33bdc (at 10.9.105.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98783c2e3400, cur 1549668984 expire 1549668834 last 1549668757 Feb 08 15:36:24 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 15:42:07 fir-io1-s1 kernel: LustreError: 96902:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.105.45@o2ib4) returned error from glimpse AST (req@ffff986783463000 x1624930128323312 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff985e92d95e80/0x49e185e91de66af5 lrc: 3/0,0 mode: PW/PW res: [0xc80000402:0x87705:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000000000 nid: 10.9.105.45@o2ib4 remote: 0xa567adbc8f1c942a expref: 5 pid: 96409 timeout: 0 lvb_type: 0 Feb 08 15:42:07 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.9.105.45@o2ib4 was evicted due to a lock glimpse callback time out: rc -107 Feb 08 15:42:07 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1549669327s: evicting client at 10.9.105.45@o2ib4 ns: filter-fir-OST000a_UUID lock: ffff986225972ac0/0x49e185e91de66ea6 lrc: 3/0,0 mode: PW/PW res: [0x580000400:0x878ce:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000000000 nid: 10.9.105.45@o2ib4 remote: 0xa567adbc8f1c9462 expref: 6 pid: 96371 timeout: 0 lvb_type: 0 Feb 08 15:42:07 fir-io1-s1 kernel: LustreError: 96902:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Feb 08 15:42:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b26341ef-ca9d-2919-f620-8a2863f6486b (at 10.8.11.22@o2ib6) in 158 seconds. I think it's dead, and I am evicting it. exp ffff986785cdd000, cur 1549669353 expire 1549669203 last 1549669195 Feb 08 15:42:33 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 15:42:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b26341ef-ca9d-2919-f620-8a2863f6486b (at 10.8.11.22@o2ib6) in 164 seconds. I think it's dead, and I am evicting it. exp ffff986785cddc00, cur 1549669359 expire 1549669209 last 1549669195 Feb 08 15:42:39 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 15:42:40 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549669353/real 1549669353] req@ffff9857640eda00 x1624930128602896/t0(0) o106->fir-OST0004@10.8.11.22@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549669360 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 08 15:42:40 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 08 15:43:01 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549669374/real 1549669374] req@ffff9857640eda00 x1624930128602896/t0(0) o106->fir-OST0004@10.8.11.22@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549669381 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 15:43:01 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 08 15:43:36 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549669409/real 1549669409] req@ffff9857640eda00 x1624930128602896/t0(0) o106->fir-OST0004@10.8.11.22@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549669416 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 08 15:43:36 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 08 15:43:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b26341ef-ca9d-2919-f620-8a2863f6486b (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cda400, cur 1549669422 expire 1549669272 last 1549669195 Feb 08 15:44:58 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d57b9eef-8689-84fb-167c-d5dc115141b7 (at 10.8.12.20@o2ib6) Feb 08 15:44:58 fir-io1-s1 kernel: Lustre: Skipped 685 previous similar messages Feb 08 15:54:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 038633ee-8c4c-96e7-df9b-09761ff242b9 (at 10.9.105.9@o2ib4) Feb 08 15:54:58 fir-io1-s1 kernel: Lustre: Skipped 430 previous similar messages Feb 08 16:05:04 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 031d42d0-894f-b37f-ebbd-45aa637dedc9 (at 10.9.107.33@o2ib4) Feb 08 16:05:04 fir-io1-s1 kernel: Lustre: Skipped 650 previous similar messages Feb 08 16:06:03 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ce5b7e91-2a31-7810-bb87-09a986016b40 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867850b5400, cur 1549670763 expire 1549670613 last 1549670536 Feb 08 16:14:35 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ee23baab-77d3-3930-74e9-c93f730e142b (at 10.8.21.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8b800, cur 1549671275 expire 1549671125 last 1549671048 Feb 08 16:14:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 16:15:11 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.29.2@o2ib6) Feb 08 16:15:11 fir-io1-s1 kernel: Lustre: Skipped 919 previous similar messages Feb 08 16:15:51 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 967be65a-99c0-69bf-ca26-5acec3add8db (at 10.8.11.22@o2ib6) in 176 seconds. I think it's dead, and I am evicting it. exp ffff98582bba6400, cur 1549671351 expire 1549671201 last 1549671175 Feb 08 16:15:51 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 16:16:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 967be65a-99c0-69bf-ca26-5acec3add8db (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f232000, cur 1549671402 expire 1549671252 last 1549671175 Feb 08 16:25:12 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 40113a8d-c41f-508e-9772-7563fba01286 (at 10.9.105.5@o2ib4) Feb 08 16:25:12 fir-io1-s1 kernel: Lustre: Skipped 617 previous similar messages Feb 08 16:27:11 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client aaf017e1-98c3-f97d-d4ea-425a6657968e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd263800, cur 1549672031 expire 1549671881 last 1549671804 Feb 08 16:27:11 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 08 16:33:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4db2fe6f-16f9-5e46-9f88-2d7ba2ebffbf (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885b400, cur 1549672401 expire 1549672251 last 1549672174 Feb 08 16:33:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 08 16:34:37 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2439cf13-2531-025e-0a67-2293ace7641f (at 10.8.18.35@o2ib6) in 225 seconds. I think it's dead, and I am evicting it. exp ffff98575d205000, cur 1549672477 expire 1549672327 last 1549672252 Feb 08 16:34:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 16:35:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 74f9852d-189b-8596-eb4f-bcf617e42f7c (at 10.8.7.22@o2ib6) Feb 08 16:35:13 fir-io1-s1 kernel: Lustre: Skipped 803 previous similar messages Feb 08 16:38:43 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 72d2301c-ddfd-4c1e-eff8-bf018aa1c0b1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835770c00, cur 1549672723 expire 1549672573 last 1549672496 Feb 08 16:38:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 16:42:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 16175f50-51b4-8177-7064-daced4c417ce (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987833e97c00, cur 1549672968 expire 1549672818 last 1549672741 Feb 08 16:42:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 16:45:14 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 33b6d949-dd04-d411-f203-1edd995e9a2f (at 10.8.13.24@o2ib6) Feb 08 16:45:14 fir-io1-s1 kernel: Lustre: Skipped 753 previous similar messages Feb 08 16:55:15 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 83e72c3d-872c-7a5f-f5c1-edf566d41d60 (at 10.9.107.1@o2ib4) Feb 08 16:55:15 fir-io1-s1 kernel: Lustre: Skipped 890 previous similar messages Feb 08 17:02:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7afff036-3532-79c1-88f9-663a849526ec (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f7fc00, cur 1549674134 expire 1549673984 last 1549673907 Feb 08 17:02:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 08 17:05:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2daf93cd-09cc-690c-e30a-05a92ce8115a (at 10.9.104.51@o2ib4) Feb 08 17:05:15 fir-io1-s1 kernel: Lustre: Skipped 854 previous similar messages Feb 08 17:10:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d3dbb03f-5efb-d900-afad-276997d212e1 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b1dec00, cur 1549674652 expire 1549674502 last 1549674425 Feb 08 17:10:52 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 17:15:16 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c685ce6c-10e4-7444-bf47-f6501a7232f0 (at 10.8.8.20@o2ib6) Feb 08 17:15:16 fir-io1-s1 kernel: Lustre: Skipped 825 previous similar messages Feb 08 17:17:27 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 662a3d82-8849-9bb2-4128-cd5192814a19 (at 10.8.21.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d203c00, cur 1549675047 expire 1549674897 last 1549674820 Feb 08 17:17:27 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 17:21:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9da106d7-0528-d74a-015e-4ade8f0421c0 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f233c00, cur 1549675311 expire 1549675161 last 1549675084 Feb 08 17:21:51 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 17:25:18 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to aead70f5-f5fc-a972-7a47-29ab30efeef2 (at 10.9.104.23@o2ib4) Feb 08 17:25:18 fir-io1-s1 kernel: Lustre: Skipped 801 previous similar messages Feb 08 17:32:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5f22821b-0472-716d-d871-bb9488893a1e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbffc00, cur 1549675921 expire 1549675771 last 1549675694 Feb 08 17:32:01 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 17:35:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to dbe81c3f-e038-b02e-a6dc-aaba56293b77 (at 10.8.2.19@o2ib6) Feb 08 17:35:33 fir-io1-s1 kernel: Lustre: Skipped 703 previous similar messages Feb 08 17:39:56 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d4425bba-c123-bb1d-3b64-f16a4c422ca0 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848372d7800, cur 1549676396 expire 1549676246 last 1549676169 Feb 08 17:39:56 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 17:45:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8402838d-f75d-af53-0850-df7f3dd913ea (at 10.9.104.58@o2ib4) Feb 08 17:45:34 fir-io1-s1 kernel: Lustre: Skipped 505 previous similar messages Feb 08 17:48:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b3d1419f-591a-7854-124c-81b9eccbde9b (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786948000, cur 1549676906 expire 1549676756 last 1549676679 Feb 08 17:48:26 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 08 17:55:42 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 08 17:55:42 fir-io1-s1 kernel: Lustre: Skipped 552 previous similar messages Feb 08 18:05:42 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8427e8d1-34c3-886c-dab9-5ccc235f41c4 (at 10.9.107.5@o2ib4) Feb 08 18:05:42 fir-io1-s1 kernel: Lustre: Skipped 816 previous similar messages Feb 08 18:15:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client eae463d3-9a7b-356d-af50-533d58b36c74 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c1a400, cur 1549678507 expire 1549678357 last 1549678280 Feb 08 18:15:07 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 08 18:15:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) Feb 08 18:15:42 fir-io1-s1 kernel: Lustre: Skipped 1011 previous similar messages Feb 08 18:25:43 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 59806ee4-869d-859c-ce86-5b2a7b506ab7 (at 10.9.107.26@o2ib4) Feb 08 18:25:43 fir-io1-s1 kernel: Lustre: Skipped 1071 previous similar messages Feb 08 18:35:45 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 031d42d0-894f-b37f-ebbd-45aa637dedc9 (at 10.9.107.33@o2ib4) Feb 08 18:35:45 fir-io1-s1 kernel: Lustre: Skipped 1033 previous similar messages Feb 08 18:40:34 fir-io1-s1 kernel: sd 0:0:173:0: device_block, handle(0x00bd) Feb 08 18:40:34 fir-io1-s1 kernel: sd 0:0:173:0: [sdfo] tag#11 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK Feb 08 18:40:34 fir-io1-s1 kernel: sd 0:0:173:0: [sdfo] tag#11 CDB: Test Unit Ready 00 00 00 00 00 00 Feb 08 18:40:34 fir-io1-s1 kernel: sd 0:0:234:0: [sdhw] tag#12 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK Feb 08 18:40:34 fir-io1-s1 kernel: sd 0:0:234:0: [sdhw] tag#12 CDB: Test Unit Ready 00 00 00 00 00 00 Feb 08 18:40:34 fir-io1-s1 kernel: sd 0:0:234:0: device_block, handle(0x0102) Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:173:0: device_unblock and setting to running, handle(0x00bd) Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:234:0: device_unblock and setting to running, handle(0x0102) Feb 08 18:40:37 fir-io1-s1 kernel: mpt3sas_cm0: log_info(0x31130000): originator(PL), code(0x13), sub_code(0x0000) Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:173:0: rejecting I/O to offline device Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:173:0: [sdfo] killing request Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:173:0: [sdfo] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:173:0: [sdfo] CDB: Test Unit Ready 00 00 00 00 00 00 Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:234:0: [sdhw] tag#11 FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:234:0: [sdhw] tag#11 CDB: Test Unit Ready 00 00 00 00 00 00 Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:173:0: [sdfo] Synchronizing SCSI cache Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:173:0: [sdfo] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:234:0: rejecting I/O to offline device Feb 08 18:40:37 fir-io1-s1 kernel: device-mapper: multipath: Failing path 134:96. Feb 08 18:40:37 fir-io1-s1 kernel: blk_update_request: I/O error, dev dm-119, sector 15628052992 Feb 08 18:40:37 fir-io1-s1 kernel: blk_update_request: I/O error, dev dm-119, sector 15628052992 Feb 08 18:40:37 fir-io1-s1 kernel: Buffer I/O error on dev dm-119, logical block 1953506624, async page read Feb 08 18:40:37 fir-io1-s1 kernel: mpt3sas_cm0: removing handle(0x00bd), sas_addr(0x5000c50095383876) Feb 08 18:40:37 fir-io1-s1 kernel: mpt3sas_cm0: enclosure logical id(0x50016360020accbd), slot(50) Feb 08 18:40:37 fir-io1-s1 kernel: mpt3sas_cm0: enclosure level(0x0000),connector name( C3 ) Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:234:0: [sdhw] Synchronizing SCSI cache Feb 08 18:40:37 fir-io1-s1 kernel: sd 0:0:234:0: [sdhw] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK Feb 08 18:40:37 fir-io1-s1 kernel: mpt3sas_cm0: removing handle(0x0102), sas_addr(0x5000c50095383875) Feb 08 18:40:37 fir-io1-s1 kernel: mpt3sas_cm0: enclosure logical id(0x5001636001a07b7d), slot(50) Feb 08 18:40:37 fir-io1-s1 kernel: mpt3sas_cm0: enclosure level(0x0000),connector name( C2 ) Feb 08 18:42:25 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:25 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:25 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:25 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:25 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:25 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:26 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:27 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:27 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:27 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:27 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:27 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:27 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:28 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:28 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:28 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:28 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:29 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:30 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:31 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:32 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:33 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:34 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:35 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:36 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:37 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:38 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:39 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:40 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:40 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:40 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:40 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:40 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:41 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:41 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:41 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:41 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:41 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:42 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:43 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:44 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:44 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:44 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:44 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:45 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:45 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:45 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:46 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:47 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:48 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:48 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:48 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:48 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:48 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010f), lun(0) Feb 08 18:42:49 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:49 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:49 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:49 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:49 fir-io1-s1 kernel: mpt3sas_cm0: START_UNIT: handle(0x010e), lun(0) Feb 08 18:42:49 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010f), sas_address(0x5000c5008dc935cd), phy(50) Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010f), retries(0) Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010f), lun(0) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:245:0: Direct-Access SEAGATE ST8000NM0075 E004 PQ: 0 ANSI: 6 Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:245:0: SSP: handle(0x010f), sas_addr(0x5000c5008dc935cd), phy(50), device_name(0x5000c5008dc935cc) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:245:0: enclosure logical id(0x5001636001a07b7d), slot(50) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:245:0: enclosure level(0x0000), connector name( C2 ) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:245:0: serial_number(ZA151EN90000R649TWEG) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:245:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:245:0: Attached scsi generic sg173 type 0 Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:245:0: [sdfo] 15628053168 512-byte logical blocks: (8.00 TB/7.27 TiB) Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:245:0: [sdfo] 4096-byte physical blocks Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: detecting: handle(0x010e), sas_address(0x5000c5008dc935ce), phy(50) Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: REPORT_LUNS: handle(0x010e), retries(0) Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:245:0: [sdfo] Write Protect is off Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:245:0: [sdfo] Mode Sense: db 00 10 08 Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:245:0: [sdfo] Write cache: enabled, read cache: enabled, supports DPO and FUA Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: TEST_UNIT_READY: handle(0x010e), lun(0) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:246:0: Direct-Access SEAGATE ST8000NM0075 E004 PQ: 0 ANSI: 6 Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:246:0: SSP: handle(0x010e), sas_addr(0x5000c5008dc935ce), phy(50), device_name(0x5000c5008dc935cc) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:246:0: enclosure logical id(0x50016360020accbd), slot(50) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:246:0: enclosure level(0x0000), connector name( C3 ) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:246:0: serial_number(ZA151EN90000R649TWEG) Feb 08 18:42:50 fir-io1-s1 kernel: scsi 0:0:246:0: qdepth(254), tagged(1), simple(0), ordered(0), scsi_level(7), cmd_que(1) Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:246:0: Attached scsi generic sg234 type 0 Feb 08 18:42:50 fir-io1-s1 kernel: mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:246:0: [sdhw] 15628053168 512-byte logical blocks: (8.00 TB/7.27 TiB) Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:245:0: [sdfo] Attached SCSI disk Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:246:0: [sdhw] 4096-byte physical blocks Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:246:0: [sdhw] Write Protect is off Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:246:0: [sdhw] Mode Sense: db 00 10 08 Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:246:0: [sdhw] Write cache: enabled, read cache: enabled, supports DPO and FUA Feb 08 18:42:50 fir-io1-s1 kernel: sd 0:0:246:0: [sdhw] Attached SCSI disk Feb 08 18:45:47 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ce2a5a1a-545c-760b-44a7-8c19aadb7a36 (at 10.9.107.71@o2ib4) Feb 08 18:45:47 fir-io1-s1 kernel: Lustre: Skipped 855 previous similar messages Feb 08 18:55:47 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c8e6faa5-8e52-627a-b276-9b1da9fb48ae (at 10.8.7.23@o2ib6) Feb 08 18:55:47 fir-io1-s1 kernel: Lustre: Skipped 846 previous similar messages Feb 08 19:05:50 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 59806ee4-869d-859c-ce86-5b2a7b506ab7 (at 10.9.107.26@o2ib4) Feb 08 19:05:50 fir-io1-s1 kernel: Lustre: Skipped 605 previous similar messages Feb 08 19:15:51 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 054c4743-619d-965f-f786-cc0afc52d348 (at 10.9.101.68@o2ib4) Feb 08 19:15:51 fir-io1-s1 kernel: Lustre: Skipped 583 previous similar messages Feb 08 19:25:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0a270a28-384b-10a4-3edd-bee22242e3ea (at 10.8.4.35@o2ib6) Feb 08 19:25:54 fir-io1-s1 kernel: Lustre: Skipped 528 previous similar messages Feb 08 19:35:54 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 04bbddc7-37a9-9b79-7fa8-c451901e5d15 (at 10.9.101.67@o2ib4) Feb 08 19:35:54 fir-io1-s1 kernel: Lustre: Skipped 592 previous similar messages Feb 08 19:45:58 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 7cba68d6-88e2-7226-0b4f-cb83c3107f8f (at 10.9.102.34@o2ib4) Feb 08 19:45:58 fir-io1-s1 kernel: Lustre: Skipped 846 previous similar messages Feb 08 19:56:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 08 19:56:05 fir-io1-s1 kernel: Lustre: Skipped 592 previous similar messages Feb 08 20:06:05 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c8e6faa5-8e52-627a-b276-9b1da9fb48ae (at 10.8.7.23@o2ib6) Feb 08 20:06:05 fir-io1-s1 kernel: Lustre: Skipped 790 previous similar messages Feb 08 20:16:06 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 2b64d06d-0a33-7dc1-7c60-2608607acb48 (at 10.9.104.50@o2ib4) Feb 08 20:16:06 fir-io1-s1 kernel: Lustre: Skipped 1027 previous similar messages Feb 08 20:26:06 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 5b322ed5-1f83-3d4a-541c-ca479ed4d108 (at 10.8.18.18@o2ib6) Feb 08 20:26:06 fir-io1-s1 kernel: Lustre: Skipped 1000 previous similar messages Feb 08 20:36:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 848db794-00ba-ef76-c052-86987458d4b8 (at 10.9.104.55@o2ib4) Feb 08 20:36:09 fir-io1-s1 kernel: Lustre: Skipped 958 previous similar messages Feb 08 20:46:10 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b5581c4c-47a8-b046-19f4-e572468b3106 (at 10.9.102.32@o2ib4) Feb 08 20:46:10 fir-io1-s1 kernel: Lustre: Skipped 955 previous similar messages Feb 08 20:56:12 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) Feb 08 20:56:12 fir-io1-s1 kernel: Lustre: Skipped 895 previous similar messages Feb 08 21:06:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3352dbfd-d63f-74b6-bc30-6f22efce7e27 (at 10.8.7.3@o2ib6) Feb 08 21:06:13 fir-io1-s1 kernel: Lustre: Skipped 899 previous similar messages Feb 08 21:16:19 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 2b64d06d-0a33-7dc1-7c60-2608607acb48 (at 10.9.104.50@o2ib4) Feb 08 21:16:19 fir-io1-s1 kernel: Lustre: Skipped 867 previous similar messages Feb 08 21:26:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f74f6780-500b-a99c-769a-05932d2be074 (at 10.9.102.10@o2ib4) Feb 08 21:26:19 fir-io1-s1 kernel: Lustre: Skipped 739 previous similar messages Feb 08 21:36:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 18be1149-584e-3095-4e7f-669b8a4c97d2 (at 10.8.29.6@o2ib6) Feb 08 21:36:22 fir-io1-s1 kernel: Lustre: Skipped 577 previous similar messages Feb 08 21:46:24 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 838e95ba-7d72-299c-b143-b6a5f04f9e78 (at 10.9.107.36@o2ib4) Feb 08 21:46:24 fir-io1-s1 kernel: Lustre: Skipped 919 previous similar messages Feb 08 21:56:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4386eb61-b821-6959-a669-9747602b9eba (at 10.9.107.20@o2ib4) Feb 08 21:56:26 fir-io1-s1 kernel: Lustre: Skipped 936 previous similar messages Feb 08 22:06:28 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8427e8d1-34c3-886c-dab9-5ccc235f41c4 (at 10.9.107.5@o2ib4) Feb 08 22:06:28 fir-io1-s1 kernel: Lustre: Skipped 725 previous similar messages Feb 08 22:16:31 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to edc5d3f2-dae7-69fe-e7fb-9c4cf59a4b4c (at 10.8.31.8@o2ib6) Feb 08 22:16:31 fir-io1-s1 kernel: Lustre: Skipped 792 previous similar messages Feb 08 22:26:32 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 59806ee4-869d-859c-ce86-5b2a7b506ab7 (at 10.9.107.26@o2ib4) Feb 08 22:26:32 fir-io1-s1 kernel: Lustre: Skipped 753 previous similar messages Feb 08 22:36:32 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 11bdb068-3a68-b486-d427-87b2a02899d5 (at 10.8.2.24@o2ib6) Feb 08 22:36:32 fir-io1-s1 kernel: Lustre: Skipped 825 previous similar messages Feb 08 22:46:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4376dd10-0b2f-912c-8c73-35fe5309aa36 (at 10.8.22.25@o2ib6) Feb 08 22:46:32 fir-io1-s1 kernel: Lustre: Skipped 690 previous similar messages Feb 08 22:56:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 08 22:56:39 fir-io1-s1 kernel: Lustre: Skipped 611 previous similar messages Feb 08 23:06:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 71eebf52-b0fc-514f-7aaf-aca66e4f2af1 (at 10.8.27.27@o2ib6) Feb 08 23:06:40 fir-io1-s1 kernel: Lustre: Skipped 458 previous similar messages Feb 08 23:16:44 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a0ecc820-bd9c-7838-4b1d-2594d114d886 (at 10.8.11.30@o2ib6) Feb 08 23:16:44 fir-io1-s1 kernel: Lustre: Skipped 586 previous similar messages Feb 08 23:26:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 85dd464d-e86b-f56a-846e-fdb51d5f1d4c (at 10.8.21.13@o2ib6) Feb 08 23:26:49 fir-io1-s1 kernel: Lustre: Skipped 831 previous similar messages Feb 08 23:36:49 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e78a174a-ae69-cb32-9b42-0306c7153992 (at 10.9.103.7@o2ib4) Feb 08 23:36:49 fir-io1-s1 kernel: Lustre: Skipped 702 previous similar messages Feb 08 23:46:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 83f7575b-b138-8f39-fa8f-a4cda38cceb7 (at 10.9.105.26@o2ib4) Feb 08 23:46:50 fir-io1-s1 kernel: Lustre: Skipped 701 previous similar messages Feb 08 23:56:50 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d14918a5-6565-e169-f50e-e38934e1ba9e (at 10.9.105.53@o2ib4) Feb 08 23:56:50 fir-io1-s1 kernel: Lustre: Skipped 768 previous similar messages Feb 09 00:06:53 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 09 00:06:53 fir-io1-s1 kernel: Lustre: Skipped 801 previous similar messages Feb 09 00:16:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b2364f4b-9129-81e8-7e2f-15aa4210b663 (at 10.9.107.12@o2ib4) Feb 09 00:16:56 fir-io1-s1 kernel: Lustre: Skipped 720 previous similar messages Feb 09 00:26:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 11bdb068-3a68-b486-d427-87b2a02899d5 (at 10.8.2.24@o2ib6) Feb 09 00:26:56 fir-io1-s1 kernel: Lustre: Skipped 656 previous similar messages Feb 09 00:36:38 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fb5e4f4f-a645-fbec-7b80-08c8d8a2fea0 (at 10.8.1.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986781f72400, cur 1549701398 expire 1549701248 last 1549701171 Feb 09 00:36:38 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 09 00:36:59 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to b9de1fd1-ccbe-721f-e4ab-c6e06447a81c (at 10.8.15.4@o2ib6) Feb 09 00:36:59 fir-io1-s1 kernel: Lustre: Skipped 847 previous similar messages Feb 09 00:47:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 74f9852d-189b-8596-eb4f-bcf617e42f7c (at 10.8.7.22@o2ib6) Feb 09 00:47:00 fir-io1-s1 kernel: Lustre: Skipped 624 previous similar messages Feb 09 00:57:00 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a908dcd0-9db1-9f70-5f22-f8c81c7a1077 (at 10.8.23.10@o2ib6) Feb 09 00:57:00 fir-io1-s1 kernel: Lustre: Skipped 709 previous similar messages Feb 09 01:07:01 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c8e6faa5-8e52-627a-b276-9b1da9fb48ae (at 10.8.7.23@o2ib6) Feb 09 01:07:01 fir-io1-s1 kernel: Lustre: Skipped 668 previous similar messages Feb 09 01:17:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b3dd093c-5019-d756-25ce-058811799d68 (at 10.9.102.46@o2ib4) Feb 09 01:17:03 fir-io1-s1 kernel: Lustre: Skipped 723 previous similar messages Feb 09 01:27:04 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 52c4baff-6963-9e99-57ca-87da8aa6d147 (at 10.8.10.13@o2ib6) Feb 09 01:27:04 fir-io1-s1 kernel: Lustre: Skipped 611 previous similar messages Feb 09 01:37:04 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c12a1684-dc00-3104-ef93-3cf20d893979 (at 10.9.106.8@o2ib4) Feb 09 01:37:04 fir-io1-s1 kernel: Lustre: Skipped 685 previous similar messages Feb 09 01:47:05 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to b2364f4b-9129-81e8-7e2f-15aa4210b663 (at 10.9.107.12@o2ib4) Feb 09 01:47:05 fir-io1-s1 kernel: Lustre: Skipped 719 previous similar messages Feb 09 01:57:07 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 59806ee4-869d-859c-ce86-5b2a7b506ab7 (at 10.9.107.26@o2ib4) Feb 09 01:57:07 fir-io1-s1 kernel: Lustre: Skipped 682 previous similar messages Feb 09 02:07:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1753156e-3e53-ffdc-53f8-4f5717fb63ea (at 10.9.102.27@o2ib4) Feb 09 02:07:07 fir-io1-s1 kernel: Lustre: Skipped 1160 previous similar messages Feb 09 02:17:07 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to e256f342-dd4a-0733-a0fa-e78c997bbd1d (at 10.8.23.27@o2ib6) Feb 09 02:17:07 fir-io1-s1 kernel: Lustre: Skipped 1008 previous similar messages Feb 09 02:18:33 fir-io1-s1 kernel: perf: interrupt took too long (2505 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Feb 09 02:27:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7d6292c2-dc0a-0082-5273-c1ff8e6163ed (at 10.9.102.25@o2ib4) Feb 09 02:27:07 fir-io1-s1 kernel: Lustre: Skipped 989 previous similar messages Feb 09 02:37:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.107.24@o2ib4) Feb 09 02:37:08 fir-io1-s1 kernel: Lustre: Skipped 1257 previous similar messages Feb 09 02:47:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c3afa774-0e39-3459-0af2-ddf264214c5b (at 10.9.104.54@o2ib4) Feb 09 02:47:09 fir-io1-s1 kernel: Lustre: Skipped 897 previous similar messages Feb 09 02:57:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.102.48@o2ib4) Feb 09 02:57:09 fir-io1-s1 kernel: Lustre: Skipped 1072 previous similar messages Feb 09 03:07:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 09 03:07:10 fir-io1-s1 kernel: Lustre: Skipped 931 previous similar messages Feb 09 03:17:10 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to d01707bf-d8db-a4c3-f544-1f9ecca8f036 (at 10.8.18.29@o2ib6) Feb 09 03:17:10 fir-io1-s1 kernel: Lustre: Skipped 369 previous similar messages Feb 09 03:27:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fdc26fb4-ee64-0351-e5c8-066e7d1f1cf3 (at 10.8.3.10@o2ib6) Feb 09 03:27:14 fir-io1-s1 kernel: Lustre: Skipped 270 previous similar messages Feb 09 03:37:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3352dbfd-d63f-74b6-bc30-6f22efce7e27 (at 10.8.7.3@o2ib6) Feb 09 03:37:23 fir-io1-s1 kernel: Lustre: Skipped 398 previous similar messages Feb 09 03:47:35 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 031d42d0-894f-b37f-ebbd-45aa637dedc9 (at 10.9.107.33@o2ib4) Feb 09 03:47:35 fir-io1-s1 kernel: Lustre: Skipped 409 previous similar messages Feb 09 03:57:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e4f08268-1c0f-53f2-ca47-f50f478ec6f9 (at 10.8.18.3@o2ib6) Feb 09 03:57:36 fir-io1-s1 kernel: Lustre: Skipped 428 previous similar messages Feb 09 04:07:37 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c91fab0b-68e8-83d1-48e6-1d31487752c1 (at 10.8.3.32@o2ib6) Feb 09 04:07:37 fir-io1-s1 kernel: Lustre: Skipped 445 previous similar messages Feb 09 04:17:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to eb0df602-8348-c011-6e89-eb251abd235e (at 10.9.102.63@o2ib4) Feb 09 04:17:37 fir-io1-s1 kernel: Lustre: Skipped 509 previous similar messages Feb 09 04:27:49 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 880716fb-6a2b-3d87-95fb-03534cabe92d (at 10.8.8.28@o2ib6) Feb 09 04:27:49 fir-io1-s1 kernel: Lustre: Skipped 542 previous similar messages Feb 09 04:37:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 09 04:37:49 fir-io1-s1 kernel: Lustre: Skipped 722 previous similar messages Feb 09 04:47:50 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b8fb3549-2789-3c8d-8664-05de4a5b5d4e (at 10.9.104.35@o2ib4) Feb 09 04:47:50 fir-io1-s1 kernel: Lustre: Skipped 655 previous similar messages Feb 09 04:57:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 880716fb-6a2b-3d87-95fb-03534cabe92d (at 10.8.8.28@o2ib6) Feb 09 04:57:50 fir-io1-s1 kernel: Lustre: Skipped 635 previous similar messages Feb 09 05:07:52 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d8dc7933-beed-019d-bb3c-58b11a857fbb (at 10.8.18.5@o2ib6) Feb 09 05:07:52 fir-io1-s1 kernel: Lustre: Skipped 866 previous similar messages Feb 09 05:17:53 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fbc61b39-e424-c398-e545-061af0049cf9 (at 10.9.107.11@o2ib4) Feb 09 05:17:53 fir-io1-s1 kernel: Lustre: Skipped 885 previous similar messages Feb 09 05:27:54 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ffcc7510-f875-7549-61d4-9f6248a33eef (at 10.9.105.7@o2ib4) Feb 09 05:27:54 fir-io1-s1 kernel: Lustre: Skipped 699 previous similar messages Feb 09 05:37:54 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 29b97f96-e09c-6114-f2fc-805b41aea072 (at 10.8.1.33@o2ib6) Feb 09 05:37:54 fir-io1-s1 kernel: Lustre: Skipped 898 previous similar messages Feb 09 05:47:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c202cb1b-dff6-0d0f-3d70-67d8f9a1aa0a (at 10.9.101.27@o2ib4) Feb 09 05:47:56 fir-io1-s1 kernel: Lustre: Skipped 864 previous similar messages Feb 09 05:57:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b5581c4c-47a8-b046-19f4-e572468b3106 (at 10.9.102.32@o2ib4) Feb 09 05:57:57 fir-io1-s1 kernel: Lustre: Skipped 683 previous similar messages Feb 09 06:07:57 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ef406a33-dbdc-c381-15c5-4fb662abecc1 (at 10.8.27.10@o2ib6) Feb 09 06:07:57 fir-io1-s1 kernel: Lustre: Skipped 737 previous similar messages Feb 09 06:17:58 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 56a7901d-9f1f-93d3-c75e-92d9b6eaca50 (at 10.9.107.48@o2ib4) Feb 09 06:17:58 fir-io1-s1 kernel: Lustre: Skipped 759 previous similar messages Feb 09 06:27:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a397e427-2a18-94f0-0f06-4c6a9e455efa (at 10.8.18.8@o2ib6) Feb 09 06:27:58 fir-io1-s1 kernel: Lustre: Skipped 630 previous similar messages Feb 09 06:37:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ef406a33-dbdc-c381-15c5-4fb662abecc1 (at 10.8.27.10@o2ib6) Feb 09 06:37:58 fir-io1-s1 kernel: Lustre: Skipped 766 previous similar messages Feb 09 06:47:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3512bfd0-06dd-989b-6d83-75d517d82937 (at 10.9.105.6@o2ib4) Feb 09 06:47:58 fir-io1-s1 kernel: Lustre: Skipped 673 previous similar messages Feb 09 06:58:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 767aeaf5-eff7-75a7-f62d-b6dcfae50462 (at 10.8.18.10@o2ib6) Feb 09 06:58:00 fir-io1-s1 kernel: Lustre: Skipped 681 previous similar messages Feb 09 07:08:04 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.8.3.17@o2ib6) Feb 09 07:08:04 fir-io1-s1 kernel: Lustre: Skipped 633 previous similar messages Feb 09 07:18:04 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 1c925920-cb23-478a-e656-d5227cb2f310 (at 10.8.18.4@o2ib6) Feb 09 07:18:04 fir-io1-s1 kernel: Lustre: Skipped 636 previous similar messages Feb 09 07:28:05 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.16.7@o2ib6) Feb 09 07:28:05 fir-io1-s1 kernel: Lustre: Skipped 602 previous similar messages Feb 09 07:38:09 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b7eb786c-4da9-9431-a597-6f5f4ba4c9ed (at 10.9.114.12@o2ib4) Feb 09 07:38:09 fir-io1-s1 kernel: Lustre: Skipped 670 previous similar messages Feb 09 07:48:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d20f9a14-81fb-ba0e-a6d1-6b2753e8ed8d (at 10.8.31.1@o2ib6) Feb 09 07:48:09 fir-io1-s1 kernel: Lustre: Skipped 689 previous similar messages Feb 09 07:58:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9633991e-ce4f-d92c-b6aa-ec983a0f2b80 (at 10.8.8.23@o2ib6) Feb 09 07:58:09 fir-io1-s1 kernel: Lustre: Skipped 662 previous similar messages Feb 09 08:08:11 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.8.4.20@o2ib6) Feb 09 08:08:11 fir-io1-s1 kernel: Lustre: Skipped 681 previous similar messages Feb 09 08:18:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.106.66@o2ib4) Feb 09 08:18:13 fir-io1-s1 kernel: Lustre: Skipped 615 previous similar messages Feb 09 08:28:14 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 202fc320-3606-cc07-c5a8-57679ec64217 (at 10.8.22.30@o2ib6) Feb 09 08:28:14 fir-io1-s1 kernel: Lustre: Skipped 562 previous similar messages Feb 09 08:38:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 41274e3f-2ebe-212c-8fae-793d010c94ef (at 10.9.105.34@o2ib4) Feb 09 08:38:16 fir-io1-s1 kernel: Lustre: Skipped 597 previous similar messages Feb 09 08:48:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to afd89a08-6b67-04f7-2303-d504e382d3cd (at 10.9.105.61@o2ib4) Feb 09 08:48:18 fir-io1-s1 kernel: Lustre: Skipped 606 previous similar messages Feb 09 08:58:19 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3dd7b3c3-369e-14ba-c881-c252e5dc17a0 (at 10.8.8.27@o2ib6) Feb 09 08:58:19 fir-io1-s1 kernel: Lustre: Skipped 527 previous similar messages Feb 09 09:08:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.112.7@o2ib4) Feb 09 09:08:26 fir-io1-s1 kernel: Lustre: Skipped 596 previous similar messages Feb 09 09:09:47 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client dcd406ad-ffdd-a7c9-489f-309957a1236e (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762861800, cur 1549732187 expire 1549732037 last 1549731960 Feb 09 09:09:47 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 09 09:18:31 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3512bfd0-06dd-989b-6d83-75d517d82937 (at 10.9.105.6@o2ib4) Feb 09 09:18:31 fir-io1-s1 kernel: Lustre: Skipped 647 previous similar messages Feb 09 09:28:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.104.4@o2ib4) Feb 09 09:28:31 fir-io1-s1 kernel: Lustre: Skipped 524 previous similar messages Feb 09 09:38:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45a12f91-6aa3-f0ae-a299-8aadf4c776a5 (at 10.8.17.5@o2ib6) Feb 09 09:38:32 fir-io1-s1 kernel: Lustre: Skipped 646 previous similar messages Feb 09 09:48:32 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b5d0fd65-06a7-fd14-4925-ef95dfe63868 (at 10.9.107.28@o2ib4) Feb 09 09:48:32 fir-io1-s1 kernel: Lustre: Skipped 848 previous similar messages Feb 09 09:58:36 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1bdafb8f-098e-e990-f183-dce8ce68db0c (at 10.9.102.5@o2ib4) Feb 09 09:58:36 fir-io1-s1 kernel: Lustre: Skipped 756 previous similar messages Feb 09 10:08:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3352dbfd-d63f-74b6-bc30-6f22efce7e27 (at 10.8.7.3@o2ib6) Feb 09 10:08:40 fir-io1-s1 kernel: Lustre: Skipped 677 previous similar messages Feb 09 10:18:46 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to d851fba9-b115-cdc7-e280-01f5ac21500a (at 10.8.13.5@o2ib6) Feb 09 10:18:46 fir-io1-s1 kernel: Lustre: Skipped 588 previous similar messages Feb 09 10:28:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 400e2c70-3670-eb05-66c0-e754ea5cd280 (at 10.8.29.7@o2ib6) Feb 09 10:28:46 fir-io1-s1 kernel: Lustre: Skipped 556 previous similar messages Feb 09 10:38:49 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a6388386-fc78-2645-7cfe-6de1bc9e10a6 (at 10.9.102.37@o2ib4) Feb 09 10:38:49 fir-io1-s1 kernel: Lustre: Skipped 495 previous similar messages Feb 09 10:48:50 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to fabd17e2-268b-b1ba-b568-7f6550034520 (at 10.9.106.16@o2ib4) Feb 09 10:48:50 fir-io1-s1 kernel: Lustre: Skipped 762 previous similar messages Feb 09 10:58:52 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 1ba5aef6-2be1-c2db-6ba0-6e3a31e32627 (at 10.9.107.41@o2ib4) Feb 09 10:58:52 fir-io1-s1 kernel: Lustre: Skipped 835 previous similar messages Feb 09 11:08:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 54df73a3-4915-b589-f8b2-dd262402c8c5 (at 10.9.107.65@o2ib4) Feb 09 11:08:54 fir-io1-s1 kernel: Lustre: Skipped 921 previous similar messages Feb 09 11:18:55 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 11bdb068-3a68-b486-d427-87b2a02899d5 (at 10.8.2.24@o2ib6) Feb 09 11:18:55 fir-io1-s1 kernel: Lustre: Skipped 936 previous similar messages Feb 09 11:28:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d4ede191-33ac-db3d-8e23-e76bd511a700 (at 10.8.28.3@o2ib6) Feb 09 11:28:56 fir-io1-s1 kernel: Lustre: Skipped 950 previous similar messages Feb 09 11:38:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3512bfd0-06dd-989b-6d83-75d517d82937 (at 10.9.105.6@o2ib4) Feb 09 11:38:57 fir-io1-s1 kernel: Lustre: Skipped 893 previous similar messages Feb 09 11:48:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 73d8b62a-8470-ee48-b43c-7b159c916ee7 (at 10.9.105.52@o2ib4) Feb 09 11:48:58 fir-io1-s1 kernel: Lustre: Skipped 778 previous similar messages Feb 09 11:58:59 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 29b97f96-e09c-6114-f2fc-805b41aea072 (at 10.8.1.33@o2ib6) Feb 09 11:58:59 fir-io1-s1 kernel: Lustre: Skipped 677 previous similar messages Feb 09 12:09:00 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7e9161e5-27d8-4cac-a415-4c23ea14bc0e (at 10.9.106.46@o2ib4) Feb 09 12:09:00 fir-io1-s1 kernel: Lustre: Skipped 569 previous similar messages Feb 09 12:19:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dbfdab11-8ec7-41b0-1485-dd07c4a504bd (at 10.9.104.47@o2ib4) Feb 09 12:19:02 fir-io1-s1 kernel: Lustre: Skipped 486 previous similar messages Feb 09 12:29:02 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b5581c4c-47a8-b046-19f4-e572468b3106 (at 10.9.102.32@o2ib4) Feb 09 12:29:02 fir-io1-s1 kernel: Lustre: Skipped 545 previous similar messages Feb 09 12:39:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 09 12:39:02 fir-io1-s1 kernel: Lustre: Skipped 645 previous similar messages Feb 09 12:49:05 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 09 12:49:05 fir-io1-s1 kernel: Lustre: Skipped 481 previous similar messages Feb 09 12:59:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c3afa774-0e39-3459-0af2-ddf264214c5b (at 10.9.104.54@o2ib4) Feb 09 12:59:06 fir-io1-s1 kernel: Lustre: Skipped 341 previous similar messages Feb 09 13:09:07 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 6d1355d0-7b33-677d-c8cf-a270e3061917 (at 10.8.7.15@o2ib6) Feb 09 13:09:07 fir-io1-s1 kernel: Lustre: Skipped 494 previous similar messages Feb 09 13:19:10 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 37327d93-dd03-80bf-ad2c-df17a42702a9 (at 10.8.18.26@o2ib6) Feb 09 13:19:10 fir-io1-s1 kernel: Lustre: Skipped 562 previous similar messages Feb 09 13:29:11 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9b6ee593-357f-08be-650d-81734979fa6c (at 10.9.107.40@o2ib4) Feb 09 13:29:11 fir-io1-s1 kernel: Lustre: Skipped 568 previous similar messages Feb 09 13:39:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 889a8bd9-6c23-c824-f979-e28b7cf41d1f (at 10.9.114.10@o2ib4) Feb 09 13:39:17 fir-io1-s1 kernel: Lustre: Skipped 437 previous similar messages Feb 09 13:49:19 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bdf3d0f5-851d-26cd-ba90-9355de313856 (at 10.8.6.19@o2ib6) Feb 09 13:49:19 fir-io1-s1 kernel: Lustre: Skipped 418 previous similar messages Feb 09 13:59:19 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ea7067de-16a0-5aef-a5e8-0c3de073c862 (at 10.9.106.72@o2ib4) Feb 09 13:59:19 fir-io1-s1 kernel: Lustre: Skipped 425 previous similar messages Feb 09 13:59:59 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ba56f8dc-0155-bbc1-f64d-9b90836aeb9c (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784becc00, cur 1549749599 expire 1549749449 last 1549749372 Feb 09 13:59:59 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 09 14:09:28 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.101.64@o2ib4) Feb 09 14:09:28 fir-io1-s1 kernel: Lustre: Skipped 624 previous similar messages Feb 09 14:19:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9fdeb7e4-dbe7-36a2-5704-e3cbd89d3a9c (at 10.9.104.25@o2ib4) Feb 09 14:19:33 fir-io1-s1 kernel: Lustre: Skipped 340 previous similar messages Feb 09 14:29:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 37aa7d65-aee3-56c5-fbe1-78e40cec7bbc (at 10.9.101.66@o2ib4) Feb 09 14:29:38 fir-io1-s1 kernel: Lustre: Skipped 521 previous similar messages Feb 09 14:36:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 38a104b5-26ce-5d2d-596d-9304083f888f (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de2800, cur 1549751760 expire 1549751610 last 1549751533 Feb 09 14:36:00 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 09 14:39:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8f530db8-807a-fdf0-2880-6c939864abc5 (at 10.9.115.8@o2ib4) Feb 09 14:39:41 fir-io1-s1 kernel: Lustre: Skipped 522 previous similar messages Feb 09 14:49:45 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 32ea102c-c8a6-9ae9-e0f7-d3fc0379beb1 (at 10.8.2.23@o2ib6) Feb 09 14:49:45 fir-io1-s1 kernel: Lustre: Skipped 507 previous similar messages Feb 09 14:59:45 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 61244de0-3d61-3ad3-fe92-f92a9f896d83 (at 10.9.107.23@o2ib4) Feb 09 14:59:45 fir-io1-s1 kernel: Lustre: Skipped 638 previous similar messages Feb 09 15:09:48 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.9.102.29@o2ib4) Feb 09 15:09:48 fir-io1-s1 kernel: Lustre: Skipped 703 previous similar messages Feb 09 15:19:48 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6) Feb 09 15:19:48 fir-io1-s1 kernel: Lustre: Skipped 557 previous similar messages Feb 09 15:29:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5a74680-e7af-ebfb-7dfd-72e2645d277b (at 10.9.101.51@o2ib4) Feb 09 15:29:51 fir-io1-s1 kernel: Lustre: Skipped 316 previous similar messages Feb 09 15:39:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 09 15:39:54 fir-io1-s1 kernel: Lustre: Skipped 328 previous similar messages Feb 09 15:49:55 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 364c8566-8b44-8def-9ccb-d070a0bfcffe (at 10.8.26.35@o2ib6) Feb 09 15:49:55 fir-io1-s1 kernel: Lustre: Skipped 418 previous similar messages Feb 09 16:00:24 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9334fe18-06e4-7654-e3bd-8d4e4ade184d (at 10.8.8.8@o2ib6) Feb 09 16:00:24 fir-io1-s1 kernel: Lustre: Skipped 386 previous similar messages Feb 09 16:10:24 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to b02dee9c-7899-81b7-2e05-7b8f99ef42cf (at 10.8.30.18@o2ib6) Feb 09 16:10:24 fir-io1-s1 kernel: Lustre: Skipped 460 previous similar messages Feb 09 16:20:31 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 02904912-fe2e-5026-bc07-2718cbca6fa6 (at 10.8.2.34@o2ib6) Feb 09 16:20:31 fir-io1-s1 kernel: Lustre: Skipped 377 previous similar messages Feb 09 16:27:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 71256bfb-0197-3e32-60b3-6d6186c065c4 (at 10.9.105.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9878386e3800, cur 1549758457 expire 1549758307 last 1549758230 Feb 09 16:27:37 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 09 16:30:37 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 56a7901d-9f1f-93d3-c75e-92d9b6eaca50 (at 10.9.107.48@o2ib4) Feb 09 16:30:37 fir-io1-s1 kernel: Lustre: Skipped 395 previous similar messages Feb 09 16:40:37 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 09 16:40:37 fir-io1-s1 kernel: Lustre: Skipped 477 previous similar messages Feb 09 16:50:38 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to f97f4802-8c0b-e9cd-e8d7-93692decf22a (at 10.9.102.8@o2ib4) Feb 09 16:50:38 fir-io1-s1 kernel: Lustre: Skipped 471 previous similar messages Feb 09 17:00:38 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a28c293f-ee5d-ad54-991b-4f6f191450b5 (at 10.9.102.57@o2ib4) Feb 09 17:00:38 fir-io1-s1 kernel: Lustre: Skipped 615 previous similar messages Feb 09 17:02:50 fir-io1-s1 kernel: Lustre: 96903:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549760563/real 1549760563] req@ffff98480446ad00 x1624930748585872/t0(0) o106->fir-OST0008@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549760570 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 09 17:03:04 fir-io1-s1 kernel: Lustre: 96903:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549760577/real 1549760577] req@ffff98480446ad00 x1624930748585872/t0(0) o106->fir-OST0008@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549760584 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 09 17:03:04 fir-io1-s1 kernel: Lustre: 96903:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 09 17:03:25 fir-io1-s1 kernel: Lustre: 96903:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549760598/real 1549760598] req@ffff98480446ad00 x1624930748585872/t0(0) o106->fir-OST0008@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549760605 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 09 17:03:25 fir-io1-s1 kernel: Lustre: 96903:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 09 17:04:00 fir-io1-s1 kernel: Lustre: 96774:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549760629/real 1549760629] req@ffff98408212f200 x1624930748585856/t0(0) o106->fir-OST0006@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549760640 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 09 17:04:00 fir-io1-s1 kernel: Lustre: 96774:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 09 17:04:49 fir-io1-s1 kernel: LustreError: 96903:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.26.33@o2ib6) returned error from glimpse AST (req@ffff98480446ad00 x1624930748585872 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff98575cc5a880/0x49e185e928fe8261 lrc: 3/0,0 mode: PW/PW res: [0xc80000400:0x27fd0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000000000 nid: 10.8.26.33@o2ib6 remote: 0xbe3dd0c23188244e expref: 5 pid: 96931 timeout: 0 lvb_type: 0 Feb 09 17:04:49 fir-io1-s1 kernel: LustreError: 96903:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Feb 09 17:04:49 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.26.33@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 09 17:04:49 fir-io1-s1 kernel: LustreError: Skipped 6 previous similar messages Feb 09 17:04:49 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1549760689s: evicting client at 10.8.26.33@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff98575cc5a880/0x49e185e928fe8261 lrc: 3/0,0 mode: PW/PW res: [0xc80000400:0x27fd0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000000000 nid: 10.8.26.33@o2ib6 remote: 0xbe3dd0c23188244e expref: 6 pid: 96931 timeout: 0 lvb_type: 0 Feb 09 17:04:49 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Feb 09 17:04:55 fir-io1-s1 kernel: LustreError: 94235:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.26.33@o2ib6) returned error from glimpse AST (req@ffff983d44bbf500 x1624930748585904 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff98575cc5f500/0x49e185e928fe82a0 lrc: 3/0,0 mode: PW/PW res: [0x5c0000401:0x2805b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.26.33@o2ib6 remote: 0xbe3dd0c2318824be expref: 6 pid: 96931 timeout: 0 lvb_type: 0 Feb 09 17:04:55 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.26.33@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 09 17:04:55 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1549760695s: evicting client at 10.8.26.33@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff98575cc59680/0x49e185e928fe823e lrc: 3/0,0 mode: PW/PW res: [0xc40000400:0x27f74:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000000000 nid: 10.8.26.33@o2ib6 remote: 0xbe3dd0c231882416 expref: 7 pid: 96931 timeout: 0 lvb_type: 0 Feb 09 17:04:55 fir-io1-s1 kernel: LustreError: 94235:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 09 17:05:06 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client dfad11f5-37c7-7014-c92a-508def843e75 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984803004400, cur 1549760706 expire 1549760556 last 1549760479 Feb 09 17:05:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 09 17:10:44 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) Feb 09 17:10:44 fir-io1-s1 kernel: Lustre: Skipped 704 previous similar messages Feb 09 17:20:47 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2354d7af-653b-aaa3-2c33-b296ff69d0d2 (at 10.8.10.15@o2ib6) Feb 09 17:20:47 fir-io1-s1 kernel: Lustre: Skipped 495 previous similar messages Feb 09 17:25:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 36b574bd-6f14-9f2c-6c78-1e23c5b6197c (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848008f2000, cur 1549761940 expire 1549761790 last 1549761713 Feb 09 17:25:40 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 09 17:30:49 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 7c312f41-52f1-f4c1-0bd5-a3e41737d0f6 (at 10.9.106.56@o2ib4) Feb 09 17:30:49 fir-io1-s1 kernel: Lustre: Skipped 452 previous similar messages Feb 09 17:32:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bdaf51fb-edf4-e82f-51be-9141ee573c83 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480041cc00, cur 1549762362 expire 1549762212 last 1549762135 Feb 09 17:32:42 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 09 17:40:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1e5f7f4c-78fa-5eb7-a0ea-e8f04fabf57f (at 10.8.30.32@o2ib6) Feb 09 17:40:53 fir-io1-s1 kernel: Lustre: Skipped 306 previous similar messages Feb 09 17:45:39 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 1d2da3ac-e307-036e-f8a3-99f8b3ab4ed7 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678694e400, cur 1549763139 expire 1549762989 last 1549762912 Feb 09 17:45:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 09 17:51:10 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5fd68af5-0c2b-1947-5fc1-6504b55b60fb (at 10.9.103.16@o2ib4) Feb 09 17:51:10 fir-io1-s1 kernel: Lustre: Skipped 399 previous similar messages Feb 09 17:55:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client e44bc79e-ea6c-0264-2357-8d89f812546a (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575be1e400, cur 1549763735 expire 1549763585 last 1549763508 Feb 09 17:55:35 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 09 18:01:20 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.18.23@o2ib6) Feb 09 18:01:20 fir-io1-s1 kernel: Lustre: Skipped 346 previous similar messages Feb 09 18:11:24 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 8d5e4f9e-ae64-4b8a-4d39-5ab33e08ed45 (at 10.9.106.67@o2ib4) Feb 09 18:11:24 fir-io1-s1 kernel: Lustre: Skipped 625 previous similar messages Feb 09 18:21:25 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) Feb 09 18:21:25 fir-io1-s1 kernel: Lustre: Skipped 1004 previous similar messages Feb 09 18:31:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1bdafb8f-098e-e990-f183-dce8ce68db0c (at 10.9.102.5@o2ib4) Feb 09 18:31:26 fir-io1-s1 kernel: Lustre: Skipped 971 previous similar messages Feb 09 18:41:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9334fe18-06e4-7654-e3bd-8d4e4ade184d (at 10.8.8.8@o2ib6) Feb 09 18:41:30 fir-io1-s1 kernel: Lustre: Skipped 680 previous similar messages Feb 09 18:51:39 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3942546e-2266-9b89-886d-2fc4af57c4cd (at 10.8.4.33@o2ib6) Feb 09 18:51:39 fir-io1-s1 kernel: Lustre: Skipped 378 previous similar messages Feb 09 19:01:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 63d43612-6a45-cc62-c015-db6d91359b53 (at 10.9.104.32@o2ib4) Feb 09 19:01:44 fir-io1-s1 kernel: Lustre: Skipped 376 previous similar messages Feb 09 19:07:31 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ca1af5b2-4b74-b03d-4a2b-13a823b2dc8f (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576184a800, cur 1549768051 expire 1549767901 last 1549767824 Feb 09 19:07:31 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 09 19:11:45 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to aa02b352-d447-011e-8454-fff3bd1add27 (at 10.9.104.41@o2ib4) Feb 09 19:11:45 fir-io1-s1 kernel: Lustre: Skipped 344 previous similar messages Feb 09 19:21:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to f854c2f0-b53f-8306-9638-bc37f75b2b94 (at 10.8.8.7@o2ib6) Feb 09 19:21:57 fir-io1-s1 kernel: Lustre: Skipped 302 previous similar messages Feb 09 19:32:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7030631e-b3d2-9eed-f765-9117cb5ba8a4 (at 10.9.103.35@o2ib4) Feb 09 19:32:09 fir-io1-s1 kernel: Lustre: Skipped 301 previous similar messages Feb 09 19:32:42 fir-io1-s1 kernel: perf: interrupt took too long (3132 > 3131), lowering kernel.perf_event_max_sample_rate to 63000 Feb 09 19:42:14 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 09 19:42:14 fir-io1-s1 kernel: Lustre: Skipped 240 previous similar messages Feb 09 19:52:15 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) Feb 09 19:52:15 fir-io1-s1 kernel: Lustre: Skipped 277 previous similar messages Feb 09 20:02:20 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7408574f-6b07-66ec-bd6d-34d66cefdfdc (at 10.8.11.28@o2ib6) Feb 09 20:02:20 fir-io1-s1 kernel: Lustre: Skipped 277 previous similar messages Feb 09 20:12:28 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) Feb 09 20:12:28 fir-io1-s1 kernel: Lustre: Skipped 349 previous similar messages Feb 09 20:22:40 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 6179b8f9-2b15-63f8-671c-eb8d9ecd9187 (at 10.8.16.3@o2ib6) Feb 09 20:22:40 fir-io1-s1 kernel: Lustre: Skipped 489 previous similar messages Feb 09 20:32:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 23391852-ef00-9fa9-e0d6-21dd26d1c3f4 (at 10.9.102.42@o2ib4) Feb 09 20:32:41 fir-io1-s1 kernel: Lustre: Skipped 362 previous similar messages Feb 09 20:42:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 09 20:42:46 fir-io1-s1 kernel: Lustre: Skipped 354 previous similar messages Feb 09 20:52:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 09 20:52:46 fir-io1-s1 kernel: Lustre: Skipped 357 previous similar messages Feb 09 21:02:46 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 09 21:02:46 fir-io1-s1 kernel: Lustre: Skipped 309 previous similar messages Feb 09 21:12:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8d89fca0-9472-dc06-65e2-fd0a61adf564 (at 10.9.106.23@o2ib4) Feb 09 21:12:57 fir-io1-s1 kernel: Lustre: Skipped 375 previous similar messages Feb 09 21:22:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 11122870-d2b4-7c10-4146-c4d58251a247 (at 10.8.16.4@o2ib6) Feb 09 21:22:57 fir-io1-s1 kernel: Lustre: Skipped 345 previous similar messages Feb 09 21:33:10 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to b0f4a89e-7973-eb31-a1c9-fdc42b6cc4f6 (at 10.8.18.25@o2ib6) Feb 09 21:33:10 fir-io1-s1 kernel: Lustre: Skipped 408 previous similar messages Feb 09 21:43:14 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6a1a7e31-f24a-25a3-4e1c-41f3ed10a783 (at 10.9.102.36@o2ib4) Feb 09 21:43:14 fir-io1-s1 kernel: Lustre: Skipped 321 previous similar messages Feb 09 21:53:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) Feb 09 21:53:22 fir-io1-s1 kernel: Lustre: Skipped 364 previous similar messages Feb 09 22:03:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ad934baa-14a0-ff83-e640-01d16d0aae6a (at 10.8.8.21@o2ib6) Feb 09 22:03:23 fir-io1-s1 kernel: Lustre: Skipped 397 previous similar messages Feb 09 22:13:25 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 09 22:13:25 fir-io1-s1 kernel: Lustre: Skipped 466 previous similar messages Feb 09 22:23:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 131aad16-4048-4b97-7d38-4c357c4e10e5 (at 10.8.10.10@o2ib6) Feb 09 22:23:26 fir-io1-s1 kernel: Lustre: Skipped 572 previous similar messages Feb 09 22:33:32 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 09 22:33:32 fir-io1-s1 kernel: Lustre: Skipped 462 previous similar messages Feb 09 22:43:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 09 22:43:38 fir-io1-s1 kernel: Lustre: Skipped 486 previous similar messages Feb 09 22:53:42 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 09 22:53:42 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 09 23:03:44 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6ba54f1c-46af-7bf3-fbb0-e98de6bd7af0 (at 10.8.30.11@o2ib6) Feb 09 23:03:44 fir-io1-s1 kernel: Lustre: Skipped 633 previous similar messages Feb 09 23:13:55 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) Feb 09 23:13:55 fir-io1-s1 kernel: Lustre: Skipped 430 previous similar messages Feb 09 23:24:03 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 09 23:24:03 fir-io1-s1 kernel: Lustre: Skipped 265 previous similar messages Feb 09 23:27:56 fir-io1-s1 kernel: Lustre: 117030:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549783669/real 1549783669] req@ffff983c5b986900 x1624931022259872/t0(0) o105->fir-OST0004@10.8.17.23@o2ib6:15/16 lens 360/224 e 0 to 1 dl 1549783676 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 09 23:27:56 fir-io1-s1 kernel: Lustre: 117030:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Feb 09 23:28:10 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549783683/real 1549783683] req@ffff984bde597800 x1624931022259968/t0(0) o104->fir-OST0004@10.8.17.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1549783690 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 09 23:28:10 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 09 23:28:31 fir-io1-s1 kernel: Lustre: 117030:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549783704/real 1549783704] req@ffff983c5b986900 x1624931022259872/t0(0) o105->fir-OST0004@10.8.17.23@o2ib6:15/16 lens 360/224 e 0 to 1 dl 1549783711 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 09 23:28:31 fir-io1-s1 kernel: Lustre: 117030:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 09 23:29:06 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549783739/real 1549783739] req@ffff984bde597800 x1624931022259968/t0(0) o104->fir-OST0004@10.8.17.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1549783746 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 09 23:29:06 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 09 23:29:34 fir-io1-s1 kernel: LustreError: 96368:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.17.23@o2ib6) failed to reply to blocking AST (req@ffff984bde597800 x1624931022259968 status 0 rc -110), evict it ns: filter-fir-OST0004_UUID lock: ffff9842155a57c0/0x49e185e92d7691a4 lrc: 5/0,0 mode: PW/PW res: [0x8c0000402:0x88c4f:0x0].0x0 rrc: 1778 type: EXT [29032448->62586879] (req 29032448->29036543) flags: 0x60000400020020 nid: 10.8.17.23@o2ib6 remote: 0x84f7b87675b27244 expref: 8 pid: 96245 timeout: 131618 lvb_type: 0 Feb 09 23:29:34 fir-io1-s1 kernel: LustreError: 96368:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 09 23:29:34 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.17.23@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 09 23:29:34 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 09 23:29:34 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.17.23@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff9842155a57c0/0x49e185e92d7691a4 lrc: 4/0,0 mode: PW/PW res: [0x8c0000402:0x88c4f:0x0].0x0 rrc: 1778 type: EXT [29032448->62586879] (req 29032448->29036543) flags: 0x60000400020020 nid: 10.8.17.23@o2ib6 remote: 0x84f7b87675b27244 expref: 9 pid: 96245 timeout: 0 lvb_type: 0 Feb 09 23:29:34 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Feb 09 23:31:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f679400, cur 1549783896 expire 1549783746 last 1549783669 Feb 09 23:31:36 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 09 23:34:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) Feb 09 23:34:09 fir-io1-s1 kernel: Lustre: Skipped 273 previous similar messages Feb 09 23:44:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) Feb 09 23:44:13 fir-io1-s1 kernel: Lustre: Skipped 405 previous similar messages Feb 09 23:54:18 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Feb 09 23:54:18 fir-io1-s1 kernel: Lustre: Skipped 537 previous similar messages Feb 10 00:04:20 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Feb 10 00:04:20 fir-io1-s1 kernel: Lustre: Skipped 365 previous similar messages Feb 10 00:14:24 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) Feb 10 00:14:24 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 10 00:24:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.106.61@o2ib4) Feb 10 00:24:36 fir-io1-s1 kernel: Lustre: Skipped 292 previous similar messages Feb 10 00:24:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0cda1ca9-b849-4b59-7ce7-48abe2de3c2e (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f7ef800, cur 1549787083 expire 1549786933 last 1549786856 Feb 10 00:24:43 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 00:34:39 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Feb 10 00:34:39 fir-io1-s1 kernel: Lustre: Skipped 219 previous similar messages Feb 10 00:44:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 342724f1-ee95-beed-9c54-15c49b83cfa7 (at 10.8.10.31@o2ib6) Feb 10 00:44:59 fir-io1-s1 kernel: Lustre: Skipped 247 previous similar messages Feb 10 00:55:17 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Feb 10 00:55:17 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 10 01:00:01 fir-io1-s1 kernel: md: data-check of RAID array md8 Feb 10 01:00:07 fir-io1-s1 kernel: md: data-check of RAID array md4 Feb 10 01:00:13 fir-io1-s1 kernel: md: data-check of RAID array md2 Feb 10 01:00:20 fir-io1-s1 kernel: md: data-check of RAID array md6 Feb 10 01:00:26 fir-io1-s1 kernel: md: data-check of RAID array md10 Feb 10 01:00:32 fir-io1-s1 kernel: md: data-check of RAID array md0 Feb 10 01:05:21 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6396b591-675d-9cab-3d7d-b05dbeda50e7 (at 10.8.26.36@o2ib6) Feb 10 01:05:21 fir-io1-s1 kernel: Lustre: Skipped 243 previous similar messages Feb 10 01:15:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 420eb17f-e3a0-1fd9-bcf2-389dfdfba340 (at 10.8.20.5@o2ib6) Feb 10 01:15:57 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 10 01:25:59 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to bd6b0907-bbf0-754e-ba62-411999a5fe50 (at 10.8.15.1@o2ib6) Feb 10 01:25:59 fir-io1-s1 kernel: Lustre: Skipped 228 previous similar messages Feb 10 01:36:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 10 01:36:01 fir-io1-s1 kernel: Lustre: Skipped 188 previous similar messages Feb 10 01:46:07 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 6179b8f9-2b15-63f8-671c-eb8d9ecd9187 (at 10.8.16.3@o2ib6) Feb 10 01:46:07 fir-io1-s1 kernel: Lustre: Skipped 242 previous similar messages Feb 10 01:56:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5fda523c-9891-cc11-7a0e-50a252e6fb83 (at 10.9.103.8@o2ib4) Feb 10 01:56:07 fir-io1-s1 kernel: Lustre: Skipped 272 previous similar messages Feb 10 02:06:08 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3d56b93c-c51e-bd16-d798-5d8639e7069c (at 10.8.11.31@o2ib6) Feb 10 02:06:08 fir-io1-s1 kernel: Lustre: Skipped 195 previous similar messages Feb 10 02:16:19 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Feb 10 02:16:19 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 10 02:26:21 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Feb 10 02:26:21 fir-io1-s1 kernel: Lustre: Skipped 172 previous similar messages Feb 10 02:36:39 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2d486816-0fee-3e02-cb25-89549e06c293 (at 10.9.104.52@o2ib4) Feb 10 02:36:39 fir-io1-s1 kernel: Lustre: Skipped 219 previous similar messages Feb 10 02:46:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8f273097-d34e-d604-e427-2da4f99ca32a (at 10.9.106.26@o2ib4) Feb 10 02:46:41 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 10 02:56:50 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 88ddc3f9-1f21-4f81-e1f1-3f396b007308 (at 10.9.101.54@o2ib4) Feb 10 02:56:50 fir-io1-s1 kernel: Lustre: Skipped 247 previous similar messages Feb 10 03:06:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e63b86ba-9595-7419-fca7-82e45f9f64cb (at 10.9.104.67@o2ib4) Feb 10 03:06:52 fir-io1-s1 kernel: Lustre: Skipped 242 previous similar messages Feb 10 03:17:05 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9334fe18-06e4-7654-e3bd-8d4e4ade184d (at 10.8.8.8@o2ib6) Feb 10 03:17:05 fir-io1-s1 kernel: Lustre: Skipped 204 previous similar messages Feb 10 03:27:10 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 87721f3b-4f03-c138-ffa3-cffa8a052df0 (at 10.8.26.5@o2ib6) Feb 10 03:27:10 fir-io1-s1 kernel: Lustre: Skipped 266 previous similar messages Feb 10 03:37:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Feb 10 03:37:15 fir-io1-s1 kernel: Lustre: Skipped 273 previous similar messages Feb 10 03:47:17 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 03:47:17 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 10 03:57:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 03:57:20 fir-io1-s1 kernel: Lustre: Skipped 236 previous similar messages Feb 10 04:07:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9468a1d3-3abd-8063-5952-288cca0f1dec (at 10.8.27.35@o2ib6) Feb 10 04:07:39 fir-io1-s1 kernel: Lustre: Skipped 253 previous similar messages Feb 10 04:17:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 26276ee4-1318-03d7-bf25-08cb51193a9d (at 10.9.102.22@o2ib4) Feb 10 04:17:40 fir-io1-s1 kernel: Lustre: Skipped 232 previous similar messages Feb 10 04:27:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 763f631c-7b84-895c-764f-d88426b5fe26 (at 10.8.1.3@o2ib6) Feb 10 04:27:40 fir-io1-s1 kernel: Lustre: Skipped 435 previous similar messages Feb 10 04:37:44 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9334fe18-06e4-7654-e3bd-8d4e4ade184d (at 10.8.8.8@o2ib6) Feb 10 04:37:44 fir-io1-s1 kernel: Lustre: Skipped 511 previous similar messages Feb 10 04:47:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 79f3f1b7-618c-6db7-a20c-7bae7242287e (at 10.8.1.16@o2ib6) Feb 10 04:47:44 fir-io1-s1 kernel: Lustre: Skipped 303 previous similar messages Feb 10 04:57:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 10 04:57:46 fir-io1-s1 kernel: Lustre: Skipped 272 previous similar messages Feb 10 05:07:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a8b0e693-902f-1f36-293b-a7b32f871782 (at 10.8.7.13@o2ib6) Feb 10 05:07:47 fir-io1-s1 kernel: Lustre: Skipped 339 previous similar messages Feb 10 05:17:52 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 05:17:52 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 10 05:28:02 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9334fe18-06e4-7654-e3bd-8d4e4ade184d (at 10.8.8.8@o2ib6) Feb 10 05:28:02 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 10 05:38:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6179b8f9-2b15-63f8-671c-eb8d9ecd9187 (at 10.8.16.3@o2ib6) Feb 10 05:38:04 fir-io1-s1 kernel: Lustre: Skipped 365 previous similar messages Feb 10 05:48:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ec1586f8-34f3-cd32-7140-123054dfbfed (at 10.9.102.30@o2ib4) Feb 10 05:48:13 fir-io1-s1 kernel: Lustre: Skipped 367 previous similar messages Feb 10 05:58:17 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 861509dc-4523-55e8-f09a-16b6fca3f713 (at 10.9.107.51@o2ib4) Feb 10 05:58:17 fir-io1-s1 kernel: Lustre: Skipped 370 previous similar messages Feb 10 06:08:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9468a1d3-3abd-8063-5952-288cca0f1dec (at 10.8.27.35@o2ib6) Feb 10 06:08:19 fir-io1-s1 kernel: Lustre: Skipped 320 previous similar messages Feb 10 06:18:20 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 5da38a2b-1f85-f985-e647-807ffb38b26c (at 10.8.30.35@o2ib6) Feb 10 06:18:20 fir-io1-s1 kernel: Lustre: Skipped 399 previous similar messages Feb 10 06:27:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 572fee18-597b-f5ad-f93d-9178ef57a0e3 (at 10.8.21.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483ed4b400, cur 1549808869 expire 1549808719 last 1549808642 Feb 10 06:27:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 06:28:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9468a1d3-3abd-8063-5952-288cca0f1dec (at 10.8.27.35@o2ib6) Feb 10 06:28:26 fir-io1-s1 kernel: Lustre: Skipped 435 previous similar messages Feb 10 06:38:29 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to e70f289f-dd27-3962-9868-2a7ca371acbb (at 10.8.10.30@o2ib6) Feb 10 06:38:29 fir-io1-s1 kernel: Lustre: Skipped 311 previous similar messages Feb 10 06:48:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 06:48:30 fir-io1-s1 kernel: Lustre: Skipped 351 previous similar messages Feb 10 06:58:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 06:58:36 fir-io1-s1 kernel: Lustre: Skipped 326 previous similar messages Feb 10 07:08:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 503151d3-e911-b92b-974a-493626aee137 (at 10.8.15.8@o2ib6) Feb 10 07:08:39 fir-io1-s1 kernel: Lustre: Skipped 286 previous similar messages Feb 10 07:18:40 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9633991e-ce4f-d92c-b6aa-ec983a0f2b80 (at 10.8.8.23@o2ib6) Feb 10 07:18:40 fir-io1-s1 kernel: Lustre: Skipped 416 previous similar messages Feb 10 07:28:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 83fafa06-153a-a1fd-dd02-4b2d3ef6c90e (at 10.9.114.2@o2ib4) Feb 10 07:28:43 fir-io1-s1 kernel: Lustre: Skipped 393 previous similar messages Feb 10 07:38:49 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.9.107.24@o2ib4) Feb 10 07:38:49 fir-io1-s1 kernel: Lustre: Skipped 361 previous similar messages Feb 10 07:48:50 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 73d8b62a-8470-ee48-b43c-7b159c916ee7 (at 10.9.105.52@o2ib4) Feb 10 07:48:50 fir-io1-s1 kernel: Lustre: Skipped 463 previous similar messages Feb 10 07:58:59 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.2.15@o2ib6) Feb 10 07:58:59 fir-io1-s1 kernel: Lustre: Skipped 435 previous similar messages Feb 10 08:08:59 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2c96ed5b-7b98-2819-9f2b-c8d6f7172439 (at 10.9.101.43@o2ib4) Feb 10 08:08:59 fir-io1-s1 kernel: Lustre: Skipped 371 previous similar messages Feb 10 08:19:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) Feb 10 08:19:09 fir-io1-s1 kernel: Lustre: Skipped 425 previous similar messages Feb 10 08:29:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Feb 10 08:29:14 fir-io1-s1 kernel: Lustre: Skipped 498 previous similar messages Feb 10 08:39:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d7f0cefd-f5dc-ae79-9fbe-8c42036c5092 (at 10.9.105.21@o2ib4) Feb 10 08:39:23 fir-io1-s1 kernel: Lustre: Skipped 363 previous similar messages Feb 10 08:46:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 08:46:17 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 10 08:46:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 08:46:18 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 10 08:46:58 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.14.7@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 10 08:46:58 fir-io1-s1 kernel: LustreError: Skipped 10 previous similar messages Feb 10 08:47:24 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 08:47:24 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 10 08:49:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 08:49:23 fir-io1-s1 kernel: Lustre: Skipped 458 previous similar messages Feb 10 08:50:12 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.14.7@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 10 08:50:12 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 10 08:50:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 08:50:37 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 10 08:54:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867832d0400, cur 1549817675 expire 1549817525 last 1549817448 Feb 10 08:54:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 08:54:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867832d4000, cur 1549817679 expire 1549817529 last 1549817452 Feb 10 08:54:45 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867832d6000, cur 1549817685 expire 1549817535 last 1549817458 Feb 10 08:59:28 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3751c7ce-a39e-44f6-65d9-394c617faca9 (at 10.8.18.22@o2ib6) Feb 10 08:59:28 fir-io1-s1 kernel: Lustre: Skipped 402 previous similar messages Feb 10 09:01:16 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.14.7@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 10 09:01:16 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Feb 10 09:01:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 09:01:41 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 10 09:05:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 09:05:58 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 10 09:06:01 fir-io1-s1 kernel: Lustre: fir-OST0004: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 09:06:01 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 10 09:09:28 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9042fb3a-c0ab-6915-0268-4626f11a023e (at 10.9.106.45@o2ib4) Feb 10 09:09:28 fir-io1-s1 kernel: Lustre: Skipped 331 previous similar messages Feb 10 09:14:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 09:14:34 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 09:19:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f2a610e8-4b67-c7b0-e65a-cefd2bd7f74e (at 10.8.13.17@o2ib6) Feb 10 09:19:30 fir-io1-s1 kernel: Lustre: Skipped 455 previous similar messages Feb 10 09:26:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 09:26:47 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 10 09:29:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9065faec-fdd8-46fc-db53-5ff36e99d790 (at 10.8.26.4@o2ib6) Feb 10 09:29:39 fir-io1-s1 kernel: Lustre: Skipped 501 previous similar messages Feb 10 09:39:40 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.29.2@o2ib6) Feb 10 09:39:40 fir-io1-s1 kernel: Lustre: Skipped 368 previous similar messages Feb 10 09:49:44 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 87a33bb7-d653-ad0d-1e59-70064b2446dd (at 10.9.105.64@o2ib4) Feb 10 09:49:44 fir-io1-s1 kernel: Lustre: Skipped 495 previous similar messages Feb 10 09:53:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98729e1cf400, cur 1549821184 expire 1549821034 last 1549820957 Feb 10 09:53:04 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 10 09:59:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 116e39d9-c33f-c321-735f-9ea512fa0ba7 (at 10.9.101.23@o2ib4) Feb 10 09:59:57 fir-io1-s1 kernel: Lustre: Skipped 567 previous similar messages Feb 10 10:09:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d9d3ac4e-0fb3-be83-7c67-dfe4c97facfb (at 10.9.114.9@o2ib4) Feb 10 10:09:57 fir-io1-s1 kernel: Lustre: Skipped 477 previous similar messages Feb 10 10:19:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 4c5b69ea-d1f1-0261-ea03-15f22270fb92 (at 10.9.101.2@o2ib4) Feb 10 10:19:58 fir-io1-s1 kernel: Lustre: Skipped 475 previous similar messages Feb 10 10:27:55 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c685ce6c-10e4-7444-bf47-f6501a7232f0 (at 10.8.8.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a14af400, cur 1549823275 expire 1549823125 last 1549823048 Feb 10 10:27:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 10:29:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5fa2be43-ef73-0106-b068-07493ef16b97 (at 10.8.24.24@o2ib6) in 197 seconds. I think it's dead, and I am evicting it. exp ffff9848000eb000, cur 1549823351 expire 1549823201 last 1549823154 Feb 10 10:29:11 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 10 10:29:27 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) reconnecting Feb 10 10:29:41 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5fa2be43-ef73-0106-b068-07493ef16b97 (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575de9e000, cur 1549823381 expire 1549823231 last 1549823154 Feb 10 10:29:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to cd4830da-6dd3-dc14-e86c-794e3eb4bb0b (at 10.8.8.24@o2ib6) Feb 10 10:29:58 fir-io1-s1 kernel: Lustre: Skipped 643 previous similar messages Feb 10 10:40:01 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f97f4802-8c0b-e9cd-e8d7-93692decf22a (at 10.9.102.8@o2ib4) Feb 10 10:40:01 fir-io1-s1 kernel: Lustre: Skipped 471 previous similar messages Feb 10 10:43:53 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.14.7@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 10 10:43:53 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 10 10:44:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df2000, cur 1549824255 expire 1549824105 last 1549824028 Feb 10 10:44:15 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 10:50:07 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bfe736a6-72da-534a-a0b9-aa8669f81433 (at 10.8.25.11@o2ib6) Feb 10 10:50:07 fir-io1-s1 kernel: Lustre: Skipped 587 previous similar messages Feb 10 10:53:09 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c9c466de-2010-da89-de6a-267ff847464e (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c26800, cur 1549824789 expire 1549824639 last 1549824562 Feb 10 10:53:09 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 10:54:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 313817d5-fec2-02c4-445d-e59ed224bf6e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a14d7400, cur 1549824884 expire 1549824734 last 1549824657 Feb 10 10:54:44 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 10:54:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 313817d5-fec2-02c4-445d-e59ed224bf6e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984804dbf800, cur 1549824895 expire 1549824745 last 1549824668 Feb 10 10:54:55 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 10:58:14 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1b7eb91b-6d34-d2d9-adec-fec070392a7e (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677bc9fc00, cur 1549825094 expire 1549824944 last 1549824867 Feb 10 11:00:11 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2e8b1a97-514f-63aa-1bc8-051eadecacf0 (at 10.9.112.9@o2ib4) Feb 10 11:00:11 fir-io1-s1 kernel: Lustre: Skipped 607 previous similar messages Feb 10 11:10:12 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 54b0700c-697e-0244-02c1-dbdca5773fee (at 10.8.27.7@o2ib6) Feb 10 11:10:12 fir-io1-s1 kernel: Lustre: Skipped 519 previous similar messages Feb 10 11:14:52 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client debffa97-2639-9c31-35f1-ecb330d410d8 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762493800, cur 1549826092 expire 1549825942 last 1549825865 Feb 10 11:14:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 11:20:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6179b8f9-2b15-63f8-671c-eb8d9ecd9187 (at 10.8.16.3@o2ib6) Feb 10 11:20:12 fir-io1-s1 kernel: Lustre: Skipped 575 previous similar messages Feb 10 11:30:14 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 013cf202-b3d4-6a22-f4d9-6c984ce87e6f (at 10.8.7.10@o2ib6) Feb 10 11:30:14 fir-io1-s1 kernel: Lustre: Skipped 511 previous similar messages Feb 10 11:40:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to fa5fc8be-f740-e109-5125-43d4f0bf6a14 (at 10.9.102.33@o2ib4) Feb 10 11:40:18 fir-io1-s1 kernel: Lustre: Skipped 451 previous similar messages Feb 10 11:41:00 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549827652/real 1549827652] req@ffff986713c22700 x1624931155671600/t0(0) o104->fir-OST0006@10.9.112.15@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549827659 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 10 11:41:00 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 10 11:41:14 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549827667/real 1549827667] req@ffff986713c22700 x1624931155671600/t0(0) o104->fir-OST0006@10.9.112.15@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549827674 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 10 11:41:14 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 10 11:41:35 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549827688/real 1549827688] req@ffff986713c22700 x1624931155671600/t0(0) o104->fir-OST0006@10.9.112.15@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549827695 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 10 11:41:35 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 10 11:42:10 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549827723/real 1549827723] req@ffff986713c22700 x1624931155671600/t0(0) o104->fir-OST0006@10.9.112.15@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549827730 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 10 11:42:10 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 10 11:43:20 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549827793/real 1549827793] req@ffff986713c22700 x1624931155671600/t0(0) o104->fir-OST0006@10.9.112.15@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1549827800 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 10 11:43:20 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 10 11:43:27 fir-io1-s1 kernel: LustreError: 96374:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.112.15@o2ib4) failed to reply to blocking AST (req@ffff986713c22700 x1624931155671600 status 0 rc -110), evict it ns: filter-fir-OST0006_UUID lock: ffff985d55713840/0x49e185e92fab217e lrc: 4/0,0 mode: PW/PW res: [0xc40000402:0x9fae5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.9.112.15@o2ib4 remote: 0x43b35d7c79a75a42 expref: 10 pid: 96566 timeout: 175700 lvb_type: 0 Feb 10 11:43:27 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.9.112.15@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Feb 10 11:43:27 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.9.112.15@o2ib4 ns: filter-fir-OST0006_UUID lock: ffff985d55713840/0x49e185e92fab217e lrc: 3/0,0 mode: PW/PW res: [0xc40000402:0x9fae5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.9.112.15@o2ib4 remote: 0x43b35d7c79a75a42 expref: 11 pid: 96566 timeout: 0 lvb_type: 0 Feb 10 11:44:25 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f29c267f-523c-8da1-3024-0978c4db5c38 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768859000, cur 1549827865 expire 1549827715 last 1549827638 Feb 10 11:44:25 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 11:44:33 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f29c267f-523c-8da1-3024-0978c4db5c38 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885b000, cur 1549827873 expire 1549827723 last 1549827646 Feb 10 11:44:35 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f29c267f-523c-8da1-3024-0978c4db5c38 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885d000, cur 1549827875 expire 1549827725 last 1549827648 Feb 10 11:44:38 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f29c267f-523c-8da1-3024-0978c4db5c38 (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576855e000, cur 1549827878 expire 1549827728 last 1549827651 Feb 10 11:50:22 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9cf6d7a8-4898-44eb-2590-b689cf0f2dd8 (at 10.8.6.11@o2ib6) Feb 10 11:50:22 fir-io1-s1 kernel: Lustre: Skipped 483 previous similar messages Feb 10 12:00:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a2c53951-d174-3b52-a3ee-252826248ac1 (at 10.9.112.6@o2ib4) Feb 10 12:00:33 fir-io1-s1 kernel: Lustre: Skipped 513 previous similar messages Feb 10 12:04:14 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b22e05ba-2b75-11fe-7710-708e418387f4 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576dc9d400, cur 1549829054 expire 1549828904 last 1549828827 Feb 10 12:04:14 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 10 12:10:33 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0afc86ec-aba3-e28e-ab47-9d17266465ab (at 10.8.21.34@o2ib6) Feb 10 12:10:33 fir-io1-s1 kernel: Lustre: Skipped 553 previous similar messages Feb 10 12:20:34 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a2c53951-d174-3b52-a3ee-252826248ac1 (at 10.9.112.6@o2ib4) Feb 10 12:20:34 fir-io1-s1 kernel: Lustre: Skipped 472 previous similar messages Feb 10 12:25:24 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9e138298-dd85-93a9-9508-40699f5ed24f (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0d400, cur 1549830324 expire 1549830174 last 1549830097 Feb 10 12:25:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 12:30:45 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 8f9a5846-92d8-9462-f5e9-bd06d534ff2f (at 10.9.107.62@o2ib4) Feb 10 12:30:45 fir-io1-s1 kernel: Lustre: Skipped 461 previous similar messages Feb 10 12:36:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ab320563-d050-ffbe-8413-f1c5b4f61fd6 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857654f0800, cur 1549831018 expire 1549830868 last 1549830791 Feb 10 12:36:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 12:39:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1d7ed545-667f-2ef8-6bba-6c20aaec9c9f (at 10.8.14.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678445c000, cur 1549831176 expire 1549831026 last 1549830949 Feb 10 12:39:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 12:40:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Feb 10 12:40:46 fir-io1-s1 kernel: Lustre: Skipped 443 previous similar messages Feb 10 12:43:18 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b7ea7940-cd9b-1ecc-5677-2e7490fe21e0 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867860b9c00, cur 1549831398 expire 1549831248 last 1549831171 Feb 10 12:43:18 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 10 12:51:11 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b932d3d7-3fae-3581-688a-6fe1d66da337 (at 10.8.13.21@o2ib6) Feb 10 12:51:11 fir-io1-s1 kernel: Lustre: Skipped 402 previous similar messages Feb 10 12:51:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a733ad7e-6dc5-d9cf-2e54-457a75b969b8 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848349e5000, cur 1549831899 expire 1549831749 last 1549831672 Feb 10 12:51:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 12:55:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 49f3f2b2-da2e-90a1-153e-8780a04b76ca (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848349e7c00, cur 1549832147 expire 1549831997 last 1549831920 Feb 10 12:55:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 12:58:36 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8008e591-85d3-189e-54a7-035ff3fafdea (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfe800, cur 1549832316 expire 1549832166 last 1549832089 Feb 10 12:58:36 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 13:01:14 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Feb 10 13:01:14 fir-io1-s1 kernel: Lustre: Skipped 465 previous similar messages Feb 10 13:07:11 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4145f9ae-ef96-c859-ef6a-37d022f944a1 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756587400, cur 1549832831 expire 1549832681 last 1549832604 Feb 10 13:07:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 13:11:19 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 156070c6-6b1a-c523-d65c-fc06e69c00b3 (at 10.9.103.39@o2ib4) Feb 10 13:11:19 fir-io1-s1 kernel: Lustre: Skipped 449 previous similar messages Feb 10 13:21:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 98b7035a-35b8-cd21-9b10-3e7f2a49b7a7 (at 10.8.13.9@o2ib6) Feb 10 13:21:26 fir-io1-s1 kernel: Lustre: Skipped 501 previous similar messages Feb 10 13:22:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c69ac8c6-44e2-66aa-8a62-befc0d012a83 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c6db400, cur 1549833724 expire 1549833574 last 1549833497 Feb 10 13:22:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 13:30:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client eae16858-c424-88aa-9e2d-b84977fedd01 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857630fb000, cur 1549834202 expire 1549834052 last 1549833975 Feb 10 13:30:02 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 13:31:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 32ea102c-c8a6-9ae9-e0f7-d3fc0379beb1 (at 10.8.2.23@o2ib6) Feb 10 13:31:27 fir-io1-s1 kernel: Lustre: Skipped 510 previous similar messages Feb 10 13:41:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f00399f0-46cf-c80f-b0b6-1b044a6fb9c6 (at 10.9.101.40@o2ib4) Feb 10 13:41:33 fir-io1-s1 kernel: Lustre: Skipped 382 previous similar messages Feb 10 13:51:34 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 282817f5-6ae5-e49a-959f-04d16934f700 (at 10.9.101.12@o2ib4) Feb 10 13:51:34 fir-io1-s1 kernel: Lustre: Skipped 365 previous similar messages Feb 10 14:01:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9b247833-563f-e9ef-1dac-d589d25b4b09 (at 10.8.1.8@o2ib6) Feb 10 14:01:37 fir-io1-s1 kernel: Lustre: Skipped 449 previous similar messages Feb 10 14:11:40 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.8.2.15@o2ib6) Feb 10 14:11:40 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 10 14:20:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 94219986-9952-af97-7955-bdd5fbf68578 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00bd800, cur 1549837253 expire 1549837103 last 1549837026 Feb 10 14:20:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 14:21:46 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 6f460c24-f6dc-99fb-dece-05ba714311b0 (at 10.8.27.15@o2ib6) Feb 10 14:21:46 fir-io1-s1 kernel: Lustre: Skipped 413 previous similar messages Feb 10 14:31:46 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Feb 10 14:31:46 fir-io1-s1 kernel: Lustre: Skipped 399 previous similar messages Feb 10 14:41:50 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e766245f-9765-a917-c767-17a05908f74d (at 10.8.20.24@o2ib6) Feb 10 14:41:50 fir-io1-s1 kernel: Lustre: Skipped 442 previous similar messages Feb 10 14:47:28 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 80186a91-5622-f710-936e-60b13bf6ba2a (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758dbd400, cur 1549838848 expire 1549838698 last 1549838621 Feb 10 14:47:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 14:51:53 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Feb 10 14:51:53 fir-io1-s1 kernel: Lustre: Skipped 345 previous similar messages Feb 10 15:01:55 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.16.7@o2ib6) Feb 10 15:01:55 fir-io1-s1 kernel: Lustre: Skipped 447 previous similar messages Feb 10 15:11:57 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 7a5d42a3-072b-ada7-8bd9-6b223c35b055 (at 10.8.20.9@o2ib6) Feb 10 15:11:57 fir-io1-s1 kernel: Lustre: Skipped 283 previous similar messages Feb 10 15:21:59 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 87721f3b-4f03-c138-ffa3-cffa8a052df0 (at 10.8.26.5@o2ib6) Feb 10 15:21:59 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 10 15:23:36 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 83785fe3-bd11-caf3-a67d-c2afc4418cf5 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575eaa7800, cur 1549841016 expire 1549840866 last 1549840789 Feb 10 15:23:36 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 10 15:32:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2336058a-aa9c-4463-bac9-a8ea66369e87 (at 10.8.11.22@o2ib6) Feb 10 15:32:01 fir-io1-s1 kernel: Lustre: Skipped 222 previous similar messages Feb 10 15:42:05 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to efc6b332-a736-88e8-194a-588aa3e05348 (at 10.8.21.36@o2ib6) Feb 10 15:42:05 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 10 15:52:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Feb 10 15:52:08 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 10 15:54:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a5dc9d81-fe89-1d0c-ba56-8e5850561c3c (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784bb9400, cur 1549842879 expire 1549842729 last 1549842652 Feb 10 15:54:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 16:02:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.105.69@o2ib4) Feb 10 16:02:21 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 10 16:12:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e18ec55f-27df-5b55-2c1d-0fb1ae5cad9b (at 10.8.27.28@o2ib6) Feb 10 16:12:23 fir-io1-s1 kernel: Lustre: Skipped 158 previous similar messages Feb 10 16:22:27 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 11122870-d2b4-7c10-4146-c4d58251a247 (at 10.8.16.4@o2ib6) Feb 10 16:22:27 fir-io1-s1 kernel: Lustre: Skipped 263 previous similar messages Feb 10 16:32:34 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 16:32:34 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 10 16:42:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Feb 10 16:42:38 fir-io1-s1 kernel: Lustre: Skipped 315 previous similar messages Feb 10 16:46:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5d54f6bb-ec60-93ce-23fe-959bd0c0b8ca (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857606b0400, cur 1549845978 expire 1549845828 last 1549845751 Feb 10 16:46:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 16:52:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 10 16:52:46 fir-io1-s1 kernel: Lustre: Skipped 213 previous similar messages Feb 10 17:02:46 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 10 17:02:46 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 10 17:05:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e40a0243-3183-a092-90fd-52eaa9dab2c5 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cb7000, cur 1549847108 expire 1549846958 last 1549846881 Feb 10 17:05:08 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 17:12:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 10 17:12:47 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 10 17:13:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f3c028c9-a3d9-d5a7-ddec-09f86ecca0cf (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8b400, cur 1549847603 expire 1549847453 last 1549847376 Feb 10 17:13:23 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 17:18:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3ab5bcaa-8d5e-27d9-5913-f9d8f76ca855 (at 10.8.11.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762f5cc00, cur 1549847893 expire 1549847743 last 1549847666 Feb 10 17:18:13 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 17:22:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8b55087e-0f17-47d4-05be-3ba21cb68a0a (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985836ac4400, cur 1549848130 expire 1549847980 last 1549847903 Feb 10 17:22:10 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 17:22:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d7f0cefd-f5dc-ae79-9fbe-8c42036c5092 (at 10.9.105.21@o2ib4) Feb 10 17:22:53 fir-io1-s1 kernel: Lustre: Skipped 208 previous similar messages Feb 10 17:30:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client df4b39e0-88f5-81ca-04a5-b693fd6c93da (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582bba5c00, cur 1549848618 expire 1549848468 last 1549848391 Feb 10 17:30:18 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 17:32:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 17:32:55 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 10 17:43:08 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c3ba6e64-2791-36bf-c905-71dc1f9569f2 (at 10.9.101.10@o2ib4) Feb 10 17:43:08 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Feb 10 17:53:19 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 17:53:19 fir-io1-s1 kernel: Lustre: Skipped 150 previous similar messages Feb 10 18:03:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Feb 10 18:03:42 fir-io1-s1 kernel: Lustre: Skipped 396 previous similar messages Feb 10 18:13:46 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a4e427fa-d968-911a-150f-37f69bc4903c (at 10.9.106.3@o2ib4) Feb 10 18:13:46 fir-io1-s1 kernel: Lustre: Skipped 431 previous similar messages Feb 10 18:23:49 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3de5156d-8e66-984f-011b-183244b66c19 (at 10.8.18.21@o2ib6) Feb 10 18:23:49 fir-io1-s1 kernel: Lustre: Skipped 208 previous similar messages Feb 10 18:34:06 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 10 18:34:06 fir-io1-s1 kernel: Lustre: Skipped 398 previous similar messages Feb 10 18:44:08 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 10 18:44:08 fir-io1-s1 kernel: Lustre: Skipped 393 previous similar messages Feb 10 18:54:08 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 2ae35386-7dbf-e9e4-77f5-0f6e19b82987 (at 10.8.30.25@o2ib6) Feb 10 18:54:08 fir-io1-s1 kernel: Lustre: Skipped 232 previous similar messages Feb 10 19:04:20 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 22505c78-b9c2-e28a-88c4-7dadc4be41e9 (at 10.9.101.28@o2ib4) Feb 10 19:04:20 fir-io1-s1 kernel: Lustre: Skipped 440 previous similar messages Feb 10 19:14:40 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Feb 10 19:14:40 fir-io1-s1 kernel: Lustre: Skipped 445 previous similar messages Feb 10 19:24:40 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 15339979-51e5-e16d-f976-ff72d24bd14f (at 10.8.9.10@o2ib6) Feb 10 19:24:40 fir-io1-s1 kernel: Lustre: Skipped 233 previous similar messages Feb 10 19:34:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 378375cf-e47b-0dfd-24f9-b821ea9c2298 (at 10.8.22.15@o2ib6) Feb 10 19:34:41 fir-io1-s1 kernel: Lustre: Skipped 535 previous similar messages Feb 10 19:44:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b298c00, cur 1549856641 expire 1549856491 last 1549856414 Feb 10 19:44:01 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 19:44:14 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576dc9ec00, cur 1549856654 expire 1549856504 last 1549856427 Feb 10 19:44:14 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 10 19:44:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c7d587fd-2878-3a53-bb0b-89a81458bb83 (at 10.8.6.5@o2ib6) Feb 10 19:44:47 fir-io1-s1 kernel: Lustre: Skipped 449 previous similar messages Feb 10 19:54:49 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7cd573f6-bbff-21be-bb91-b3a1cd4e1cd3 (at 10.8.7.19@o2ib6) Feb 10 19:54:49 fir-io1-s1 kernel: Lustre: Skipped 367 previous similar messages Feb 10 20:04:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Feb 10 20:04:57 fir-io1-s1 kernel: Lustre: Skipped 480 previous similar messages Feb 10 20:15:06 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 10 20:15:06 fir-io1-s1 kernel: Lustre: Skipped 518 previous similar messages Feb 10 20:25:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 20:25:10 fir-io1-s1 kernel: Lustre: Skipped 259 previous similar messages Feb 10 20:35:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Feb 10 20:35:13 fir-io1-s1 kernel: Lustre: Skipped 453 previous similar messages Feb 10 20:45:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 720160b3-b510-fd4a-7aef-667fe71d4b1d (at 10.8.27.20@o2ib6) Feb 10 20:45:15 fir-io1-s1 kernel: Lustre: Skipped 467 previous similar messages Feb 10 20:55:32 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 10 20:55:32 fir-io1-s1 kernel: Lustre: Skipped 296 previous similar messages Feb 10 21:05:35 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 21:05:35 fir-io1-s1 kernel: Lustre: Skipped 436 previous similar messages Feb 10 21:15:35 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 10 21:15:35 fir-io1-s1 kernel: Lustre: Skipped 468 previous similar messages Feb 10 21:25:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 16fe1f06-91d4-6364-b5d9-1d6caad6f915 (at 10.8.22.22@o2ib6) Feb 10 21:25:36 fir-io1-s1 kernel: Lustre: Skipped 236 previous similar messages Feb 10 21:35:38 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9468a1d3-3abd-8063-5952-288cca0f1dec (at 10.8.27.35@o2ib6) Feb 10 21:35:38 fir-io1-s1 kernel: Lustre: Skipped 489 previous similar messages Feb 10 21:45:43 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to cb55eece-21bc-a42b-f0ac-74d5957e3321 (at 10.8.21.35@o2ib6) Feb 10 21:45:43 fir-io1-s1 kernel: Lustre: Skipped 562 previous similar messages Feb 10 21:55:45 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f2a610e8-4b67-c7b0-e65a-cefd2bd7f74e (at 10.8.13.17@o2ib6) Feb 10 21:55:45 fir-io1-s1 kernel: Lustre: Skipped 213 previous similar messages Feb 10 22:05:45 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 173446cc-39b1-333f-81fc-6684fb678e20 (at 10.8.3.19@o2ib6) Feb 10 22:05:45 fir-io1-s1 kernel: Lustre: Skipped 708 previous similar messages Feb 10 22:15:52 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 254b94e8-a059-c2e7-f0ce-7ccba72fe51c (at 10.8.1.34@o2ib6) Feb 10 22:15:52 fir-io1-s1 kernel: Lustre: Skipped 470 previous similar messages Feb 10 22:19:21 fir-io1-s1 kernel: Lustre: 96760:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549865954/real 1549865954] req@ffff984302e83600 x1624931196538144/t0(0) o106->fir-OST0006@10.8.15.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549865961 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 10 22:19:21 fir-io1-s1 kernel: Lustre: 96760:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 10 22:19:42 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549865975/real 1549865975] req@ffff984abb013c00 x1624931196538160/t0(0) o106->fir-OST0008@10.8.15.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549865982 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 10 22:19:42 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549865975/real 1549865975] req@ffff984b639d3900 x1624931196538112/t0(0) o106->fir-OST0004@10.8.15.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549865982 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 10 22:19:42 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Feb 10 22:19:42 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 10 22:20:52 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549866045/real 1549866045] req@ffff984b639d3900 x1624931196538112/t0(0) o106->fir-OST0004@10.8.15.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549866052 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 10 22:20:52 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 67 previous similar messages Feb 10 22:21:39 fir-io1-s1 kernel: LustreError: 96785:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.15.7@o2ib6) returned error from glimpse AST (req@ffff983d2179ef00 x1624931196538128 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff984d0a5c18c0/0x49e185e9310a8a44 lrc: 3/0,0 mode: PW/PW res: [0xad11e:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x40000000020000 nid: 10.8.15.7@o2ib6 remote: 0x74cb7365634f2667 expref: 6 pid: 96566 timeout: 0 lvb_type: 0 Feb 10 22:21:39 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.15.7@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 10 22:21:39 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1549866099s: evicting client at 10.8.15.7@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff98668dd53840/0x49e185e9310a8a36 lrc: 3/0,0 mode: PW/PW res: [0xac7f5:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.15.7@o2ib6 remote: 0x74cb7365634f25e9 expref: 7 pid: 96566 timeout: 0 lvb_type: 0 Feb 10 22:21:39 fir-io1-s1 kernel: LustreError: 96785:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages Feb 10 22:25:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f854c2f0-b53f-8306-9638-bc37f75b2b94 (at 10.8.8.7@o2ib6) Feb 10 22:25:54 fir-io1-s1 kernel: Lustre: Skipped 334 previous similar messages Feb 10 22:35:55 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 10 22:35:55 fir-io1-s1 kernel: Lustre: Skipped 302 previous similar messages Feb 10 22:41:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5b70eaeb-9b1d-7d91-4f54-a3b1ba65e969 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e44400, cur 1549867267 expire 1549867117 last 1549867040 Feb 10 22:41:07 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 10 22:45:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 10 22:45:58 fir-io1-s1 kernel: Lustre: Skipped 524 previous similar messages Feb 10 22:55:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 10 22:55:58 fir-io1-s1 kernel: Lustre: Skipped 406 previous similar messages Feb 10 22:58:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 02370cb9-c120-e247-f0d2-7797a4f951b4 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4c000, cur 1549868337 expire 1549868187 last 1549868110 Feb 10 22:58:57 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 10 23:03:23 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 846fc7c1-9fac-9eea-ec19-a0c2ec37db81 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848349e2c00, cur 1549868603 expire 1549868453 last 1549868376 Feb 10 23:03:23 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 23:05:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Feb 10 23:05:58 fir-io1-s1 kernel: Lustre: Skipped 364 previous similar messages Feb 10 23:16:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e93ce35d-84e4-3b34-458f-23d98921edba (at 10.8.24.33@o2ib6) Feb 10 23:16:04 fir-io1-s1 kernel: Lustre: Skipped 500 previous similar messages Feb 10 23:18:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 14b8a5cc-c311-1ba9-3b8d-0cba9e5112d8 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575be1f000, cur 1549869524 expire 1549869374 last 1549869297 Feb 10 23:18:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 10 23:26:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 49092156-6580-6ae9-b01b-066339b65a21 (at 10.8.21.33@o2ib6) Feb 10 23:26:09 fir-io1-s1 kernel: Lustre: Skipped 538 previous similar messages Feb 10 23:28:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4134d24b-e8b5-caf8-c4be-35abdb4083c3 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811215800, cur 1549870084 expire 1549869934 last 1549869857 Feb 10 23:28:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 10 23:36:10 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 4b161527-b7d4-cb96-22b6-3891827325ad (at 10.9.112.2@o2ib4) Feb 10 23:36:10 fir-io1-s1 kernel: Lustre: Skipped 323 previous similar messages Feb 10 23:46:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fb5e4f4f-a645-fbec-7b80-08c8d8a2fea0 (at 10.8.1.12@o2ib6) Feb 10 23:46:12 fir-io1-s1 kernel: Lustre: Skipped 493 previous similar messages Feb 10 23:56:12 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 801c5583-df50-ef54-ebf8-d76e7be7922a (at 10.8.21.18@o2ib6) Feb 10 23:56:12 fir-io1-s1 kernel: Lustre: Skipped 462 previous similar messages Feb 10 23:56:24 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 047894be-2394-9241-fd94-a892fc926bf2 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780d5cc00, cur 1549871784 expire 1549871634 last 1549871557 Feb 10 23:56:24 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 00:06:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 11 00:06:16 fir-io1-s1 kernel: Lustre: Skipped 584 previous similar messages Feb 11 00:07:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9b9fd958-7168-f6e9-10dc-dbc17783e1de (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c3ec00, cur 1549872476 expire 1549872326 last 1549872249 Feb 11 00:07:56 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 00:16:17 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6) Feb 11 00:16:17 fir-io1-s1 kernel: Lustre: Skipped 375 previous similar messages Feb 11 00:25:59 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8e73ad7d-038e-2a91-afb3-265714da253a (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858388a5800, cur 1549873559 expire 1549873409 last 1549873332 Feb 11 00:25:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 00:26:17 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.101.7@o2ib4) Feb 11 00:26:17 fir-io1-s1 kernel: Lustre: Skipped 390 previous similar messages Feb 11 00:36:19 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 11 00:36:19 fir-io1-s1 kernel: Lustre: Skipped 245 previous similar messages Feb 11 00:43:11 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client aa69108d-2cd6-ac93-4ae7-f875db9a6dc0 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780862000, cur 1549874591 expire 1549874441 last 1549874364 Feb 11 00:43:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 00:46:20 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 11 00:46:20 fir-io1-s1 kernel: Lustre: Skipped 287 previous similar messages Feb 11 00:53:36 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d2f1ad7b-d247-38be-c80e-5fd306ed0938 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a0ac400, cur 1549875216 expire 1549875066 last 1549874989 Feb 11 00:53:36 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 00:56:25 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8a305ec8-58cd-38d7-7085-a98f5d22aa5b (at 10.9.107.47@o2ib4) Feb 11 00:56:25 fir-io1-s1 kernel: Lustre: Skipped 425 previous similar messages Feb 11 01:06:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1790ad01-c111-7e22-129d-270ee03f91c0 (at 10.8.10.8@o2ib6) Feb 11 01:06:26 fir-io1-s1 kernel: Lustre: Skipped 237 previous similar messages Feb 11 01:16:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 135b49e4-9bbb-43fc-66e9-1f7ec8c75a96 (at 10.9.113.3@o2ib4) Feb 11 01:16:27 fir-io1-s1 kernel: Lustre: Skipped 325 previous similar messages Feb 11 01:18:15 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c48f0410-9307-d7ff-8701-741433c8b30e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832798000, cur 1549876695 expire 1549876545 last 1549876468 Feb 11 01:18:15 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 01:26:27 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 935e7eb3-1ff6-7dda-ab9a-d14a4b5f1855 (at 10.9.103.32@o2ib4) Feb 11 01:26:27 fir-io1-s1 kernel: Lustre: Skipped 285 previous similar messages Feb 11 01:36:33 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) Feb 11 01:36:33 fir-io1-s1 kernel: Lustre: Skipped 242 previous similar messages Feb 11 01:46:44 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3352dbfd-d63f-74b6-bc30-6f22efce7e27 (at 10.8.7.3@o2ib6) Feb 11 01:46:44 fir-io1-s1 kernel: Lustre: Skipped 465 previous similar messages Feb 11 01:56:44 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 503151d3-e911-b92b-974a-493626aee137 (at 10.8.15.8@o2ib6) Feb 11 01:56:44 fir-io1-s1 kernel: Lustre: Skipped 387 previous similar messages Feb 11 01:59:13 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b17149e0-f4d0-7861-8f04-293d4bf02102 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ea46400, cur 1549879153 expire 1549879003 last 1549878926 Feb 11 02:06:52 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 503151d3-e911-b92b-974a-493626aee137 (at 10.8.15.8@o2ib6) Feb 11 02:06:52 fir-io1-s1 kernel: Lustre: Skipped 263 previous similar messages Feb 11 02:08:22 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 79c829d7-b69b-d592-0b18-ba944f3d3d45 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf08000, cur 1549879702 expire 1549879552 last 1549879475 Feb 11 02:08:22 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 11 02:17:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 11 02:17:00 fir-io1-s1 kernel: Lustre: Skipped 340 previous similar messages Feb 11 02:17:07 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7b79d0e2-7d96-494a-23c5-d664fb94353a (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00ec400, cur 1549880227 expire 1549880077 last 1549880000 Feb 11 02:17:07 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 02:27:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2336058a-aa9c-4463-bac9-a8ea66369e87 (at 10.8.11.22@o2ib6) Feb 11 02:27:01 fir-io1-s1 kernel: Lustre: Skipped 345 previous similar messages Feb 11 02:33:45 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a09d7781-bfe6-b29a-d4a6-237ce4594fd1 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985763316000, cur 1549881225 expire 1549881075 last 1549880998 Feb 11 02:33:45 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 02:37:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 81ece3a3-4ce6-c10f-0f1a-dfedb115d731 (at 10.8.21.2@o2ib6) Feb 11 02:37:09 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 11 02:41:51 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client dba19b0d-a73a-9666-ab00-14e2c4360649 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ea43c00, cur 1549881711 expire 1549881561 last 1549881484 Feb 11 02:41:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 02:47:10 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 5f9695c8-8953-f70d-cc1c-5dc8656027f2 (at 10.9.101.20@o2ib4) Feb 11 02:47:10 fir-io1-s1 kernel: Lustre: Skipped 364 previous similar messages Feb 11 02:50:19 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e5cf781a-b2b2-298e-962f-c2788920b59d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00bcc00, cur 1549882219 expire 1549882069 last 1549881992 Feb 11 02:50:19 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 02:57:12 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 11 02:57:12 fir-io1-s1 kernel: Lustre: Skipped 391 previous similar messages Feb 11 03:07:29 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 83fafa06-153a-a1fd-dd02-4b2d3ef6c90e (at 10.9.114.2@o2ib4) Feb 11 03:07:29 fir-io1-s1 kernel: Lustre: Skipped 211 previous similar messages Feb 11 03:07:50 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ce537b46-fded-c277-fe11-f1c21a81918b (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984835172800, cur 1549883270 expire 1549883120 last 1549883043 Feb 11 03:07:50 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 03:17:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 03:17:41 fir-io1-s1 kernel: Lustre: Skipped 364 previous similar messages Feb 11 03:24:35 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 43c2ca0b-6906-1031-26a5-d79023293574 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c04400, cur 1549884275 expire 1549884125 last 1549884048 Feb 11 03:24:35 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 03:27:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 03:27:41 fir-io1-s1 kernel: Lustre: Skipped 358 previous similar messages Feb 11 03:37:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 03:37:41 fir-io1-s1 kernel: Lustre: Skipped 293 previous similar messages Feb 11 03:47:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 03:47:41 fir-io1-s1 kernel: Lustre: Skipped 327 previous similar messages Feb 11 03:53:06 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 245a049e-9b3f-8f27-db3b-53ec4ff00d6a (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cdc800, cur 1549885986 expire 1549885836 last 1549885759 Feb 11 03:53:06 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 03:57:41 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 03:57:41 fir-io1-s1 kernel: Lustre: Skipped 321 previous similar messages Feb 11 04:07:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 04:07:41 fir-io1-s1 kernel: Lustre: Skipped 187 previous similar messages Feb 11 04:11:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6b80a9a6-a6c1-58c4-7b30-59f57c9c0b92 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c672400, cur 1549887084 expire 1549886934 last 1549886857 Feb 11 04:13:45 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 975ea4f2-c7a8-67b2-2e53-2ddb935cc0e9 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f7000, cur 1549887225 expire 1549887075 last 1549886998 Feb 11 04:13:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 04:17:43 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to c99c064f-83fa-682c-8def-f011ca1d2686 (at 10.9.113.12@o2ib4) Feb 11 04:17:43 fir-io1-s1 kernel: Lustre: Skipped 269 previous similar messages Feb 11 04:27:44 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 65ad79ee-001c-5939-0b4a-f0cbeb92b2c0 (at 10.8.22.1@o2ib6) Feb 11 04:27:44 fir-io1-s1 kernel: Lustre: Skipped 250 previous similar messages Feb 11 04:37:49 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 11 04:37:49 fir-io1-s1 kernel: Lustre: Skipped 220 previous similar messages Feb 11 04:47:50 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 11 04:47:50 fir-io1-s1 kernel: Lustre: Skipped 252 previous similar messages Feb 11 04:54:57 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 66c67ebd-4385-f2d6-e9cf-17c29bb89b59 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986806758800, cur 1549889697 expire 1549889547 last 1549889470 Feb 11 04:57:59 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 11 04:57:59 fir-io1-s1 kernel: Lustre: Skipped 201 previous similar messages Feb 11 05:08:00 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b0f4a89e-7973-eb31-a1c9-fdc42b6cc4f6 (at 10.8.18.25@o2ib6) Feb 11 05:08:00 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 11 05:18:10 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8427e8d1-34c3-886c-dab9-5ccc235f41c4 (at 10.9.107.5@o2ib4) Feb 11 05:18:10 fir-io1-s1 kernel: Lustre: Skipped 220 previous similar messages Feb 11 05:28:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 11 05:28:18 fir-io1-s1 kernel: Lustre: Skipped 263 previous similar messages Feb 11 05:31:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 53f44e1f-5e71-cb4c-6e10-bb9a56755b69 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe373c00, cur 1549891893 expire 1549891743 last 1549891666 Feb 11 05:31:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 05:38:18 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6) Feb 11 05:38:18 fir-io1-s1 kernel: Lustre: Skipped 276 previous similar messages Feb 11 05:47:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 6ef69a54-2164-5a52-b2ce-fca07a329ed2 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848349e5800, cur 1549892824 expire 1549892674 last 1549892597 Feb 11 05:47:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 05:48:21 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ec7c15c1-1122-48df-e09c-cacc05cb75a8 (at 10.8.1.15@o2ib6) Feb 11 05:48:21 fir-io1-s1 kernel: Lustre: Skipped 269 previous similar messages Feb 11 05:55:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 028d6433-9e7d-1b84-c8b7-1bb2a8570ec4 (at 10.8.1.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581bfe8400, cur 1549893316 expire 1549893166 last 1549893089 Feb 11 05:58:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 11 05:58:23 fir-io1-s1 kernel: Lustre: Skipped 286 previous similar messages Feb 11 06:08:27 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9f0fc7f1-3017-a377-d5ca-3f45fac7d96c (at 10.9.106.2@o2ib4) Feb 11 06:08:27 fir-io1-s1 kernel: Lustre: Skipped 259 previous similar messages Feb 11 06:09:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d863f6a6-22b5-d6d1-9e85-e87fb7a2072d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987823a6b000, cur 1549894151 expire 1549894001 last 1549893924 Feb 11 06:09:11 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 06:09:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d863f6a6-22b5-d6d1-9e85-e87fb7a2072d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987823a69000, cur 1549894166 expire 1549894016 last 1549893939 Feb 11 06:18:29 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ef406a33-dbdc-c381-15c5-4fb662abecc1 (at 10.8.27.10@o2ib6) Feb 11 06:18:29 fir-io1-s1 kernel: Lustre: Skipped 188 previous similar messages Feb 11 06:28:35 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.27.30@o2ib6) Feb 11 06:28:35 fir-io1-s1 kernel: Lustre: Skipped 206 previous similar messages Feb 11 06:37:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d2d1ad32-0bae-d301-940f-331201d0cf4d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b917400, cur 1549895876 expire 1549895726 last 1549895649 Feb 11 06:37:56 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 06:38:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d2d1ad32-0bae-d301-940f-331201d0cf4d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8aa000, cur 1549895887 expire 1549895737 last 1549895660 Feb 11 06:38:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d6335190-14e9-cc30-2081-663fdf52e20a (at 10.9.102.12@o2ib4) Feb 11 06:38:36 fir-io1-s1 kernel: Lustre: Skipped 256 previous similar messages Feb 11 06:47:04 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549896417/real 1549896417] req@ffff9877a13b9b00 x1624931206299792/t0(0) o106->fir-OST0006@10.8.11.22@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549896424 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 11 06:47:04 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Feb 11 06:47:25 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549896438/real 1549896438] req@ffff9877a13b9b00 x1624931206299792/t0(0) o106->fir-OST0006@10.8.11.22@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549896445 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 11 06:47:25 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 11 06:47:38 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a897fd4a-55d1-835c-1959-c63e18a93dfc (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762865400, cur 1549896458 expire 1549896308 last 1549896231 Feb 11 06:47:38 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 06:48:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 11 06:48:38 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 11 06:56:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 17407aee-6ef6-fd16-00b0-40f05e3e5420 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b9c800, cur 1549896960 expire 1549896810 last 1549896733 Feb 11 06:56:00 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 06:58:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 4e48592f-b97d-5c93-9da4-86c872d7a486 (at 10.9.107.43@o2ib4) Feb 11 06:58:38 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 11 07:08:44 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6) Feb 11 07:08:44 fir-io1-s1 kernel: Lustre: Skipped 262 previous similar messages Feb 11 07:18:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 07:18:55 fir-io1-s1 kernel: Lustre: Skipped 299 previous similar messages Feb 11 07:29:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 82bcfbd3-d6e5-0967-d3f2-c921c94e988c (at 10.9.105.71@o2ib4) Feb 11 07:29:07 fir-io1-s1 kernel: Lustre: Skipped 358 previous similar messages Feb 11 07:36:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c22e763b-2712-8624-a4bb-1c3145d32fd9 (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811fb000, cur 1549899378 expire 1549899228 last 1549899151 Feb 11 07:36:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 07:36:22 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c22e763b-2712-8624-a4bb-1c3145d32fd9 (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f67a400, cur 1549899382 expire 1549899232 last 1549899155 Feb 11 07:36:22 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 07:39:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.112.17@o2ib4) Feb 11 07:39:08 fir-io1-s1 kernel: Lustre: Skipped 290 previous similar messages Feb 11 07:49:15 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 04c67229-21fb-0235-15ed-cccc9063a531 (at 10.8.27.18@o2ib6) Feb 11 07:49:15 fir-io1-s1 kernel: Lustre: Skipped 280 previous similar messages Feb 11 07:59:15 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 31f9f1e5-0053-c9bc-655f-d68cfd64847e (at 10.8.21.32@o2ib6) Feb 11 07:59:15 fir-io1-s1 kernel: Lustre: Skipped 441 previous similar messages Feb 11 08:01:23 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 03a37b2d-0fa0-08c5-eb22-42785e11e92d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd266c00, cur 1549900883 expire 1549900733 last 1549900656 Feb 11 08:09:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 08:09:17 fir-io1-s1 kernel: Lustre: Skipped 344 previous similar messages Feb 11 08:13:08 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 76a65351-c8d5-a05c-00d8-8bde0063ecac (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800f57000, cur 1549901588 expire 1549901438 last 1549901361 Feb 11 08:13:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 08:13:41 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549901614/real 1549901614] req@ffff9852a9220600 x1624931208076752/t0(0) o106->fir-OST0002@10.8.22.28@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549901621 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 11 08:13:41 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 11 08:13:48 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549901621/real 1549901621] req@ffff98629690a400 x1624931208076768/t0(0) o106->fir-OST0000@10.8.22.28@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549901628 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 11 08:14:02 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549901635/real 1549901635] req@ffff98629690a400 x1624931208076768/t0(0) o106->fir-OST0000@10.8.22.28@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549901642 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 11 08:14:02 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 11 08:14:23 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549901656/real 1549901656] req@ffff9852a9220600 x1624931208076752/t0(0) o106->fir-OST0002@10.8.22.28@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549901663 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 11 08:14:23 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 11 08:14:58 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549901691/real 1549901691] req@ffff98629690a400 x1624931208076768/t0(0) o106->fir-OST0000@10.8.22.28@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549901698 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 11 08:14:58 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 11 08:16:08 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549901761/real 1549901761] req@ffff9852a9220600 x1624931208076752/t0(0) o106->fir-OST0002@10.8.22.28@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549901768 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 11 08:16:08 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Feb 11 08:16:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ab1f55c6-25d8-b2cf-818d-4cc69ca36dd0 (at 10.8.22.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811210c00, cur 1549901781 expire 1549901631 last 1549901554 Feb 11 08:16:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 08:18:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4b8453cd-aefc-d81d-671f-39121770f943 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98680476dc00, cur 1549901934 expire 1549901784 last 1549901707 Feb 11 08:18:54 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 11 08:19:21 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 08:19:21 fir-io1-s1 kernel: Lustre: Skipped 392 previous similar messages Feb 11 08:21:13 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client eedb9fc4-eb23-5c2e-0221-01d21528a876 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785e42c00, cur 1549902073 expire 1549901923 last 1549901846 Feb 11 08:21:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 08:21:38 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client eedb9fc4-eb23-5c2e-0221-01d21528a876 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762b63000, cur 1549902098 expire 1549901948 last 1549901871 Feb 11 08:27:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ebc348ae-5a1a-6bcc-1254-ed696c20d527 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677c02c800, cur 1549902464 expire 1549902314 last 1549902237 Feb 11 08:29:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 11 08:29:27 fir-io1-s1 kernel: Lustre: Skipped 494 previous similar messages Feb 11 08:35:24 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 37bf9c8c-52f6-ddcc-ad24-ef4d27fc2542 (at 10.8.1.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804769c00, cur 1549902924 expire 1549902774 last 1549902697 Feb 11 08:35:24 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 08:39:31 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3b95c04d-b4e9-7c45-bdb5-b89e27b35d9e (at 10.9.105.27@o2ib4) Feb 11 08:39:31 fir-io1-s1 kernel: Lustre: Skipped 369 previous similar messages Feb 11 08:49:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to df48e9ec-bd5c-a138-66b8-99e9c0451d54 (at 10.9.107.46@o2ib4) Feb 11 08:49:32 fir-io1-s1 kernel: Lustre: Skipped 310 previous similar messages Feb 11 08:59:34 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 11 08:59:34 fir-io1-s1 kernel: Lustre: Skipped 302 previous similar messages Feb 11 09:09:34 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 20d572d4-8f70-51c7-3e05-1d3dc78a02cc (at 10.8.27.16@o2ib6) Feb 11 09:09:34 fir-io1-s1 kernel: Lustre: Skipped 492 previous similar messages Feb 11 09:11:46 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 460ae552-4e39-8e7b-d7f5-5ba3d2e4cf2e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fbda2400, cur 1549905106 expire 1549904956 last 1549904879 Feb 11 09:11:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 09:19:36 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 031d5ef2-efa4-2e84-d94c-8ad2acba90a2 (at 10.8.27.17@o2ib6) Feb 11 09:19:36 fir-io1-s1 kernel: Lustre: Skipped 636 previous similar messages Feb 11 09:29:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 39b74f02-0c4c-fd51-e621-4bd6eb7173c0 (at 10.9.103.18@o2ib4) Feb 11 09:29:37 fir-io1-s1 kernel: Lustre: Skipped 530 previous similar messages Feb 11 09:31:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 71866cd3-749e-3d62-ea2c-232151da7f87 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581ef41000, cur 1549906260 expire 1549906110 last 1549906033 Feb 11 09:39:39 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 7030631e-b3d2-9eed-f765-9117cb5ba8a4 (at 10.9.103.35@o2ib4) Feb 11 09:39:39 fir-io1-s1 kernel: Lustre: Skipped 662 previous similar messages Feb 11 09:49:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 95777d32-4667-7d7f-8bf0-f1108b8f2ac8 (at 10.8.2.31@o2ib6) Feb 11 09:49:41 fir-io1-s1 kernel: Lustre: Skipped 473 previous similar messages Feb 11 09:59:41 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c370136b-e8a1-fb50-4c1f-974b289a34b5 (at 10.8.23.22@o2ib6) Feb 11 09:59:41 fir-io1-s1 kernel: Lustre: Skipped 417 previous similar messages Feb 11 10:08:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d5d3317b-c244-e390-c667-44821c70095e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f4d000, cur 1549908490 expire 1549908340 last 1549908263 Feb 11 10:08:10 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 11 10:09:44 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 16db5e03-e2d9-f103-4a0b-78f283c497a4 (at 10.8.3.2@o2ib6) Feb 11 10:09:44 fir-io1-s1 kernel: Lustre: Skipped 393 previous similar messages Feb 11 10:19:44 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 37ae9ee6-0608-3719-0e67-b987ecb945fa (at 10.8.6.10@o2ib6) Feb 11 10:19:44 fir-io1-s1 kernel: Lustre: Skipped 432 previous similar messages Feb 11 10:20:27 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 7747e685-e761-7cc3-41b6-aa87888ca308 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768457000, cur 1549909227 expire 1549909077 last 1549909000 Feb 11 10:28:47 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6ac883fb-a075-ce29-9340-b3f63fbb31e4 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe374000, cur 1549909727 expire 1549909577 last 1549909500 Feb 11 10:28:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 10:29:55 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 10:29:55 fir-io1-s1 kernel: Lustre: Skipped 619 previous similar messages Feb 11 10:38:34 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client d0863b2f-1d55-6e08-151d-db6e01d8478e (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e52c00, cur 1549910314 expire 1549910164 last 1549910087 Feb 11 10:40:04 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3cc732cd-cd8a-e01a-1ca8-de6a3b7d52cc (at 10.8.25.8@o2ib6) Feb 11 10:40:04 fir-io1-s1 kernel: Lustre: Skipped 408 previous similar messages Feb 11 10:50:16 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to cf9d17c1-1ffb-39b7-f814-1f005dbcb1a0 (at 10.9.101.24@o2ib4) Feb 11 10:50:16 fir-io1-s1 kernel: Lustre: Skipped 363 previous similar messages Feb 11 10:53:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d7b5cf6d-76c4-c724-0a7d-d27d9c5017d1 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b264400, cur 1549911217 expire 1549911067 last 1549910990 Feb 11 10:58:40 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7e48463b-3270-7d7a-36ee-975286e3213a (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef12c00, cur 1549911520 expire 1549911370 last 1549911293 Feb 11 10:59:43 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 60086963-2a35-ce86-8df1-420855a847a8 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857630f9800, cur 1549911583 expire 1549911433 last 1549911356 Feb 11 10:59:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 11:00:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Feb 11 11:00:20 fir-io1-s1 kernel: Lustre: Skipped 443 previous similar messages Feb 11 11:10:31 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e4f08268-1c0f-53f2-ca47-f50f478ec6f9 (at 10.8.18.3@o2ib6) Feb 11 11:10:31 fir-io1-s1 kernel: Lustre: Skipped 469 previous similar messages Feb 11 11:13:55 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0c34da0d-723f-4adc-7101-0a36b660b545 (at 10.8.24.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986813a73c00, cur 1549912435 expire 1549912285 last 1549912208 Feb 11 11:13:55 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 11 11:20:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 11 11:20:33 fir-io1-s1 kernel: Lustre: Skipped 455 previous similar messages Feb 11 11:30:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to d9657556-3698-de72-acc2-cb9f2581779e (at 10.9.106.5@o2ib4) Feb 11 11:30:33 fir-io1-s1 kernel: Lustre: Skipped 380 previous similar messages Feb 11 11:40:33 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3d122a91-53f0-f449-1f10-d08490897e63 (at 10.9.106.65@o2ib4) Feb 11 11:40:33 fir-io1-s1 kernel: Lustre: Skipped 446 previous similar messages Feb 11 11:47:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7617920e-f4f2-3764-2e41-ab08f91e63ae (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985838d8e800, cur 1549914459 expire 1549914309 last 1549914232 Feb 11 11:47:39 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 11 11:50:38 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 054698ea-f84b-bd00-f4ed-c64e725d9902 (at 10.8.1.2@o2ib6) Feb 11 11:50:38 fir-io1-s1 kernel: Lustre: Skipped 486 previous similar messages Feb 11 12:00:45 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 12:00:45 fir-io1-s1 kernel: Lustre: Skipped 355 previous similar messages Feb 11 12:06:20 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1c165ad6-3d34-e17c-f84d-cefa71258e01 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a14a9400, cur 1549915580 expire 1549915430 last 1549915353 Feb 11 12:06:20 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 11 12:10:46 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dbe81c3f-e038-b02e-a6dc-aaba56293b77 (at 10.8.2.19@o2ib6) Feb 11 12:10:46 fir-io1-s1 kernel: Lustre: Skipped 412 previous similar messages Feb 11 12:19:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c5fd14ab-5a8b-cc5a-3993-ae003dfa190e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583bbaec00, cur 1549916350 expire 1549916200 last 1549916123 Feb 11 12:19:10 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 11 12:20:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2c0acc5d-6891-c1a2-a8f5-ae544f6710b0 (at 10.8.1.20@o2ib6) Feb 11 12:20:50 fir-io1-s1 kernel: Lustre: Skipped 362 previous similar messages Feb 11 12:30:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d01707bf-d8db-a4c3-f544-1f9ecca8f036 (at 10.8.18.29@o2ib6) Feb 11 12:30:50 fir-io1-s1 kernel: Lustre: Skipped 355 previous similar messages Feb 11 12:33:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0e96810b-d4c3-0ba3-47fc-1e6ff4eeb00f (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677c028800, cur 1549917180 expire 1549917030 last 1549916953 Feb 11 12:33:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 12:40:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1c418863-78cc-8f23-893e-27e5ce2dfd94 (at 10.9.101.71@o2ib4) Feb 11 12:40:50 fir-io1-s1 kernel: Lustre: Skipped 303 previous similar messages Feb 11 12:43:10 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client de755408-acf3-b883-333b-5a9368ce9896 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b29a800, cur 1549917790 expire 1549917640 last 1549917563 Feb 11 12:43:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 12:43:11 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client de755408-acf3-b883-333b-5a9368ce9896 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784642c00, cur 1549917791 expire 1549917641 last 1549917564 Feb 11 12:43:11 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 11 12:43:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client de755408-acf3-b883-333b-5a9368ce9896 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768453800, cur 1549917793 expire 1549917643 last 1549917566 Feb 11 12:43:13 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 11 12:50:51 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 94c2dba8-b225-c91a-753d-91a7f0495a0f (at 10.9.101.8@o2ib4) Feb 11 12:50:51 fir-io1-s1 kernel: Lustre: Skipped 445 previous similar messages Feb 11 13:00:52 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4600638c-d686-27e8-2646-fd49b60d2ae1 (at 10.9.102.13@o2ib4) Feb 11 13:00:52 fir-io1-s1 kernel: Lustre: Skipped 547 previous similar messages Feb 11 13:08:05 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client fb87ca38-1da5-4edb-617b-5f2345c093fd (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678448f000, cur 1549919285 expire 1549919135 last 1549919058 Feb 11 13:08:05 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 11 13:10:55 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.104.4@o2ib4) Feb 11 13:10:55 fir-io1-s1 kernel: Lustre: Skipped 515 previous similar messages Feb 11 13:14:25 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a3f211ad-b5e5-e436-db7e-d3de45f9861d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985815684000, cur 1549919665 expire 1549919515 last 1549919438 Feb 11 13:14:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 13:20:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1b71b1a8-fdc3-550f-13e6-42b0376dd743 (at 10.9.112.8@o2ib4) Feb 11 13:20:56 fir-io1-s1 kernel: Lustre: Skipped 454 previous similar messages Feb 11 13:27:43 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d5e4ed35-a88c-665e-f99f-2b1c5dab2580 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868f9400, cur 1549920463 expire 1549920313 last 1549920236 Feb 11 13:27:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 13:30:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3dd7b3c3-369e-14ba-c881-c252e5dc17a0 (at 10.8.8.27@o2ib6) Feb 11 13:30:58 fir-io1-s1 kernel: Lustre: Skipped 449 previous similar messages Feb 11 13:40:59 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fbe187bd-3c7e-1c1e-2397-90b673b213a7 (at 10.9.115.3@o2ib4) Feb 11 13:40:59 fir-io1-s1 kernel: Lustre: Skipped 549 previous similar messages Feb 11 13:51:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 414dcc40-1a1f-dafe-b9b7-84383e8013e5 (at 10.8.4.29@o2ib6) Feb 11 13:51:09 fir-io1-s1 kernel: Lustre: Skipped 341 previous similar messages Feb 11 13:58:01 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 30604a56-9dd6-1986-ffe4-f914f7c90bb5 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786831400, cur 1549922281 expire 1549922131 last 1549922054 Feb 11 13:58:01 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 14:01:15 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 70c3879e-e355-8a86-c7f6-86725077b527 (at 10.8.3.28@o2ib6) Feb 11 14:01:15 fir-io1-s1 kernel: Lustre: Skipped 394 previous similar messages Feb 11 14:11:16 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to cf9d17c1-1ffb-39b7-f814-1f005dbcb1a0 (at 10.9.101.24@o2ib4) Feb 11 14:11:16 fir-io1-s1 kernel: Lustre: Skipped 353 previous similar messages Feb 11 14:21:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c694e053-04d0-ee79-c9a4-0ace9e2f2c9a (at 10.8.3.29@o2ib6) Feb 11 14:21:17 fir-io1-s1 kernel: Lustre: Skipped 421 previous similar messages Feb 11 14:24:13 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4a72c43c-b639-3519-2806-50e34cdc9168 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786835800, cur 1549923853 expire 1549923703 last 1549923626 Feb 11 14:24:13 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 14:24:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4a72c43c-b639-3519-2806-50e34cdc9168 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762860000, cur 1549923855 expire 1549923705 last 1549923628 Feb 11 14:24:15 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 14:31:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ca76035c-fba3-91d6-92f5-1fd97493e3e8 (at 10.9.107.4@o2ib4) Feb 11 14:31:17 fir-io1-s1 kernel: Lustre: Skipped 456 previous similar messages Feb 11 14:41:24 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 8f530db8-807a-fdf0-2880-6c939864abc5 (at 10.9.115.8@o2ib4) Feb 11 14:41:24 fir-io1-s1 kernel: Lustre: Skipped 532 previous similar messages Feb 11 14:51:27 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 045f5984-f159-09cd-da98-0d87730fa119 (at 10.8.4.16@o2ib6) Feb 11 14:51:27 fir-io1-s1 kernel: Lustre: Skipped 443 previous similar messages Feb 11 15:01:27 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3d8dfe83-25e5-247f-c731-de21f1e85c71 (at 10.8.12.10@o2ib6) Feb 11 15:01:27 fir-io1-s1 kernel: Lustre: Skipped 594 previous similar messages Feb 11 15:11:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 34fa37c4-98f8-0ae7-6c7a-1133b926560e (at 10.9.101.19@o2ib4) Feb 11 15:11:30 fir-io1-s1 kernel: Lustre: Skipped 423 previous similar messages Feb 11 15:21:30 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 70c3879e-e355-8a86-c7f6-86725077b527 (at 10.8.3.28@o2ib6) Feb 11 15:21:30 fir-io1-s1 kernel: Lustre: Skipped 360 previous similar messages Feb 11 15:31:36 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 15:31:36 fir-io1-s1 kernel: Lustre: Skipped 484 previous similar messages Feb 11 15:36:20 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e53c6d05-da5b-3806-db03-80005c8cadeb (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785781400, cur 1549928180 expire 1549928030 last 1549927953 Feb 11 15:36:20 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 11 15:41:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 25268aa7-8aba-977f-1700-754dcdcbd041 (at 10.9.107.14@o2ib4) Feb 11 15:41:38 fir-io1-s1 kernel: Lustre: Skipped 328 previous similar messages Feb 11 15:51:42 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to e2d946d5-12d9-285b-e4ca-825370f3d1df (at 10.9.101.22@o2ib4) Feb 11 15:51:42 fir-io1-s1 kernel: Lustre: Skipped 422 previous similar messages Feb 11 16:01:45 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to cf27932c-5cfb-509a-c7ce-6753e8ed5f45 (at 10.8.0.66@o2ib6) Feb 11 16:01:45 fir-io1-s1 kernel: Lustre: Skipped 533 previous similar messages Feb 11 16:11:47 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Feb 11 16:11:47 fir-io1-s1 kernel: Lustre: Skipped 448 previous similar messages Feb 11 16:21:29 fir-io1-s1 kernel: Lustre: 96255:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549930878/real 1549930878] req@ffff9862d15b0f00 x1624931411016192/t0(0) o106->fir-OST0008@10.8.11.22@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549930889 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 11 16:21:29 fir-io1-s1 kernel: Lustre: 96255:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 11 16:21:49 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) Feb 11 16:21:49 fir-io1-s1 kernel: Lustre: Skipped 547 previous similar messages Feb 11 16:21:51 fir-io1-s1 kernel: Lustre: 96569:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1549930900/real 1549930900] req@ffff9847d66e0c00 x1624931411016208/t0(0) o106->fir-OST000a@10.8.11.22@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1549930911 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 11 16:21:51 fir-io1-s1 kernel: Lustre: 96569:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 11 16:22:07 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f936710e-e7d8-651c-2b1b-7cc8a68ad371 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a37f2800, cur 1549930927 expire 1549930777 last 1549930700 Feb 11 16:31:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 4c5b69ea-d1f1-0261-ea03-15f22270fb92 (at 10.9.101.2@o2ib4) Feb 11 16:31:57 fir-io1-s1 kernel: Lustre: Skipped 700 previous similar messages Feb 11 16:38:18 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2d279b6b-ae49-37ac-0a12-0938de9dc4ca (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583bb59c00, cur 1549931898 expire 1549931748 last 1549931671 Feb 11 16:38:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 16:41:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4c9919a3-1838-af9d-fa66-ce9474087e70 (at 10.8.21.27@o2ib6) Feb 11 16:41:57 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 11 16:51:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ad95c58-a291-a852-e763-1230aae68b67 (at 10.8.13.6@o2ib6) Feb 11 16:51:57 fir-io1-s1 kernel: Lustre: Skipped 494 previous similar messages Feb 11 17:02:14 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ec7b3c21-b807-d2d6-877f-ad536dfb41e4 (at 10.8.27.26@o2ib6) Feb 11 17:02:14 fir-io1-s1 kernel: Lustre: Skipped 402 previous similar messages Feb 11 17:12:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 11 17:12:22 fir-io1-s1 kernel: Lustre: Skipped 340 previous similar messages Feb 11 17:22:25 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ca78cddb-ea39-0254-4028-7b0b6c7c780d (at 10.8.19.2@o2ib6) Feb 11 17:22:25 fir-io1-s1 kernel: Lustre: Skipped 367 previous similar messages Feb 11 17:24:50 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a8449bff-56f7-51ba-5adc-6abf8e452713 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf2a400, cur 1549934690 expire 1549934540 last 1549934463 Feb 11 17:24:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 17:32:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f0ff72b7-1a07-1d54-c66c-24d3bb719dda (at 10.9.102.58@o2ib4) Feb 11 17:32:26 fir-io1-s1 kernel: Lustre: Skipped 606 previous similar messages Feb 11 17:32:55 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 93c777d8-19c8-6356-41d2-9bba7f65eb8f (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811214000, cur 1549935175 expire 1549935025 last 1549934948 Feb 11 17:32:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 17:39:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4716fe2e-b162-9071-e50d-24885b29871a (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984836f2fc00, cur 1549935576 expire 1549935426 last 1549935349 Feb 11 17:39:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 17:40:52 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 670fc807-dd14-9be9-2373-4bbdc84964c5 (at 10.8.11.22@o2ib6) in 219 seconds. I think it's dead, and I am evicting it. exp ffff986785cb7400, cur 1549935652 expire 1549935502 last 1549935433 Feb 11 17:40:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 17:42:26 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to d2fd779c-157e-52b0-c4f2-2f7aaa06aac8 (at 10.8.23.7@o2ib6) Feb 11 17:42:26 fir-io1-s1 kernel: Lustre: Skipped 361 previous similar messages Feb 11 17:48:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 266a2d9e-b528-ee9e-45f8-a1ce8f70560a (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c0000, cur 1549936138 expire 1549935988 last 1549935911 Feb 11 17:48:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 17:52:27 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 62b8e745-55d6-06c6-f56c-66b6a8e58cb8 (at 10.8.20.26@o2ib6) Feb 11 17:52:27 fir-io1-s1 kernel: Lustre: Skipped 304 previous similar messages Feb 11 17:55:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e617b155-aa3a-8262-58f4-1d9ab77dbd2c (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768559000, cur 1549936546 expire 1549936396 last 1549936319 Feb 11 17:55:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 18:02:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Feb 11 18:02:29 fir-io1-s1 kernel: Lustre: Skipped 309 previous similar messages Feb 11 18:04:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b48cb1cb-71f0-84ae-e350-3e2bfee85624 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867818a9c00, cur 1549937041 expire 1549936891 last 1549936814 Feb 11 18:04:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 18:12:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f425d37e-c125-4f6b-235a-7086b8c4eff9 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f67b800, cur 1549937540 expire 1549937390 last 1549937313 Feb 11 18:12:20 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 18:12:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 18:12:30 fir-io1-s1 kernel: Lustre: Skipped 277 previous similar messages Feb 11 18:19:58 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4ae4897a-8bdc-7a34-cbef-1e528133e141 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c6c00, cur 1549937998 expire 1549937848 last 1549937771 Feb 11 18:19:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 18:22:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 18:22:30 fir-io1-s1 kernel: Lustre: Skipped 319 previous similar messages Feb 11 18:32:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 18:32:41 fir-io1-s1 kernel: Lustre: Skipped 332 previous similar messages Feb 11 18:42:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 18:42:41 fir-io1-s1 kernel: Lustre: Skipped 263 previous similar messages Feb 11 18:46:17 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ecc6f109-15ce-20bc-40e5-4957d4ec1dab (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a9c381000, cur 1549939577 expire 1549939427 last 1549939350 Feb 11 18:46:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 18:52:41 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 18:52:41 fir-io1-s1 kernel: Lustre: Skipped 286 previous similar messages Feb 11 18:53:45 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b6d9ac42-6245-c8c8-28e4-f4f90666c36d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a152b800, cur 1549940025 expire 1549939875 last 1549939798 Feb 11 18:53:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 19:02:39 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a6f3ed12-cfa6-928e-c47b-b2decb830de0 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0d800, cur 1549940559 expire 1549940409 last 1549940332 Feb 11 19:02:39 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 19:02:41 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 19:02:41 fir-io1-s1 kernel: Lustre: Skipped 425 previous similar messages Feb 11 19:10:08 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9375d0c9-c5ee-bf70-22e3-e96d55a1c3f7 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a0af400, cur 1549941008 expire 1549940858 last 1549940781 Feb 11 19:10:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 19:12:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 19:12:42 fir-io1-s1 kernel: Lustre: Skipped 288 previous similar messages Feb 11 19:16:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 403ce4c1-c8fa-6a2c-d007-5d4f4be9e234 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15c8400, cur 1549941397 expire 1549941247 last 1549941170 Feb 11 19:16:37 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 11 19:22:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 11 19:22:42 fir-io1-s1 kernel: Lustre: Skipped 421 previous similar messages Feb 11 19:23:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f841bf45-d0c7-2331-4e02-f9aac83d70db (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a059fc00, cur 1549941822 expire 1549941672 last 1549941595 Feb 11 19:23:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 19:32:44 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 11 19:32:44 fir-io1-s1 kernel: Lustre: Skipped 400 previous similar messages Feb 11 19:39:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e1d674ee-a2b9-c264-88c9-2f1a865d5b09 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfdc00, cur 1549942749 expire 1549942599 last 1549942522 Feb 11 19:39:09 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 11 19:42:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c8e6faa5-8e52-627a-b276-9b1da9fb48ae (at 10.8.7.23@o2ib6) Feb 11 19:42:47 fir-io1-s1 kernel: Lustre: Skipped 326 previous similar messages Feb 11 19:52:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 91c3019e-2880-7f92-0b14-6eb0f2cbe1dd (at 10.9.101.65@o2ib4) Feb 11 19:52:58 fir-io1-s1 kernel: Lustre: Skipped 288 previous similar messages Feb 11 19:59:17 fir-io1-s1 kernel: LustreError: 48579:0:(brw_test.c:389:brw_server_rpc_done()) Bulk transfer to 12345-10.8.0.66@o2ib6 has failed: -108 Feb 11 20:03:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 38a104b5-26ce-5d2d-596d-9304083f888f (at 10.9.112.14@o2ib4) Feb 11 20:03:09 fir-io1-s1 kernel: Lustre: Skipped 247 previous similar messages Feb 11 20:11:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b4cdb74d-1373-4e5e-0458-2fab1774f617 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904994800, cur 1549944684 expire 1549944534 last 1549944457 Feb 11 20:11:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 20:13:10 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) Feb 11 20:13:10 fir-io1-s1 kernel: Lustre: Skipped 241 previous similar messages Feb 11 20:19:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5049d3eb-3c1a-afa0-eb07-45fcb7b6a2b7 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762491000, cur 1549945156 expire 1549945006 last 1549944929 Feb 11 20:19:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 20:23:21 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1c418863-78cc-8f23-893e-27e5ce2dfd94 (at 10.9.101.71@o2ib4) Feb 11 20:23:21 fir-io1-s1 kernel: Lustre: Skipped 280 previous similar messages Feb 11 20:25:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c8e59f32-a41f-125e-b0df-9091342dd2b5 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481df05c00, cur 1549945548 expire 1549945398 last 1549945321 Feb 11 20:25:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 20:33:22 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Feb 11 20:33:22 fir-io1-s1 kernel: Lustre: Skipped 308 previous similar messages Feb 11 20:34:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1132224d-7fa8-98e5-e8a6-1c7847fb622d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a9c384400, cur 1549946052 expire 1549945902 last 1549945825 Feb 11 20:34:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 20:43:23 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 37327d93-dd03-80bf-ad2c-df17a42702a9 (at 10.8.18.26@o2ib6) Feb 11 20:43:23 fir-io1-s1 kernel: Lustre: Skipped 298 previous similar messages Feb 11 20:51:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client eb9f1238-5a7e-bbe3-ed9b-13d4a55a928d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985769981400, cur 1549947082 expire 1549946932 last 1549946855 Feb 11 20:51:22 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 11 20:53:29 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 11 20:53:29 fir-io1-s1 kernel: Lustre: Skipped 294 previous similar messages Feb 11 21:03:31 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ed540f5a-5df6-5998-8f5c-40181564f690 (at 10.9.106.50@o2ib4) Feb 11 21:03:31 fir-io1-s1 kernel: Lustre: Skipped 346 previous similar messages Feb 11 21:06:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5e8a09e9-5518-63e3-4864-b26fd545b7b6 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800e4dc00, cur 1549948013 expire 1549947863 last 1549947786 Feb 11 21:06:53 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 11 21:13:33 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 49507785-4778-3df3-6428-f7e53034ffec (at 10.9.107.32@o2ib4) Feb 11 21:13:33 fir-io1-s1 kernel: Lustre: Skipped 411 previous similar messages Feb 11 21:22:27 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client dba63adb-6a8e-6cd9-54a6-c84c12865402 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f4ac00, cur 1549948947 expire 1549948797 last 1549948720 Feb 11 21:22:27 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 11 21:23:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2336058a-aa9c-4463-bac9-a8ea66369e87 (at 10.8.11.22@o2ib6) Feb 11 21:23:34 fir-io1-s1 kernel: Lustre: Skipped 339 previous similar messages Feb 11 21:33:39 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to bf62c7df-fe55-c26d-63f8-adbf89ed0ecb (at 10.8.3.34@o2ib6) Feb 11 21:33:39 fir-io1-s1 kernel: Lustre: Skipped 485 previous similar messages Feb 11 21:34:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 61f112c2-83b6-0e81-d768-2d21fef76a1a (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596bec00, cur 1549949695 expire 1549949545 last 1549949468 Feb 11 21:34:55 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 11 21:43:47 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b41950de-0614-6c1a-0d53-d43c60fe0f33 (at 10.9.102.1@o2ib4) Feb 11 21:43:47 fir-io1-s1 kernel: Lustre: Skipped 434 previous similar messages Feb 11 21:47:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4ed7c160-349d-241e-70dd-58bbe640a453 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9878332cd800, cur 1549950458 expire 1549950308 last 1549950231 Feb 11 21:47:38 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 11 21:54:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bf62c7df-fe55-c26d-63f8-adbf89ed0ecb (at 10.8.3.34@o2ib6) Feb 11 21:54:08 fir-io1-s1 kernel: Lustre: Skipped 348 previous similar messages Feb 11 22:04:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 11 22:04:15 fir-io1-s1 kernel: Lustre: Skipped 206 previous similar messages Feb 11 22:14:27 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.0.64@o2ib4) Feb 11 22:14:27 fir-io1-s1 kernel: Lustre: Skipped 178 previous similar messages Feb 11 22:20:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0da9192a-02bc-f90a-2f6b-6cdde33f9aec (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756584800, cur 1549952435 expire 1549952285 last 1549952208 Feb 11 22:20:35 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Feb 11 22:24:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) Feb 11 22:24:42 fir-io1-s1 kernel: Lustre: Skipped 198 previous similar messages Feb 11 22:28:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 55625a39-821d-1d3c-a0f3-d15d0bc47864 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784488000, cur 1549952890 expire 1549952740 last 1549952663 Feb 11 22:28:10 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 22:34:48 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 21ad58eb-b0eb-378b-d1d7-0646aa1b95cf (at 10.9.102.28@o2ib4) Feb 11 22:34:48 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 11 22:45:03 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to da7f09b2-cefc-256e-025a-17645b86ee8d (at 10.9.112.1@o2ib4) Feb 11 22:45:03 fir-io1-s1 kernel: Lustre: Skipped 266 previous similar messages Feb 11 22:50:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c49e9713-be58-954f-804e-7a03164cd71b (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e45c00, cur 1549954227 expire 1549954077 last 1549954000 Feb 11 22:50:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 22:55:07 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 11 22:55:07 fir-io1-s1 kernel: Lustre: Skipped 323 previous similar messages Feb 11 22:58:02 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7f4c0d2b-87b2-4817-11fc-35386bfabbe0 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c68800, cur 1549954682 expire 1549954532 last 1549954455 Feb 11 22:58:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 23:04:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ee4f158b-fd50-0cd9-0490-f2444de640f2 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985757bae000, cur 1549955078 expire 1549954928 last 1549954851 Feb 11 23:04:38 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 23:05:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 11 23:05:08 fir-io1-s1 kernel: Lustre: Skipped 211 previous similar messages Feb 11 23:12:19 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 19cf490f-03ee-9fc2-04c3-2c1eae55f005 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de7c00, cur 1549955539 expire 1549955389 last 1549955312 Feb 11 23:12:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 11 23:15:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to dcd406ad-ffdd-a7c9-489f-309957a1236e (at 10.8.15.7@o2ib6) Feb 11 23:15:09 fir-io1-s1 kernel: Lustre: Skipped 216 previous similar messages Feb 11 23:19:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 14fa8e28-05e8-da5c-f4f8-3d5000bd3f2d (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678448a400, cur 1549955987 expire 1549955837 last 1549955760 Feb 11 23:19:47 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 11 23:25:12 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7796eaa1-24c1-f1a6-996a-1af3c662e968 (at 10.8.7.2@o2ib6) Feb 11 23:25:12 fir-io1-s1 kernel: Lustre: Skipped 284 previous similar messages Feb 11 23:26:46 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 73de8ba2-b841-92c8-7916-67aed8a86373 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985836ac4000, cur 1549956406 expire 1549956256 last 1549956179 Feb 11 23:26:46 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 11 23:35:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1b71b1a8-fdc3-550f-13e6-42b0376dd743 (at 10.9.112.8@o2ib4) Feb 11 23:35:16 fir-io1-s1 kernel: Lustre: Skipped 320 previous similar messages Feb 11 23:45:19 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 83fafa06-153a-a1fd-dd02-4b2d3ef6c90e (at 10.9.114.2@o2ib4) Feb 11 23:45:19 fir-io1-s1 kernel: Lustre: Skipped 330 previous similar messages Feb 11 23:47:15 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ce363aa6-1c10-0fe8-eab0-7cf3f41dfcfa (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b97c00, cur 1549957635 expire 1549957485 last 1549957408 Feb 11 23:47:15 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 11 23:55:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 48c8053a-b65c-fab0-0c2e-a095d7828758 (at 10.8.6.29@o2ib6) Feb 11 23:55:30 fir-io1-s1 kernel: Lustre: Skipped 285 previous similar messages Feb 11 23:57:15 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fe6d5948-0153-0ad3-9cf7-388515c5d491 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283dd800, cur 1549958235 expire 1549958085 last 1549958008 Feb 11 23:57:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 00:05:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 83fafa06-153a-a1fd-dd02-4b2d3ef6c90e (at 10.9.114.2@o2ib4) Feb 12 00:05:35 fir-io1-s1 kernel: Lustre: Skipped 223 previous similar messages Feb 12 00:15:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 318de70a-4b49-6572-6064-ea964a3568c4 (at 10.9.107.37@o2ib4) Feb 12 00:15:37 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 12 00:18:43 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 1c5bfc10-7978-c1d8-7305-a696dcfa2db9 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d648000, cur 1549959523 expire 1549959373 last 1549959296 Feb 12 00:18:43 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 00:25:38 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) Feb 12 00:25:38 fir-io1-s1 kernel: Lustre: Skipped 261 previous similar messages Feb 12 00:36:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 861509dc-4523-55e8-f09a-16b6fca3f713 (at 10.9.107.51@o2ib4) Feb 12 00:36:09 fir-io1-s1 kernel: Lustre: Skipped 202 previous similar messages Feb 12 00:46:09 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.101.44@o2ib4) Feb 12 00:46:09 fir-io1-s1 kernel: Lustre: Skipped 208 previous similar messages Feb 12 00:56:12 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 090fafdf-b851-44a0-92d1-bfda03f3741e (at 10.8.8.29@o2ib6) Feb 12 00:56:12 fir-io1-s1 kernel: Lustre: Skipped 270 previous similar messages Feb 12 01:06:28 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a397e427-2a18-94f0-0f06-4c6a9e455efa (at 10.8.18.8@o2ib6) Feb 12 01:06:28 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 12 01:16:29 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 78a94154-9621-e13c-f5a0-60f10c4683a5 (at 10.9.102.71@o2ib4) Feb 12 01:16:29 fir-io1-s1 kernel: Lustre: Skipped 170 previous similar messages Feb 12 01:26:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to d9f51967-967f-e724-b4d7-c6424894c591 (at 10.8.30.2@o2ib6) Feb 12 01:26:37 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 12 01:34:52 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2ff14a70-611f-f900-540e-b4a3a631a01e (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c1ac00, cur 1549964092 expire 1549963942 last 1549963865 Feb 12 01:34:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 01:36:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 12 01:36:45 fir-io1-s1 kernel: Lustre: Skipped 156 previous similar messages Feb 12 01:41:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ae8c7931-eb93-9db6-aac4-6598d9a5d7a3 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b9e000, cur 1549964482 expire 1549964332 last 1549964255 Feb 12 01:41:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 01:46:20 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 275a07e3-5074-a2be-55af-a08dee65a2ef (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5a400, cur 1549964780 expire 1549964630 last 1549964553 Feb 12 01:46:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 01:46:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 12 01:46:56 fir-io1-s1 kernel: Lustre: Skipped 223 previous similar messages Feb 12 01:56:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.4.32@o2ib6) Feb 12 01:56:56 fir-io1-s1 kernel: Lustre: Skipped 251 previous similar messages Feb 12 02:07:00 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to bef16258-699d-0e14-bdeb-b454fac00d89 (at 10.9.112.15@o2ib4) Feb 12 02:07:00 fir-io1-s1 kernel: Lustre: Skipped 237 previous similar messages Feb 12 02:17:01 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a26bda41-9e3a-f8bb-aa63-ba992cd69aad (at 10.9.0.62@o2ib4) Feb 12 02:17:01 fir-io1-s1 kernel: Lustre: Skipped 174 previous similar messages Feb 12 02:27:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 12 02:27:17 fir-io1-s1 kernel: Lustre: Skipped 122 previous similar messages Feb 12 02:37:33 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 74576cb8-d6cd-cd3f-fbd2-32131b2925d8 (at 10.9.107.19@o2ib4) Feb 12 02:37:33 fir-io1-s1 kernel: Lustre: Skipped 267 previous similar messages Feb 12 02:47:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 12 02:47:38 fir-io1-s1 kernel: Lustre: Skipped 116 previous similar messages Feb 12 02:57:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 12 02:57:41 fir-io1-s1 kernel: Lustre: Skipped 101 previous similar messages Feb 12 03:07:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 12 03:07:42 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Feb 12 03:17:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 12 03:17:56 fir-io1-s1 kernel: Lustre: Skipped 197 previous similar messages Feb 12 03:28:37 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 12 03:28:37 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 12 03:38:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to c46b7f3d-ec55-7fd7-e207-e9c6f5525b60 (at 10.8.0.68@o2ib6) Feb 12 03:38:39 fir-io1-s1 kernel: Lustre: Skipped 231 previous similar messages Feb 12 03:48:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 12 03:48:39 fir-io1-s1 kernel: Lustre: Skipped 227 previous similar messages Feb 12 03:58:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ed8a9d5a-3c64-88a2-4de9-5c7913d6ef08 (at 10.9.101.15@o2ib4) Feb 12 03:58:58 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Feb 12 04:08:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2336058a-aa9c-4463-bac9-a8ea66369e87 (at 10.8.11.22@o2ib6) Feb 12 04:08:59 fir-io1-s1 kernel: Lustre: Skipped 216 previous similar messages Feb 12 04:19:10 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c19cae2a-5b5b-b935-b49e-0f0362e17271 (at 10.9.105.33@o2ib4) Feb 12 04:19:10 fir-io1-s1 kernel: Lustre: Skipped 191 previous similar messages Feb 12 04:29:10 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3e4de392-7948-5bb8-1b41-4df53ad748a7 (at 10.8.8.10@o2ib6) Feb 12 04:29:10 fir-io1-s1 kernel: Lustre: Skipped 178 previous similar messages Feb 12 04:39:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 7e9161e5-27d8-4cac-a415-4c23ea14bc0e (at 10.9.106.46@o2ib4) Feb 12 04:39:18 fir-io1-s1 kernel: Lustre: Skipped 230 previous similar messages Feb 12 04:49:34 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 7b84967c-82fe-7749-1773-42f1fdeff446 (at 10.8.18.1@o2ib6) Feb 12 04:49:34 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 12 04:59:45 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 641e9c16-9f49-3085-a67f-ec8077cf8e17 (at 10.9.107.21@o2ib4) Feb 12 04:59:45 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 12 05:09:48 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 4b161527-b7d4-cb96-22b6-3891827325ad (at 10.9.112.2@o2ib4) Feb 12 05:09:48 fir-io1-s1 kernel: Lustre: Skipped 230 previous similar messages Feb 12 05:20:03 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677c23f800, cur 1549977603 expire 1549977453 last 1549977376 Feb 12 05:20:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 05:20:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 084ae2e7-32eb-f718-134b-ac7a3c2328d4 (at 10.9.101.9@o2ib4) Feb 12 05:20:26 fir-io1-s1 kernel: Lustre: Skipped 178 previous similar messages Feb 12 05:30:27 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e527b419-86b9-ff93-48e0-b15e55994667 (at 10.9.106.28@o2ib4) Feb 12 05:30:27 fir-io1-s1 kernel: Lustre: Skipped 182 previous similar messages Feb 12 05:40:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.9.112.17@o2ib4) Feb 12 05:40:50 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 12 05:50:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 12 05:50:50 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 12 06:00:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to baf8fd69-3c06-48ef-44c1-991a98b1784d (at 10.8.13.14@o2ib6) Feb 12 06:00:56 fir-io1-s1 kernel: Lustre: Skipped 231 previous similar messages Feb 12 06:10:59 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1ca40c95-5615-186b-162f-92f0324c3c09 (at 10.8.26.21@o2ib6) Feb 12 06:10:59 fir-io1-s1 kernel: Lustre: Skipped 197 previous similar messages Feb 12 06:21:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Feb 12 06:21:09 fir-io1-s1 kernel: Lustre: Skipped 217 previous similar messages Feb 12 06:31:13 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to cc107cab-3544-aeaa-6b27-e00a056fcf80 (at 10.8.1.26@o2ib6) Feb 12 06:31:13 fir-io1-s1 kernel: Lustre: Skipped 297 previous similar messages Feb 12 06:35:30 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 72688305-1dc5-39ee-20c6-95dee4029423 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756865400, cur 1549982130 expire 1549981980 last 1549981903 Feb 12 06:35:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 06:41:36 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d3157baf-de90-86ed-7c87-5f5f5c909a71 (at 10.9.105.15@o2ib4) Feb 12 06:41:36 fir-io1-s1 kernel: Lustre: Skipped 182 previous similar messages Feb 12 06:51:41 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 7cba68d6-88e2-7226-0b4f-cb83c3107f8f (at 10.9.102.34@o2ib4) Feb 12 06:51:41 fir-io1-s1 kernel: Lustre: Skipped 257 previous similar messages Feb 12 07:01:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) Feb 12 07:01:45 fir-io1-s1 kernel: Lustre: Skipped 249 previous similar messages Feb 12 07:11:47 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 12 07:11:47 fir-io1-s1 kernel: Lustre: Skipped 198 previous similar messages Feb 12 07:22:25 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 037cf541-a575-03b3-3aed-140164784d71 (at 10.9.107.61@o2ib4) Feb 12 07:22:25 fir-io1-s1 kernel: Lustre: Skipped 213 previous similar messages Feb 12 07:32:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 12 07:32:41 fir-io1-s1 kernel: Lustre: Skipped 223 previous similar messages Feb 12 07:42:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 12 07:42:41 fir-io1-s1 kernel: Lustre: Skipped 272 previous similar messages Feb 12 07:52:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 12 07:52:41 fir-io1-s1 kernel: Lustre: Skipped 327 previous similar messages Feb 12 08:02:49 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4975f97a-dade-f6fe-b4b2-2a6a6e1e4710 (at 10.9.102.52@o2ib4) Feb 12 08:02:49 fir-io1-s1 kernel: Lustre: Skipped 314 previous similar messages Feb 12 08:09:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c703af01-ea2f-81b6-cba4-3f1b6304e904 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986816f3c000, cur 1549987741 expire 1549987591 last 1549987514 Feb 12 08:09:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 08:12:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 56a35e3b-d984-54d7-931b-bb4aae4648e4 (at 10.8.11.5@o2ib6) Feb 12 08:12:56 fir-io1-s1 kernel: Lustre: Skipped 309 previous similar messages Feb 12 08:16:21 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ca2418c8-2810-0138-dee7-6b4d135fc41f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848324a2c00, cur 1549988181 expire 1549988031 last 1549987954 Feb 12 08:16:21 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 08:23:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.16.7@o2ib6) Feb 12 08:23:03 fir-io1-s1 kernel: Lustre: Skipped 294 previous similar messages Feb 12 08:25:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 70c3209a-a03c-80b3-cdcb-cba5d14b68cc (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801bfdc00, cur 1549988713 expire 1549988563 last 1549988486 Feb 12 08:25:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 08:33:04 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to b7a3d707-0c8e-49e8-1e76-06f174330f00 (at 10.8.8.2@o2ib6) Feb 12 08:33:04 fir-io1-s1 kernel: Lustre: Skipped 301 previous similar messages Feb 12 08:43:05 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 037cf541-a575-03b3-3aed-140164784d71 (at 10.9.107.61@o2ib4) Feb 12 08:43:05 fir-io1-s1 kernel: Lustre: Skipped 339 previous similar messages Feb 12 08:46:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e74cc2ee-8963-d22d-fcb9-add9dd65caea (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800f56c00, cur 1549989961 expire 1549989811 last 1549989734 Feb 12 08:46:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 08:52:31 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2d0cd6c9-20a2-72b1-d64d-aed2ae13e5bd (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582bba2400, cur 1549990351 expire 1549990201 last 1549990124 Feb 12 08:52:31 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 08:53:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 12 08:53:06 fir-io1-s1 kernel: Lustre: Skipped 496 previous similar messages Feb 12 09:03:16 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2196625d-9992-1b8a-5a12-40751a9cdd4e (at 10.9.107.2@o2ib4) Feb 12 09:03:16 fir-io1-s1 kernel: Lustre: Skipped 515 previous similar messages Feb 12 09:13:16 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9e7dc1a5-746c-8e56-5ad9-e239237ff7d7 (at 10.8.24.22@o2ib6) Feb 12 09:13:16 fir-io1-s1 kernel: Lustre: Skipped 507 previous similar messages Feb 12 09:23:25 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3e1e4361-b591-1d07-eae5-f7fddaf6a7a9 (at 10.8.17.14@o2ib6) Feb 12 09:23:25 fir-io1-s1 kernel: Lustre: Skipped 659 previous similar messages Feb 12 09:33:37 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) Feb 12 09:33:37 fir-io1-s1 kernel: Lustre: Skipped 482 previous similar messages Feb 12 09:39:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2b7fe363-8432-dffa-9e17-ff01de7cd5a1 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581ef41800, cur 1549993146 expire 1549992996 last 1549992919 Feb 12 09:39:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 09:43:37 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c8bbd681-6d42-3196-83fc-d269a276781d (at 10.9.105.55@o2ib4) Feb 12 09:43:37 fir-io1-s1 kernel: Lustre: Skipped 722 previous similar messages Feb 12 09:45:46 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 174f480f-c55e-5b39-3618-a781391d0bc2 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ea47c00, cur 1549993546 expire 1549993396 last 1549993319 Feb 12 09:45:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 09:53:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) Feb 12 09:53:37 fir-io1-s1 kernel: Lustre: Skipped 609 previous similar messages Feb 12 09:56:29 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b81f0187-91e5-066d-74b4-50740b410ac6 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678694c800, cur 1549994189 expire 1549994039 last 1549993962 Feb 12 09:59:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 47ce4289-7e25-8c66-9590-6b36cdee8e22 (at 10.9.101.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9878386e6000, cur 1549994384 expire 1549994234 last 1549994157 Feb 12 09:59:44 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 12 10:03:34 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3327954e-8c7f-0ead-8ba3-049efd5cfb2e (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f476800, cur 1549994614 expire 1549994464 last 1549994387 Feb 12 10:03:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 10:03:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 12 10:03:40 fir-io1-s1 kernel: Lustre: Skipped 371 previous similar messages Feb 12 10:09:50 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8ef62543-b176-d2fe-0816-32c8ba131459 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b293800, cur 1549994990 expire 1549994840 last 1549994763 Feb 12 10:09:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 10:13:50 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 72da9a6e-2827-1c9d-1aa6-7b398153fee1 (at 10.9.106.71@o2ib4) Feb 12 10:13:50 fir-io1-s1 kernel: Lustre: Skipped 473 previous similar messages Feb 12 10:15:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c644ce5b-1df4-b5d0-d9fd-fefb5c1f7171 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575eaa0400, cur 1549995327 expire 1549995177 last 1549995100 Feb 12 10:15:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 10:23:52 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to e9560223-f857-8af8-8e66-18924c1e4b0e (at 10.8.3.22@o2ib6) Feb 12 10:23:52 fir-io1-s1 kernel: Lustre: Skipped 531 previous similar messages Feb 12 10:26:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c34176e6-7800-27ea-91ad-9ce77e309cca (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d53400, cur 1549995960 expire 1549995810 last 1549995733 Feb 12 10:26:00 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 12 10:33:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 29b97f96-e09c-6114-f2fc-805b41aea072 (at 10.8.1.33@o2ib6) Feb 12 10:33:56 fir-io1-s1 kernel: Lustre: Skipped 443 previous similar messages Feb 12 10:43:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 40010d6e-d29b-c686-3f5e-dc2316139f55 (at 10.9.107.27@o2ib4) Feb 12 10:43:57 fir-io1-s1 kernel: Lustre: Skipped 660 previous similar messages Feb 12 10:54:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0b22cd50-4b3f-cc55-9158-e0958bde4beb (at 10.8.18.27@o2ib6) Feb 12 10:54:03 fir-io1-s1 kernel: Lustre: Skipped 1022 previous similar messages Feb 12 11:04:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6d1355d0-7b33-677d-c8cf-a270e3061917 (at 10.8.7.15@o2ib6) Feb 12 11:04:06 fir-io1-s1 kernel: Lustre: Skipped 753 previous similar messages Feb 12 11:08:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 11:08:52 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:943801 to 0x580000400:943937 Feb 12 11:09:43 fir-io1-s1 kernel: Lustre: fir-OST0006: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 11:09:43 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 11:09:45 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:778958 to 0x0:779041 Feb 12 11:09:45 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:943247 to 0xc40000402:943329 Feb 12 11:10:27 fir-io1-s1 kernel: Lustre: fir-OST0006: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 11:10:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 11:10:34 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:779614 to 0x0:779681 Feb 12 11:10:34 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 11:10:37 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:779245 to 0x0:779297 Feb 12 11:10:37 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:779009 to 0x0:779041 Feb 12 11:10:37 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:943331 to 0xc40000402:943361 Feb 12 11:10:39 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:943939 to 0x580000400:943969 Feb 12 11:10:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 11:10:57 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 11:10:58 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:779745 to 0x0:779809 Feb 12 11:12:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d389dc2f-a4ea-0582-21f5-cd6c5e60379e (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfd000, cur 1549998733 expire 1549998583 last 1549998506 Feb 12 11:12:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 11:13:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7d6c5577-524c-8a26-5793-add4b9806463 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a059c800, cur 1549998802 expire 1549998652 last 1549998575 Feb 12 11:13:22 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 12 11:14:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to d9d3ac4e-0fb3-be83-7c67-dfe4c97facfb (at 10.9.114.9@o2ib4) Feb 12 11:14:06 fir-io1-s1 kernel: Lustre: Skipped 449 previous similar messages Feb 12 11:24:07 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 12 11:24:07 fir-io1-s1 kernel: Lustre: Skipped 791 previous similar messages Feb 12 11:33:52 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 38063fd0-8914-d8d4-6ba8-da9d20cf8181 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b611800, cur 1550000032 expire 1549999882 last 1549999805 Feb 12 11:33:52 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 11:34:18 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to d35217f0-5ade-6000-33a4-4fb011efe6c5 (at 10.8.4.8@o2ib6) Feb 12 11:34:18 fir-io1-s1 kernel: Lustre: Skipped 580 previous similar messages Feb 12 11:44:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b3fc52f5-cc19-f1e2-5d13-43190203fae8 (at 10.9.106.22@o2ib4) Feb 12 11:44:21 fir-io1-s1 kernel: Lustre: Skipped 469 previous similar messages Feb 12 11:45:50 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9e77b693-63df-849b-a228-a6c1f81932bb (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762b64800, cur 1550000750 expire 1550000600 last 1550000523 Feb 12 11:45:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 11:53:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a1af19c9-6240-5b50-a172-f2b4e5930368 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786975400, cur 1550001187 expire 1550001037 last 1550000960 Feb 12 11:53:07 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 11:54:24 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c8bbd681-6d42-3196-83fc-d269a276781d (at 10.9.105.55@o2ib4) Feb 12 11:54:24 fir-io1-s1 kernel: Lustre: Skipped 649 previous similar messages Feb 12 12:02:02 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7453866c-ba6a-7bbb-051e-5cef18c08a04 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849be95cc00, cur 1550001722 expire 1550001572 last 1550001495 Feb 12 12:02:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 12:04:24 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 12 12:04:24 fir-io1-s1 kernel: Lustre: Skipped 403 previous similar messages Feb 12 12:10:50 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 36a9956b-003d-7e94-9d03-d77be3735083 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576dc9e800, cur 1550002250 expire 1550002100 last 1550002023 Feb 12 12:10:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 12:14:24 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to fb5e4f4f-a645-fbec-7b80-08c8d8a2fea0 (at 10.8.1.12@o2ib6) Feb 12 12:14:24 fir-io1-s1 kernel: Lustre: Skipped 576 previous similar messages Feb 12 12:19:15 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 576087f9-5682-a505-77bf-c479536e3b51 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801241000, cur 1550002755 expire 1550002605 last 1550002528 Feb 12 12:19:15 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 12:24:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) Feb 12 12:24:32 fir-io1-s1 kernel: Lustre: Skipped 490 previous similar messages Feb 12 12:34:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c243879d-6590-e58d-10d6-105c5b7b4def (at 10.8.28.1@o2ib6) Feb 12 12:34:33 fir-io1-s1 kernel: Lustre: Skipped 398 previous similar messages Feb 12 12:44:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.3.27@o2ib6) Feb 12 12:44:36 fir-io1-s1 kernel: Lustre: Skipped 361 previous similar messages Feb 12 12:54:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5c820789-8e24-ad89-a0df-b1759dd671b0 (at 10.9.101.56@o2ib4) Feb 12 12:54:53 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 12 12:58:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987833e90000, cur 1550005135 expire 1550004985 last 1550004908 Feb 12 12:58:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 13:05:13 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.18.23@o2ib6) Feb 12 13:05:13 fir-io1-s1 kernel: Lustre: Skipped 95 previous similar messages Feb 12 13:15:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 12 13:15:23 fir-io1-s1 kernel: Lustre: Skipped 79 previous similar messages Feb 12 13:25:36 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 12 13:25:36 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 12 13:26:03 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 4 seconds Feb 12 13:26:03 fir-io1-s1 kernel: Lustre: 91455:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550006761/real 1550006763] req@ffff985d80e92d00 x1624932069627888/t0(0) o400->fir-MDT0001-lwp-OST0006@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550007517 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 12 13:26:03 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 12 13:26:03 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 13:26:03 fir-io1-s1 kernel: Lustre: 91455:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 12 13:26:03 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Feb 12 13:26:03 fir-io1-s1 kernel: Lustre: Skipped 84 previous similar messages Feb 12 13:26:04 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 64 seconds Feb 12 13:26:04 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Feb 12 13:26:04 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0002: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 12 13:26:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 12 13:26:29 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 12 13:26:29 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Feb 12 13:26:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 13:26:54 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Feb 12 13:26:54 fir-io1-s1 kernel: LustreError: Skipped 17 previous similar messages Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:780097 to 0x0:780129 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:780096 to 0x0:780129 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:780862 to 0x0:780897 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:780353 to 0x0:780417 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:780013 to 0x0:780129 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:780738 to 0x0:780769 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:948759 to 0xc80000402:948833 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:948379 to 0x8c0000402:948545 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:948694 to 0xc40000402:948737 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:948650 to 0x5c0000400:948833 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:948819 to 0x6c0000400:949121 Feb 12 13:26:55 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:949295 to 0x580000400:949377 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:186264 to 0xc40000400:186337 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:186225 to 0x6c0000401:186305 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:186538 to 0x5c0000401:186593 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:186237 to 0x8c0000400:186337 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:186582 to 0x580000401:186657 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:186334 to 0xc80000400:186497 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:301250 to 0x6c0000402:301313 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:302391 to 0x5c0000402:302465 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:301090 to 0xc40000401:301153 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:301144 to 0xc80000401:301185 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:302188 to 0x580000402:302337 Feb 12 13:26:56 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:301141 to 0x8c0000401:301313 Feb 12 13:31:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 35e43627-4559-6e68-aa0d-2f084505361b (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583baf5400, cur 1550007064 expire 1550006914 last 1550006837 Feb 12 13:31:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 35e43627-4559-6e68-aa0d-2f084505361b (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98782763d000, cur 1550007065 expire 1550006915 last 1550006838 Feb 12 13:31:05 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 13:31:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 35e43627-4559-6e68-aa0d-2f084505361b (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9878323e9000, cur 1550007066 expire 1550006916 last 1550006839 Feb 12 13:31:06 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 13:36:04 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e9560223-f857-8af8-8e66-18924c1e4b0e (at 10.8.3.22@o2ib6) Feb 12 13:36:04 fir-io1-s1 kernel: Lustre: Skipped 1109 previous similar messages Feb 12 13:46:05 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 20cb63a2-4619-420f-25f6-c09916b5cd24 (at 10.8.17.16@o2ib6) Feb 12 13:46:05 fir-io1-s1 kernel: Lustre: Skipped 641 previous similar messages Feb 12 13:56:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3f2e0558-152b-3ca0-be2b-5080fc19b5c2 (at 10.8.21.14@o2ib6) Feb 12 13:56:09 fir-io1-s1 kernel: Lustre: Skipped 384 previous similar messages Feb 12 14:06:10 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.9.106.19@o2ib4) Feb 12 14:06:10 fir-io1-s1 kernel: Lustre: Skipped 276 previous similar messages Feb 12 14:16:10 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 12 14:16:10 fir-io1-s1 kernel: Lustre: Skipped 512 previous similar messages Feb 12 14:26:18 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 45a12f91-6aa3-f0ae-a299-8aadf4c776a5 (at 10.8.17.5@o2ib6) Feb 12 14:26:18 fir-io1-s1 kernel: Lustre: Skipped 1045 previous similar messages Feb 12 14:36:20 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 95777d32-4667-7d7f-8bf0-f1108b8f2ac8 (at 10.8.2.31@o2ib6) Feb 12 14:36:20 fir-io1-s1 kernel: Lustre: Skipped 846 previous similar messages Feb 12 14:46:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to feac05a4-716f-c34a-fd9d-1220a521af0c (at 10.9.107.69@o2ib4) Feb 12 14:46:22 fir-io1-s1 kernel: Lustre: Skipped 886 previous similar messages Feb 12 14:54:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 14:54:17 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 14:54:17 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:781336 to 0x0:781377 Feb 12 14:54:17 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:959314 to 0x580000400:959329 Feb 12 14:54:51 fir-io1-s1 kernel: Lustre: fir-OST0004: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 14:54:51 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:958487 to 0x8c0000402:958529 Feb 12 14:55:17 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000b_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 12 14:55:17 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Feb 12 14:56:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) Feb 12 14:56:22 fir-io1-s1 kernel: Lustre: Skipped 625 previous similar messages Feb 12 15:06:27 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 276945c0-0d00-01d6-84a8-1f1c209c1b8f (at 10.9.103.36@o2ib4) Feb 12 15:06:27 fir-io1-s1 kernel: Lustre: Skipped 740 previous similar messages Feb 12 15:07:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 15:07:56 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 15:08:21 fir-io1-s1 kernel: Lustre: fir-OST0002: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 15:08:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 15:08:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 12 15:08:21 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0005_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 12 15:08:31 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:781148 to 0x0:781185 Feb 12 15:08:34 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:963434 to 0x6c0000400:963457 Feb 12 15:08:36 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:781790 to 0x0:781825 Feb 12 15:08:42 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 15:08:42 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 12 15:08:42 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 15:08:45 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:962903 to 0x8c0000402:962945 Feb 12 15:08:45 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:963697 to 0x580000400:963713 Feb 12 15:08:45 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:781148 to 0x0:781185 Feb 12 15:08:45 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:963145 to 0xc80000402:963169 Feb 12 15:09:09 fir-io1-s1 kernel: Lustre: fir-OST0002: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 15:09:09 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:781142 to 0x0:781185 Feb 12 15:09:09 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:781478 to 0x0:781505 Feb 12 15:09:09 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 15:09:09 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:963148 to 0x5c0000400:963233 Feb 12 15:10:21 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 1cc72755-0b18-692a-013c-e5abb0ad9b59 (at 10.9.106.44@o2ib4) reconnecting Feb 12 15:10:48 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 1cc72755-0b18-692a-013c-e5abb0ad9b59 (at 10.9.106.44@o2ib4) reconnecting Feb 12 15:16:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2cfcad01-8df5-2887-8f5c-a2aec6d77cee (at 10.9.107.6@o2ib4) Feb 12 15:16:30 fir-io1-s1 kernel: Lustre: Skipped 662 previous similar messages Feb 12 15:26:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 720160b3-b510-fd4a-7aef-667fe71d4b1d (at 10.8.27.20@o2ib6) Feb 12 15:26:30 fir-io1-s1 kernel: Lustre: Skipped 679 previous similar messages Feb 12 15:29:03 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 68217227-70ae-290e-e4df-14b7516db509 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885dc00, cur 1550014143 expire 1550013993 last 1550013916 Feb 12 15:29:03 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 15:29:09 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 68217227-70ae-290e-e4df-14b7516db509 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1552000, cur 1550014149 expire 1550013999 last 1550013922 Feb 12 15:29:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 68217227-70ae-290e-e4df-14b7516db509 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848000edc00, cur 1550014156 expire 1550014006 last 1550013929 Feb 12 15:29:16 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 15:29:18 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 68217227-70ae-290e-e4df-14b7516db509 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848000ee800, cur 1550014158 expire 1550014008 last 1550013931 Feb 12 15:29:18 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 15:36:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 44ee34be-64b9-b529-f92c-8f8496b513f8 (at 10.9.104.16@o2ib4) Feb 12 15:36:33 fir-io1-s1 kernel: Lustre: Skipped 530 previous similar messages Feb 12 15:42:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fa3988d8-312e-baa0-298b-1666a8960425 (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801bfe800, cur 1550014925 expire 1550014775 last 1550014698 Feb 12 15:42:28 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 12 15:42:28 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 12 15:42:28 fir-io1-s1 kernel: Lustre: 91463:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550014942/real 1550014948] req@ffff985e5b436300 x1624932080373104/t0(0) o400->fir-MDT0000-lwp-OST0004@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550014949 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 12 15:42:28 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0004: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 12 15:42:28 fir-io1-s1 kernel: Lustre: 91463:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 12 15:42:29 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds Feb 12 15:42:29 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Feb 12 15:42:29 fir-io1-s1 kernel: Lustre: 91454:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550014942/real 1550014942] req@ffff985e5b431200 x1624932080372768/t0(0) o400->fir-MDT0002-lwp-OST0000@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550014949 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Feb 12 15:42:29 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST000a: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 12 15:42:29 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 12 15:42:29 fir-io1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail Feb 12 15:42:29 fir-io1-s1 kernel: Lustre: 91454:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 12 15:42:30 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 2 seconds Feb 12 15:42:30 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 5 previous similar messages Feb 12 15:43:21 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 39 seconds Feb 12 15:43:21 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages Feb 12 15:44:11 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 38 seconds Feb 12 15:44:11 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 9 previous similar messages Feb 12 15:45:02 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 42 seconds Feb 12 15:45:02 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 12 previous similar messages Feb 12 15:45:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784417400, cur 1550015140 expire 1550014990 last 1550014913 Feb 12 15:45:40 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 15:45:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a5fc00, cur 1550015143 expire 1550014993 last 1550014916 Feb 12 15:45:43 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 12 15:45:51 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 8 seconds Feb 12 15:45:51 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 10 previous similar messages Feb 12 15:46:46 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 12 15:46:46 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 12 15:46:59 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 33893189-0abb-4d15-5624-4b6aa533cb69 (at 10.8.8.4@o2ib6) Feb 12 15:46:59 fir-io1-s1 kernel: Lustre: Skipped 462 previous similar messages Feb 12 15:48:01 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 12 15:48:01 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 12 15:48:51 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 12 15:48:51 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 12 15:50:07 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 12 15:50:07 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 12 15:52:31 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST0000: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Feb 12 15:52:31 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 12 15:53:47 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Feb 12 15:53:47 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 194ae90b-dda5-aeec-e623-aa1c27f6c383 (at 10.8.17.21@o2ib6) Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: Skipped 91 previous similar messages Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:781443 to 0x0:781473 Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:781443 to 0x0:781473 Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:781765 to 0x0:781793 Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:782083 to 0x0:782113 Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:782170 to 0x0:782209 Feb 12 15:57:37 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:781443 to 0x0:781473 Feb 12 15:58:51 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:974127 to 0x6c0000400:974145 Feb 12 15:58:51 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:973904 to 0x5c0000400:973921 Feb 12 15:58:51 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:973601 to 0x8c0000402:973633 Feb 12 15:58:51 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:973724 to 0xc40000402:973761 Feb 12 15:58:51 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:973832 to 0xc80000402:973857 Feb 12 15:58:51 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:974371 to 0x580000400:974401 Feb 12 15:59:13 fir-io1-s1 kernel: Lustre: 96251:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550015946/real 1550015946] req@ffff985647ddb900 x1624932080549840/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550015953 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 12 15:59:13 fir-io1-s1 kernel: Lustre: 96251:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 12 15:59:20 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550015953/real 1550015953] req@ffff9854c8ed1b00 x1624932080549824/t0(0) o106->fir-OST0008@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550015960 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 15:59:20 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 12 15:59:27 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550015960/real 1550015960] req@ffff984a60b8c800 x1624932080549856/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550015967 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 15:59:27 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 12 15:59:34 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550015967/real 1550015967] req@ffff9862d15b6c00 x1624932080549808/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550015974 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 15:59:34 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 12 15:59:41 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550015974/real 1550015974] req@ffff9854c8ed1b00 x1624932080549824/t0(0) o106->fir-OST0008@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550015981 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 15:59:41 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550015974/real 1550015974] req@ffff984a60b8c800 x1624932080549856/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550015981 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 15:59:41 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 12 15:59:55 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550015988/real 1550015988] req@ffff984a60b8c800 x1624932080549856/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550015995 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 15:59:55 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 12 16:00:16 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550016009/real 1550016009] req@ffff9862d15b6c00 x1624932080549808/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550016016 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 16:00:16 fir-io1-s1 kernel: Lustre: 96328:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 12 16:00:51 fir-io1-s1 kernel: Lustre: 96251:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550016044/real 1550016044] req@ffff985647ddb900 x1624932080549840/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550016051 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 16:00:51 fir-io1-s1 kernel: Lustre: 96251:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 12 16:02:00 fir-io1-s1 kernel: Lustre: 96950:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550016113/real 1550016113] req@ffff986c617d6f00 x1624932080678496/t0(0) o104->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550016120 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 16:02:00 fir-io1-s1 kernel: Lustre: 96950:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Feb 12 16:02:26 fir-io1-s1 kernel: LNet: Service thread pid 96516 was inactive for 200.31s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 12 16:02:26 fir-io1-s1 kernel: Pid: 96516, comm: ll_ost01_060 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 12 16:02:26 fir-io1-s1 kernel: Call Trace: Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 12 16:02:26 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 12 16:02:26 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550016146.96516 Feb 12 16:02:27 fir-io1-s1 kernel: LNet: Service thread pid 96778 was inactive for 201.78s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 12 16:02:27 fir-io1-s1 kernel: Pid: 96778, comm: ll_ost01_073 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 12 16:02:27 fir-io1-s1 kernel: Call Trace: Feb 12 16:02:27 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 12 16:02:27 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 12 16:02:28 fir-io1-s1 kernel: Pid: 96251, comm: ll_ost01_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 12 16:02:28 fir-io1-s1 kernel: Call Trace: Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 12 16:02:28 fir-io1-s1 kernel: Pid: 96328, comm: ll_ost02_022 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 12 16:02:28 fir-io1-s1 kernel: Call Trace: Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 12 16:02:28 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 12 16:02:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1ac6a39a-851f-0b86-b0cc-450029b6e9a4 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678448f400, cur 1550016162 expire 1550016012 last 1550015935 Feb 12 16:02:42 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 16:02:42 fir-io1-s1 kernel: LNet: Service thread pid 96251 completed after 215.86s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 12 16:02:42 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 12 16:07:44 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8d1694af-d32c-3288-00b1-fe38a72175fb (at 10.9.104.37@o2ib4) Feb 12 16:07:44 fir-io1-s1 kernel: Lustre: Skipped 878 previous similar messages Feb 12 16:15:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Client fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) reconnecting Feb 12 16:15:08 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 16:15:09 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:304845 to 0x6c0000402:304865 Feb 12 16:15:09 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:304854 to 0x8c0000401:304897 Feb 12 16:15:09 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:188327 to 0x6c0000401:188353 Feb 12 16:17:45 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.4.20@o2ib6) Feb 12 16:17:45 fir-io1-s1 kernel: Lustre: Skipped 654 previous similar messages Feb 12 16:25:03 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client d08cee71-06f7-ee6a-b86d-21b77ac0f41b (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7ec400, cur 1550017503 expire 1550017353 last 1550017276 Feb 12 16:25:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 16:27:48 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 33893189-0abb-4d15-5624-4b6aa533cb69 (at 10.8.8.4@o2ib6) Feb 12 16:27:48 fir-io1-s1 kernel: Lustre: Skipped 567 previous similar messages Feb 12 16:37:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to db555f2c-d8fc-5eb9-ee6a-aef92ac58ee4 (at 10.8.28.5@o2ib6) Feb 12 16:37:52 fir-io1-s1 kernel: Lustre: Skipped 478 previous similar messages Feb 12 16:47:53 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 74576cb8-d6cd-cd3f-fbd2-32131b2925d8 (at 10.9.107.19@o2ib4) Feb 12 16:47:53 fir-io1-s1 kernel: Lustre: Skipped 544 previous similar messages Feb 12 16:58:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e25277b-1ad0-ec5f-2777-56d7cafdcd31 (at 10.8.17.6@o2ib6) Feb 12 16:58:08 fir-io1-s1 kernel: Lustre: Skipped 879 previous similar messages Feb 12 17:08:11 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.1.36@o2ib6) Feb 12 17:08:11 fir-io1-s1 kernel: Lustre: Skipped 783 previous similar messages Feb 12 17:18:15 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) Feb 12 17:18:15 fir-io1-s1 kernel: Lustre: Skipped 737 previous similar messages Feb 12 17:28:15 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 64d3667d-1c77-7867-b1f3-ec9c4a6035ad (at 10.9.104.56@o2ib4) Feb 12 17:28:15 fir-io1-s1 kernel: Lustre: Skipped 626 previous similar messages Feb 12 17:38:16 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 71eebf52-b0fc-514f-7aaf-aca66e4f2af1 (at 10.8.27.27@o2ib6) Feb 12 17:38:16 fir-io1-s1 kernel: Lustre: Skipped 801 previous similar messages Feb 12 17:40:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 17:40:07 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 17:40:20 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 17:40:20 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 17:40:26 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:995827 to 0x580000400:995905 Feb 12 17:40:35 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:995364 to 0x5c0000400:995393 Feb 12 17:40:36 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:995185 to 0xc40000402:995201 Feb 12 17:40:36 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:995578 to 0x6c0000400:995617 Feb 12 17:40:39 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:995274 to 0xc80000402:995297 Feb 12 17:40:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 17:40:41 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:995395 to 0x5c0000400:995425 Feb 12 17:40:42 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:783087 to 0x0:783105 Feb 12 17:40:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 12 17:40:52 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 17:40:52 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:782668 to 0x0:782689 Feb 12 17:48:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 283673e9-8136-ebb7-35e9-2d12f60edf66 (at 10.9.105.23@o2ib4) Feb 12 17:48:16 fir-io1-s1 kernel: Lustre: Skipped 942 previous similar messages Feb 12 17:51:32 fir-io1-s1 kernel: Lustre: 96905:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550022685/real 1550022685] req@ffff9841155df500 x1624932087445328/t0(0) o106->fir-OST0004@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550022692 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 12 17:51:32 fir-io1-s1 kernel: Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550022685/real 1550022685] req@ffff983b08781800 x1624932087445424/t0(0) o106->fir-OST0008@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550022692 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 12 17:51:32 fir-io1-s1 kernel: Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Feb 12 17:51:32 fir-io1-s1 kernel: Lustre: 96905:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 12 17:51:41 fir-io1-s1 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x1a4b7ac34b47bad2 to 0x6bcd99f60a0d4c23 Feb 12 17:51:53 fir-io1-s1 kernel: Lustre: 96252:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550022706/real 1550022706] req@ffff983d1d6a1e00 x1624932087445408/t0(0) o106->fir-OST0006@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550022713 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 17:51:53 fir-io1-s1 kernel: Lustre: 96252:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 12 17:52:28 fir-io1-s1 kernel: Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550022741/real 1550022741] req@ffff983b08781800 x1624932087445424/t0(0) o106->fir-OST0008@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550022748 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 17:52:28 fir-io1-s1 kernel: Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 12 17:53:38 fir-io1-s1 kernel: Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550022811/real 1550022811] req@ffff983b08781800 x1624932087445424/t0(0) o106->fir-OST0008@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550022818 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 17:53:38 fir-io1-s1 kernel: Lustre: 94314:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550022811/real 1550022811] req@ffff9841155d9500 x1624932087445440/t0(0) o106->fir-OST000a@10.8.26.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550022818 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 17:53:38 fir-io1-s1 kernel: Lustre: 94314:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 36 previous similar messages Feb 12 17:53:38 fir-io1-s1 kernel: Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 12 17:54:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client dcbd3055-70f8-e9e4-add1-7062ac39fbe8 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832bd8800, cur 1550022856 expire 1550022706 last 1550022629 Feb 12 17:54:16 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 17:58:16 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Feb 12 17:58:16 fir-io1-s1 kernel: Lustre: Skipped 1052 previous similar messages Feb 12 18:08:16 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8f86419d-3c7d-f8e0-fb5d-facc0f493f73 (at 10.8.27.31@o2ib6) Feb 12 18:08:16 fir-io1-s1 kernel: Lustre: Skipped 1621 previous similar messages Feb 12 18:18:16 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to edc5d3f2-dae7-69fe-e7fb-9c4cf59a4b4c (at 10.8.31.8@o2ib6) Feb 12 18:18:16 fir-io1-s1 kernel: Lustre: Skipped 2343 previous similar messages Feb 12 18:28:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to fb00bf6b-d767-c6bd-b589-d3608416a604 (at 10.9.105.44@o2ib4) Feb 12 18:28:17 fir-io1-s1 kernel: Lustre: Skipped 1853 previous similar messages Feb 12 18:30:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ee3e4832-6062-2522-4b30-5123e13593ba (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98783bf56800, cur 1550025036 expire 1550024886 last 1550024809 Feb 12 18:30:36 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 12 18:38:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 57269b50-0465-38e8-cbda-995a4a22296e (at 10.8.8.1@o2ib6) Feb 12 18:38:17 fir-io1-s1 kernel: Lustre: Skipped 1648 previous similar messages Feb 12 18:48:17 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 57269b50-0465-38e8-cbda-995a4a22296e (at 10.8.8.1@o2ib6) Feb 12 18:48:17 fir-io1-s1 kernel: Lustre: Skipped 2339 previous similar messages Feb 12 18:54:28 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8e017b11-002d-91d8-bdba-87be112df3ff (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043fa800, cur 1550026468 expire 1550026318 last 1550026241 Feb 12 18:54:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 18:58:17 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to c7981bef-8624-1b06-32b3-1f88bc1711f2 (at 10.8.8.16@o2ib6) Feb 12 18:58:17 fir-io1-s1 kernel: Lustre: Skipped 1678 previous similar messages Feb 12 19:08:17 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 72311aa4-1605-5f7b-7c5e-b154cc799618 (at 10.8.25.23@o2ib6) Feb 12 19:08:17 fir-io1-s1 kernel: Lustre: Skipped 1391 previous similar messages Feb 12 19:13:10 fir-io1-s1 kernel: LustreError: 96592:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1561908(2610484) req@ffff987823a6d850 x1625108336999136/t0(0) o4->97812235-e854-1397-0493-02d3b2b70974@10.9.112.14@o2ib4:526/0 lens 488/448 e 3 to 0 dl 1550027606 ref 1 fl Interpret:/0/0 rc 0/0 Feb 12 19:13:10 fir-io1-s1 kernel: Lustre: fir-OST0002: Bulk IO write error with 97812235-e854-1397-0493-02d3b2b70974 (at 10.9.112.14@o2ib4), client will retry: rc = -110 Feb 12 19:15:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 97812235-e854-1397-0493-02d3b2b70974 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785e19c00, cur 1550027733 expire 1550027583 last 1550027506 Feb 12 19:15:51 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 97812235-e854-1397-0493-02d3b2b70974 (at 10.9.112.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b90ec00, cur 1550027751 expire 1550027601 last 1550027524 Feb 12 19:15:51 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 12 19:18:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 31adb3b3-0369-4500-50b5-f7d1316c2b2a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d5d800, cur 1550027888 expire 1550027738 last 1550027661 Feb 12 19:18:12 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 31adb3b3-0369-4500-50b5-f7d1316c2b2a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283d9400, cur 1550027892 expire 1550027742 last 1550027665 Feb 12 19:18:12 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 19:18:17 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) Feb 12 19:18:17 fir-io1-s1 kernel: Lustre: Skipped 1005 previous similar messages Feb 12 19:18:27 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 31adb3b3-0369-4500-50b5-f7d1316c2b2a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d5cc00, cur 1550027907 expire 1550027757 last 1550027680 Feb 12 19:18:38 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 31adb3b3-0369-4500-50b5-f7d1316c2b2a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786887800, cur 1550027918 expire 1550027768 last 1550027691 Feb 12 19:18:38 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 19:28:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1cc72755-0b18-692a-013c-e5abb0ad9b59 (at 10.9.106.44@o2ib4) Feb 12 19:28:18 fir-io1-s1 kernel: Lustre: Skipped 1378 previous similar messages Feb 12 19:38:18 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 12 19:38:18 fir-io1-s1 kernel: Lustre: Skipped 1425 previous similar messages Feb 12 19:46:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9cc869db-86b5-83f4-3d1d-91c30da3920c (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d649c00, cur 1550029569 expire 1550029419 last 1550029342 Feb 12 19:48:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to bab702f8-b44a-da13-8f71-e38d2f6bf022 (at 10.8.1.13@o2ib6) Feb 12 19:48:19 fir-io1-s1 kernel: Lustre: Skipped 1415 previous similar messages Feb 12 19:58:22 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 26276ee4-1318-03d7-bf25-08cb51193a9d (at 10.9.102.22@o2ib4) Feb 12 19:58:22 fir-io1-s1 kernel: Lustre: Skipped 1377 previous similar messages Feb 12 20:01:24 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5ea7cc5a-0450-8799-326b-cbf513f8c9b6 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ca34c00, cur 1550030484 expire 1550030334 last 1550030257 Feb 12 20:01:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 20:02:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b7631cb3-40a3-08be-ae74-cf548ae0665c (at 10.8.14.8@o2ib6) in 152 seconds. I think it's dead, and I am evicting it. exp ffff984998f83400, cur 1550030560 expire 1550030410 last 1550030408 Feb 12 20:02:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 20:03:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b7631cb3-40a3-08be-ae74-cf548ae0665c (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811213400, cur 1550030635 expire 1550030485 last 1550030408 Feb 12 20:03:55 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 20:08:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 58531951-dcfc-2dad-91c4-688aefd85811 (at 10.9.104.5@o2ib4) Feb 12 20:08:23 fir-io1-s1 kernel: Lustre: Skipped 1813 previous similar messages Feb 12 20:18:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) Feb 12 20:18:23 fir-io1-s1 kernel: Lustre: Skipped 1943 previous similar messages Feb 12 20:20:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 964de41a-4f96-b1b4-c925-03a276fa8d60 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f5400, cur 1550031658 expire 1550031508 last 1550031431 Feb 12 20:20:58 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 12 20:28:25 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 02904912-fe2e-5026-bc07-2718cbca6fa6 (at 10.8.2.34@o2ib6) Feb 12 20:28:25 fir-io1-s1 kernel: Lustre: Skipped 1421 previous similar messages Feb 12 20:38:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4f44a399-702a-8d9a-166c-3e54590e6073 (at 10.8.10.16@o2ib6) Feb 12 20:38:26 fir-io1-s1 kernel: Lustre: Skipped 1781 previous similar messages Feb 12 20:48:26 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to b0a96e4d-c160-6440-3ca8-b81b22c6458c (at 10.8.7.11@o2ib6) Feb 12 20:48:26 fir-io1-s1 kernel: Lustre: Skipped 1819 previous similar messages Feb 12 20:53:50 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 356dbaf4-5dce-52ea-ea28-c77e8bb74769 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677be37800, cur 1550033630 expire 1550033480 last 1550033403 Feb 12 20:53:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 20:54:09 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 356dbaf4-5dce-52ea-ea28-c77e8bb74769 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848324a7400, cur 1550033649 expire 1550033499 last 1550033422 Feb 12 20:54:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 356dbaf4-5dce-52ea-ea28-c77e8bb74769 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f67ac00, cur 1550033659 expire 1550033509 last 1550033432 Feb 12 20:54:19 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 20:54:19 fir-io1-s1 kernel: LustreError: 96914:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) returned error from glimpse AST (req@ffff983e56474e00 x1624932119436688 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff984c49f33cc0/0x49e185e94e530e3c lrc: 3/0,0 mode: PW/PW res: [0x5c0000402:0x4b582:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x40000000000000 nid: 10.8.3.11@o2ib6 remote: 0x41ff132a621ea3ea expref: 110 pid: 96916 timeout: 0 lvb_type: 0 Feb 12 20:54:19 fir-io1-s1 kernel: LustreError: 96914:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 12 20:54:19 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 12 20:54:19 fir-io1-s1 kernel: LustreError: Skipped 6 previous similar messages Feb 12 20:54:19 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550033659s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff984c49f33cc0/0x49e185e94e530e3c lrc: 3/0,0 mode: PW/PW res: [0x5c0000402:0x4b582:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x40000000000000 nid: 10.8.3.11@o2ib6 remote: 0x41ff132a621ea3ea expref: 111 pid: 96916 timeout: 0 lvb_type: 0 Feb 12 20:54:19 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Feb 12 20:54:21 fir-io1-s1 kernel: LustreError: 96772:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) returned error from glimpse AST (req@ffff9845e78dc500 x1624932119463616 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff9847806b8d80/0x49e185e94e529d78 lrc: 3/0,0 mode: PW/PW res: [0x6c0000402:0x4b10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.3.11@o2ib6 remote: 0x41ff132a621e2ab3 expref: 125 pid: 96899 timeout: 0 lvb_type: 0 Feb 12 20:54:21 fir-io1-s1 kernel: LustreError: 96772:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Feb 12 20:54:21 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 12 20:54:21 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 12 20:54:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550033661s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff9847806b8d80/0x49e185e94e529d78 lrc: 3/0,0 mode: PW/PW res: [0x6c0000402:0x4b10d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.3.11@o2ib6 remote: 0x41ff132a621e2ab3 expref: 126 pid: 96899 timeout: 0 lvb_type: 0 Feb 12 20:54:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Feb 12 20:58:28 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 97bc8e0c-1614-4de0-a593-98b585b7fd0b (at 10.9.103.30@o2ib4) Feb 12 20:58:28 fir-io1-s1 kernel: Lustre: Skipped 1718 previous similar messages Feb 12 21:08:28 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to fb00bf6b-d767-c6bd-b589-d3608416a604 (at 10.9.105.44@o2ib4) Feb 12 21:08:28 fir-io1-s1 kernel: Lustre: Skipped 2587 previous similar messages Feb 12 21:15:43 fir-io1-s1 kernel: Lustre: 96518:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550034936/real 1550034936] req@ffff9847f1c72700 x1624932122242256/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550034943 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 12 21:15:43 fir-io1-s1 kernel: Lustre: 96518:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Feb 12 21:16:18 fir-io1-s1 kernel: Lustre: 96764:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550034971/real 1550034971] req@ffff984302e80300 x1624932122242272/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550034978 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 21:16:18 fir-io1-s1 kernel: Lustre: 96764:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 12 21:17:28 fir-io1-s1 kernel: Lustre: 96240:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550035041/real 1550035041] req@ffff983aa180d100 x1624932122242320/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550035048 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 21:17:28 fir-io1-s1 kernel: Lustre: 96240:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 36 previous similar messages Feb 12 21:17:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 955ff48b-8b0e-bb9b-72ea-64873490b461 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7ed800, cur 1550035078 expire 1550034928 last 1550034851 Feb 12 21:18:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 955ff48b-8b0e-bb9b-72ea-64873490b461 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986838666c00, cur 1550035081 expire 1550034931 last 1550034854 Feb 12 21:18:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 955ff48b-8b0e-bb9b-72ea-64873490b461 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986838660000, cur 1550035086 expire 1550034936 last 1550034859 Feb 12 21:18:06 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 21:18:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 56f157dc-9f05-a349-697a-ac16ba31313e (at 10.9.104.59@o2ib4) Feb 12 21:18:32 fir-io1-s1 kernel: Lustre: Skipped 2151 previous similar messages Feb 12 21:28:36 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 50290e53-c65d-a70c-6960-ed601e5d1ddb (at 10.8.1.35@o2ib6) Feb 12 21:28:36 fir-io1-s1 kernel: Lustre: Skipped 2499 previous similar messages Feb 12 21:38:36 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 2513721a-1a7a-beeb-ce96-babfef130551 (at 10.8.18.32@o2ib6) Feb 12 21:38:36 fir-io1-s1 kernel: Lustre: Skipped 2346 previous similar messages Feb 12 21:48:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 8ee660c6-3e54-f054-0509-b448082e8dec (at 10.9.105.32@o2ib4) Feb 12 21:48:37 fir-io1-s1 kernel: Lustre: Skipped 2199 previous similar messages Feb 12 21:58:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 82a1e830-469b-4bea-5c90-29fa850618a7 (at 10.8.24.13@o2ib6) Feb 12 21:58:38 fir-io1-s1 kernel: Lustre: Skipped 2606 previous similar messages Feb 12 22:08:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.8.9@o2ib6) Feb 12 22:08:38 fir-io1-s1 kernel: Lustre: Skipped 2748 previous similar messages Feb 12 22:18:38 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f3023e70-967c-cdca-d170-324afefea199 (at 10.9.106.49@o2ib4) Feb 12 22:18:38 fir-io1-s1 kernel: Lustre: Skipped 2704 previous similar messages Feb 12 22:28:42 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 283673e9-8136-ebb7-35e9-2d12f60edf66 (at 10.9.105.23@o2ib4) Feb 12 22:28:42 fir-io1-s1 kernel: Lustre: Skipped 2713 previous similar messages Feb 12 22:38:43 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 85cfcf77-29ac-d755-b385-af543ebdafc6 (at 10.9.101.31@o2ib4) Feb 12 22:38:43 fir-io1-s1 kernel: Lustre: Skipped 2645 previous similar messages Feb 12 22:48:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1b71b1a8-fdc3-550f-13e6-42b0376dd743 (at 10.9.112.8@o2ib4) Feb 12 22:48:43 fir-io1-s1 kernel: Lustre: Skipped 2332 previous similar messages Feb 12 22:58:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fbb7e0da-3603-8dfb-de71-fd8cea5618ef (at 10.9.106.69@o2ib4) Feb 12 22:58:43 fir-io1-s1 kernel: Lustre: Skipped 2379 previous similar messages Feb 12 23:04:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client eaf59f85-a4cb-323d-5061-5e1bf30e1e75 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986813a77c00, cur 1550041479 expire 1550041329 last 1550041252 Feb 12 23:04:39 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 23:08:43 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ddfc24dc-ab35-b5d1-5ce6-6e97aa901210 (at 10.9.107.16@o2ib4) Feb 12 23:08:43 fir-io1-s1 kernel: Lustre: Skipped 2337 previous similar messages Feb 12 23:18:43 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to be293349-45a5-91c6-c8b7-456ff508fdc0 (at 10.8.1.31@o2ib6) Feb 12 23:18:43 fir-io1-s1 kernel: Lustre: Skipped 2257 previous similar messages Feb 12 23:28:44 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 23f9a8de-c6a0-76ea-9378-57e47966c02c (at 10.8.21.23@o2ib6) Feb 12 23:28:44 fir-io1-s1 kernel: Lustre: Skipped 2002 previous similar messages Feb 12 23:36:15 fir-io1-s1 kernel: Lustre: 96768:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550043368/real 1550043368] req@ffff9840b6c7f800 x1624932137845312/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550043375 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 12 23:36:15 fir-io1-s1 kernel: Lustre: 96768:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 12 23:36:36 fir-io1-s1 kernel: Lustre: 96760:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550043389/real 1550043389] req@ffff983b08785700 x1624932137845360/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550043396 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 23:36:36 fir-io1-s1 kernel: Lustre: 96760:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 12 23:37:11 fir-io1-s1 kernel: Lustre: 96329:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550043424/real 1550043424] req@ffff9847f1c72100 x1624932137845328/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550043431 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 23:37:11 fir-io1-s1 kernel: Lustre: 96329:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 12 23:38:21 fir-io1-s1 kernel: Lustre: 96770:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550043494/real 1550043494] req@ffff984302e84800 x1624932137845344/t0(0) o106->fir-OST0000@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550043501 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 12 23:38:21 fir-io1-s1 kernel: Lustre: 96770:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 38 previous similar messages Feb 12 23:38:44 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to cb55eece-21bc-a42b-f0ac-74d5957e3321 (at 10.8.21.35@o2ib6) Feb 12 23:38:44 fir-io1-s1 kernel: Lustre: Skipped 1793 previous similar messages Feb 12 23:39:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client fbf97e21-d7bf-78c2-a67f-f6e04220bc0c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576855b800, cur 1550043542 expire 1550043392 last 1550043315 Feb 12 23:39:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 12 23:39:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fbf97e21-d7bf-78c2-a67f-f6e04220bc0c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5e800, cur 1550043548 expire 1550043398 last 1550043321 Feb 12 23:39:08 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 12 23:39:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fbf97e21-d7bf-78c2-a67f-f6e04220bc0c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5bc00, cur 1550043553 expire 1550043403 last 1550043326 Feb 12 23:39:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fbf97e21-d7bf-78c2-a67f-f6e04220bc0c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c6dfc00, cur 1550043556 expire 1550043406 last 1550043329 Feb 12 23:48:43 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9fcee8f7-97bf-b7d8-e8fc-46d398ed949c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c2800, cur 1550044123 expire 1550043973 last 1550043896 Feb 12 23:48:44 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 135b49e4-9bbb-43fc-66e9-1f7ec8c75a96 (at 10.9.113.3@o2ib4) Feb 12 23:48:44 fir-io1-s1 kernel: Lustre: Skipped 2055 previous similar messages Feb 12 23:48:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9fcee8f7-97bf-b7d8-e8fc-46d398ed949c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b6f400, cur 1550044131 expire 1550043981 last 1550043904 Feb 12 23:48:51 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 12 23:58:48 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to f74f6780-500b-a99c-769a-05932d2be074 (at 10.9.102.10@o2ib4) Feb 12 23:58:48 fir-io1-s1 kernel: Lustre: Skipped 2263 previous similar messages Feb 13 00:08:49 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b7a4f2fb-7087-2da1-fe97-a6b3f3961662 (at 10.9.103.38@o2ib4) Feb 13 00:08:49 fir-io1-s1 kernel: Lustre: Skipped 2400 previous similar messages Feb 13 00:17:25 fir-io1-s1 kernel: Lustre: 96518:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045838/real 1550045838] req@ffff984302e87b00 x1624932142167408/t0(0) o106->fir-OST0004@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045845 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 00:17:25 fir-io1-s1 kernel: Lustre: 94235:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045838/real 1550045838] req@ffff9840b6c7da00 x1624932142167392/t0(0) o106->fir-OST0000@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045845 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 00:17:25 fir-io1-s1 kernel: Lustre: 96290:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045838/real 1550045838] req@ffff9840b6c7b600 x1624932142167360/t0(0) o106->fir-OST000a@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045845 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 00:17:25 fir-io1-s1 kernel: Lustre: 94235:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Feb 13 00:17:25 fir-io1-s1 kernel: Lustre: 96290:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Feb 13 00:17:32 fir-io1-s1 kernel: Lustre: 94235:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045845/real 1550045845] req@ffff9840b6c7da00 x1624932142167392/t0(0) o106->fir-OST0000@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045852 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 00:17:32 fir-io1-s1 kernel: Lustre: 96290:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045845/real 1550045845] req@ffff9840b6c7b600 x1624932142167360/t0(0) o106->fir-OST000a@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045852 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 00:17:32 fir-io1-s1 kernel: Lustre: 94235:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 13 00:17:39 fir-io1-s1 kernel: Lustre: 96290:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045852/real 1550045852] req@ffff9840b6c7b600 x1624932142167360/t0(0) o106->fir-OST000a@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045859 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 00:17:39 fir-io1-s1 kernel: Lustre: 96290:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 13 00:17:53 fir-io1-s1 kernel: Lustre: 96754:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045866/real 1550045866] req@ffff983bdaf80f00 x1624932142167376/t0(0) o106->fir-OST0002@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045873 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 00:17:53 fir-io1-s1 kernel: Lustre: 96754:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 13 00:18:14 fir-io1-s1 kernel: Lustre: 96754:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550045887/real 1550045887] req@ffff983bdaf80f00 x1624932142167376/t0(0) o106->fir-OST0002@10.8.11.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550045894 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 00:18:14 fir-io1-s1 kernel: Lustre: 96754:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Feb 13 00:18:36 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 807d7fbf-b301-cb35-ff6f-385b0544f088 (at 10.8.11.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c211000, cur 1550045916 expire 1550045766 last 1550045689 Feb 13 00:18:49 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 4f44a399-702a-8d9a-166c-3e54590e6073 (at 10.8.10.16@o2ib6) Feb 13 00:18:49 fir-io1-s1 kernel: Lustre: Skipped 2323 previous similar messages Feb 13 00:28:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 60c991c9-bbb6-b45f-4869-dff10ed664d7 (at 10.8.3.25@o2ib6) Feb 13 00:28:50 fir-io1-s1 kernel: Lustre: Skipped 2407 previous similar messages Feb 13 00:29:04 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0016f7ca-8f34-8f96-ecfb-03f782a1cda7 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784ac5000, cur 1550046544 expire 1550046394 last 1550046317 Feb 13 00:29:04 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 00:38:52 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e73046bd-a1a7-71eb-f27a-0923a1632ebd (at 10.8.23.23@o2ib6) Feb 13 00:38:52 fir-io1-s1 kernel: Lustre: Skipped 2516 previous similar messages Feb 13 00:48:52 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 61244de0-3d61-3ad3-fe92-f92a9f896d83 (at 10.9.107.23@o2ib4) Feb 13 00:48:52 fir-io1-s1 kernel: Lustre: Skipped 2157 previous similar messages Feb 13 00:58:53 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 09e96f09-88f3-17cc-ba26-88dee3b61d1c (at 10.8.7.12@o2ib6) Feb 13 00:58:53 fir-io1-s1 kernel: Lustre: Skipped 2187 previous similar messages Feb 13 01:08:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15339979-51e5-e16d-f976-ff72d24bd14f (at 10.8.9.10@o2ib6) Feb 13 01:08:53 fir-io1-s1 kernel: Lustre: Skipped 2361 previous similar messages Feb 13 01:18:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4944a2ce-d92c-784e-9536-c95b01530191 (at 10.9.101.48@o2ib4) Feb 13 01:18:53 fir-io1-s1 kernel: Lustre: Skipped 2019 previous similar messages Feb 13 01:28:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 82ffae03-c02c-86e8-2dc8-ed4f97ac9c9d (at 10.8.25.9@o2ib6) Feb 13 01:28:54 fir-io1-s1 kernel: Lustre: Skipped 2413 previous similar messages Feb 13 01:38:55 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 4cdb4c1f-c631-73f5-0cc6-576f1959d1eb (at 10.8.17.8@o2ib6) Feb 13 01:38:55 fir-io1-s1 kernel: Lustre: Skipped 1989 previous similar messages Feb 13 01:48:55 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 874d84ab-2918-f27e-a1fe-cdc3435eb5ad (at 10.8.2.18@o2ib6) Feb 13 01:48:55 fir-io1-s1 kernel: Lustre: Skipped 1761 previous similar messages Feb 13 01:58:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 02904912-fe2e-5026-bc07-2718cbca6fa6 (at 10.8.2.34@o2ib6) Feb 13 01:58:56 fir-io1-s1 kernel: Lustre: Skipped 2231 previous similar messages Feb 13 02:08:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.106.61@o2ib4) Feb 13 02:08:58 fir-io1-s1 kernel: Lustre: Skipped 1594 previous similar messages Feb 13 02:16:16 fir-io1-s1 kernel: Lustre: 96245:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550052969/real 1550052969] req@ffff983b08784200 x1624932154877456/t0(0) o106->fir-OST000a@10.8.12.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550052976 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 02:16:16 fir-io1-s1 kernel: Lustre: 96245:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Feb 13 02:16:23 fir-io1-s1 kernel: Lustre: 96899:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550052976/real 1550052976] req@ffff984302e85a00 x1624932154877488/t0(0) o106->fir-OST0002@10.8.12.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550052983 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 02:16:23 fir-io1-s1 kernel: Lustre: 96569:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550052976/real 1550052976] req@ffff983d1d6a4e00 x1624932154877424/t0(0) o106->fir-OST0008@10.8.12.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550052983 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 02:16:23 fir-io1-s1 kernel: Lustre: 96899:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 13 02:16:44 fir-io1-s1 kernel: Lustre: 96569:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550052997/real 1550052997] req@ffff983d1d6a4e00 x1624932154877424/t0(0) o106->fir-OST0008@10.8.12.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550053004 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 02:16:44 fir-io1-s1 kernel: Lustre: 96889:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550052997/real 1550052997] req@ffff983ee8a53c00 x1624932154877536/t0(0) o106->fir-OST0000@10.8.12.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550053004 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 02:16:44 fir-io1-s1 kernel: Lustre: 96889:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 13 02:16:44 fir-io1-s1 kernel: Lustre: 96569:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 13 02:17:54 fir-io1-s1 kernel: Lustre: 96889:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550053067/real 1550053067] req@ffff983ee8a53c00 x1624932154877536/t0(0) o106->fir-OST0000@10.8.12.33@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550053074 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 02:17:54 fir-io1-s1 kernel: Lustre: 96889:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 36 previous similar messages Feb 13 02:18:59 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) Feb 13 02:18:59 fir-io1-s1 kernel: Lustre: Skipped 838 previous similar messages Feb 13 02:19:27 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 22ee14eb-5a96-ad04-6e5f-188b7aec897d (at 10.8.12.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1550400, cur 1550053167 expire 1550053017 last 1550052940 Feb 13 02:19:27 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 02:29:00 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ef406a33-dbdc-c381-15c5-4fb662abecc1 (at 10.8.27.10@o2ib6) Feb 13 02:29:00 fir-io1-s1 kernel: Lustre: Skipped 1741 previous similar messages Feb 13 02:39:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f34aebb6-59fc-4cef-28dc-df15ded97223 (at 10.9.105.62@o2ib4) Feb 13 02:39:02 fir-io1-s1 kernel: Lustre: Skipped 978 previous similar messages Feb 13 02:49:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 121431d1-d11c-6cf2-7fd0-eaf8c3ca6b3e (at 10.9.103.12@o2ib4) Feb 13 02:49:03 fir-io1-s1 kernel: Lustre: Skipped 1424 previous similar messages Feb 13 02:59:05 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 38ab0677-0c51-35a5-8e38-bb5f254042a2 (at 10.8.17.4@o2ib6) Feb 13 02:59:05 fir-io1-s1 kernel: Lustre: Skipped 992 previous similar messages Feb 13 03:09:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a18587f4-6669-8a89-c311-224538b5a6f2 (at 10.8.27.32@o2ib6) Feb 13 03:09:08 fir-io1-s1 kernel: Lustre: Skipped 1347 previous similar messages Feb 13 03:19:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7a4f2fb-7087-2da1-fe97-a6b3f3961662 (at 10.9.103.38@o2ib4) Feb 13 03:19:10 fir-io1-s1 kernel: Lustre: Skipped 1048 previous similar messages Feb 13 03:29:11 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 111255c5-0b7f-306e-408c-6abf6623385a (at 10.9.104.46@o2ib4) Feb 13 03:29:11 fir-io1-s1 kernel: Lustre: Skipped 902 previous similar messages Feb 13 03:39:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e6e1afb6-7acc-3808-2f05-02b79c99637e (at 10.8.23.5@o2ib6) Feb 13 03:39:11 fir-io1-s1 kernel: Lustre: Skipped 903 previous similar messages Feb 13 03:49:11 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 4cdb4c1f-c631-73f5-0cc6-576f1959d1eb (at 10.8.17.8@o2ib6) Feb 13 03:49:11 fir-io1-s1 kernel: Lustre: Skipped 721 previous similar messages Feb 13 03:59:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.104.60@o2ib4) Feb 13 03:59:14 fir-io1-s1 kernel: Lustre: Skipped 1251 previous similar messages Feb 13 04:09:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 4965ee7b-9b83-7eaf-bb5b-67e7cf8abc63 (at 10.8.20.35@o2ib6) Feb 13 04:09:18 fir-io1-s1 kernel: Lustre: Skipped 908 previous similar messages Feb 13 04:19:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e7658bee-b529-b857-21ef-217c5e9fe7b7 (at 10.9.113.9@o2ib4) Feb 13 04:19:19 fir-io1-s1 kernel: Lustre: Skipped 985 previous similar messages Feb 13 04:29:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c694e053-04d0-ee79-c9a4-0ace9e2f2c9a (at 10.8.3.29@o2ib6) Feb 13 04:29:22 fir-io1-s1 kernel: Lustre: Skipped 1219 previous similar messages Feb 13 04:39:22 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Feb 13 04:39:22 fir-io1-s1 kernel: Lustre: Skipped 631 previous similar messages Feb 13 04:49:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dfe698bc-81af-2a92-fd5f-f1174b08b3ad (at 10.9.107.7@o2ib4) Feb 13 04:49:22 fir-io1-s1 kernel: Lustre: Skipped 1070 previous similar messages Feb 13 04:59:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f6844397-c58e-1716-47c7-98dc229eec16 (at 10.8.21.11@o2ib6) Feb 13 04:59:24 fir-io1-s1 kernel: Lustre: Skipped 803 previous similar messages Feb 13 05:09:24 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 13 05:09:24 fir-io1-s1 kernel: Lustre: Skipped 1149 previous similar messages Feb 13 05:19:31 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.27.30@o2ib6) Feb 13 05:19:31 fir-io1-s1 kernel: Lustre: Skipped 709 previous similar messages Feb 13 05:29:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.24.2@o2ib6) Feb 13 05:29:33 fir-io1-s1 kernel: Lustre: Skipped 784 previous similar messages Feb 13 05:39:34 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ee98715-4bcf-b4e8-27bc-89f9f237369b (at 10.8.30.5@o2ib6) Feb 13 05:39:34 fir-io1-s1 kernel: Lustre: Skipped 883 previous similar messages Feb 13 05:49:39 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) Feb 13 05:49:39 fir-io1-s1 kernel: Lustre: Skipped 947 previous similar messages Feb 13 05:59:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 691c85d2-0e39-9e6d-1bfd-ecbaccae5366 (at 10.8.2.27@o2ib6) Feb 13 05:59:39 fir-io1-s1 kernel: Lustre: Skipped 1011 previous similar messages Feb 13 06:09:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 78f38211-beb8-aca3-b985-9281f7d5f62c (at 10.8.29.8@o2ib6) Feb 13 06:09:41 fir-io1-s1 kernel: Lustre: Skipped 844 previous similar messages Feb 13 06:19:41 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 032aae30-e439-a130-1f18-efe924baca21 (at 10.9.106.70@o2ib4) Feb 13 06:19:41 fir-io1-s1 kernel: Lustre: Skipped 646 previous similar messages Feb 13 06:29:42 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 720160b3-b510-fd4a-7aef-667fe71d4b1d (at 10.8.27.20@o2ib6) Feb 13 06:29:42 fir-io1-s1 kernel: Lustre: Skipped 699 previous similar messages Feb 13 06:39:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5b896ca8-2947-8e31-025e-233ef4d66e00 (at 10.8.17.11@o2ib6) Feb 13 06:39:42 fir-io1-s1 kernel: Lustre: Skipped 803 previous similar messages Feb 13 06:45:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3724d9a0-17fd-0e0a-31ff-19cac289c2d8 (at 10.8.21.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877c6f03400, cur 1550069113 expire 1550068963 last 1550068886 Feb 13 06:45:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 13 06:49:43 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 95d24266-18da-6c0e-7cc6-3ed980208315 (at 10.8.27.1@o2ib6) Feb 13 06:49:43 fir-io1-s1 kernel: Lustre: Skipped 883 previous similar messages Feb 13 06:59:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 4cdb4c1f-c631-73f5-0cc6-576f1959d1eb (at 10.8.17.8@o2ib6) Feb 13 06:59:50 fir-io1-s1 kernel: Lustre: Skipped 747 previous similar messages Feb 13 07:09:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c95418d8-abb0-d4fc-c763-439a35e76a6c (at 10.9.102.50@o2ib4) Feb 13 07:09:50 fir-io1-s1 kernel: Lustre: Skipped 634 previous similar messages Feb 13 07:11:35 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 54ce30c9-3bed-a43b-b985-a5c2f449d263 (at 10.8.21.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2a400, cur 1550070695 expire 1550070545 last 1550070468 Feb 13 07:19:50 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6db7d291-7912-a662-9ec2-f76af6a57200 (at 10.8.1.7@o2ib6) Feb 13 07:19:50 fir-io1-s1 kernel: Lustre: Skipped 633 previous similar messages Feb 13 07:29:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) Feb 13 07:29:53 fir-io1-s1 kernel: Lustre: Skipped 726 previous similar messages Feb 13 07:39:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 13 07:39:53 fir-io1-s1 kernel: Lustre: Skipped 770 previous similar messages Feb 13 07:49:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 16b26052-afa3-f913-3f13-3b653d2521a8 (at 10.8.30.23@o2ib6) Feb 13 07:49:55 fir-io1-s1 kernel: Lustre: Skipped 671 previous similar messages Feb 13 08:00:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 202fc320-3606-cc07-c5a8-57679ec64217 (at 10.8.22.30@o2ib6) Feb 13 08:00:00 fir-io1-s1 kernel: Lustre: Skipped 731 previous similar messages Feb 13 08:10:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to cd2f196e-9e42-81f8-3052-f3c348cb8b16 (at 10.8.20.32@o2ib6) Feb 13 08:10:03 fir-io1-s1 kernel: Lustre: Skipped 723 previous similar messages Feb 13 08:20:05 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 91d645ed-86a8-bf9b-39ba-6e32b02e94c5 (at 10.9.102.18@o2ib4) Feb 13 08:20:05 fir-io1-s1 kernel: Lustre: Skipped 698 previous similar messages Feb 13 08:30:06 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to fc0522a2-e86f-7812-92f9-18c8c5b33bdc (at 10.9.105.45@o2ib4) Feb 13 08:30:06 fir-io1-s1 kernel: Lustre: Skipped 815 previous similar messages Feb 13 08:40:12 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to cb411050-e808-fe95-2c54-f6cec5d6dc3c (at 10.8.27.9@o2ib6) Feb 13 08:40:12 fir-io1-s1 kernel: Lustre: Skipped 773 previous similar messages Feb 13 08:45:26 fir-io1-s1 kernel: Lustre: 96910:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076319/real 1550076319] req@ffff983ee8a53c00 x1624932177129040/t0(0) o106->fir-OST000a@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076326 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 08:45:26 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076319/real 1550076319] req@ffff984302e81500 x1624932177128960/t0(0) o106->fir-OST0004@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076326 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 08:45:26 fir-io1-s1 kernel: Lustre: 96241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076319/real 1550076319] req@ffff984302e86f00 x1624932177129024/t0(0) o106->fir-OST0008@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076326 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 08:45:26 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 55 previous similar messages Feb 13 08:45:26 fir-io1-s1 kernel: Lustre: 96241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 55 previous similar messages Feb 13 08:45:33 fir-io1-s1 kernel: Lustre: 96241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076326/real 1550076326] req@ffff984302e86f00 x1624932177129024/t0(0) o106->fir-OST0008@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076333 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 08:45:33 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076326/real 1550076326] req@ffff984302e81500 x1624932177128960/t0(0) o106->fir-OST0004@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076333 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 08:45:33 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 13 08:45:47 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076340/real 1550076340] req@ffff983e56474200 x1624932177129008/t0(0) o106->fir-OST0006@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076347 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 08:45:47 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 13 08:46:08 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076361/real 1550076361] req@ffff984302e81500 x1624932177128960/t0(0) o106->fir-OST0004@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076368 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 08:46:08 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 13 08:46:43 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076396/real 1550076396] req@ffff983e56474200 x1624932177129008/t0(0) o106->fir-OST0006@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076403 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 08:46:43 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Feb 13 08:47:33 fir-io1-s1 kernel: LustreError: 96335:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.14.4@o2ib6) returned error from glimpse AST (req@ffff983e56474200 x1624932177129008 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff986569aaca40/0x49e185e95054f665 lrc: 3/0,0 mode: PW/PW res: [0xc40000402:0xff826:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000020000 nid: 10.8.14.4@o2ib6 remote: 0x363af5d5c6130076 expref: 5 pid: 96371 timeout: 0 lvb_type: 0 Feb 13 08:47:33 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.14.4@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 13 08:47:33 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550076453s: evicting client at 10.8.14.4@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff986569aa9200/0x49e185e95054f66c lrc: 3/0,0 mode: PW/PW res: [0xc80000402:0xff879:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.14.4@o2ib6 remote: 0x363af5d5c61300ae expref: 6 pid: 96371 timeout: 0 lvb_type: 0 Feb 13 08:47:33 fir-io1-s1 kernel: LustreError: 96335:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Feb 13 08:47:53 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550076466/real 1550076466] req@ffff984302e81500 x1624932177128960/t0(0) o106->fir-OST0004@10.8.14.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550076473 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 08:47:53 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Feb 13 08:48:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 488cfd93-1121-504d-019d-485c13be114d (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83c000, cur 1550076495 expire 1550076345 last 1550076268 Feb 13 08:50:12 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 82ffae03-c02c-86e8-2dc8-ed4f97ac9c9d (at 10.8.25.9@o2ib6) Feb 13 08:50:12 fir-io1-s1 kernel: Lustre: Skipped 686 previous similar messages Feb 13 09:00:12 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 7589c81e-2d34-67a0-8460-0dfbb24f93e8 (at 10.8.18.6@o2ib6) Feb 13 09:00:12 fir-io1-s1 kernel: Lustre: Skipped 917 previous similar messages Feb 13 09:02:15 fir-io1-s1 kernel: Lustre: fir-OST0004: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:02:15 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:797547 to 0x0:797601 Feb 13 09:02:15 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 09:02:15 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1046880 to 0x8c0000402:1046945 Feb 13 09:02:19 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:02:19 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1047228 to 0x5c0000400:1047265 Feb 13 09:02:19 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 09:02:19 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1047690 to 0x580000400:1047745 Feb 13 09:02:24 fir-io1-s1 kernel: Lustre: fir-OST0004: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:02:24 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:797542 to 0x0:797601 Feb 13 09:03:15 fir-io1-s1 kernel: Lustre: fir-OST000a: Client 82bcfbd3-d6e5-0967-d3f2-c921c94e988c (at 10.9.105.71@o2ib4) reconnecting Feb 13 09:03:20 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:797547 to 0x0:797633 Feb 13 09:04:04 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds Feb 13 09:04:04 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 12 previous similar messages Feb 13 09:04:04 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550077443/real 1550077444] req@ffff985bed875a00 x1624932177868496/t0(0) o400->fir-MDT0002-lwp-OST0002@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550077536 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 13 09:04:04 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0006: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 09:04:04 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0006: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 09:04:04 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 09:04:04 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 09:04:04 fir-io1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail Feb 13 09:04:04 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 13 09:04:05 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0008: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 09:04:05 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:797547 to 0x0:797665 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:797542 to 0x0:797633 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:797895 to 0x0:797985 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:798223 to 0x0:798305 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:798309 to 0x0:798401 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1047750 to 0x580000400:1047777 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1047269 to 0x5c0000400:1047297 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1047433 to 0x6c0000400:1047489 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1046949 to 0x8c0000402:1046977 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1047007 to 0xc40000402:1047073 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1047087 to 0xc80000402:1047105 Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 13 09:04:54 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:797543 to 0x0:797633 Feb 13 09:10:13 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ec7c15c1-1122-48df-e09c-cacc05cb75a8 (at 10.8.1.15@o2ib6) Feb 13 09:10:13 fir-io1-s1 kernel: Lustre: Skipped 767 previous similar messages Feb 13 09:20:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4f2a8620-81f0-e31b-fbad-a029c3256423 (at 10.9.105.43@o2ib4) Feb 13 09:20:14 fir-io1-s1 kernel: Lustre: Skipped 644 previous similar messages Feb 13 09:30:20 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 5bd1312d-d4d9-79f2-e6c0-fb1a0e35eed8 (at 10.8.12.17@o2ib6) Feb 13 09:30:20 fir-io1-s1 kernel: Lustre: Skipped 344 previous similar messages Feb 13 09:38:59 fir-io1-s1 kernel: Lustre: fir-OST0008: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:38:59 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:798573 to 0x0:798593 Feb 13 09:39:02 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:39:03 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:797805 to 0x0:797825 Feb 13 09:39:11 fir-io1-s1 kernel: Lustre: fir-OST0008: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:39:12 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:798573 to 0x0:798625 Feb 13 09:39:13 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST000a: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 09:39:13 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 13 09:39:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:39:20 fir-io1-s1 kernel: Lustre: fir-OST0008: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:39:20 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 09:39:20 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 13 09:39:44 fir-io1-s1 kernel: Lustre: fir-OST0004: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:39:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 09:39:48 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:797806 to 0x0:797825 Feb 13 09:40:09 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1047684 to 0xc40000402:1047713 Feb 13 09:40:11 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:798478 to 0x0:798497 Feb 13 09:40:11 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1047716 to 0xc80000402:1047745 Feb 13 09:40:12 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:798156 to 0x0:798177 Feb 13 09:40:12 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1048390 to 0x580000400:1048417 Feb 13 09:40:14 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1047590 to 0x8c0000402:1047617 Feb 13 09:40:40 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0007_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 13 09:40:40 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 13 09:40:40 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0003_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 13 09:40:40 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 13 09:41:05 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0000-mdtlov_UUID (at 10.0.10.51@o2ib7) reconnecting Feb 13 09:41:05 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 13 09:41:05 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Feb 13 09:41:05 fir-io1-s1 kernel: Lustre: Skipped 121 previous similar messages Feb 13 09:41:05 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:797805 to 0x0:797825 Feb 13 09:50:54 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST000a: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Feb 13 09:50:54 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 04bbddc7-37a9-9b79-7fa8-c451901e5d15 (at 10.9.101.67@o2ib4) Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: Skipped 333 previous similar messages Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1048103 to 0x6c0000400:1048129 Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1047590 to 0x8c0000402:1047649 Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1047912 to 0x5c0000400:1047937 Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1048390 to 0x580000400:1048449 Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1047684 to 0xc40000402:1047745 Feb 13 09:51:07 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1047716 to 0xc80000402:1047777 Feb 13 10:01:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 824f0587-167c-fe30-e5f5-4a8a8b3eb359 (at 10.8.26.17@o2ib6) Feb 13 10:01:08 fir-io1-s1 kernel: Lustre: Skipped 2305 previous similar messages Feb 13 10:11:10 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 50290e53-c65d-a70c-6960-ed601e5d1ddb (at 10.8.1.35@o2ib6) Feb 13 10:11:10 fir-io1-s1 kernel: Lustre: Skipped 596 previous similar messages Feb 13 10:21:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fed507e8-5435-f949-539f-6cb9d563cc12 (at 10.9.106.52@o2ib4) Feb 13 10:21:11 fir-io1-s1 kernel: Lustre: Skipped 658 previous similar messages Feb 13 10:31:11 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 58a24522-e5c5-5b7c-258b-500f9e3166ed (at 10.9.107.49@o2ib4) Feb 13 10:31:11 fir-io1-s1 kernel: Lustre: Skipped 611 previous similar messages Feb 13 10:34:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 59210099-21b5-cecf-8f5b-dbe56f2d7a8e (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786779800, cur 1550082863 expire 1550082713 last 1550082636 Feb 13 10:41:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ed8a9d5a-3c64-88a2-4de9-5c7913d6ef08 (at 10.9.101.15@o2ib4) Feb 13 10:41:12 fir-io1-s1 kernel: Lustre: Skipped 718 previous similar messages Feb 13 10:50:42 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 10:50:42 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 13 10:51:07 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Feb 13 10:51:07 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:193074 to 0x6c0000401:193153 Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:193072 to 0xc40000400:193089 Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:193334 to 0x5c0000401:193377 Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:193087 to 0x8c0000400:193121 Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:193376 to 0x580000401:193473 Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:193229 to 0xc80000400:193249 Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 725d787f-ad1b-18f8-1091-f9fac1b511d9 (at 10.9.106.62@o2ib4) Feb 13 10:51:40 fir-io1-s1 kernel: Lustre: Skipped 1044 previous similar messages Feb 13 10:51:43 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:317260 to 0x5c0000402:317281 Feb 13 10:51:43 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:316188 to 0x8c0000401:316225 Feb 13 10:51:43 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:316147 to 0x6c0000402:316193 Feb 13 10:51:43 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:315980 to 0xc40000401:316033 Feb 13 10:51:43 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:316009 to 0xc80000401:316065 Feb 13 10:51:43 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:317171 to 0x580000402:317217 Feb 13 11:01:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 08958bbc-0f90-1cbd-61ae-768cfa6c9459 (at 10.9.104.69@o2ib4) Feb 13 11:01:41 fir-io1-s1 kernel: Lustre: Skipped 1289 previous similar messages Feb 13 11:11:42 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 13 11:11:42 fir-io1-s1 kernel: Lustre: Skipped 1107 previous similar messages Feb 13 11:21:46 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 13 11:21:46 fir-io1-s1 kernel: Lustre: Skipped 896 previous similar messages Feb 13 11:30:45 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e18eed75-ce52-cc42-be69-772ded053e90 (at 10.8.13.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e02f000, cur 1550086245 expire 1550086095 last 1550086018 Feb 13 11:31:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 85810729-2f82-b4f2-1241-3806d86f03d3 (at 10.8.30.6@o2ib6) Feb 13 11:31:50 fir-io1-s1 kernel: Lustre: Skipped 441 previous similar messages Feb 13 11:32:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 61244de0-3d61-3ad3-fe92-f92a9f896d83 (at 10.9.107.23@o2ib4) reconnecting Feb 13 11:32:33 fir-io1-s1 kernel: Lustre: 91453:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550086351/real 1550086353] req@ffff98661a1aa700 x1624932189012544/t0(0) o400->fir-MDT0002-lwp-OST000a@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550086358 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 13 11:32:33 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 11:32:33 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 11:32:33 fir-io1-s1 kernel: Lustre: 91453:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 13 11:32:35 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Feb 13 11:32:35 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 17 previous similar messages Feb 13 11:32:35 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0004: Connection to fir-MDT0002 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 11:32:35 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 11:32:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 2 seconds Feb 13 11:32:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Feb 13 11:32:37 fir-io1-s1 kernel: Lustre: 91453:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550086351/real 1550086357] req@ffff98661a1aaa00 x1624932189012736/t0(0) o400->fir-MDT0002-lwp-OST0008@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550086358 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 13 11:32:37 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0006: Connection to fir-MDT0002 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 11:32:37 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 11:32:37 fir-io1-s1 kernel: Lustre: 91453:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 13 11:32:38 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds Feb 13 11:32:38 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 13 11:32:41 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Feb 13 11:32:41 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message Feb 13 11:33:08 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000b_UUID: not available for connect from 10.0.10.52@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 13 11:33:08 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: fir-OST000a: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) reconnecting Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1051575 to 0x5c0000400:1051617 Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1051771 to 0x6c0000400:1051809 Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1051268 to 0x8c0000402:1051297 Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1051384 to 0xc40000402:1051425 Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1051421 to 0xc80000402:1051457 Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 13 11:33:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1052079 to 0x580000400:1052097 Feb 13 11:34:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) reconnecting Feb 13 11:34:30 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 11:34:30 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1051771 to 0x6c0000400:1051841 Feb 13 11:34:30 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1051575 to 0x5c0000400:1051649 Feb 13 11:34:30 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1051384 to 0xc40000402:1051457 Feb 13 11:34:48 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 13 11:34:48 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (2): c: 0, oc: 2, rc: 8 Feb 13 11:34:48 fir-io1-s1 kernel: Lustre: 91461:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550086482/real 1550086488] req@ffff985b64803f00 x1624932189027968/t0(0) o400->fir-MDT0002-lwp-OST0004@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550086615 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 13 11:34:48 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0002: Connection to fir-MDT0002 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 11:34:48 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 11:34:48 fir-io1-s1 kernel: Lustre: 91461:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 13 11:34:50 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 2 seconds Feb 13 11:34:50 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message Feb 13 11:34:59 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1052079 to 0x580000400:1052129 Feb 13 11:34:59 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1051421 to 0xc80000402:1051489 Feb 13 11:35:00 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1051268 to 0x8c0000402:1051329 Feb 13 11:35:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) reconnecting Feb 13 11:35:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 13 11:35:15 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1051771 to 0x6c0000400:1051841 Feb 13 11:35:39 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1052079 to 0x580000400:1052129 Feb 13 11:35:39 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1051384 to 0xc40000402:1051489 Feb 13 11:35:39 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1051651 to 0x5c0000400:1051681 Feb 13 11:36:55 fir-io1-s1 kernel: Lustre: 91454:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550086482/real 1550086482] req@ffff985b64804200 x1624932189027648/t0(0) o400->fir-MDT0002-lwp-OST0000@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550086615 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Feb 13 11:36:55 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 11:36:55 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 11:41:50 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 13 11:41:50 fir-io1-s1 kernel: Lustre: Skipped 428 previous similar messages Feb 13 11:51:51 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 83f7575b-b138-8f39-fa8f-a4cda38cceb7 (at 10.9.105.26@o2ib4) Feb 13 11:51:51 fir-io1-s1 kernel: Lustre: Skipped 671 previous similar messages Feb 13 12:01:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9041b005-ca79-5425-d710-65376539b634 (at 10.9.107.59@o2ib4) Feb 13 12:01:52 fir-io1-s1 kernel: Lustre: Skipped 472 previous similar messages Feb 13 12:11:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 37ae9ee6-0608-3719-0e67-b987ecb945fa (at 10.8.6.10@o2ib6) Feb 13 12:11:56 fir-io1-s1 kernel: Lustre: Skipped 391 previous similar messages Feb 13 12:18:38 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5784eab3-8e81-fe4a-0169-374f84d16684 (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800fdf000, cur 1550089118 expire 1550088968 last 1550088891 Feb 13 12:18:38 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 13 12:22:01 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 13 12:22:01 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 13 12:32:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d3b133e8-4ec8-ebe3-7fc5-79aa16e59c0b (at 10.9.106.35@o2ib4) Feb 13 12:32:02 fir-io1-s1 kernel: Lustre: Skipped 308 previous similar messages Feb 13 12:42:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 83f7575b-b138-8f39-fa8f-a4cda38cceb7 (at 10.9.105.26@o2ib4) Feb 13 12:42:11 fir-io1-s1 kernel: Lustre: Skipped 479 previous similar messages Feb 13 12:52:13 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fc738f4c-fd89-da60-f357-76c857400e3c (at 10.9.115.6@o2ib4) Feb 13 12:52:13 fir-io1-s1 kernel: Lustre: Skipped 577 previous similar messages Feb 13 13:02:13 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0127092e-ab70-ddef-6a66-286028d84f5d (at 10.9.102.43@o2ib4) Feb 13 13:02:13 fir-io1-s1 kernel: Lustre: Skipped 561 previous similar messages Feb 13 13:12:15 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 57269b50-0465-38e8-cbda-995a4a22296e (at 10.8.8.1@o2ib6) Feb 13 13:12:15 fir-io1-s1 kernel: Lustre: Skipped 580 previous similar messages Feb 13 13:22:21 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e18ec55f-27df-5b55-2c1d-0fb1ae5cad9b (at 10.8.27.28@o2ib6) Feb 13 13:22:21 fir-io1-s1 kernel: Lustre: Skipped 640 previous similar messages Feb 13 13:32:28 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fb00bf6b-d767-c6bd-b589-d3608416a604 (at 10.9.105.44@o2ib4) Feb 13 13:32:28 fir-io1-s1 kernel: Lustre: Skipped 548 previous similar messages Feb 13 13:42:28 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 76e809db-e90a-7c07-20d4-e3130ed3be85 (at 10.9.104.30@o2ib4) Feb 13 13:42:28 fir-io1-s1 kernel: Lustre: Skipped 744 previous similar messages Feb 13 13:52:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e18ec55f-27df-5b55-2c1d-0fb1ae5cad9b (at 10.8.27.28@o2ib6) Feb 13 13:52:29 fir-io1-s1 kernel: Lustre: Skipped 805 previous similar messages Feb 13 14:02:31 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 71eebf52-b0fc-514f-7aaf-aca66e4f2af1 (at 10.8.27.27@o2ib6) Feb 13 14:02:31 fir-io1-s1 kernel: Lustre: Skipped 620 previous similar messages Feb 13 14:12:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 254b94e8-a059-c2e7-f0ce-7ccba72fe51c (at 10.8.1.34@o2ib6) Feb 13 14:12:35 fir-io1-s1 kernel: Lustre: Skipped 774 previous similar messages Feb 13 14:22:40 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4aa7fc02-5c7e-708e-e06d-9e4448eb0e1b (at 10.9.104.48@o2ib4) Feb 13 14:22:40 fir-io1-s1 kernel: Lustre: Skipped 633 previous similar messages Feb 13 14:23:26 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3775d9cc-6810-2460-c906-100e1a7ecab2 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867814da400, cur 1550096606 expire 1550096456 last 1550096379 Feb 13 14:23:26 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 14:32:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 13 14:32:42 fir-io1-s1 kernel: Lustre: Skipped 275 previous similar messages Feb 13 14:42:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2cfcad01-8df5-2887-8f5c-a2aec6d77cee (at 10.9.107.6@o2ib4) Feb 13 14:42:42 fir-io1-s1 kernel: Lustre: Skipped 305 previous similar messages Feb 13 14:52:49 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2105faed-3f3f-f302-d9e7-f8bce33a4b72 (at 10.8.3.23@o2ib6) Feb 13 14:52:49 fir-io1-s1 kernel: Lustre: Skipped 280 previous similar messages Feb 13 15:02:55 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b15137cf-311a-3865-c282-8f1cad0a5e07 (at 10.8.30.14@o2ib6) Feb 13 15:02:55 fir-io1-s1 kernel: Lustre: Skipped 174 previous similar messages Feb 13 15:12:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to d4dd6b0c-9843-b272-918e-a7e2aa547f92 (at 10.8.17.1@o2ib6) Feb 13 15:12:56 fir-io1-s1 kernel: Lustre: Skipped 300 previous similar messages Feb 13 15:22:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Feb 13 15:22:58 fir-io1-s1 kernel: Lustre: Skipped 445 previous similar messages Feb 13 15:32:59 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to e5be9ff2-873f-0542-6c5f-13af50413057 (at 10.8.30.1@o2ib6) Feb 13 15:32:59 fir-io1-s1 kernel: Lustre: Skipped 363 previous similar messages Feb 13 15:43:02 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 28529dc2-a4f3-77ae-28a6-713b2825ee6a (at 10.9.106.27@o2ib4) Feb 13 15:43:02 fir-io1-s1 kernel: Lustre: Skipped 419 previous similar messages Feb 13 15:51:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c46b7f3d-ec55-7fd7-e207-e9c6f5525b60 (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0c400, cur 1550101913 expire 1550101763 last 1550101686 Feb 13 15:51:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 13 15:53:02 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ed8fb36c-f8db-c3bc-c229-78172cc42522 (at 10.9.107.10@o2ib4) Feb 13 15:53:02 fir-io1-s1 kernel: Lustre: Skipped 401 previous similar messages Feb 13 15:54:56 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 13 15:54:56 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 15:55:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) reconnecting Feb 13 15:55:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 13 15:55:36 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.0.10.52@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 13 15:55:36 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 13 15:56:05 fir-io1-s1 kernel: Lustre: fir-OST0006: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) reconnecting Feb 13 15:56:05 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0003_UUID: not available for connect from 10.0.10.52@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 13 15:56:05 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 13 15:56:07 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1056897 to 0xc80000402:1056929 Feb 13 15:56:27 fir-io1-s1 kernel: Lustre: fir-OST0008: Client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) reconnecting Feb 13 15:56:28 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000b_UUID: not available for connect from 10.0.10.52@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 13 15:56:28 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 13 15:59:03 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762867000, cur 1550102343 expire 1550102193 last 1550102116 Feb 13 15:59:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 13 15:59:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762864800, cur 1550102392 expire 1550102242 last 1550102165 Feb 13 15:59:52 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 16:00:14 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client fir-MDT0002-mdtlov_UUID (at 10.0.10.52@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762866800, cur 1550102414 expire 1550102264 last 1550102187 Feb 13 16:01:36 fir-io1-s1 kernel: LNetError: 81923:0:(o2iblnd_cb.c:2935:kiblnd_rejected()) 10.0.10.52@o2ib7 rejected: o2iblnd fatal error Feb 13 16:03:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d3157baf-de90-86ed-7c87-5f5f5c909a71 (at 10.9.105.15@o2ib4) Feb 13 16:03:24 fir-io1-s1 kernel: Lustre: Skipped 365 previous similar messages Feb 13 16:04:58 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds Feb 13 16:04:58 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 13 16:05:03 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds Feb 13 16:05:03 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 13 16:07:33 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 13 16:07:33 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 13 16:10:03 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 13 16:10:03 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 13 16:10:48 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0008: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Feb 13 16:10:48 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Feb 13 16:11:38 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0006: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Feb 13 16:11:38 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 13 16:11:54 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1057249 to 0x6c0000400:1057281 Feb 13 16:11:54 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1057087 to 0x5c0000400:1057121 Feb 13 16:11:54 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1057528 to 0x580000400:1057569 Feb 13 16:11:54 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1056723 to 0x8c0000402:1056769 Feb 13 16:11:54 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1056897 to 0xc80000402:1056929 Feb 13 16:11:54 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1056899 to 0xc40000402:1056929 Feb 13 16:13:24 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8f86419d-3c7d-f8e0-fb5d-facc0f493f73 (at 10.8.27.31@o2ib6) Feb 13 16:13:24 fir-io1-s1 kernel: Lustre: Skipped 468 previous similar messages Feb 13 16:23:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f9ba6989-1b37-0945-75ce-916db80d0755 (at 10.9.106.29@o2ib4) Feb 13 16:23:27 fir-io1-s1 kernel: Lustre: Skipped 512 previous similar messages Feb 13 16:23:29 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 24c16cc7-a7a7-a924-aae9-a81f5b5dffef (at 10.8.21.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857620a2400, cur 1550103809 expire 1550103659 last 1550103582 Feb 13 16:23:29 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 13 16:33:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 276945c0-0d00-01d6-84a8-1f1c209c1b8f (at 10.9.103.36@o2ib4) Feb 13 16:33:28 fir-io1-s1 kernel: Lustre: Skipped 528 previous similar messages Feb 13 16:43:29 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.106.66@o2ib4) Feb 13 16:43:29 fir-io1-s1 kernel: Lustre: Skipped 531 previous similar messages Feb 13 16:47:57 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550105270/real 1550105270] req@ffff984e0da39800 x1624932309522960/t0(0) o104->fir-OST0000@10.8.11.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550105277 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 13 16:47:57 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 13 16:48:04 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550105277/real 1550105277] req@ffff985647dda400 x1624932309523456/t0(0) o104->fir-OST0002@10.8.11.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550105284 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 16:48:11 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550105284/real 1550105284] req@ffff985647dda400 x1624932309523456/t0(0) o104->fir-OST0002@10.8.11.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550105291 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 16:48:11 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 13 16:48:25 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550105298/real 1550105298] req@ffff984e0da39800 x1624932309522960/t0(0) o104->fir-OST0000@10.8.11.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550105305 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 16:48:25 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 13 16:48:46 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550105319/real 1550105319] req@ffff985647dda400 x1624932309523456/t0(0) o104->fir-OST0002@10.8.11.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550105326 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 16:48:46 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 13 16:49:21 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550105354/real 1550105354] req@ffff984e0da39800 x1624932309522960/t0(0) o104->fir-OST0000@10.8.11.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550105361 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 13 16:49:21 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 13 16:50:24 fir-io1-s1 kernel: LustreError: 96524:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.11.22@o2ib6) failed to reply to blocking AST (req@ffff985647dda400 x1624932309523456 status 0 rc -110), evict it ns: filter-fir-OST0002_UUID lock: ffff984753d69200/0x49e185e954296c93 lrc: 4/0,0 mode: PW/PW res: [0x5c0000400:0x10253f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.11.22@o2ib6 remote: 0x906bdc199f92f8dc expref: 11 pid: 96405 timeout: 453318 lvb_type: 0 Feb 13 16:50:24 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.11.22@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 13 16:50:24 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 13 16:50:24 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.11.22@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff98420ae2a880/0x49e185e9542973e0 lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x1025e3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.11.22@o2ib6 remote: 0x906bdc199f92fb36 expref: 12 pid: 96891 timeout: 0 lvb_type: 0 Feb 13 16:50:24 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Feb 13 16:50:24 fir-io1-s1 kernel: LustreError: 96524:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 13 16:51:21 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2c17b79b-0e07-5124-88ca-dcc2be55cc01 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986816f39800, cur 1550105481 expire 1550105331 last 1550105254 Feb 13 16:51:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 13 16:51:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2c17b79b-0e07-5124-88ca-dcc2be55cc01 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e02fc00, cur 1550105487 expire 1550105337 last 1550105260 Feb 13 16:51:27 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 13 16:51:32 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2c17b79b-0e07-5124-88ca-dcc2be55cc01 (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857590d2400, cur 1550105492 expire 1550105342 last 1550105265 Feb 13 16:53:31 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e7658bee-b529-b857-21ef-217c5e9fe7b7 (at 10.9.113.9@o2ib4) Feb 13 16:53:31 fir-io1-s1 kernel: Lustre: Skipped 486 previous similar messages Feb 13 17:03:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.102.49@o2ib4) Feb 13 17:03:39 fir-io1-s1 kernel: Lustre: Skipped 367 previous similar messages Feb 13 17:13:49 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.106.19@o2ib4) Feb 13 17:13:49 fir-io1-s1 kernel: Lustre: Skipped 387 previous similar messages Feb 13 17:23:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 88f11576-1898-ee78-29af-b93b7778dcb7 (at 10.8.13.26@o2ib6) Feb 13 17:23:50 fir-io1-s1 kernel: Lustre: Skipped 285 previous similar messages Feb 13 17:33:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 054698ea-f84b-bd00-f4ed-c64e725d9902 (at 10.8.1.2@o2ib6) Feb 13 17:33:59 fir-io1-s1 kernel: Lustre: Skipped 257 previous similar messages Feb 13 17:44:04 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 16214717-e51a-197b-25e8-75bbe23d8ed2 (at 10.9.101.63@o2ib4) Feb 13 17:44:04 fir-io1-s1 kernel: Lustre: Skipped 328 previous similar messages Feb 13 17:54:29 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d0986368-6b42-a7a1-d471-712579f07716 (at 10.8.27.34@o2ib6) Feb 13 17:54:29 fir-io1-s1 kernel: Lustre: Skipped 307 previous similar messages Feb 13 18:04:32 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to d1eafd0d-f5d5-63d1-f545-e28e22ce25f0 (at 10.9.106.21@o2ib4) Feb 13 18:04:32 fir-io1-s1 kernel: Lustre: Skipped 387 previous similar messages Feb 13 18:14:33 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 276945c0-0d00-01d6-84a8-1f1c209c1b8f (at 10.9.103.36@o2ib4) Feb 13 18:14:33 fir-io1-s1 kernel: Lustre: Skipped 324 previous similar messages Feb 13 18:24:34 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 77963223-bc75-0922-f3f9-87c125865623 (at 10.8.31.5@o2ib6) Feb 13 18:24:34 fir-io1-s1 kernel: Lustre: Skipped 332 previous similar messages Feb 13 18:34:34 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 2e8b1a97-514f-63aa-1bc8-051eadecacf0 (at 10.9.112.9@o2ib4) Feb 13 18:34:34 fir-io1-s1 kernel: Lustre: Skipped 354 previous similar messages Feb 13 18:44:42 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 71464f83-f435-3a33-e9d6-ef54166e95b7 (at 10.8.30.36@o2ib6) Feb 13 18:44:42 fir-io1-s1 kernel: Lustre: Skipped 369 previous similar messages Feb 13 18:54:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 13 18:54:44 fir-io1-s1 kernel: Lustre: Skipped 403 previous similar messages Feb 13 19:04:54 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.8.18.23@o2ib6) Feb 13 19:04:54 fir-io1-s1 kernel: Lustre: Skipped 313 previous similar messages Feb 13 19:15:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 5c665c4f-c7f8-f45f-f420-c3aac6430af6 (at 10.8.27.4@o2ib6) Feb 13 19:15:09 fir-io1-s1 kernel: Lustre: Skipped 401 previous similar messages Feb 13 19:25:24 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3dbbb061-d93c-c7e8-88c8-e262ff513397 (at 10.8.14.6@o2ib6) Feb 13 19:25:24 fir-io1-s1 kernel: Lustre: Skipped 327 previous similar messages Feb 13 19:35:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) Feb 13 19:35:26 fir-io1-s1 kernel: Lustre: Skipped 229 previous similar messages Feb 13 19:45:26 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 25262cb2-e449-1554-b5d1-0b6e448154f1 (at 10.9.106.18@o2ib4) Feb 13 19:45:26 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 13 19:55:45 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to fb00bf6b-d767-c6bd-b589-d3608416a604 (at 10.9.105.44@o2ib4) Feb 13 19:55:45 fir-io1-s1 kernel: Lustre: Skipped 244 previous similar messages Feb 13 20:05:49 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.18.23@o2ib6) Feb 13 20:05:49 fir-io1-s1 kernel: Lustre: Skipped 269 previous similar messages Feb 13 20:15:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8ee660c6-3e54-f054-0509-b448082e8dec (at 10.9.105.32@o2ib4) Feb 13 20:15:51 fir-io1-s1 kernel: Lustre: Skipped 206 previous similar messages Feb 13 20:26:10 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d1eafd0d-f5d5-63d1-f545-e28e22ce25f0 (at 10.9.106.21@o2ib4) Feb 13 20:26:10 fir-io1-s1 kernel: Lustre: Skipped 199 previous similar messages Feb 13 20:36:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fed507e8-5435-f949-539f-6cb9d563cc12 (at 10.9.106.52@o2ib4) Feb 13 20:36:11 fir-io1-s1 kernel: Lustre: Skipped 224 previous similar messages Feb 13 20:46:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) Feb 13 20:46:26 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 13 20:56:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 1c47c083-1697-95bd-3469-0636ee21aa42 (at 10.8.2.32@o2ib6) Feb 13 20:56:50 fir-io1-s1 kernel: Lustre: Skipped 144 previous similar messages Feb 13 21:06:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 676b9462-d0c7-96e9-ddb9-5790c315c2e9 (at 10.9.103.26@o2ib4) Feb 13 21:06:51 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 13 21:17:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0935f868-2cb6-7be4-32e4-6f8243d37d7c (at 10.9.0.1@o2ib4) Feb 13 21:17:09 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Feb 13 21:27:28 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 1815d69a-de12-0777-af99-1fa63af02a98 (at 10.9.106.31@o2ib4) Feb 13 21:27:28 fir-io1-s1 kernel: Lustre: Skipped 217 previous similar messages Feb 13 21:37:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5188c60b-8e1f-a67c-e214-04fbb301fd99 (at 10.9.106.14@o2ib4) Feb 13 21:37:31 fir-io1-s1 kernel: Lustre: Skipped 187 previous similar messages Feb 13 21:47:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a811a23e-a19d-35f5-63e7-a6dd4ef128fb (at 10.8.1.28@o2ib6) Feb 13 21:47:32 fir-io1-s1 kernel: Lustre: Skipped 187 previous similar messages Feb 13 21:57:34 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 342724f1-ee95-beed-9c54-15c49b83cfa7 (at 10.8.10.31@o2ib6) Feb 13 21:57:34 fir-io1-s1 kernel: Lustre: Skipped 245 previous similar messages Feb 13 22:07:37 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d1eafd0d-f5d5-63d1-f545-e28e22ce25f0 (at 10.9.106.21@o2ib4) Feb 13 22:07:37 fir-io1-s1 kernel: Lustre: Skipped 201 previous similar messages Feb 13 22:17:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 13 22:17:41 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 13 22:27:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 13 22:27:41 fir-io1-s1 kernel: Lustre: Skipped 160 previous similar messages Feb 13 22:37:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 13 22:37:41 fir-io1-s1 kernel: Lustre: Skipped 211 previous similar messages Feb 13 22:47:41 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 13 22:47:41 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 13 22:57:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 13 22:57:41 fir-io1-s1 kernel: Lustre: Skipped 193 previous similar messages Feb 13 23:07:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 13 23:07:41 fir-io1-s1 kernel: Lustre: Skipped 254 previous similar messages Feb 13 23:18:20 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2019e0f4-a199-9a37-e710-fbb1c8ccd2aa (at 10.9.107.60@o2ib4) Feb 13 23:18:20 fir-io1-s1 kernel: Lustre: Skipped 209 previous similar messages Feb 13 23:28:28 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 20cb63a2-4619-420f-25f6-c09916b5cd24 (at 10.8.17.16@o2ib6) Feb 13 23:28:28 fir-io1-s1 kernel: Lustre: Skipped 246 previous similar messages Feb 13 23:38:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 97bc8e0c-1614-4de0-a593-98b585b7fd0b (at 10.9.103.30@o2ib4) Feb 13 23:38:30 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 13 23:48:41 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 13 23:48:41 fir-io1-s1 kernel: Lustre: Skipped 383 previous similar messages Feb 13 23:59:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3608e712-e3ad-8999-51dd-65547ce62d69 (at 10.8.7.25@o2ib6) Feb 13 23:59:09 fir-io1-s1 kernel: Lustre: Skipped 412 previous similar messages Feb 14 00:09:13 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 14 00:09:13 fir-io1-s1 kernel: Lustre: Skipped 316 previous similar messages Feb 14 00:19:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 135b49e4-9bbb-43fc-66e9-1f7ec8c75a96 (at 10.9.113.3@o2ib4) Feb 14 00:19:17 fir-io1-s1 kernel: Lustre: Skipped 316 previous similar messages Feb 14 00:29:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to cceca428-8e88-8ef2-bb18-acd6818a2d01 (at 10.8.23.1@o2ib6) Feb 14 00:29:23 fir-io1-s1 kernel: Lustre: Skipped 225 previous similar messages Feb 14 00:39:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7751f34c-206a-edf3-4c97-12b3ac7dcb0e (at 10.8.23.14@o2ib6) Feb 14 00:39:43 fir-io1-s1 kernel: Lustre: Skipped 351 previous similar messages Feb 14 00:49:49 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.107.22@o2ib4) Feb 14 00:49:49 fir-io1-s1 kernel: Lustre: Skipped 183 previous similar messages Feb 14 01:00:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) Feb 14 01:00:12 fir-io1-s1 kernel: Lustre: Skipped 239 previous similar messages Feb 14 01:10:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 23f9a8de-c6a0-76ea-9378-57e47966c02c (at 10.8.21.23@o2ib6) Feb 14 01:10:23 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 14 01:20:26 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.9.106.61@o2ib4) Feb 14 01:20:26 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Feb 14 01:20:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 06b73631-7983-a387-a520-1052e0544f95 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987638038000, cur 1550136033 expire 1550135883 last 1550135806 Feb 14 01:30:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Feb 14 01:30:28 fir-io1-s1 kernel: Lustre: Skipped 273 previous similar messages Feb 14 01:40:29 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f3c26b94-2261-c91e-b422-79918936510b (at 10.9.115.4@o2ib4) Feb 14 01:40:29 fir-io1-s1 kernel: Lustre: Skipped 350 previous similar messages Feb 14 01:50:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 04bbddc7-37a9-9b79-7fa8-c451901e5d15 (at 10.9.101.67@o2ib4) Feb 14 01:50:30 fir-io1-s1 kernel: Lustre: Skipped 421 previous similar messages Feb 14 02:00:34 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 14 02:00:34 fir-io1-s1 kernel: Lustre: Skipped 370 previous similar messages Feb 14 02:10:34 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3608e712-e3ad-8999-51dd-65547ce62d69 (at 10.8.7.25@o2ib6) Feb 14 02:10:34 fir-io1-s1 kernel: Lustre: Skipped 289 previous similar messages Feb 14 02:20:35 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to bef16258-699d-0e14-bdeb-b454fac00d89 (at 10.9.112.15@o2ib4) Feb 14 02:20:35 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 14 02:30:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to bef16258-699d-0e14-bdeb-b454fac00d89 (at 10.9.112.15@o2ib4) Feb 14 02:30:37 fir-io1-s1 kernel: Lustre: Skipped 368 previous similar messages Feb 14 02:40:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 14 02:40:39 fir-io1-s1 kernel: Lustre: Skipped 377 previous similar messages Feb 14 02:50:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 14 02:50:48 fir-io1-s1 kernel: Lustre: Skipped 329 previous similar messages Feb 14 02:58:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1665083d-38af-ab09-1c24-d258049879dc (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aab3000, cur 1550141934 expire 1550141784 last 1550141707 Feb 14 02:58:54 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 14 03:00:52 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 14 03:00:52 fir-io1-s1 kernel: Lustre: Skipped 335 previous similar messages Feb 14 03:10:52 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 14 03:10:52 fir-io1-s1 kernel: Lustre: Skipped 302 previous similar messages Feb 14 03:20:52 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Feb 14 03:20:52 fir-io1-s1 kernel: Lustre: Skipped 285 previous similar messages Feb 14 03:30:53 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to d1755442-892e-a805-5fa2-c61746c310b0 (at 10.9.113.7@o2ib4) Feb 14 03:30:53 fir-io1-s1 kernel: Lustre: Skipped 355 previous similar messages Feb 14 03:40:53 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 14 03:40:53 fir-io1-s1 kernel: Lustre: Skipped 360 previous similar messages Feb 14 03:45:15 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c8293d92-ca0f-5892-5fe5-c2f02d9ed234 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a9c386c00, cur 1550144715 expire 1550144565 last 1550144488 Feb 14 03:45:15 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 14 03:50:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 14 03:50:56 fir-io1-s1 kernel: Lustre: Skipped 392 previous similar messages Feb 14 04:00:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 14 04:00:57 fir-io1-s1 kernel: Lustre: Skipped 389 previous similar messages Feb 14 04:10:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 276945c0-0d00-01d6-84a8-1f1c209c1b8f (at 10.9.103.36@o2ib4) Feb 14 04:10:57 fir-io1-s1 kernel: Lustre: Skipped 382 previous similar messages Feb 14 04:20:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 028d6433-9e7d-1b84-c8b7-1bb2a8570ec4 (at 10.8.1.4@o2ib6) Feb 14 04:20:58 fir-io1-s1 kernel: Lustre: Skipped 354 previous similar messages Feb 14 04:30:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 8d89fca0-9472-dc06-65e2-fd0a61adf564 (at 10.9.106.23@o2ib4) Feb 14 04:30:58 fir-io1-s1 kernel: Lustre: Skipped 337 previous similar messages Feb 14 04:41:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a3c57bef-a739-0ea9-6582-283914517ba2 (at 10.9.103.31@o2ib4) Feb 14 04:41:00 fir-io1-s1 kernel: Lustre: Skipped 309 previous similar messages Feb 14 04:51:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 14 04:51:01 fir-io1-s1 kernel: Lustre: Skipped 299 previous similar messages Feb 14 05:01:02 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to aa9249a7-30cd-d7bd-380e-44a3180f72ff (at 10.8.27.2@o2ib6) Feb 14 05:01:02 fir-io1-s1 kernel: Lustre: Skipped 286 previous similar messages Feb 14 05:11:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 037cf541-a575-03b3-3aed-140164784d71 (at 10.9.107.61@o2ib4) Feb 14 05:11:03 fir-io1-s1 kernel: Lustre: Skipped 370 previous similar messages Feb 14 05:21:11 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 5dafe1c8-3c93-e104-2f19-ffcaca2d90cd (at 10.8.16.5@o2ib6) Feb 14 05:21:11 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 14 05:31:12 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Feb 14 05:31:12 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 14 05:41:17 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 14 05:41:17 fir-io1-s1 kernel: Lustre: Skipped 379 previous similar messages Feb 14 05:51:20 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fe789eb3-1cd9-3594-b889-6606ba1b8e4a (at 10.9.113.2@o2ib4) Feb 14 05:51:20 fir-io1-s1 kernel: Lustre: Skipped 405 previous similar messages Feb 14 06:01:25 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) Feb 14 06:01:25 fir-io1-s1 kernel: Lustre: Skipped 484 previous similar messages Feb 14 06:04:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b269ffa7-7781-a750-ae9a-09b3f1536950 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857620a3400, cur 1550153049 expire 1550152899 last 1550152822 Feb 14 06:11:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 14 06:11:26 fir-io1-s1 kernel: Lustre: Skipped 399 previous similar messages Feb 14 06:21:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d1eafd0d-f5d5-63d1-f545-e28e22ce25f0 (at 10.9.106.21@o2ib4) Feb 14 06:21:28 fir-io1-s1 kernel: Lustre: Skipped 477 previous similar messages Feb 14 06:31:29 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 72da9a6e-2827-1c9d-1aa6-7b398153fee1 (at 10.9.106.71@o2ib4) Feb 14 06:31:29 fir-io1-s1 kernel: Lustre: Skipped 409 previous similar messages Feb 14 06:41:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c5f414d6-b086-f007-4ceb-21404d074992 (at 10.8.1.1@o2ib6) Feb 14 06:41:30 fir-io1-s1 kernel: Lustre: Skipped 356 previous similar messages Feb 14 06:51:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 14 06:51:30 fir-io1-s1 kernel: Lustre: Skipped 579 previous similar messages Feb 14 07:01:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 14 07:01:35 fir-io1-s1 kernel: Lustre: Skipped 356 previous similar messages Feb 14 07:11:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a58d34d6-2c4a-1753-b4fb-33e6ffbafbc0 (at 10.8.8.31@o2ib6) Feb 14 07:11:36 fir-io1-s1 kernel: Lustre: Skipped 382 previous similar messages Feb 14 07:21:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Feb 14 07:21:38 fir-io1-s1 kernel: Lustre: Skipped 430 previous similar messages Feb 14 07:31:41 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.112.17@o2ib4) Feb 14 07:31:41 fir-io1-s1 kernel: Lustre: Skipped 399 previous similar messages Feb 14 07:41:43 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to d3157baf-de90-86ed-7c87-5f5f5c909a71 (at 10.9.105.15@o2ib4) Feb 14 07:41:43 fir-io1-s1 kernel: Lustre: Skipped 585 previous similar messages Feb 14 07:51:43 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9468a1d3-3abd-8063-5952-288cca0f1dec (at 10.8.27.35@o2ib6) Feb 14 07:51:43 fir-io1-s1 kernel: Lustre: Skipped 540 previous similar messages Feb 14 08:01:46 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Feb 14 08:01:46 fir-io1-s1 kernel: Lustre: Skipped 541 previous similar messages Feb 14 08:11:47 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 83fafa06-153a-a1fd-dd02-4b2d3ef6c90e (at 10.9.114.2@o2ib4) Feb 14 08:11:47 fir-io1-s1 kernel: Lustre: Skipped 698 previous similar messages Feb 14 08:21:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a18587f4-6669-8a89-c311-224538b5a6f2 (at 10.8.27.32@o2ib6) Feb 14 08:21:52 fir-io1-s1 kernel: Lustre: Skipped 611 previous similar messages Feb 14 08:28:28 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 793f3b7f-d2e3-9619-2aac-755693b12f48 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da0400, cur 1550161708 expire 1550161558 last 1550161481 Feb 14 08:28:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 14 08:31:54 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 14 08:31:54 fir-io1-s1 kernel: Lustre: Skipped 554 previous similar messages Feb 14 08:41:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Feb 14 08:41:54 fir-io1-s1 kernel: Lustre: Skipped 444 previous similar messages Feb 14 08:42:28 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c3ee8e29-24b2-60ad-b950-c5ea318742ba (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e55c00, cur 1550162548 expire 1550162398 last 1550162321 Feb 14 08:42:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 14 08:51:54 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d14918a5-6565-e169-f50e-e38934e1ba9e (at 10.9.105.53@o2ib4) Feb 14 08:51:54 fir-io1-s1 kernel: Lustre: Skipped 375 previous similar messages Feb 14 09:01:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 14 09:01:59 fir-io1-s1 kernel: Lustre: Skipped 446 previous similar messages Feb 14 09:12:08 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) Feb 14 09:12:08 fir-io1-s1 kernel: Lustre: Skipped 400 previous similar messages Feb 14 09:22:10 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 276945c0-0d00-01d6-84a8-1f1c209c1b8f (at 10.9.103.36@o2ib4) Feb 14 09:22:10 fir-io1-s1 kernel: Lustre: Skipped 535 previous similar messages Feb 14 09:32:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Feb 14 09:32:13 fir-io1-s1 kernel: Lustre: Skipped 487 previous similar messages Feb 14 09:42:14 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 426838dc-cbfd-f2b9-3d71-31406b483e24 (at 10.8.25.14@o2ib6) Feb 14 09:42:14 fir-io1-s1 kernel: Lustre: Skipped 432 previous similar messages Feb 14 09:46:54 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 85452ac2-1766-40f9-2bd9-027444f6ab48 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848346a2400, cur 1550166414 expire 1550166264 last 1550166187 Feb 14 09:46:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 14 09:46:56 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 85452ac2-1766-40f9-2bd9-027444f6ab48 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7e8800, cur 1550166416 expire 1550166266 last 1550166189 Feb 14 09:46:56 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 14 09:47:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 85452ac2-1766-40f9-2bd9-027444f6ab48 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582bba3000, cur 1550166424 expire 1550166274 last 1550166197 Feb 14 09:52:22 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 14 09:52:22 fir-io1-s1 kernel: Lustre: Skipped 398 previous similar messages Feb 14 10:02:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fe789eb3-1cd9-3594-b889-6606ba1b8e4a (at 10.9.113.2@o2ib4) Feb 14 10:02:22 fir-io1-s1 kernel: Lustre: Skipped 416 previous similar messages Feb 14 10:12:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1e03043d-571f-f709-9076-72fb7d056ac3 (at 10.9.103.43@o2ib4) Feb 14 10:12:22 fir-io1-s1 kernel: Lustre: Skipped 513 previous similar messages Feb 14 10:22:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 14 10:22:41 fir-io1-s1 kernel: Lustre: Skipped 333 previous similar messages Feb 14 10:24:49 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 74d85d1f-4cb0-10d6-b1f7-d9ffd79a89ab (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b746c00, cur 1550168689 expire 1550168539 last 1550168462 Feb 14 10:24:49 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 14 10:24:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 74d85d1f-4cb0-10d6-b1f7-d9ffd79a89ab (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b744c00, cur 1550168692 expire 1550168542 last 1550168465 Feb 14 10:32:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 14 10:32:41 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 14 10:42:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 14 10:42:41 fir-io1-s1 kernel: Lustre: Skipped 327 previous similar messages Feb 14 10:52:42 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 14 10:52:42 fir-io1-s1 kernel: Lustre: Skipped 418 previous similar messages Feb 14 10:53:50 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6108a9dd-4582-0df2-623b-37620f8c2adb (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986cce294000, cur 1550170430 expire 1550170280 last 1550170203 Feb 14 10:53:50 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 14 11:02:44 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e854ebb7-895a-1a12-528f-bdc84232856c (at 10.8.10.23@o2ib6) Feb 14 11:02:44 fir-io1-s1 kernel: Lustre: Skipped 487 previous similar messages Feb 14 11:11:33 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f0784ac9-943f-0104-a8b2-0ad14931e10f (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767573400, cur 1550171493 expire 1550171343 last 1550171266 Feb 14 11:11:33 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 14 11:11:35 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f0784ac9-943f-0104-a8b2-0ad14931e10f (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848845fe000, cur 1550171495 expire 1550171345 last 1550171268 Feb 14 11:12:46 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 14 11:12:46 fir-io1-s1 kernel: Lustre: Skipped 559 previous similar messages Feb 14 11:22:51 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 342c2f4a-aa20-f4a0-baab-aeaf28be9232 (at 10.8.21.4@o2ib6) Feb 14 11:22:51 fir-io1-s1 kernel: Lustre: Skipped 498 previous similar messages Feb 14 11:32:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c62ed1f1-3793-7a2f-05b0-df79888e04df (at 10.8.0.65@o2ib6) Feb 14 11:32:54 fir-io1-s1 kernel: Lustre: Skipped 495 previous similar messages Feb 14 11:42:55 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d6636cab-7aff-905c-7654-df6ea400308a (at 10.8.7.21@o2ib6) Feb 14 11:42:55 fir-io1-s1 kernel: Lustre: Skipped 606 previous similar messages Feb 14 11:53:02 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 14 11:53:02 fir-io1-s1 kernel: Lustre: Skipped 282 previous similar messages Feb 14 12:03:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 45a162ef-0459-9991-8b7a-2377aa3c8022 (at 10.9.101.32@o2ib4) Feb 14 12:03:06 fir-io1-s1 kernel: Lustre: Skipped 344 previous similar messages Feb 14 12:10:38 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c41de332-253e-3b19-4afa-15e43e480d02 (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b6a800, cur 1550175038 expire 1550174888 last 1550174811 Feb 14 12:10:38 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 14 12:13:07 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) Feb 14 12:13:07 fir-io1-s1 kernel: Lustre: Skipped 475 previous similar messages Feb 14 12:23:12 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to dcf54687-a6e1-03a6-825a-92830ad9b551 (at 10.8.7.31@o2ib6) Feb 14 12:23:12 fir-io1-s1 kernel: Lustre: Skipped 387 previous similar messages Feb 14 12:33:12 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5dafe1c8-3c93-e104-2f19-ffcaca2d90cd (at 10.8.16.5@o2ib6) Feb 14 12:33:12 fir-io1-s1 kernel: Lustre: Skipped 410 previous similar messages Feb 14 12:43:12 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) Feb 14 12:43:12 fir-io1-s1 kernel: Lustre: Skipped 346 previous similar messages Feb 14 12:53:13 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9d602dc8-8051-8982-51eb-1bb4250b93cd (at 10.8.22.18@o2ib6) Feb 14 12:53:13 fir-io1-s1 kernel: Lustre: Skipped 326 previous similar messages Feb 14 13:03:13 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to b2753dc4-19d1-7b06-13e5-9ceb58fcc4d7 (at 10.9.113.4@o2ib4) Feb 14 13:03:13 fir-io1-s1 kernel: Lustre: Skipped 475 previous similar messages Feb 14 13:13:13 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 14 13:13:13 fir-io1-s1 kernel: Lustre: Skipped 448 previous similar messages Feb 14 13:23:16 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 14 13:23:16 fir-io1-s1 kernel: Lustre: Skipped 463 previous similar messages Feb 14 13:33:18 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 14 13:33:18 fir-io1-s1 kernel: Lustre: Skipped 456 previous similar messages Feb 14 13:43:21 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d3b31001-f420-4435-d40a-bc3c2a4e6f40 (at 10.8.2.13@o2ib6) Feb 14 13:43:21 fir-io1-s1 kernel: Lustre: Skipped 455 previous similar messages Feb 14 13:53:22 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.112.7@o2ib4) Feb 14 13:53:22 fir-io1-s1 kernel: Lustre: Skipped 375 previous similar messages Feb 14 14:03:27 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 14 14:03:27 fir-io1-s1 kernel: Lustre: Skipped 402 previous similar messages Feb 14 14:13:34 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Feb 14 14:13:34 fir-io1-s1 kernel: Lustre: Skipped 305 previous similar messages Feb 14 14:23:42 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) Feb 14 14:23:42 fir-io1-s1 kernel: Lustre: Skipped 300 previous similar messages Feb 14 14:33:57 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2cfcad01-8df5-2887-8f5c-a2aec6d77cee (at 10.9.107.6@o2ib4) Feb 14 14:33:57 fir-io1-s1 kernel: Lustre: Skipped 429 previous similar messages Feb 14 14:44:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 513e4107-cd39-8515-1387-2ee9a5768f3f (at 10.8.12.29@o2ib6) Feb 14 14:44:00 fir-io1-s1 kernel: Lustre: Skipped 349 previous similar messages Feb 14 14:54:16 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) Feb 14 14:54:16 fir-io1-s1 kernel: Lustre: Skipped 334 previous similar messages Feb 14 15:04:17 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 78ee81c2-542f-3aeb-f6e6-8f441a9da343 (at 10.8.20.27@o2ib6) Feb 14 15:04:17 fir-io1-s1 kernel: Lustre: Skipped 554 previous similar messages Feb 14 15:14:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 14 15:14:24 fir-io1-s1 kernel: Lustre: Skipped 415 previous similar messages Feb 14 15:24:33 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0a36a787-0369-4905-fbcd-23c2377575ca (at 10.8.2.30@o2ib6) Feb 14 15:24:33 fir-io1-s1 kernel: Lustre: Skipped 294 previous similar messages Feb 14 15:34:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) Feb 14 15:34:33 fir-io1-s1 kernel: Lustre: Skipped 434 previous similar messages Feb 14 15:44:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ebee093c-8e69-4e39-2895-94a502f715cc (at 10.9.113.6@o2ib4) Feb 14 15:44:33 fir-io1-s1 kernel: Lustre: Skipped 474 previous similar messages Feb 14 15:54:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 39eb100c-0d25-d4de-9dbe-7a71ed238778 (at 10.8.2.28@o2ib6) Feb 14 15:54:33 fir-io1-s1 kernel: Lustre: Skipped 461 previous similar messages Feb 14 16:04:41 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Feb 14 16:04:41 fir-io1-s1 kernel: Lustre: Skipped 359 previous similar messages Feb 14 16:14:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3512bfd0-06dd-989b-6d83-75d517d82937 (at 10.9.105.6@o2ib4) Feb 14 16:14:47 fir-io1-s1 kernel: Lustre: Skipped 347 previous similar messages Feb 14 16:24:54 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c3993171-24c0-89a4-7cb1-27b4ddbf15a6 (at 10.8.6.36@o2ib6) Feb 14 16:24:54 fir-io1-s1 kernel: Lustre: Skipped 270 previous similar messages Feb 14 16:35:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) Feb 14 16:35:03 fir-io1-s1 kernel: Lustre: Skipped 395 previous similar messages Feb 14 16:45:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3d122a91-53f0-f449-1f10-d08490897e63 (at 10.9.106.65@o2ib4) Feb 14 16:45:17 fir-io1-s1 kernel: Lustre: Skipped 311 previous similar messages Feb 14 16:55:18 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f34aebb6-59fc-4cef-28dc-df15ded97223 (at 10.9.105.62@o2ib4) Feb 14 16:55:18 fir-io1-s1 kernel: Lustre: Skipped 296 previous similar messages Feb 14 17:05:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 14 17:05:26 fir-io1-s1 kernel: Lustre: Skipped 379 previous similar messages Feb 14 17:15:34 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fc738f4c-fd89-da60-f357-76c857400e3c (at 10.9.115.6@o2ib4) Feb 14 17:15:34 fir-io1-s1 kernel: Lustre: Skipped 338 previous similar messages Feb 14 17:25:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f3023e70-967c-cdca-d170-324afefea199 (at 10.9.106.49@o2ib4) Feb 14 17:25:48 fir-io1-s1 kernel: Lustre: Skipped 233 previous similar messages Feb 14 17:31:11 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2d374da6-b461-ad6a-bac0-9da0a46f19de (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cdf800, cur 1550194271 expire 1550194121 last 1550194044 Feb 14 17:31:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 14 17:35:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7d6292c2-dc0a-0082-5273-c1ff8e6163ed (at 10.9.102.25@o2ib4) Feb 14 17:35:53 fir-io1-s1 kernel: Lustre: Skipped 530 previous similar messages Feb 14 17:46:14 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 14 17:46:14 fir-io1-s1 kernel: Lustre: Skipped 457 previous similar messages Feb 14 17:56:16 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 2c2d2f6b-3086-1f70-68ed-98263873eaff (at 10.9.107.25@o2ib4) Feb 14 17:56:16 fir-io1-s1 kernel: Lustre: Skipped 262 previous similar messages Feb 14 18:06:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6d1355d0-7b33-677d-c8cf-a270e3061917 (at 10.8.7.15@o2ib6) Feb 14 18:06:16 fir-io1-s1 kernel: Lustre: Skipped 376 previous similar messages Feb 14 18:16:16 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Feb 14 18:16:16 fir-io1-s1 kernel: Lustre: Skipped 441 previous similar messages Feb 14 18:26:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 255adf8b-e7cc-8914-212c-11fb3c6a4274 (at 10.9.105.35@o2ib4) Feb 14 18:26:18 fir-io1-s1 kernel: Lustre: Skipped 286 previous similar messages Feb 14 18:36:23 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3e4de392-7948-5bb8-1b41-4df53ad748a7 (at 10.8.8.10@o2ib6) Feb 14 18:36:23 fir-io1-s1 kernel: Lustre: Skipped 325 previous similar messages Feb 14 18:46:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 14 18:46:30 fir-io1-s1 kernel: Lustre: Skipped 279 previous similar messages Feb 14 18:56:31 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) Feb 14 18:56:31 fir-io1-s1 kernel: Lustre: Skipped 254 previous similar messages Feb 14 19:06:25 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0f9b3a96-fc1c-c69c-6e3e-2c0e68790443 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576855d800, cur 1550199985 expire 1550199835 last 1550199758 Feb 14 19:06:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 1b71b1a8-fdc3-550f-13e6-42b0376dd743 (at 10.9.112.8@o2ib4) Feb 14 19:06:36 fir-io1-s1 kernel: Lustre: Skipped 452 previous similar messages Feb 14 19:16:39 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 50f28a0e-eb03-3ed4-df6b-96db06d3f42b (at 10.9.107.34@o2ib4) Feb 14 19:16:39 fir-io1-s1 kernel: Lustre: Skipped 405 previous similar messages Feb 14 19:26:39 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6c7afae7-804f-5837-942b-b1962fecb1db (at 10.8.8.33@o2ib6) Feb 14 19:26:39 fir-io1-s1 kernel: Lustre: Skipped 411 previous similar messages Feb 14 19:36:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 14 19:36:43 fir-io1-s1 kernel: Lustre: Skipped 285 previous similar messages Feb 14 19:46:43 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0405d36b-1dfe-417d-33da-88f65ca0bd9f (at 10.8.28.4@o2ib6) Feb 14 19:46:43 fir-io1-s1 kernel: Lustre: Skipped 412 previous similar messages Feb 14 19:56:59 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 2019e0f4-a199-9a37-e710-fbb1c8ccd2aa (at 10.9.107.60@o2ib4) Feb 14 19:56:59 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 14 20:07:00 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ceafb54-89ce-8961-d103-913efe379d81 (at 10.8.21.7@o2ib6) Feb 14 20:07:00 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 14 20:10:06 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Feb 14 20:10:06 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (105): c: 8, oc: 0, rc: 8 Feb 14 20:17:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca8fb1e1-3fd5-5ac5-d9b6-afbc41df25e2 (at 10.8.26.32@o2ib6) Feb 14 20:17:46 fir-io1-s1 kernel: Lustre: Skipped 76 previous similar messages Feb 14 20:27:46 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 2aa90518-227d-4557-9238-cd8cd884ba59 (at 10.8.26.19@o2ib6) Feb 14 20:27:46 fir-io1-s1 kernel: Lustre: Skipped 92 previous similar messages Feb 14 20:37:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 11122870-d2b4-7c10-4146-c4d58251a247 (at 10.8.16.4@o2ib6) Feb 14 20:37:53 fir-io1-s1 kernel: Lustre: Skipped 129 previous similar messages Feb 14 20:48:01 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 78ac1cbc-f34c-9a00-1315-0c6dfe7ad07f (at 10.8.13.4@o2ib6) Feb 14 20:48:01 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 14 20:58:09 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 14 20:58:09 fir-io1-s1 kernel: Lustre: Skipped 136 previous similar messages Feb 14 21:08:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 11122870-d2b4-7c10-4146-c4d58251a247 (at 10.8.16.4@o2ib6) Feb 14 21:08:53 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Feb 14 21:19:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4130b965-27d0-7407-e546-10c8630df6b2 (at 10.8.11.6@o2ib6) Feb 14 21:19:03 fir-io1-s1 kernel: Lustre: Skipped 106 previous similar messages Feb 14 21:29:20 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 57bd9565-249c-08fa-b75e-115d9c0f2fee (at 10.9.104.26@o2ib4) Feb 14 21:29:20 fir-io1-s1 kernel: Lustre: Skipped 156 previous similar messages Feb 14 21:39:22 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 14 21:39:22 fir-io1-s1 kernel: Lustre: Skipped 127 previous similar messages Feb 14 21:49:25 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to baf8fd69-3c06-48ef-44c1-991a98b1784d (at 10.8.13.14@o2ib6) Feb 14 21:49:25 fir-io1-s1 kernel: Lustre: Skipped 126 previous similar messages Feb 14 21:59:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 14 21:59:26 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 14 22:09:27 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9468a1d3-3abd-8063-5952-288cca0f1dec (at 10.8.27.35@o2ib6) Feb 14 22:09:27 fir-io1-s1 kernel: Lustre: Skipped 196 previous similar messages Feb 14 22:19:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a6a757ad-2ece-7189-1c8a-4459b2a0b330 (at 10.8.18.2@o2ib6) Feb 14 22:19:30 fir-io1-s1 kernel: Lustre: Skipped 295 previous similar messages Feb 14 22:29:32 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.101.64@o2ib4) Feb 14 22:29:32 fir-io1-s1 kernel: Lustre: Skipped 300 previous similar messages Feb 14 22:39:33 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 14 22:39:33 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 14 22:49:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 44c34e5e-d358-e5f1-f032-e5118620e81b (at 10.8.24.9@o2ib6) Feb 14 22:49:37 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Feb 14 22:59:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 14 22:59:47 fir-io1-s1 kernel: Lustre: Skipped 102 previous similar messages Feb 14 23:09:52 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ffcc7510-f875-7549-61d4-9f6248a33eef (at 10.9.105.7@o2ib4) Feb 14 23:09:52 fir-io1-s1 kernel: Lustre: Skipped 101 previous similar messages Feb 14 23:15:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c6acdf6f-0a09-7a4d-7dce-dbc1d585c32d (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7ec800, cur 1550214901 expire 1550214751 last 1550214674 Feb 14 23:15:01 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 14 23:20:13 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9d602dc8-8051-8982-51eb-1bb4250b93cd (at 10.8.22.18@o2ib6) Feb 14 23:20:13 fir-io1-s1 kernel: Lustre: Skipped 111 previous similar messages Feb 14 23:30:13 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1dbbeb8f-09ef-c969-e797-86b35c4732a3 (at 10.8.24.12@o2ib6) Feb 14 23:30:13 fir-io1-s1 kernel: Lustre: Skipped 125 previous similar messages Feb 14 23:40:16 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to de71e293-4bab-6a9f-f864-40636c6dd616 (at 10.8.23.12@o2ib6) Feb 14 23:40:16 fir-io1-s1 kernel: Lustre: Skipped 100 previous similar messages Feb 14 23:50:18 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 14 23:50:18 fir-io1-s1 kernel: Lustre: Skipped 119 previous similar messages Feb 15 00:00:45 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 725d787f-ad1b-18f8-1091-f9fac1b511d9 (at 10.9.106.62@o2ib4) Feb 15 00:00:45 fir-io1-s1 kernel: Lustre: Skipped 133 previous similar messages Feb 15 00:10:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 15 00:10:57 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Feb 15 00:20:58 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 11bdb068-3a68-b486-d427-87b2a02899d5 (at 10.8.2.24@o2ib6) Feb 15 00:20:58 fir-io1-s1 kernel: Lustre: Skipped 278 previous similar messages Feb 15 00:31:00 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to eb304df2-55f5-4c00-5490-ecefc431af89 (at 10.8.12.18@o2ib6) Feb 15 00:31:00 fir-io1-s1 kernel: Lustre: Skipped 197 previous similar messages Feb 15 00:41:02 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 7cba68d6-88e2-7226-0b4f-cb83c3107f8f (at 10.9.102.34@o2ib4) Feb 15 00:41:02 fir-io1-s1 kernel: Lustre: Skipped 160 previous similar messages Feb 15 00:51:13 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 926fa24d-f3ab-7ad6-dbc7-f8a15bdf8c5a (at 10.8.19.8@o2ib6) Feb 15 00:51:13 fir-io1-s1 kernel: Lustre: Skipped 209 previous similar messages Feb 15 01:01:19 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a6a757ad-2ece-7189-1c8a-4459b2a0b330 (at 10.8.18.2@o2ib6) Feb 15 01:01:19 fir-io1-s1 kernel: Lustre: Skipped 125 previous similar messages Feb 15 01:11:20 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.101.64@o2ib4) Feb 15 01:11:20 fir-io1-s1 kernel: Lustre: Skipped 222 previous similar messages Feb 15 01:13:52 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000b_UUID: not available for connect from 10.0.10.52@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 15 01:13:52 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 15 01:21:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b7404664-4513-1975-142a-e2289416f002 (at 10.8.22.14@o2ib6) Feb 15 01:21:23 fir-io1-s1 kernel: Lustre: Skipped 217 previous similar messages Feb 15 01:31:24 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 948ddf77-30e8-6f7f-29c3-d9d6fe8d8435 (at 10.9.101.4@o2ib4) Feb 15 01:31:24 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 15 01:41:28 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 15 01:41:28 fir-io1-s1 kernel: Lustre: Skipped 227 previous similar messages Feb 15 01:51:39 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a2fcf67a-09d3-5d40-e9ba-95f310029e79 (at 10.9.102.21@o2ib4) Feb 15 01:51:39 fir-io1-s1 kernel: Lustre: Skipped 239 previous similar messages Feb 15 02:01:48 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a05ee6b2-bd86-6a30-4f7d-45fc6c076a48 (at 10.9.107.57@o2ib4) Feb 15 02:01:48 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 15 02:11:52 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Feb 15 02:11:52 fir-io1-s1 kernel: Lustre: Skipped 226 previous similar messages Feb 15 02:20:36 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 14fc2dba-9aec-2ff7-cbad-c1fd119632c0 (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762c71800, cur 1550226036 expire 1550225886 last 1550225809 Feb 15 02:20:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 15 02:21:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 15 02:21:53 fir-io1-s1 kernel: Lustre: Skipped 269 previous similar messages Feb 15 02:31:54 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a3e043df-71e9-4562-072f-6b8c6c088b8f (at 10.9.105.49@o2ib4) Feb 15 02:31:54 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 15 02:41:55 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 8b819495-2088-ce05-abe6-a051f7fc0b48 (at 10.9.104.7@o2ib4) Feb 15 02:41:55 fir-io1-s1 kernel: Lustre: Skipped 252 previous similar messages Feb 15 02:51:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 02:51:57 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 15 03:01:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 03:01:57 fir-io1-s1 kernel: Lustre: Skipped 160 previous similar messages Feb 15 03:11:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 03:11:57 fir-io1-s1 kernel: Lustre: Skipped 273 previous similar messages Feb 15 03:21:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 03:21:57 fir-io1-s1 kernel: Lustre: Skipped 340 previous similar messages Feb 15 03:31:57 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 03:31:57 fir-io1-s1 kernel: Lustre: Skipped 361 previous similar messages Feb 15 03:41:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 03:41:57 fir-io1-s1 kernel: Lustre: Skipped 287 previous similar messages Feb 15 03:51:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 03:51:57 fir-io1-s1 kernel: Lustre: Skipped 267 previous similar messages Feb 15 04:02:06 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.107.22@o2ib4) Feb 15 04:02:06 fir-io1-s1 kernel: Lustre: Skipped 209 previous similar messages Feb 15 04:12:08 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a51ee541-bf68-f6d3-6ffe-0b75e0908e4b (at 10.9.106.39@o2ib4) Feb 15 04:12:08 fir-io1-s1 kernel: Lustre: Skipped 254 previous similar messages Feb 15 04:22:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a58d34d6-2c4a-1753-b4fb-33e6ffbafbc0 (at 10.8.8.31@o2ib6) Feb 15 04:22:08 fir-io1-s1 kernel: Lustre: Skipped 275 previous similar messages Feb 15 04:32:34 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 86944b63-e282-0425-5312-520ee6361734 (at 10.8.22.10@o2ib6) Feb 15 04:32:34 fir-io1-s1 kernel: Lustre: Skipped 212 previous similar messages Feb 15 04:42:34 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6bfa800a-c0c5-71b7-464d-e3efc0c0229b (at 10.8.8.11@o2ib6) Feb 15 04:42:34 fir-io1-s1 kernel: Lustre: Skipped 271 previous similar messages Feb 15 04:52:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9d602dc8-8051-8982-51eb-1bb4250b93cd (at 10.8.22.18@o2ib6) Feb 15 04:52:56 fir-io1-s1 kernel: Lustre: Skipped 302 previous similar messages Feb 15 05:03:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.8.8.3@o2ib6) Feb 15 05:03:03 fir-io1-s1 kernel: Lustre: Skipped 227 previous similar messages Feb 15 05:13:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 15 05:13:17 fir-io1-s1 kernel: Lustre: Skipped 290 previous similar messages Feb 15 05:23:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 4f2a8620-81f0-e31b-fbad-a029c3256423 (at 10.9.105.43@o2ib4) Feb 15 05:23:22 fir-io1-s1 kernel: Lustre: Skipped 329 previous similar messages Feb 15 05:33:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0405d36b-1dfe-417d-33da-88f65ca0bd9f (at 10.8.28.4@o2ib6) Feb 15 05:33:39 fir-io1-s1 kernel: Lustre: Skipped 257 previous similar messages Feb 15 05:43:46 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 15 05:43:46 fir-io1-s1 kernel: Lustre: Skipped 283 previous similar messages Feb 15 05:53:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 054c4743-619d-965f-f786-cc0afc52d348 (at 10.9.101.68@o2ib4) Feb 15 05:53:52 fir-io1-s1 kernel: Lustre: Skipped 271 previous similar messages Feb 15 06:03:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9b6ee593-357f-08be-650d-81734979fa6c (at 10.9.107.40@o2ib4) Feb 15 06:03:54 fir-io1-s1 kernel: Lustre: Skipped 242 previous similar messages Feb 15 06:13:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Feb 15 06:13:58 fir-io1-s1 kernel: Lustre: Skipped 322 previous similar messages Feb 15 06:24:00 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 254b94e8-a059-c2e7-f0ce-7ccba72fe51c (at 10.8.1.34@o2ib6) Feb 15 06:24:00 fir-io1-s1 kernel: Lustre: Skipped 307 previous similar messages Feb 15 06:34:01 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a437cdb1-0624-3608-42e3-2ea1301fd34c (at 10.9.102.3@o2ib4) Feb 15 06:34:01 fir-io1-s1 kernel: Lustre: Skipped 276 previous similar messages Feb 15 06:40:32 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 19f61579-cd44-4d64-7654-1b016006b4d8 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aabd800, cur 1550241632 expire 1550241482 last 1550241405 Feb 15 06:40:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 15 06:44:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d91de257-d515-e16e-d341-03d1203ae5b3 (at 10.8.24.5@o2ib6) Feb 15 06:44:04 fir-io1-s1 kernel: Lustre: Skipped 298 previous similar messages Feb 15 06:54:07 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to b1c1324f-29ea-b1e6-d3ab-a5c4feafdaa0 (at 10.8.22.3@o2ib6) Feb 15 06:54:07 fir-io1-s1 kernel: Lustre: Skipped 374 previous similar messages Feb 15 07:04:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 028d6433-9e7d-1b84-c8b7-1bb2a8570ec4 (at 10.8.1.4@o2ib6) Feb 15 07:04:09 fir-io1-s1 kernel: Lustre: Skipped 262 previous similar messages Feb 15 07:14:17 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 32ea102c-c8a6-9ae9-e0f7-d3fc0379beb1 (at 10.8.2.23@o2ib6) Feb 15 07:14:17 fir-io1-s1 kernel: Lustre: Skipped 262 previous similar messages Feb 15 07:24:23 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 37aa7d65-aee3-56c5-fbe1-78e40cec7bbc (at 10.9.101.66@o2ib4) Feb 15 07:24:23 fir-io1-s1 kernel: Lustre: Skipped 300 previous similar messages Feb 15 07:34:41 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a437cdb1-0624-3608-42e3-2ea1301fd34c (at 10.9.102.3@o2ib4) Feb 15 07:34:41 fir-io1-s1 kernel: Lustre: Skipped 215 previous similar messages Feb 15 07:44:44 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 4386eb61-b821-6959-a669-9747602b9eba (at 10.9.107.20@o2ib4) Feb 15 07:44:44 fir-io1-s1 kernel: Lustre: Skipped 230 previous similar messages Feb 15 07:54:47 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 153a9818-8d26-403e-7a91-27bec80982b8 (at 10.8.23.35@o2ib6) Feb 15 07:54:47 fir-io1-s1 kernel: Lustre: Skipped 274 previous similar messages Feb 15 08:04:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) Feb 15 08:04:55 fir-io1-s1 kernel: Lustre: Skipped 274 previous similar messages Feb 15 08:15:00 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) Feb 15 08:15:00 fir-io1-s1 kernel: Lustre: Skipped 321 previous similar messages Feb 15 08:25:02 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a6a757ad-2ece-7189-1c8a-4459b2a0b330 (at 10.8.18.2@o2ib6) Feb 15 08:25:02 fir-io1-s1 kernel: Lustre: Skipped 437 previous similar messages Feb 15 08:35:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c9ad05ea-846e-fcaa-4bcc-aa3c8bd45ad4 (at 10.9.107.9@o2ib4) Feb 15 08:35:08 fir-io1-s1 kernel: Lustre: Skipped 382 previous similar messages Feb 15 08:45:08 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3d122a91-53f0-f449-1f10-d08490897e63 (at 10.9.106.65@o2ib4) Feb 15 08:45:08 fir-io1-s1 kernel: Lustre: Skipped 395 previous similar messages Feb 15 08:55:15 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 87a33bb7-d653-ad0d-1e59-70064b2446dd (at 10.9.105.64@o2ib4) Feb 15 08:55:15 fir-io1-s1 kernel: Lustre: Skipped 427 previous similar messages Feb 15 09:05:15 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to d3b133e8-4ec8-ebe3-7fc5-79aa16e59c0b (at 10.9.106.35@o2ib4) Feb 15 09:05:15 fir-io1-s1 kernel: Lustre: Skipped 365 previous similar messages Feb 15 09:15:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.64@o2ib4) Feb 15 09:15:17 fir-io1-s1 kernel: Lustre: Skipped 332 previous similar messages Feb 15 09:25:21 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Feb 15 09:25:21 fir-io1-s1 kernel: Lustre: Skipped 365 previous similar messages Feb 15 09:35:27 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2354d7af-653b-aaa3-2c33-b296ff69d0d2 (at 10.8.10.15@o2ib6) Feb 15 09:35:27 fir-io1-s1 kernel: Lustre: Skipped 330 previous similar messages Feb 15 09:45:27 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3ba1ab01-3e0e-7c6e-7b3b-01278da6f049 (at 10.8.20.31@o2ib6) Feb 15 09:45:27 fir-io1-s1 kernel: Lustre: Skipped 373 previous similar messages Feb 15 09:55:28 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 15 09:55:28 fir-io1-s1 kernel: Lustre: Skipped 365 previous similar messages Feb 15 10:05:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a8fc24fc-68d9-a5c4-e6f4-94cbb871c8dc (at 10.9.114.8@o2ib4) Feb 15 10:05:33 fir-io1-s1 kernel: Lustre: Skipped 356 previous similar messages Feb 15 10:15:46 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to cae45eb3-9c26-68be-7b4d-a987a0c2c715 (at 10.8.26.3@o2ib6) Feb 15 10:15:46 fir-io1-s1 kernel: Lustre: Skipped 304 previous similar messages Feb 15 10:25:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 1e5f7f4c-78fa-5eb7-a0ea-e8f04fabf57f (at 10.8.30.32@o2ib6) Feb 15 10:25:58 fir-io1-s1 kernel: Lustre: Skipped 376 previous similar messages Feb 15 10:35:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 830537ef-7de7-09e8-8833-72f2240ed68c (at 10.9.114.1@o2ib4) Feb 15 10:35:58 fir-io1-s1 kernel: Lustre: Skipped 315 previous similar messages Feb 15 10:46:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d78401ad-d7f9-70f0-e382-808abf23c9bd (at 10.9.103.10@o2ib4) Feb 15 10:46:02 fir-io1-s1 kernel: Lustre: Skipped 299 previous similar messages Feb 15 10:56:22 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 2b64d06d-0a33-7dc1-7c60-2608607acb48 (at 10.9.104.50@o2ib4) Feb 15 10:56:22 fir-io1-s1 kernel: Lustre: Skipped 345 previous similar messages Feb 15 11:06:26 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3e106753-f830-b044-4cf1-87da4380b4a5 (at 10.8.6.12@o2ib6) Feb 15 11:06:26 fir-io1-s1 kernel: Lustre: Skipped 314 previous similar messages Feb 15 11:16:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 15 11:16:36 fir-io1-s1 kernel: Lustre: Skipped 444 previous similar messages Feb 15 11:26:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c46b7f3d-ec55-7fd7-e207-e9c6f5525b60 (at 10.8.0.68@o2ib6) Feb 15 11:26:42 fir-io1-s1 kernel: Lustre: Skipped 276 previous similar messages Feb 15 11:36:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 26276ee4-1318-03d7-bf25-08cb51193a9d (at 10.9.102.22@o2ib4) Feb 15 11:36:42 fir-io1-s1 kernel: Lustre: Skipped 329 previous similar messages Feb 15 11:46:45 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to de25656c-fa14-1cbc-c67d-a4c1e6d26c7a (at 10.8.7.34@o2ib6) Feb 15 11:46:45 fir-io1-s1 kernel: Lustre: Skipped 328 previous similar messages Feb 15 11:56:46 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b41950de-0614-6c1a-0d53-d43c60fe0f33 (at 10.9.102.1@o2ib4) Feb 15 11:56:46 fir-io1-s1 kernel: Lustre: Skipped 387 previous similar messages Feb 15 12:06:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 12:06:57 fir-io1-s1 kernel: Lustre: Skipped 308 previous similar messages Feb 15 12:16:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 12:16:57 fir-io1-s1 kernel: Lustre: Skipped 192 previous similar messages Feb 15 12:26:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 12:26:57 fir-io1-s1 kernel: Lustre: Skipped 245 previous similar messages Feb 15 12:36:57 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 12:36:57 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 15 12:46:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 12:46:57 fir-io1-s1 kernel: Lustre: Skipped 256 previous similar messages Feb 15 12:57:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.0.64@o2ib4) Feb 15 12:57:04 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 15 13:07:10 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 763f631c-7b84-895c-764f-d88426b5fe26 (at 10.8.1.3@o2ib6) Feb 15 13:07:10 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 15 13:17:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 2c2d2f6b-3086-1f70-68ed-98263873eaff (at 10.9.107.25@o2ib4) Feb 15 13:17:18 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 15 13:27:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 15 13:27:23 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 15 13:37:36 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 89311c25-5811-466c-95ed-d7a183bd4753 (at 10.9.113.15@o2ib4) Feb 15 13:37:36 fir-io1-s1 kernel: Lustre: Skipped 180 previous similar messages Feb 15 13:47:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 899cd11a-4996-4751-56a3-fe3c17225e3d (at 10.9.105.29@o2ib4) Feb 15 13:47:39 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Feb 15 13:57:39 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.9.101.44@o2ib4) Feb 15 13:57:39 fir-io1-s1 kernel: Lustre: Skipped 348 previous similar messages Feb 15 13:58:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fbda2400, cur 1550267903 expire 1550267753 last 1550267676 Feb 15 13:58:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 15 14:07:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 98b4b957-f72b-06df-b2c8-094f2a70e1ca (at 10.9.102.39@o2ib4) Feb 15 14:07:41 fir-io1-s1 kernel: Lustre: Skipped 204 previous similar messages Feb 15 14:17:43 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 15a46d11-bb13-9263-b09b-73d01716030b (at 10.9.104.1@o2ib4) Feb 15 14:17:43 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 15 14:27:43 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d6335190-14e9-cc30-2081-663fdf52e20a (at 10.9.102.12@o2ib4) Feb 15 14:27:43 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 15 14:37:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fca0f45e-b434-84ac-1ed2-ce278e523445 (at 10.8.8.15@o2ib6) Feb 15 14:37:47 fir-io1-s1 kernel: Lustre: Skipped 316 previous similar messages Feb 15 14:47:48 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b5d0fd65-06a7-fd14-4925-ef95dfe63868 (at 10.9.107.28@o2ib4) Feb 15 14:47:48 fir-io1-s1 kernel: Lustre: Skipped 197 previous similar messages Feb 15 14:58:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b7a4f2fb-7087-2da1-fe97-a6b3f3961662 (at 10.9.103.38@o2ib4) Feb 15 14:58:08 fir-io1-s1 kernel: Lustre: Skipped 226 previous similar messages Feb 15 15:01:55 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b7a4f2fb-7087-2da1-fe97-a6b3f3961662 (at 10.9.103.38@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786927c00, cur 1550271715 expire 1550271565 last 1550271488 Feb 15 15:01:55 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 15 15:08:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Feb 15 15:08:14 fir-io1-s1 kernel: Lustre: Skipped 298 previous similar messages Feb 15 15:18:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f4b03aa2-d5b7-4f9a-0875-baaa698d022e (at 10.8.25.19@o2ib6) Feb 15 15:18:15 fir-io1-s1 kernel: Lustre: Skipped 410 previous similar messages Feb 15 15:28:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9b6ee593-357f-08be-650d-81734979fa6c (at 10.9.107.40@o2ib4) Feb 15 15:28:18 fir-io1-s1 kernel: Lustre: Skipped 319 previous similar messages Feb 15 15:38:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ed8fb36c-f8db-c3bc-c229-78172cc42522 (at 10.9.107.10@o2ib4) Feb 15 15:38:19 fir-io1-s1 kernel: Lustre: Skipped 429 previous similar messages Feb 15 15:48:24 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 57bd9565-249c-08fa-b75e-115d9c0f2fee (at 10.9.104.26@o2ib4) Feb 15 15:48:24 fir-io1-s1 kernel: Lustre: Skipped 328 previous similar messages Feb 15 15:53:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 27c4e23c-8f2c-7d4d-507f-8f8c7d058007 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c217400, cur 1550274819 expire 1550274669 last 1550274592 Feb 15 15:53:39 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 15 15:58:27 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ec1586f8-34f3-cd32-7140-123054dfbfed (at 10.9.102.30@o2ib4) Feb 15 15:58:27 fir-io1-s1 kernel: Lustre: Skipped 311 previous similar messages Feb 15 16:08:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9fdeb7e4-dbe7-36a2-5704-e3cbd89d3a9c (at 10.9.104.25@o2ib4) Feb 15 16:08:36 fir-io1-s1 kernel: Lustre: Skipped 195 previous similar messages Feb 15 16:18:38 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fb273c41-c272-402d-98b5-3e5f91dba50e (at 10.9.114.15@o2ib4) Feb 15 16:18:38 fir-io1-s1 kernel: Lustre: Skipped 254 previous similar messages Feb 15 16:19:21 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 572ef7cf-289a-fcb2-170a-766a28d2eaa2 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aabc400, cur 1550276361 expire 1550276211 last 1550276134 Feb 15 16:19:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 15 16:28:40 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 8427e8d1-34c3-886c-dab9-5ccc235f41c4 (at 10.9.107.5@o2ib4) Feb 15 16:28:40 fir-io1-s1 kernel: Lustre: Skipped 225 previous similar messages Feb 15 16:35:18 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6c5ac4b7-7b1b-b37a-4b63-edb3719ae54a (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868147b7c00, cur 1550277318 expire 1550277168 last 1550277091 Feb 15 16:35:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 15 16:38:44 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to be64bf26-6087-8b97-22e6-f82a5a07a7ba (at 10.8.30.20@o2ib6) Feb 15 16:38:44 fir-io1-s1 kernel: Lustre: Skipped 223 previous similar messages Feb 15 16:48:45 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f25e95fd-ca39-a936-c3ea-af6c0e743e71 (at 10.8.31.6@o2ib6) Feb 15 16:48:45 fir-io1-s1 kernel: Lustre: Skipped 164 previous similar messages Feb 15 16:57:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9ed66d12-a580-5aed-755f-4474cef71f97 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833e41c00, cur 1550278677 expire 1550278527 last 1550278450 Feb 15 16:57:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 15 16:58:51 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to efbcbe9e-a3dd-829d-5712-c2cc27d7fd18 (at 10.8.23.34@o2ib6) Feb 15 16:58:51 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 15 17:08:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 85cfcf77-29ac-d755-b385-af543ebdafc6 (at 10.9.101.31@o2ib4) Feb 15 17:08:56 fir-io1-s1 kernel: Lustre: Skipped 170 previous similar messages Feb 15 17:19:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a05ee6b2-bd86-6a30-4f7d-45fc6c076a48 (at 10.9.107.57@o2ib4) Feb 15 17:19:00 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 15 17:29:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 37327d93-dd03-80bf-ad2c-df17a42702a9 (at 10.8.18.26@o2ib6) Feb 15 17:29:19 fir-io1-s1 kernel: Lustre: Skipped 261 previous similar messages Feb 15 17:39:36 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f9896111-f5c5-60ad-1338-45ff52f63664 (at 10.8.8.13@o2ib6) Feb 15 17:39:36 fir-io1-s1 kernel: Lustre: Skipped 248 previous similar messages Feb 15 17:49:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3825d956-b97e-6239-e799-7bf6492ff2c9 (at 10.9.105.56@o2ib4) Feb 15 17:49:52 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 15 17:57:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 449ab5a0-1671-c50c-bbbe-21371abb55d6 (at 10.9.107.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480073c800, cur 1550282240 expire 1550282090 last 1550282013 Feb 15 17:57:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 15 17:57:34 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 449ab5a0-1671-c50c-bbbe-21371abb55d6 (at 10.9.107.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872ffc16000, cur 1550282254 expire 1550282104 last 1550282027 Feb 15 17:57:34 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 15 18:00:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c62ed1f1-3793-7a2f-05b0-df79888e04df (at 10.8.0.65@o2ib6) Feb 15 18:00:13 fir-io1-s1 kernel: Lustre: Skipped 114 previous similar messages Feb 15 18:04:24 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 58a24522-e5c5-5b7c-258b-500f9e3166ed (at 10.9.107.49@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481e2c4400, cur 1550282664 expire 1550282514 last 1550282437 Feb 15 18:04:24 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 15 18:10:52 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Feb 15 18:10:52 fir-io1-s1 kernel: Lustre: Skipped 192 previous similar messages Feb 15 18:20:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Feb 15 18:20:57 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 15 18:31:26 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6e436ee1-56e5-e1f3-3459-58b43a359102 (at 10.9.108.17@o2ib4) Feb 15 18:31:26 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 15 18:36:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2b160d7e-8d51-b48d-ec95-d0ac3846be82 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984830fb5800, cur 1550284601 expire 1550284451 last 1550284374 Feb 15 18:36:41 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 15 18:41:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f25e95fd-ca39-a936-c3ea-af6c0e743e71 (at 10.8.31.6@o2ib6) Feb 15 18:41:35 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 15 18:51:46 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 67ef383a-b6cc-67d1-a82b-0713a3b720e4 (at 10.8.22.5@o2ib6) Feb 15 18:51:46 fir-io1-s1 kernel: Lustre: Skipped 222 previous similar messages Feb 15 19:01:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 19:01:57 fir-io1-s1 kernel: Lustre: Skipped 142 previous similar messages Feb 15 19:11:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 19:11:57 fir-io1-s1 kernel: Lustre: Skipped 161 previous similar messages Feb 15 19:21:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 15 19:21:57 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 15 19:32:20 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b0f4a89e-7973-eb31-a1c9-fdc42b6cc4f6 (at 10.8.18.25@o2ib6) Feb 15 19:32:20 fir-io1-s1 kernel: Lustre: Skipped 136 previous similar messages Feb 15 19:42:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 70103d89-ec34-ce17-2e6c-cc604ff9ae8a (at 10.8.27.29@o2ib6) Feb 15 19:42:23 fir-io1-s1 kernel: Lustre: Skipped 144 previous similar messages Feb 15 19:52:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4c9919a3-1838-af9d-fa66-ce9474087e70 (at 10.8.21.27@o2ib6) Feb 15 19:52:42 fir-io1-s1 kernel: Lustre: Skipped 300 previous similar messages Feb 15 20:02:48 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9f0fc7f1-3017-a377-d5ca-3f45fac7d96c (at 10.9.106.2@o2ib4) Feb 15 20:02:48 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 15 20:02:58 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9878323e9400, cur 1550289778 expire 1550289628 last 1550289551 Feb 15 20:02:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 15 20:03:01 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784ae8400, cur 1550289781 expire 1550289631 last 1550289554 Feb 15 20:03:01 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 15 20:03:03 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1414000, cur 1550289783 expire 1550289633 last 1550289556 Feb 15 20:03:03 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 15 20:12:54 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 7c312f41-52f1-f4c1-0bd5-a3e41737d0f6 (at 10.9.106.56@o2ib4) Feb 15 20:12:54 fir-io1-s1 kernel: Lustre: Skipped 191 previous similar messages Feb 15 20:22:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 2336058a-aa9c-4463-bac9-a8ea66369e87 (at 10.8.11.22@o2ib6) Feb 15 20:22:58 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 15 20:33:16 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 15 20:33:16 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 15 20:43:16 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to c8bbd681-6d42-3196-83fc-d269a276781d (at 10.9.105.55@o2ib4) Feb 15 20:43:16 fir-io1-s1 kernel: Lustre: Skipped 116 previous similar messages Feb 15 20:53:22 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to bd525d3e-a429-d051-4dce-9b431ca4655f (at 10.8.7.33@o2ib6) Feb 15 20:53:22 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 15 21:03:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d01707bf-d8db-a4c3-f544-1f9ecca8f036 (at 10.8.18.29@o2ib6) Feb 15 21:03:26 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 15 21:14:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 37327d93-dd03-80bf-ad2c-df17a42702a9 (at 10.8.18.26@o2ib6) Feb 15 21:14:19 fir-io1-s1 kernel: Lustre: Skipped 130 previous similar messages Feb 15 21:24:24 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to b7eb786c-4da9-9431-a597-6f5f4ba4c9ed (at 10.9.114.12@o2ib4) Feb 15 21:24:24 fir-io1-s1 kernel: Lustre: Skipped 170 previous similar messages Feb 15 21:34:43 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 378375cf-e47b-0dfd-24f9-b821ea9c2298 (at 10.8.22.15@o2ib6) Feb 15 21:34:43 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 15 21:44:48 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fe8f9169-1b75-ba67-02e9-ac6ba53a9586 (at 10.8.24.32@o2ib6) Feb 15 21:44:48 fir-io1-s1 kernel: Lustre: Skipped 241 previous similar messages Feb 15 21:54:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd33c9b1-bb5e-c24e-0675-22654fcc67c5 (at 10.8.24.26@o2ib6) Feb 15 21:54:51 fir-io1-s1 kernel: Lustre: Skipped 261 previous similar messages Feb 15 22:02:22 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550296935/real 1550296935] req@ffff9858ba3ef800 x1624933388884704/t0(0) o104->fir-OST0008@10.9.107.51@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550296942 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 15 22:02:22 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Feb 15 22:02:36 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550296949/real 1550296949] req@ffff9858ba3ef800 x1624933388884704/t0(0) o104->fir-OST0008@10.9.107.51@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550296956 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 15 22:02:36 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 15 22:02:57 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550296970/real 1550296970] req@ffff9858ba3ef800 x1624933388884704/t0(0) o104->fir-OST0008@10.9.107.51@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550296977 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 15 22:02:57 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 15 22:03:32 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550297005/real 1550297005] req@ffff9858ba3ef800 x1624933388884704/t0(0) o104->fir-OST0008@10.9.107.51@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550297012 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 15 22:03:32 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 15 22:04:42 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550297075/real 1550297075] req@ffff9858ba3ef800 x1624933388884704/t0(0) o104->fir-OST0008@10.9.107.51@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550297082 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 15 22:04:42 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 15 22:04:49 fir-io1-s1 kernel: LustreError: 96933:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.107.51@o2ib4) failed to reply to blocking AST (req@ffff9858ba3ef800 x1624933388884704 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff984309f77980/0x49e185e970fbcab8 lrc: 4/0,0 mode: PW/PW res: [0xc80000401:0x5bb40:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->268439551) flags: 0x60000400010020 nid: 10.9.107.51@o2ib4 remote: 0xf5ed45b8ef7efece expref: 6 pid: 96887 timeout: 644983 lvb_type: 0 Feb 15 22:04:49 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.9.107.51@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Feb 15 22:04:49 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 15 22:04:49 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.9.107.51@o2ib4 ns: filter-fir-OST0008_UUID lock: ffff984309f77980/0x49e185e970fbcab8 lrc: 3/0,0 mode: PW/PW res: [0xc80000401:0x5bb40:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->268439551) flags: 0x60000400010020 nid: 10.9.107.51@o2ib4 remote: 0xf5ed45b8ef7efece expref: 7 pid: 96887 timeout: 0 lvb_type: 0 Feb 15 22:04:50 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Feb 15 22:04:51 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 15 22:04:51 fir-io1-s1 kernel: Lustre: Skipped 390 previous similar messages Feb 15 22:05:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 861509dc-4523-55e8-f09a-16b6fca3f713 (at 10.9.107.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848019c3400, cur 1550297157 expire 1550297007 last 1550296930 Feb 15 22:14:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 15 22:14:57 fir-io1-s1 kernel: Lustre: Skipped 438 previous similar messages Feb 15 22:25:01 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 15 22:25:01 fir-io1-s1 kernel: Lustre: Skipped 307 previous similar messages Feb 15 22:35:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a8fc24fc-68d9-a5c4-e6f4-94cbb871c8dc (at 10.9.114.8@o2ib4) Feb 15 22:35:06 fir-io1-s1 kernel: Lustre: Skipped 238 previous similar messages Feb 15 22:45:44 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 15 22:45:44 fir-io1-s1 kernel: Lustre: Skipped 231 previous similar messages Feb 15 22:55:46 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 72ef71e7-ecde-a7f2-d85d-452a22011f5b (at 10.9.101.14@o2ib4) Feb 15 22:55:46 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 15 23:05:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f25e95fd-ca39-a936-c3ea-af6c0e743e71 (at 10.8.31.6@o2ib6) Feb 15 23:05:53 fir-io1-s1 kernel: Lustre: Skipped 179 previous similar messages Feb 15 23:12:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 302c6952-5b6c-1588-cbd4-2c54f063f559 (at 10.9.103.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fbda7c00, cur 1550301126 expire 1550300976 last 1550300899 Feb 15 23:12:06 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 15 23:12:15 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 302c6952-5b6c-1588-cbd4-2c54f063f559 (at 10.9.103.37@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a37dd800, cur 1550301135 expire 1550300985 last 1550300908 Feb 15 23:12:15 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 15 23:15:53 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 77a364b6-c2d7-052b-2332-0b2d1cfdace4 (at 10.9.106.54@o2ib4) Feb 15 23:15:53 fir-io1-s1 kernel: Lustre: Skipped 236 previous similar messages Feb 15 23:25:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ed72929c-d604-56ff-ec0a-5b1f9da0af84 (at 10.8.17.9@o2ib6) Feb 15 23:25:56 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 15 23:36:09 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 15 23:36:09 fir-io1-s1 kernel: Lustre: Skipped 230 previous similar messages Feb 15 23:46:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9d90a6a6-e463-02e9-3fef-fe0fa60e4307 (at 10.9.114.13@o2ib4) Feb 15 23:46:12 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 15 23:56:20 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 7d64396e-7a86-2e01-38b5-8f4fd2cfeb04 (at 10.8.19.4@o2ib6) Feb 15 23:56:20 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 16 00:06:22 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ec7c15c1-1122-48df-e09c-cacc05cb75a8 (at 10.8.1.15@o2ib6) Feb 16 00:06:22 fir-io1-s1 kernel: Lustre: Skipped 488 previous similar messages Feb 16 00:16:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 58a24522-e5c5-5b7c-258b-500f9e3166ed (at 10.9.107.49@o2ib4) Feb 16 00:16:23 fir-io1-s1 kernel: Lustre: Skipped 241 previous similar messages Feb 16 00:26:23 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 7d6292c2-dc0a-0082-5273-c1ff8e6163ed (at 10.9.102.25@o2ib4) Feb 16 00:26:23 fir-io1-s1 kernel: Lustre: Skipped 202 previous similar messages Feb 16 00:36:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) Feb 16 00:36:26 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 16 00:46:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to eb0b17ca-746e-7622-abd4-371c493253d0 (at 10.9.103.24@o2ib4) Feb 16 00:46:26 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 16 00:56:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 16 00:56:38 fir-io1-s1 kernel: Lustre: Skipped 143 previous similar messages Feb 16 01:06:40 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) Feb 16 01:06:40 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 16 01:16:51 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 924a230f-7a9e-90ca-0838-4eb0790eb9f6 (at 10.8.20.16@o2ib6) Feb 16 01:16:51 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 16 01:26:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 01:26:57 fir-io1-s1 kernel: Lustre: Skipped 156 previous similar messages Feb 16 01:36:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 01:36:57 fir-io1-s1 kernel: Lustre: Skipped 199 previous similar messages Feb 16 01:46:57 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 01:46:57 fir-io1-s1 kernel: Lustre: Skipped 192 previous similar messages Feb 16 01:57:02 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) Feb 16 01:57:02 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 16 02:07:05 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6d1355d0-7b33-677d-c8cf-a270e3061917 (at 10.8.7.15@o2ib6) Feb 16 02:07:05 fir-io1-s1 kernel: Lustre: Skipped 180 previous similar messages Feb 16 02:13:38 fir-io1-s1 kernel: Lustre: 96915:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550312011/real 1550312011] req@ffff98545b17ec00 x1624933434394304/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550312018 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 16 02:13:38 fir-io1-s1 kernel: Lustre: 96915:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 16 02:13:59 fir-io1-s1 kernel: Lustre: 96942:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550312032/real 1550312032] req@ffff9849298cb000 x1624933434394288/t0(0) o106->fir-OST0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550312039 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 16 02:13:59 fir-io1-s1 kernel: Lustre: 96942:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 16 02:14:34 fir-io1-s1 kernel: Lustre: 96573:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550312067/real 1550312067] req@ffff984d30599500 x1624933434394320/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550312074 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 16 02:14:34 fir-io1-s1 kernel: Lustre: 96573:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 16 02:15:30 fir-io1-s1 kernel: LustreError: 96362:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) returned error from glimpse AST (req@ffff985647ddc200 x1624933434394272 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff9856815be0c0/0x49e185e975c7b6fb lrc: 3/0,0 mode: PW/PW res: [0xa570e:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0x9029d0e0bddd351b expref: 5 pid: 96782 timeout: 0 lvb_type: 0 Feb 16 02:15:30 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 16 02:15:30 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550312130s: evicting client at 10.8.9.8@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff9856815b9200/0x49e185e975c7b709 lrc: 3/0,0 mode: PW/PW res: [0xa571c:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0x9029d0e0bddd358b expref: 6 pid: 96782 timeout: 0 lvb_type: 0 Feb 16 02:15:30 fir-io1-s1 kernel: LustreError: 96362:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Feb 16 02:15:37 fir-io1-s1 kernel: LustreError: 96573:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) returned error from glimpse AST (req@ffff984d30599500 x1624933434394320 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff9856815b8240/0x49e185e975c7b710 lrc: 3/0,0 mode: PW/PW res: [0xa5732:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0x9029d0e0bddd35c3 expref: 5 pid: 96782 timeout: 0 lvb_type: 0 Feb 16 02:15:37 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 16 02:15:37 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 16 02:15:37 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550312137s: evicting client at 10.8.9.8@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff9856815b8240/0x49e185e975c7b710 lrc: 3/0,0 mode: PW/PW res: [0xa5732:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0x9029d0e0bddd35c3 expref: 6 pid: 96782 timeout: 0 lvb_type: 0 Feb 16 02:15:37 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Feb 16 02:17:30 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 37aa7d65-aee3-56c5-fbe1-78e40cec7bbc (at 10.9.101.66@o2ib4) Feb 16 02:17:30 fir-io1-s1 kernel: Lustre: Skipped 201 previous similar messages Feb 16 02:27:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) Feb 16 02:27:30 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Feb 16 02:37:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 16 02:37:35 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 16 02:47:40 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e6e1afb6-7acc-3808-2f05-02b79c99637e (at 10.8.23.5@o2ib6) Feb 16 02:47:40 fir-io1-s1 kernel: Lustre: Skipped 152 previous similar messages Feb 16 02:55:16 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4de04c00-036a-d169-5318-daba6e51697c (at 10.9.102.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825fac00, cur 1550314516 expire 1550314366 last 1550314289 Feb 16 02:55:16 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 16 02:57:43 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3dd7b3c3-369e-14ba-c881-c252e5dc17a0 (at 10.8.8.27@o2ib6) Feb 16 02:57:43 fir-io1-s1 kernel: Lustre: Skipped 148 previous similar messages Feb 16 03:07:45 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) Feb 16 03:07:45 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 16 03:17:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) Feb 16 03:17:56 fir-io1-s1 kernel: Lustre: Skipped 174 previous similar messages Feb 16 03:28:11 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 749ba667-6b1b-7176-05ce-9fcec3751e62 (at 10.9.108.18@o2ib4) Feb 16 03:28:11 fir-io1-s1 kernel: Lustre: Skipped 144 previous similar messages Feb 16 03:38:27 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 16 03:38:27 fir-io1-s1 kernel: Lustre: Skipped 229 previous similar messages Feb 16 03:48:29 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) Feb 16 03:48:29 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 16 03:58:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9bd541c8-5e18-2470-8262-fd1a455e43c1 (at 10.9.102.35@o2ib4) Feb 16 03:58:33 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 16 04:08:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 16 04:08:34 fir-io1-s1 kernel: Lustre: Skipped 192 previous similar messages Feb 16 04:18:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 16 04:18:44 fir-io1-s1 kernel: Lustre: Skipped 148 previous similar messages Feb 16 04:29:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9ad9001-d2d9-466a-37c8-3f54fd94183d (at 10.9.101.26@o2ib4) Feb 16 04:29:03 fir-io1-s1 kernel: Lustre: Skipped 148 previous similar messages Feb 16 04:39:36 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c95418d8-abb0-d4fc-c763-439a35e76a6c (at 10.9.102.50@o2ib4) Feb 16 04:39:36 fir-io1-s1 kernel: Lustre: Skipped 159 previous similar messages Feb 16 04:49:46 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ee9ee40a-8c5b-a83c-3054-d54662cf8892 (at 10.9.107.29@o2ib4) Feb 16 04:49:46 fir-io1-s1 kernel: Lustre: Skipped 135 previous similar messages Feb 16 04:59:49 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) Feb 16 04:59:49 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 16 05:09:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c46b7f3d-ec55-7fd7-e207-e9c6f5525b60 (at 10.8.0.68@o2ib6) Feb 16 05:09:51 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 16 05:19:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4f2a8620-81f0-e31b-fbad-a029c3256423 (at 10.9.105.43@o2ib4) Feb 16 05:19:56 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Feb 16 05:30:12 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5fd68af5-0c2b-1947-5fc1-6504b55b60fb (at 10.9.103.16@o2ib4) Feb 16 05:30:12 fir-io1-s1 kernel: Lustre: Skipped 120 previous similar messages Feb 16 05:40:13 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 16 05:40:13 fir-io1-s1 kernel: Lustre: Skipped 146 previous similar messages Feb 16 05:50:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8f86419d-3c7d-f8e0-fb5d-facc0f493f73 (at 10.8.27.31@o2ib6) Feb 16 05:50:16 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 16 06:00:23 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 56f157dc-9f05-a349-697a-ac16ba31313e (at 10.9.104.59@o2ib4) Feb 16 06:00:23 fir-io1-s1 kernel: Lustre: Skipped 180 previous similar messages Feb 16 06:10:42 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) Feb 16 06:10:42 fir-io1-s1 kernel: Lustre: Skipped 138 previous similar messages Feb 16 06:20:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a58d34d6-2c4a-1753-b4fb-33e6ffbafbc0 (at 10.8.8.31@o2ib6) Feb 16 06:20:53 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 16 06:31:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 74f9852d-189b-8596-eb4f-bcf617e42f7c (at 10.8.7.22@o2ib6) Feb 16 06:31:00 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 16 06:41:00 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.10.27@o2ib6) Feb 16 06:41:00 fir-io1-s1 kernel: Lustre: Skipped 322 previous similar messages Feb 16 06:51:01 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 8b819495-2088-ce05-abe6-a051f7fc0b48 (at 10.9.104.7@o2ib4) Feb 16 06:51:01 fir-io1-s1 kernel: Lustre: Skipped 291 previous similar messages Feb 16 07:01:05 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a58d34d6-2c4a-1753-b4fb-33e6ffbafbc0 (at 10.8.8.31@o2ib6) Feb 16 07:01:05 fir-io1-s1 kernel: Lustre: Skipped 233 previous similar messages Feb 16 07:11:13 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 56f157dc-9f05-a349-697a-ac16ba31313e (at 10.9.104.59@o2ib4) Feb 16 07:11:13 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 16 07:21:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2513721a-1a7a-beeb-ce96-babfef130551 (at 10.8.18.32@o2ib6) Feb 16 07:21:19 fir-io1-s1 kernel: Lustre: Skipped 270 previous similar messages Feb 16 07:31:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 254b94e8-a059-c2e7-f0ce-7ccba72fe51c (at 10.8.1.34@o2ib6) Feb 16 07:31:37 fir-io1-s1 kernel: Lustre: Skipped 327 previous similar messages Feb 16 07:41:45 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3b2e850b-830c-c045-0e53-e91a4da0ae80 (at 10.9.108.6@o2ib4) Feb 16 07:41:45 fir-io1-s1 kernel: Lustre: Skipped 256 previous similar messages Feb 16 07:51:46 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 2bda397b-d7a2-3060-3dca-1fd04dabf1e9 (at 10.8.30.8@o2ib6) Feb 16 07:51:46 fir-io1-s1 kernel: Lustre: Skipped 187 previous similar messages Feb 16 08:01:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 08:01:56 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 16 08:11:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 08:11:56 fir-io1-s1 kernel: Lustre: Skipped 158 previous similar messages Feb 16 08:21:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 08:21:56 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 16 08:31:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 08:31:56 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 16 08:41:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 08:41:57 fir-io1-s1 kernel: Lustre: Skipped 217 previous similar messages Feb 16 08:52:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 880716fb-6a2b-3d87-95fb-03534cabe92d (at 10.8.8.28@o2ib6) Feb 16 08:52:03 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 16 09:02:07 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 94c2dba8-b225-c91a-753d-91a7f0495a0f (at 10.9.101.8@o2ib4) Feb 16 09:02:07 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 16 09:12:07 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3dd7b3c3-369e-14ba-c881-c252e5dc17a0 (at 10.8.8.27@o2ib6) Feb 16 09:12:07 fir-io1-s1 kernel: Lustre: Skipped 275 previous similar messages Feb 16 09:22:11 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4f2a8620-81f0-e31b-fbad-a029c3256423 (at 10.9.105.43@o2ib4) Feb 16 09:22:11 fir-io1-s1 kernel: Lustre: Skipped 224 previous similar messages Feb 16 09:32:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 16 09:32:11 fir-io1-s1 kernel: Lustre: Skipped 274 previous similar messages Feb 16 09:42:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2d862fb5-44ea-7d60-cb0c-d7ea36856c36 (at 10.8.26.9@o2ib6) Feb 16 09:42:23 fir-io1-s1 kernel: Lustre: Skipped 219 previous similar messages Feb 16 09:52:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3b2e850b-830c-c045-0e53-e91a4da0ae80 (at 10.9.108.6@o2ib4) Feb 16 09:52:39 fir-io1-s1 kernel: Lustre: Skipped 217 previous similar messages Feb 16 10:02:45 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ac2dd52e-30a9-6491-144f-abbf56256ba7 (at 10.8.8.14@o2ib6) Feb 16 10:02:45 fir-io1-s1 kernel: Lustre: Skipped 249 previous similar messages Feb 16 10:12:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) Feb 16 10:12:50 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 16 10:19:54 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ee671f31-da00-d1b0-f303-04caaad3111c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f78800, cur 1550341194 expire 1550341044 last 1550340967 Feb 16 10:19:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 10:22:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d4ede191-33ac-db3d-8e23-e76bd511a700 (at 10.8.28.3@o2ib6) Feb 16 10:22:50 fir-io1-s1 kernel: Lustre: Skipped 228 previous similar messages Feb 16 10:28:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e2391796-4a42-0517-5057-c9949fd5edc5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4e000, cur 1550341713 expire 1550341563 last 1550341486 Feb 16 10:28:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 10:32:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f39378cd-1ea8-6f18-7368-d4c58159a6f7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867851e0800, cur 1550341956 expire 1550341806 last 1550341729 Feb 16 10:32:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 10:32:51 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.8.8.9@o2ib6) Feb 16 10:32:51 fir-io1-s1 kernel: Lustre: Skipped 260 previous similar messages Feb 16 10:36:00 fir-io1-s1 kernel: Lustre: 96757:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342153/real 1550342153] req@ffff9862d15b3600 x1624933530763696/t0(0) o106->fir-OST0002@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342160 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 16 10:36:00 fir-io1-s1 kernel: Lustre: 96752:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342153/real 1550342153] req@ffff985ccb0a4800 x1624933530763728/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342160 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 16 10:36:00 fir-io1-s1 kernel: Lustre: 96752:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 34 previous similar messages Feb 16 10:36:08 fir-io1-s1 kernel: Lustre: 94527:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342161/real 1550342161] req@ffff9861a8080900 x1624933530786592/t0(0) o106->fir-OST0008@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342168 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 16 10:36:08 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342161/real 1550342161] req@ffff9862a3c6d100 x1624933530786608/t0(0) o106->fir-OST000a@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342168 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 16 10:36:08 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 16 10:36:26 fir-io1-s1 kernel: Lustre: 96892:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342179/real 1550342179] req@ffff98545b179e00 x1624933530780432/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342186 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 16 10:36:26 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342179/real 1550342179] req@ffff985185b24e00 x1624933530780416/t0(0) o106->fir-OST0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342186 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 16 10:36:26 fir-io1-s1 kernel: Lustre: 96516:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Feb 16 10:36:26 fir-io1-s1 kernel: Lustre: 96892:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 16 10:37:31 fir-io1-s1 kernel: Lustre: 96752:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342244/real 1550342244] req@ffff985ccb0a4800 x1624933530763728/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342251 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 16 10:37:31 fir-io1-s1 kernel: Lustre: 96409:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550342244/real 1550342244] req@ffff98669bf98300 x1624933530763712/t0(0) o106->fir-OST0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550342251 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 16 10:37:31 fir-io1-s1 kernel: Lustre: 96409:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 107 previous similar messages Feb 16 10:37:31 fir-io1-s1 kernel: Lustre: 96752:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 16 10:38:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 46f91c5b-d968-31fa-6d36-f3b9ad14eabd (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857620a0000, cur 1550342316 expire 1550342166 last 1550342089 Feb 16 10:38:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 10:43:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) Feb 16 10:43:13 fir-io1-s1 kernel: Lustre: Skipped 298 previous similar messages Feb 16 10:53:15 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 935e7eb3-1ff6-7dda-ab9a-d14a4b5f1855 (at 10.9.103.32@o2ib4) Feb 16 10:53:15 fir-io1-s1 kernel: Lustre: Skipped 225 previous similar messages Feb 16 11:03:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d01707bf-d8db-a4c3-f544-1f9ecca8f036 (at 10.8.18.29@o2ib6) Feb 16 11:03:26 fir-io1-s1 kernel: Lustre: Skipped 241 previous similar messages Feb 16 11:13:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2d279b6b-ae49-37ac-0a12-0938de9dc4ca (at 10.8.1.29@o2ib6) Feb 16 11:13:38 fir-io1-s1 kernel: Lustre: Skipped 206 previous similar messages Feb 16 11:23:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 16 11:23:42 fir-io1-s1 kernel: Lustre: Skipped 196 previous similar messages Feb 16 11:29:07 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 317a25ea-7b33-6026-889e-f59bd4b2e4b1 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a37ed400, cur 1550345347 expire 1550345197 last 1550345120 Feb 16 11:29:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 11:33:43 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 87721f3b-4f03-c138-ffa3-cffa8a052df0 (at 10.8.26.5@o2ib6) Feb 16 11:33:43 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 16 11:43:51 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 1b2e0ae8-73cd-62bb-e10f-cb09e6b5f49b (at 10.8.31.7@o2ib6) Feb 16 11:43:51 fir-io1-s1 kernel: Lustre: Skipped 222 previous similar messages Feb 16 11:53:53 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 50290e53-c65d-a70c-6960-ed601e5d1ddb (at 10.8.1.35@o2ib6) Feb 16 11:53:53 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 16 12:03:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 56f157dc-9f05-a349-697a-ac16ba31313e (at 10.9.104.59@o2ib4) Feb 16 12:03:56 fir-io1-s1 kernel: Lustre: Skipped 219 previous similar messages Feb 16 12:13:59 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 16 12:13:59 fir-io1-s1 kernel: Lustre: Skipped 170 previous similar messages Feb 16 12:20:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client badc1944-bcba-0dc4-789f-128a2c69684a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848000ed400, cur 1550348449 expire 1550348299 last 1550348222 Feb 16 12:20:49 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 16 12:24:03 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bf62c7df-fe55-c26d-63f8-adbf89ed0ecb (at 10.8.3.34@o2ib6) Feb 16 12:24:03 fir-io1-s1 kernel: Lustre: Skipped 216 previous similar messages Feb 16 12:34:06 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a7acab70-6d4b-a155-287b-afe8188570a1 (at 10.8.21.16@o2ib6) Feb 16 12:34:06 fir-io1-s1 kernel: Lustre: Skipped 164 previous similar messages Feb 16 12:44:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fced5f19-499d-5f3d-efe5-faf9d4f8cdcd (at 10.9.107.50@o2ib4) Feb 16 12:44:09 fir-io1-s1 kernel: Lustre: Skipped 217 previous similar messages Feb 16 12:54:15 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 16 12:54:15 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 16 13:04:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 16 13:04:15 fir-io1-s1 kernel: Lustre: Skipped 319 previous similar messages Feb 16 13:14:19 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5f9695c8-8953-f70d-cc1c-5dc8656027f2 (at 10.9.101.20@o2ib4) Feb 16 13:14:19 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 16 13:24:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 090fafdf-b851-44a0-92d1-bfda03f3741e (at 10.8.8.29@o2ib6) Feb 16 13:24:23 fir-io1-s1 kernel: Lustre: Skipped 233 previous similar messages Feb 16 13:34:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fda7a4af-47c0-0068-cddf-309c3a9c784c (at 10.9.101.13@o2ib4) Feb 16 13:34:23 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 16 13:44:24 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 16 13:44:24 fir-io1-s1 kernel: Lustre: Skipped 201 previous similar messages Feb 16 13:54:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 71eebf52-b0fc-514f-7aaf-aca66e4f2af1 (at 10.8.27.27@o2ib6) Feb 16 13:54:28 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 16 14:04:32 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 7d6292c2-dc0a-0082-5273-c1ff8e6163ed (at 10.9.102.25@o2ib4) Feb 16 14:04:32 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 16 14:14:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) Feb 16 14:14:41 fir-io1-s1 kernel: Lustre: Skipped 387 previous similar messages Feb 16 14:25:06 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b3ef6690-dd23-2eef-0dcf-441d88950a4a (at 10.8.28.2@o2ib6) Feb 16 14:25:06 fir-io1-s1 kernel: Lustre: Skipped 146 previous similar messages Feb 16 14:26:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8af5284a-1e39-6217-098a-1610faab046e (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576dc9d000, cur 1550356002 expire 1550355852 last 1550355775 Feb 16 14:26:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 14:35:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8227297d-ff05-ae05-a096-01f468e47718 (at 10.8.21.24@o2ib6) Feb 16 14:35:07 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 16 14:45:18 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fbc61b39-e424-c398-e545-061af0049cf9 (at 10.9.107.11@o2ib4) Feb 16 14:45:18 fir-io1-s1 kernel: Lustre: Skipped 190 previous similar messages Feb 16 14:55:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 16 14:55:30 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 16 15:05:43 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 84178e66-d089-b59a-e089-73356cd9e03b (at 10.8.7.28@o2ib6) Feb 16 15:05:43 fir-io1-s1 kernel: Lustre: Skipped 108 previous similar messages Feb 16 15:15:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.8.7.17@o2ib6) Feb 16 15:15:47 fir-io1-s1 kernel: Lustre: Skipped 196 previous similar messages Feb 16 15:17:54 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client daeb388f-c3fa-fee9-b182-e2f48aacc876 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581bfeec00, cur 1550359074 expire 1550358924 last 1550358847 Feb 16 15:17:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 15:26:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c7d587fd-2878-3a53-bb0b-89a81458bb83 (at 10.8.6.5@o2ib6) Feb 16 15:26:09 fir-io1-s1 kernel: Lustre: Skipped 206 previous similar messages Feb 16 15:36:17 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7b3b6f2b-764c-4642-1783-263f87e59249 (at 10.8.1.10@o2ib6) Feb 16 15:36:17 fir-io1-s1 kernel: Lustre: Skipped 248 previous similar messages Feb 16 15:41:09 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7c5310d8-e3b1-6e97-c5a4-e046c89a7b88 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762600800, cur 1550360469 expire 1550360319 last 1550360242 Feb 16 15:41:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 15:46:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 033127f9-d684-a263-9e10-535f772c4f1a (at 10.9.106.25@o2ib4) Feb 16 15:46:35 fir-io1-s1 kernel: Lustre: Skipped 263 previous similar messages Feb 16 15:56:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 38ab0677-0c51-35a5-8e38-bb5f254042a2 (at 10.8.17.4@o2ib6) Feb 16 15:56:37 fir-io1-s1 kernel: Lustre: Skipped 376 previous similar messages Feb 16 16:06:37 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 495369c4-40aa-b2ab-a0e0-f943478581b7 (at 10.8.20.23@o2ib6) Feb 16 16:06:37 fir-io1-s1 kernel: Lustre: Skipped 497 previous similar messages Feb 16 16:16:42 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.3.27@o2ib6) Feb 16 16:16:42 fir-io1-s1 kernel: Lustre: Skipped 555 previous similar messages Feb 16 16:26:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 16:26:56 fir-io1-s1 kernel: Lustre: Skipped 274 previous similar messages Feb 16 16:30:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3319a925-b8d9-98a4-4961-c4c7b163e899 (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de6c00, cur 1550363435 expire 1550363285 last 1550363208 Feb 16 16:30:35 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 16 16:30:46 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3319a925-b8d9-98a4-4961-c4c7b163e899 (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f26a04800, cur 1550363446 expire 1550363296 last 1550363219 Feb 16 16:30:46 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 16 16:36:57 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 16:36:57 fir-io1-s1 kernel: Lustre: Skipped 206 previous similar messages Feb 16 16:43:38 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b20de432-e88a-d55e-b974-b470758f81fd (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b91000, cur 1550364218 expire 1550364068 last 1550363991 Feb 16 16:47:01 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 16 16:47:01 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Feb 16 16:57:01 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 38a104b5-26ce-5d2d-596d-9304083f888f (at 10.9.112.14@o2ib4) Feb 16 16:57:01 fir-io1-s1 kernel: Lustre: Skipped 197 previous similar messages Feb 16 17:07:19 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 16 17:07:19 fir-io1-s1 kernel: Lustre: Skipped 179 previous similar messages Feb 16 17:17:24 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 37b4854b-e93e-85a5-e644-9d0c6be8cc09 (at 10.8.2.29@o2ib6) Feb 16 17:17:24 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Feb 16 17:27:25 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 47ce4289-7e25-8c66-9590-6b36cdee8e22 (at 10.9.101.1@o2ib4) Feb 16 17:27:25 fir-io1-s1 kernel: Lustre: Skipped 188 previous similar messages Feb 16 17:37:33 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0f6eddca-58e8-c961-ebc9-8961d8390dc3 (at 10.8.7.18@o2ib6) Feb 16 17:37:33 fir-io1-s1 kernel: Lustre: Skipped 169 previous similar messages Feb 16 17:47:46 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 16 17:47:46 fir-io1-s1 kernel: Lustre: Skipped 174 previous similar messages Feb 16 17:57:48 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 861509dc-4523-55e8-f09a-16b6fca3f713 (at 10.9.107.51@o2ib4) Feb 16 17:57:48 fir-io1-s1 kernel: Lustre: Skipped 161 previous similar messages Feb 16 18:07:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to d6636cab-7aff-905c-7654-df6ea400308a (at 10.8.7.21@o2ib6) Feb 16 18:07:54 fir-io1-s1 kernel: Lustre: Skipped 183 previous similar messages Feb 16 18:18:27 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d6636cab-7aff-905c-7654-df6ea400308a (at 10.8.7.21@o2ib6) Feb 16 18:18:27 fir-io1-s1 kernel: Lustre: Skipped 181 previous similar messages Feb 16 18:28:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d0986368-6b42-a7a1-d471-712579f07716 (at 10.8.27.34@o2ib6) Feb 16 18:28:27 fir-io1-s1 kernel: Lustre: Skipped 181 previous similar messages Feb 16 18:38:36 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 5b896ca8-2947-8e31-025e-233ef4d66e00 (at 10.8.17.11@o2ib6) Feb 16 18:38:36 fir-io1-s1 kernel: Lustre: Skipped 268 previous similar messages Feb 16 18:48:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dbe81c3f-e038-b02e-a6dc-aaba56293b77 (at 10.8.2.19@o2ib6) Feb 16 18:48:37 fir-io1-s1 kernel: Lustre: Skipped 224 previous similar messages Feb 16 18:58:45 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to b9d5ee48-d92d-0b8f-b217-c073d8cf4946 (at 10.9.103.2@o2ib4) Feb 16 18:58:45 fir-io1-s1 kernel: Lustre: Skipped 159 previous similar messages Feb 16 19:08:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Feb 16 19:08:50 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 16 19:18:51 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1c47c083-1697-95bd-3469-0636ee21aa42 (at 10.8.2.32@o2ib6) Feb 16 19:18:51 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 16 19:28:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dcf54687-a6e1-03a6-825a-92830ad9b551 (at 10.8.7.31@o2ib6) Feb 16 19:28:55 fir-io1-s1 kernel: Lustre: Skipped 473 previous similar messages Feb 16 19:39:07 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 469c4428-ebc2-c887-175b-661c91237c7b (at 10.9.105.31@o2ib4) Feb 16 19:39:07 fir-io1-s1 kernel: Lustre: Skipped 850 previous similar messages Feb 16 19:49:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 16 19:49:09 fir-io1-s1 kernel: Lustre: Skipped 140 previous similar messages Feb 16 19:59:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 16 19:59:26 fir-io1-s1 kernel: Lustre: Skipped 99 previous similar messages Feb 16 20:10:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 00cd381d-5246-e5cd-af5e-792229d3fea2 (at 10.9.104.63@o2ib4) Feb 16 20:10:08 fir-io1-s1 kernel: Lustre: Skipped 99 previous similar messages Feb 16 20:20:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c3993171-24c0-89a4-7cb1-27b4ddbf15a6 (at 10.8.6.36@o2ib6) Feb 16 20:20:14 fir-io1-s1 kernel: Lustre: Skipped 99 previous similar messages Feb 16 20:30:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) Feb 16 20:30:15 fir-io1-s1 kernel: Lustre: Skipped 111 previous similar messages Feb 16 20:40:15 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 16 20:40:15 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 16 20:50:16 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.1.11@o2ib6) Feb 16 20:50:16 fir-io1-s1 kernel: Lustre: Skipped 191 previous similar messages Feb 16 21:00:42 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0e25277b-1ad0-ec5f-2777-56d7cafdcd31 (at 10.8.17.6@o2ib6) Feb 16 21:00:42 fir-io1-s1 kernel: Lustre: Skipped 174 previous similar messages Feb 16 21:11:03 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Feb 16 21:11:03 fir-io1-s1 kernel: Lustre: Skipped 260 previous similar messages Feb 16 21:20:47 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 374a74fb-0949-592c-d695-97a1fc0685d5 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f26a03c00, cur 1550380847 expire 1550380697 last 1550380620 Feb 16 21:20:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 21:21:06 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to c99c064f-83fa-682c-8def-f011ca1d2686 (at 10.9.113.12@o2ib4) Feb 16 21:21:06 fir-io1-s1 kernel: Lustre: Skipped 178 previous similar messages Feb 16 21:31:12 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 277b3821-f326-dcfd-7723-1e7e3e6174f1 (at 10.8.1.21@o2ib6) Feb 16 21:31:12 fir-io1-s1 kernel: Lustre: Skipped 122 previous similar messages Feb 16 21:41:12 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 17c20a65-2808-2c2b-e989-b585830f80fe (at 10.8.1.18@o2ib6) Feb 16 21:41:12 fir-io1-s1 kernel: Lustre: Skipped 298 previous similar messages Feb 16 21:51:13 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 16 21:51:13 fir-io1-s1 kernel: Lustre: Skipped 211 previous similar messages Feb 16 22:01:15 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d5a74680-e7af-ebfb-7dfd-72e2645d277b (at 10.9.101.51@o2ib4) Feb 16 22:01:15 fir-io1-s1 kernel: Lustre: Skipped 188 previous similar messages Feb 16 22:11:23 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to b419620b-9042-d31f-97b4-4ea59764e48d (at 10.8.24.21@o2ib6) Feb 16 22:11:23 fir-io1-s1 kernel: Lustre: Skipped 116 previous similar messages Feb 16 22:21:32 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 1127e5ab-6d36-a097-38c2-cf0f21f5ec17 (at 10.8.26.8@o2ib6) Feb 16 22:21:32 fir-io1-s1 kernel: Lustre: Skipped 199 previous similar messages Feb 16 22:31:35 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) Feb 16 22:31:35 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 16 22:41:51 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 16 22:41:51 fir-io1-s1 kernel: Lustre: Skipped 172 previous similar messages Feb 16 22:51:24 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3285910f-c7fd-2caf-6bff-9f21a0923c66 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767573000, cur 1550386284 expire 1550386134 last 1550386057 Feb 16 22:51:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 16 22:51:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) Feb 16 22:51:56 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 16 23:01:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 23:01:56 fir-io1-s1 kernel: Lustre: Skipped 159 previous similar messages Feb 16 23:11:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 23:11:56 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Feb 16 23:21:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 23:21:56 fir-io1-s1 kernel: Lustre: Skipped 160 previous similar messages Feb 16 23:31:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 23:31:56 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 16 23:41:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 23:41:56 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 16 23:51:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 16 23:51:56 fir-io1-s1 kernel: Lustre: Skipped 156 previous similar messages Feb 17 00:01:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 00:01:56 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 17 00:09:12 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1794292c-19e4-fd03-78b3-836bb9f7f6c0 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aaba800, cur 1550390952 expire 1550390802 last 1550390725 Feb 17 00:09:12 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 17 00:12:09 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.8.1.9@o2ib6) Feb 17 00:12:09 fir-io1-s1 kernel: Lustre: Skipped 134 previous similar messages Feb 17 00:22:13 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 66de0a73-cb41-c788-e30e-7505e7f80015 (at 10.9.106.41@o2ib4) Feb 17 00:22:13 fir-io1-s1 kernel: Lustre: Skipped 250 previous similar messages Feb 17 00:32:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 57269b50-0465-38e8-cbda-995a4a22296e (at 10.8.8.1@o2ib6) Feb 17 00:32:14 fir-io1-s1 kernel: Lustre: Skipped 197 previous similar messages Feb 17 00:42:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 7796eaa1-24c1-f1a6-996a-1af3c662e968 (at 10.8.7.2@o2ib6) Feb 17 00:42:26 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 17 00:52:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0edeac5b-ee1e-024f-de97-9e0fc3efb1af (at 10.8.6.2@o2ib6) Feb 17 00:52:50 fir-io1-s1 kernel: Lustre: Skipped 117 previous similar messages Feb 17 01:03:22 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 6da7219f-139c-94b7-44d5-3e81a85a248c (at 10.8.6.21@o2ib6) Feb 17 01:03:22 fir-io1-s1 kernel: Lustre: Skipped 130 previous similar messages Feb 17 01:13:23 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 43c491ee-b68b-5359-c63e-5195c978bbc4 (at 10.8.18.28@o2ib6) Feb 17 01:13:23 fir-io1-s1 kernel: Lustre: Skipped 174 previous similar messages Feb 17 01:23:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 17 01:23:25 fir-io1-s1 kernel: Lustre: Skipped 203 previous similar messages Feb 17 01:33:25 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ecc5b628-5efd-fbfd-7392-a1abe17de407 (at 10.9.101.36@o2ib4) Feb 17 01:33:25 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 17 01:43:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2e8b1a97-514f-63aa-1bc8-051eadecacf0 (at 10.9.112.9@o2ib4) Feb 17 01:43:33 fir-io1-s1 kernel: Lustre: Skipped 272 previous similar messages Feb 17 01:53:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.1.9@o2ib6) Feb 17 01:53:38 fir-io1-s1 kernel: Lustre: Skipped 204 previous similar messages Feb 17 02:03:38 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.1.11@o2ib6) Feb 17 02:03:38 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 17 02:07:56 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 351a9437-a846-568e-1f71-d0594aada5f6 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a6d800, cur 1550398076 expire 1550397926 last 1550397849 Feb 17 02:07:56 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 17 02:13:44 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Feb 17 02:13:44 fir-io1-s1 kernel: Lustre: Skipped 124 previous similar messages Feb 17 02:23:46 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to f74f6780-500b-a99c-769a-05932d2be074 (at 10.9.102.10@o2ib4) Feb 17 02:23:46 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 17 02:33:49 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2d8e1d81-8c01-f081-9434-1558c3a99426 (at 10.8.26.27@o2ib6) Feb 17 02:33:49 fir-io1-s1 kernel: Lustre: Skipped 96 previous similar messages Feb 17 02:41:03 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ce237f0f-3ab9-2f5f-e40e-1c9950d73c8d (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575e61a800, cur 1550400063 expire 1550399913 last 1550399836 Feb 17 02:41:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 02:43:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) Feb 17 02:43:49 fir-io1-s1 kernel: Lustre: Skipped 133 previous similar messages Feb 17 02:53:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 045f5984-f159-09cd-da98-0d87730fa119 (at 10.8.4.16@o2ib6) Feb 17 02:53:58 fir-io1-s1 kernel: Lustre: Skipped 103 previous similar messages Feb 17 03:04:29 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 17 03:04:29 fir-io1-s1 kernel: Lustre: Skipped 114 previous similar messages Feb 17 03:14:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b29100f0-7f9b-f4f6-2c85-7505f2641dbf (at 10.8.6.22@o2ib6) Feb 17 03:14:32 fir-io1-s1 kernel: Lustre: Skipped 92 previous similar messages Feb 17 03:24:36 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 43c491ee-b68b-5359-c63e-5195c978bbc4 (at 10.8.18.28@o2ib6) Feb 17 03:24:36 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Feb 17 03:34:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 179e0b48-b58d-d9b1-7e3f-f996ca06f525 (at 10.8.10.6@o2ib6) Feb 17 03:34:38 fir-io1-s1 kernel: Lustre: Skipped 119 previous similar messages Feb 17 03:44:49 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) Feb 17 03:44:49 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 17 03:54:51 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3364cbf9-a01c-3bb2-78b9-5e8955b36f20 (at 10.9.101.30@o2ib4) Feb 17 03:54:51 fir-io1-s1 kernel: Lustre: Skipped 141 previous similar messages Feb 17 04:04:55 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ed540f5a-5df6-5998-8f5c-40181564f690 (at 10.9.106.50@o2ib4) Feb 17 04:04:55 fir-io1-s1 kernel: Lustre: Skipped 101 previous similar messages Feb 17 04:14:59 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 17 04:14:59 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Feb 17 04:19:06 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client bd1394f9-d27f-aa21-e710-e264065e4c41 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885a800, cur 1550405946 expire 1550405796 last 1550405719 Feb 17 04:19:06 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 17 04:25:00 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 7b3b6f2b-764c-4642-1783-263f87e59249 (at 10.8.1.10@o2ib6) Feb 17 04:25:00 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Feb 17 04:35:01 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 10f5ca51-c1e4-ea2c-70bb-5e5b8eb0ed33 (at 10.9.115.7@o2ib4) Feb 17 04:35:01 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 17 04:45:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 5874979c-2d42-d4b8-b0f7-48cd970c494d (at 10.8.4.28@o2ib6) Feb 17 04:45:03 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 17 04:55:24 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3b95c04d-b4e9-7c45-bdb5-b89e27b35d9e (at 10.9.105.27@o2ib4) Feb 17 04:55:24 fir-io1-s1 kernel: Lustre: Skipped 180 previous similar messages Feb 17 05:05:42 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 824bd29d-65a6-aa3d-238a-0a9d670705d8 (at 10.9.101.35@o2ib4) Feb 17 05:05:42 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 17 05:15:47 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 17c20a65-2808-2c2b-e989-b585830f80fe (at 10.8.1.18@o2ib6) Feb 17 05:15:47 fir-io1-s1 kernel: Lustre: Skipped 161 previous similar messages Feb 17 05:18:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7f6c8cc5-34b6-e75b-f702-452235d4e072 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481e2c5400, cur 1550409510 expire 1550409360 last 1550409283 Feb 17 05:18:30 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 17 05:25:55 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fb0e351b-5ab8-b43d-813c-60db20cd78c1 (at 10.9.101.45@o2ib4) Feb 17 05:25:55 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Feb 17 05:35:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 50ea5c7c-97f1-e023-d007-18f98dd5bb05 (at 10.9.101.6@o2ib4) Feb 17 05:35:58 fir-io1-s1 kernel: Lustre: Skipped 73 previous similar messages Feb 17 05:46:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 17 05:46:26 fir-io1-s1 kernel: Lustre: Skipped 94 previous similar messages Feb 17 05:56:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 88ddc3f9-1f21-4f81-e1f1-3f396b007308 (at 10.9.101.54@o2ib4) Feb 17 05:56:33 fir-io1-s1 kernel: Lustre: Skipped 108 previous similar messages Feb 17 06:02:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2786a70a-33e3-89a1-2786-77636efb7775 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b88ec00, cur 1550412154 expire 1550412004 last 1550411927 Feb 17 06:02:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 06:06:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to baf8fd69-3c06-48ef-44c1-991a98b1784d (at 10.8.13.14@o2ib6) Feb 17 06:06:40 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Feb 17 06:16:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 06:16:57 fir-io1-s1 kernel: Lustre: Skipped 88 previous similar messages Feb 17 06:27:29 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c243879d-6590-e58d-10d6-105c5b7b4def (at 10.8.28.1@o2ib6) Feb 17 06:27:29 fir-io1-s1 kernel: Lustre: Skipped 99 previous similar messages Feb 17 06:37:45 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 219823d8-b457-54ef-a05b-a82af09f0709 (at 10.9.105.58@o2ib4) Feb 17 06:37:45 fir-io1-s1 kernel: Lustre: Skipped 73 previous similar messages Feb 17 06:48:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c95ae3a3-cc0b-6e03-007e-3f43096cb7c1 (at 10.9.101.21@o2ib4) Feb 17 06:48:03 fir-io1-s1 kernel: Lustre: Skipped 97 previous similar messages Feb 17 06:58:08 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2f9cee0a-253f-1406-d834-acd14e4c65fc (at 10.8.4.17@o2ib6) Feb 17 06:58:08 fir-io1-s1 kernel: Lustre: Skipped 74 previous similar messages Feb 17 06:58:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c0f5bbda-6bc9-f998-63c4-1a999fb1adbd (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15a6800, cur 1550415500 expire 1550415350 last 1550415273 Feb 17 06:58:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 07:08:13 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 17 07:08:13 fir-io1-s1 kernel: Lustre: Skipped 66 previous similar messages Feb 17 07:18:21 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 7b3b6f2b-764c-4642-1783-263f87e59249 (at 10.8.1.10@o2ib6) Feb 17 07:18:21 fir-io1-s1 kernel: Lustre: Skipped 123 previous similar messages Feb 17 07:28:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 78f38211-beb8-aca3-b985-9281f7d5f62c (at 10.8.29.8@o2ib6) Feb 17 07:28:30 fir-io1-s1 kernel: Lustre: Skipped 78 previous similar messages Feb 17 07:38:31 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Feb 17 07:38:31 fir-io1-s1 kernel: Lustre: Skipped 136 previous similar messages Feb 17 07:38:50 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 804c2951-114b-c71b-a023-84555e52040a (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f4a6a2800, cur 1550417930 expire 1550417780 last 1550417703 Feb 17 07:38:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 07:48:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 17c20a65-2808-2c2b-e989-b585830f80fe (at 10.8.1.18@o2ib6) Feb 17 07:48:55 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 17 07:58:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.1.11@o2ib6) Feb 17 07:58:56 fir-io1-s1 kernel: Lustre: Skipped 123 previous similar messages Feb 17 08:08:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7493901d-f880-8ef7-649f-e7b75304612f (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4e9000, cur 1550419735 expire 1550419585 last 1550419508 Feb 17 08:08:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 08:09:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ddfc24dc-ab35-b5d1-5ce6-6e97aa901210 (at 10.9.107.16@o2ib4) Feb 17 08:09:09 fir-io1-s1 kernel: Lustre: Skipped 84 previous similar messages Feb 17 08:19:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f97f4802-8c0b-e9cd-e8d7-93692decf22a (at 10.9.102.8@o2ib4) Feb 17 08:19:37 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Feb 17 08:21:02 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c9a5d24c-181b-f573-d1b1-75c5f22eca41 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c21c00, cur 1550420462 expire 1550420312 last 1550420235 Feb 17 08:21:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 08:29:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.4.32@o2ib6) Feb 17 08:29:38 fir-io1-s1 kernel: Lustre: Skipped 156 previous similar messages Feb 17 08:39:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to baf8fd69-3c06-48ef-44c1-991a98b1784d (at 10.8.13.14@o2ib6) Feb 17 08:39:45 fir-io1-s1 kernel: Lustre: Skipped 106 previous similar messages Feb 17 08:49:55 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to f88d3e4f-b8ad-7e3f-e052-b857e571de2a (at 10.9.107.13@o2ib4) Feb 17 08:49:55 fir-io1-s1 kernel: Lustre: Skipped 120 previous similar messages Feb 17 08:55:16 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b5393ed5-c6b6-9c08-727c-3321b78b5712 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833d7f400, cur 1550422516 expire 1550422366 last 1550422289 Feb 17 08:55:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 08:59:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 56f157dc-9f05-a349-697a-ac16ba31313e (at 10.9.104.59@o2ib4) Feb 17 08:59:56 fir-io1-s1 kernel: Lustre: Skipped 54 previous similar messages Feb 17 09:10:15 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9f1de1b3-7e56-c812-677b-e9a4e7cdbca5 (at 10.8.8.5@o2ib6) Feb 17 09:10:15 fir-io1-s1 kernel: Lustre: Skipped 95 previous similar messages Feb 17 09:20:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 98b7035a-35b8-cd21-9b10-3e7f2a49b7a7 (at 10.8.13.9@o2ib6) Feb 17 09:20:16 fir-io1-s1 kernel: Lustre: Skipped 117 previous similar messages Feb 17 09:22:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2173002e-5d34-631d-6a59-e4e2e74acf50 (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762a5fc00, cur 1550424168 expire 1550424018 last 1550423941 Feb 17 09:22:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 09:30:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 17 09:30:32 fir-io1-s1 kernel: Lustre: Skipped 84 previous similar messages Feb 17 09:40:37 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 32ea102c-c8a6-9ae9-e0f7-d3fc0379beb1 (at 10.8.2.23@o2ib6) Feb 17 09:40:37 fir-io1-s1 kernel: Lustre: Skipped 91 previous similar messages Feb 17 09:50:59 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 2b067c7d-d2f3-4c8d-f9fb-acfb69dda9ca (at 10.9.104.24@o2ib4) Feb 17 09:50:59 fir-io1-s1 kernel: Lustre: Skipped 76 previous similar messages Feb 17 10:01:01 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 67f5c0a5-d6e0-715a-6a48-d2b2401623ab (at 10.9.101.46@o2ib4) Feb 17 10:01:01 fir-io1-s1 kernel: Lustre: Skipped 140 previous similar messages Feb 17 10:11:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9089d3ee-aece-a348-093f-f0c793badf5d (at 10.8.10.1@o2ib6) Feb 17 10:11:07 fir-io1-s1 kernel: Lustre: Skipped 118 previous similar messages Feb 17 10:21:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.8.21.8@o2ib6) Feb 17 10:21:47 fir-io1-s1 kernel: Lustre: Skipped 122 previous similar messages Feb 17 10:31:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 23d2bcaf-f181-5f6e-6636-b07b46e525e0 (at 10.8.3.3@o2ib6) Feb 17 10:31:55 fir-io1-s1 kernel: Lustre: Skipped 127 previous similar messages Feb 17 10:35:03 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 30c339af-3f0c-1415-a6e6-ac884dd25bc5 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984803158c00, cur 1550428503 expire 1550428353 last 1550428276 Feb 17 10:35:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 10:41:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 10:41:56 fir-io1-s1 kernel: Lustre: Skipped 181 previous similar messages Feb 17 10:51:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 10:51:56 fir-io1-s1 kernel: Lustre: Skipped 75 previous similar messages Feb 17 11:01:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 11:01:56 fir-io1-s1 kernel: Lustre: Skipped 239 previous similar messages Feb 17 11:11:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 11:11:56 fir-io1-s1 kernel: Lustre: Skipped 141 previous similar messages Feb 17 11:22:20 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to cc107cab-3544-aeaa-6b27-e00a056fcf80 (at 10.8.1.26@o2ib6) Feb 17 11:22:20 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Feb 17 11:22:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 93980c64-27a9-93b5-96b8-9ea297f029ce (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583f51e400, cur 1550431377 expire 1550431227 last 1550431150 Feb 17 11:22:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 11:32:28 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 2a27f76c-78b6-7e1a-cff3-64717b5ae1ff (at 10.9.106.59@o2ib4) Feb 17 11:32:28 fir-io1-s1 kernel: Lustre: Skipped 86 previous similar messages Feb 17 11:39:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client dcabc36f-16f5-3663-ba70-ddd7266e18e9 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aab3800, cur 1550432346 expire 1550432196 last 1550432119 Feb 17 11:39:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 11:42:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fca0f45e-b434-84ac-1ed2-ce278e523445 (at 10.8.8.15@o2ib6) Feb 17 11:42:39 fir-io1-s1 kernel: Lustre: Skipped 101 previous similar messages Feb 17 11:52:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 5b896ca8-2947-8e31-025e-233ef4d66e00 (at 10.8.17.11@o2ib6) Feb 17 11:52:56 fir-io1-s1 kernel: Lustre: Skipped 223 previous similar messages Feb 17 12:03:05 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e43da944-d239-923f-8f68-10646264727b (at 10.8.21.20@o2ib6) Feb 17 12:03:05 fir-io1-s1 kernel: Lustre: Skipped 154 previous similar messages Feb 17 12:13:38 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.102.48@o2ib4) Feb 17 12:13:38 fir-io1-s1 kernel: Lustre: Skipped 103 previous similar messages Feb 17 12:23:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6db7d291-7912-a662-9ec2-f76af6a57200 (at 10.8.1.7@o2ib6) Feb 17 12:23:52 fir-io1-s1 kernel: Lustre: Skipped 103 previous similar messages Feb 17 12:34:19 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 749ba667-6b1b-7176-05ce-9fcec3751e62 (at 10.9.108.18@o2ib4) Feb 17 12:34:19 fir-io1-s1 kernel: Lustre: Skipped 128 previous similar messages Feb 17 12:44:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) Feb 17 12:44:23 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Feb 17 12:54:31 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 16fe1f06-91d4-6364-b5d9-1d6caad6f915 (at 10.8.22.22@o2ib6) Feb 17 12:54:31 fir-io1-s1 kernel: Lustre: Skipped 117 previous similar messages Feb 17 13:04:34 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 926fa24d-f3ab-7ad6-dbc7-f8a15bdf8c5a (at 10.8.19.8@o2ib6) Feb 17 13:04:34 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Feb 17 13:15:02 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e9560223-f857-8af8-8e66-18924c1e4b0e (at 10.8.3.22@o2ib6) Feb 17 13:15:02 fir-io1-s1 kernel: Lustre: Skipped 94 previous similar messages Feb 17 13:25:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ac16def5-1a59-80e5-2e16-45b58fcd0330 (at 10.8.2.8@o2ib6) Feb 17 13:25:27 fir-io1-s1 kernel: Lustre: Skipped 108 previous similar messages Feb 17 13:29:01 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4d616aaa-efcd-0230-2a4d-16090f222fae (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b99000, cur 1550438941 expire 1550438791 last 1550438714 Feb 17 13:29:01 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 17 13:35:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15339979-51e5-e16d-f976-ff72d24bd14f (at 10.8.9.10@o2ib6) Feb 17 13:35:32 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 17 13:43:52 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 819a23c7-dd1e-3003-8f04-45355c7307ea (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868053ea400, cur 1550439832 expire 1550439682 last 1550439605 Feb 17 13:43:52 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 17 13:45:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 17 13:45:58 fir-io1-s1 kernel: Lustre: Skipped 122 previous similar messages Feb 17 13:56:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6db7d291-7912-a662-9ec2-f76af6a57200 (at 10.8.1.7@o2ib6) Feb 17 13:56:13 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 17 14:06:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d7f0cefd-f5dc-ae79-9fbe-8c42036c5092 (at 10.9.105.21@o2ib4) Feb 17 14:06:27 fir-io1-s1 kernel: Lustre: Skipped 134 previous similar messages Feb 17 14:16:40 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f74f6780-500b-a99c-769a-05932d2be074 (at 10.9.102.10@o2ib4) Feb 17 14:16:40 fir-io1-s1 kernel: Lustre: Skipped 143 previous similar messages Feb 17 14:26:49 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e43da944-d239-923f-8f68-10646264727b (at 10.8.21.20@o2ib6) Feb 17 14:26:49 fir-io1-s1 kernel: Lustre: Skipped 120 previous similar messages Feb 17 14:36:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 14:36:56 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Feb 17 14:46:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 14:46:56 fir-io1-s1 kernel: Lustre: Skipped 232 previous similar messages Feb 17 14:56:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 14:56:56 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 17 15:06:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 15:06:56 fir-io1-s1 kernel: Lustre: Skipped 226 previous similar messages Feb 17 15:16:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 15:16:56 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 17 15:22:27 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 78c20038-e973-23a3-6231-43b08f647c2f (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f59800, cur 1550445747 expire 1550445597 last 1550445520 Feb 17 15:22:27 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 17 15:26:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 15:26:56 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 17 15:36:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 15:36:56 fir-io1-s1 kernel: Lustre: Skipped 172 previous similar messages Feb 17 15:46:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 15:46:56 fir-io1-s1 kernel: Lustre: Skipped 135 previous similar messages Feb 17 15:56:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 15:56:56 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Feb 17 16:07:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a4e427fa-d968-911a-150f-37f69bc4903c (at 10.9.106.3@o2ib4) Feb 17 16:07:08 fir-io1-s1 kernel: Lustre: Skipped 136 previous similar messages Feb 17 16:17:12 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bf617e6a-e2c4-2972-fed8-58b2cb638da2 (at 10.9.101.25@o2ib4) Feb 17 16:17:12 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Feb 17 16:27:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4e48592f-b97d-5c93-9da4-86c872d7a486 (at 10.9.107.43@o2ib4) Feb 17 16:27:32 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 17 16:37:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d5a74680-e7af-ebfb-7dfd-72e2645d277b (at 10.9.101.51@o2ib4) Feb 17 16:37:58 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 17 16:48:03 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 449ab5a0-1671-c50c-bbbe-21371abb55d6 (at 10.9.107.56@o2ib4) Feb 17 16:48:03 fir-io1-s1 kernel: Lustre: Skipped 202 previous similar messages Feb 17 16:49:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c6cf861a-9b92-6980-4a2a-c5c106c1c11c (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801a9400, cur 1550450984 expire 1550450834 last 1550450757 Feb 17 16:49:44 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 17 16:50:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c6cf861a-9b92-6980-4a2a-c5c106c1c11c (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2fc00, cur 1550451005 expire 1550450855 last 1550450778 Feb 17 16:50:05 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 17 16:58:07 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.106.61@o2ib4) Feb 17 16:58:07 fir-io1-s1 kernel: Lustre: Skipped 261 previous similar messages Feb 17 17:05:16 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 58405f8f-c50c-875f-7d6d-79d34c699323 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867851e1000, cur 1550451916 expire 1550451766 last 1550451689 Feb 17 17:08:08 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8f440af3-cd92-379d-a078-f053f705469f (at 10.9.106.58@o2ib4) Feb 17 17:08:08 fir-io1-s1 kernel: Lustre: Skipped 265 previous similar messages Feb 17 17:18:35 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d165ae17-8944-f365-7713-d429fbe1daab (at 10.8.1.23@o2ib6) Feb 17 17:18:35 fir-io1-s1 kernel: Lustre: Skipped 589 previous similar messages Feb 17 17:19:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6ebac30e-9418-b7ca-ff7a-6873c1a5400d (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780d5a400, cur 1550452794 expire 1550452644 last 1550452567 Feb 17 17:19:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 17:28:45 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d3157baf-de90-86ed-7c87-5f5f5c909a71 (at 10.9.105.15@o2ib4) Feb 17 17:28:45 fir-io1-s1 kernel: Lustre: Skipped 252 previous similar messages Feb 17 17:39:01 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b3ef6690-dd23-2eef-0dcf-441d88950a4a (at 10.8.28.2@o2ib6) Feb 17 17:39:01 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Feb 17 17:39:49 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 366b13c0-553b-6c3e-7e48-48921d2cd330 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2a800, cur 1550453989 expire 1550453839 last 1550453762 Feb 17 17:39:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 17:49:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aa32c16f-39e8-d913-ba83-00090f0ccc84 (at 10.8.23.8@o2ib6) Feb 17 17:49:02 fir-io1-s1 kernel: Lustre: Skipped 218 previous similar messages Feb 17 17:59:16 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d3b133e8-4ec8-ebe3-7fc5-79aa16e59c0b (at 10.9.106.35@o2ib4) Feb 17 17:59:16 fir-io1-s1 kernel: Lustre: Skipped 233 previous similar messages Feb 17 18:09:19 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 899cd11a-4996-4751-56a3-fe3c17225e3d (at 10.9.105.29@o2ib4) Feb 17 18:09:19 fir-io1-s1 kernel: Lustre: Skipped 123 previous similar messages Feb 17 18:19:35 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8a305ec8-58cd-38d7-7085-a98f5d22aa5b (at 10.9.107.47@o2ib4) Feb 17 18:19:35 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Feb 17 18:30:25 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.106.61@o2ib4) Feb 17 18:30:25 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 17 18:40:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b3c5bc63-aa36-b5eb-1ad9-6c8f48fdb4c3 (at 10.9.106.68@o2ib4) Feb 17 18:40:44 fir-io1-s1 kernel: Lustre: Skipped 128 previous similar messages Feb 17 18:50:45 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 6a1a7e31-f24a-25a3-4e1c-41f3ed10a783 (at 10.9.102.36@o2ib4) Feb 17 18:50:45 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 17 19:00:50 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6f43d80a-d4ed-b27e-b29c-0d22cee6d831 (at 10.9.105.8@o2ib4) Feb 17 19:00:50 fir-io1-s1 kernel: Lustre: Skipped 256 previous similar messages Feb 17 19:04:27 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5e1f6b60-5ab3-b9bf-ff93-17d068112255 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575de9c000, cur 1550459067 expire 1550458917 last 1550458840 Feb 17 19:04:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 19:04:44 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5e1f6b60-5ab3-b9bf-ff93-17d068112255 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800e4a000, cur 1550459084 expire 1550458934 last 1550458857 Feb 17 19:04:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 17 19:11:11 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a16eacd1-a921-5af0-cd75-234f1ff94647 (at 10.8.23.9@o2ib6) Feb 17 19:11:11 fir-io1-s1 kernel: Lustre: Skipped 224 previous similar messages Feb 17 19:21:36 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 156070c6-6b1a-c523-d65c-fc06e69c00b3 (at 10.9.103.39@o2ib4) Feb 17 19:21:36 fir-io1-s1 kernel: Lustre: Skipped 254 previous similar messages Feb 17 19:30:50 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 443ea72c-1898-1eea-08e2-ef32dd8ab833 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8ab800, cur 1550460650 expire 1550460500 last 1550460423 Feb 17 19:30:50 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 17 19:31:48 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 866bfb2e-3715-8e7b-5fdd-befaff184f50 (at 10.8.17.19@o2ib6) Feb 17 19:31:48 fir-io1-s1 kernel: Lustre: Skipped 127 previous similar messages Feb 17 19:41:54 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 4b161527-b7d4-cb96-22b6-3891827325ad (at 10.9.112.2@o2ib4) Feb 17 19:41:54 fir-io1-s1 kernel: Lustre: Skipped 154 previous similar messages Feb 17 19:44:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 85a4037c-fded-7c70-124c-a4b6eafb9caf (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986781f73800, cur 1550461482 expire 1550461332 last 1550461255 Feb 17 19:44:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 19:51:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 19:51:56 fir-io1-s1 kernel: Lustre: Skipped 196 previous similar messages Feb 17 20:01:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 20:01:56 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 17 20:11:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 20:11:56 fir-io1-s1 kernel: Lustre: Skipped 205 previous similar messages Feb 17 20:21:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 20:21:56 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 17 20:31:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 20:31:56 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 17 20:41:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 20:41:56 fir-io1-s1 kernel: Lustre: Skipped 122 previous similar messages Feb 17 20:51:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 17 20:51:56 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 17 20:55:03 fir-io1-s1 kernel: LustreError: 96288:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.26.33@o2ib6) returned error from blocking AST (req@ffff984d6f502700 x1624934231670768 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff983fa11de0c0/0x49e185e98db01880 lrc: 4/0,0 mode: PR/PR res: [0x5c0000401:0x38696:0x0].0x0 rrc: 121 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400000020 nid: 10.8.26.33@o2ib6 remote: 0xa2e5105b6d30e6d2 expref: 369 pid: 96763 timeout: 813604 lvb_type: 1 Feb 17 20:55:03 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.26.33@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Feb 17 20:55:03 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.26.33@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff983fa11de0c0/0x49e185e98db01880 lrc: 3/0,0 mode: PR/PR res: [0x5c0000401:0x38696:0x0].0x0 rrc: 118 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400000020 nid: 10.8.26.33@o2ib6 remote: 0xa2e5105b6d30e6d2 expref: 370 pid: 96763 timeout: 0 lvb_type: 1 Feb 17 20:55:04 fir-io1-s1 kernel: LustreError: 94513:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.26.33@o2ib6) returned error from blocking AST (req@ffff9875c4a2bf00 x1624934231682608 status -107 rc -107), evict it ns: filter-fir-OST0004_UUID lock: ffff983fa11d8d80/0x49e185e98db0188e lrc: 4/0,0 mode: PR/PR res: [0x8c0000400:0x38584:0x0].0x0 rrc: 120 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.26.33@o2ib6 remote: 0xa2e5105b6d30e742 expref: 367 pid: 96763 timeout: 813604 lvb_type: 1 Feb 17 20:55:04 fir-io1-s1 kernel: LustreError: 94513:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Feb 17 20:55:04 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.26.33@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Feb 17 20:55:04 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 17 20:55:04 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.26.33@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff9867eb142ac0/0x49e185e98db047ff lrc: 3/0,0 mode: PR/PR res: [0xc40000400:0x38500:0x0].0x0 rrc: 132 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.26.33@o2ib6 remote: 0xa2e5105b6d32635d expref: 357 pid: 82278 timeout: 0 lvb_type: 1 Feb 17 20:55:04 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Feb 17 21:02:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 467e6b58-056a-e4b1-944f-400a71d631aa (at 10.8.16.6@o2ib6) Feb 17 21:02:09 fir-io1-s1 kernel: Lustre: Skipped 206 previous similar messages Feb 17 21:12:12 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 71eebf52-b0fc-514f-7aaf-aca66e4f2af1 (at 10.8.27.27@o2ib6) Feb 17 21:12:12 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 17 21:22:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f03c187f-2a55-47d4-d485-c19c17624703 (at 10.9.107.45@o2ib4) Feb 17 21:22:28 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Feb 17 21:32:28 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c950625e-5b2e-4c61-ec5e-db6c5bcc2bf4 (at 10.9.102.23@o2ib4) Feb 17 21:32:28 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Feb 17 21:42:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 874d84ab-2918-f27e-a1fe-cdc3435eb5ad (at 10.8.2.18@o2ib6) Feb 17 21:42:29 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 17 21:53:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 58887631-07e5-7b7d-4bf3-4fd4db49c156 (at 10.9.103.44@o2ib4) Feb 17 21:53:15 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 17 22:03:32 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9e7dc1a5-746c-8e56-5ad9-e239237ff7d7 (at 10.8.24.22@o2ib6) Feb 17 22:03:32 fir-io1-s1 kernel: Lustre: Skipped 129 previous similar messages Feb 17 22:07:12 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4f2cebb1-8e77-e8c0-5a30-fd6206287ad3 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848807f3000, cur 1550470032 expire 1550469882 last 1550469805 Feb 17 22:07:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 22:13:40 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fc0522a2-e86f-7812-92f9-18c8c5b33bdc (at 10.9.105.45@o2ib4) Feb 17 22:13:40 fir-io1-s1 kernel: Lustre: Skipped 212 previous similar messages Feb 17 22:23:41 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3512bfd0-06dd-989b-6d83-75d517d82937 (at 10.9.105.6@o2ib4) Feb 17 22:23:41 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Feb 17 22:34:35 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.27.30@o2ib6) Feb 17 22:34:35 fir-io1-s1 kernel: Lustre: Skipped 118 previous similar messages Feb 17 22:45:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to e18ec55f-27df-5b55-2c1d-0fb1ae5cad9b (at 10.8.27.28@o2ib6) Feb 17 22:45:18 fir-io1-s1 kernel: Lustre: Skipped 99 previous similar messages Feb 17 22:55:38 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 255adf8b-e7cc-8914-212c-11fb3c6a4274 (at 10.9.105.35@o2ib4) Feb 17 22:55:38 fir-io1-s1 kernel: Lustre: Skipped 143 previous similar messages Feb 17 23:04:45 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1b23bfbd-6470-7e89-e16f-5dac5d9bd182 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986816f3c400, cur 1550473485 expire 1550473335 last 1550473258 Feb 17 23:04:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 17 23:05:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ea7067de-16a0-5aef-a5e8-0c3de073c862 (at 10.9.106.72@o2ib4) Feb 17 23:05:42 fir-io1-s1 kernel: Lustre: Skipped 140 previous similar messages Feb 17 23:15:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 17 23:15:58 fir-io1-s1 kernel: Lustre: Skipped 141 previous similar messages Feb 17 23:26:02 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to e3025880-856b-b6ea-a1a7-a7e183e1dd60 (at 10.8.8.22@o2ib6) Feb 17 23:26:02 fir-io1-s1 kernel: Lustre: Skipped 254 previous similar messages Feb 17 23:36:04 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to be5b5c3f-91d7-81de-8f26-0a39412de9ac (at 10.9.108.1@o2ib4) Feb 17 23:36:04 fir-io1-s1 kernel: Lustre: Skipped 227 previous similar messages Feb 17 23:46:07 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 34fa37c4-98f8-0ae7-6c7a-1133b926560e (at 10.9.101.19@o2ib4) Feb 17 23:46:07 fir-io1-s1 kernel: Lustre: Skipped 303 previous similar messages Feb 17 23:56:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 4199656a-fc55-4ace-07e9-a689e8e8d80b (at 10.8.10.7@o2ib6) Feb 17 23:56:23 fir-io1-s1 kernel: Lustre: Skipped 253 previous similar messages Feb 18 00:06:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client e9f12f7a-7484-8a2d-9e0d-f938935b66b1 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480052a000, cur 1550477172 expire 1550477022 last 1550476945 Feb 18 00:06:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 00:06:25 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 42c94472-12c2-a403-5462-682da14baa8e (at 10.9.107.54@o2ib4) Feb 18 00:06:25 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Feb 18 00:16:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5d333fe8-2d2e-778b-34c9-702cc9f2963f (at 10.9.106.57@o2ib4) Feb 18 00:16:33 fir-io1-s1 kernel: Lustre: Skipped 151 previous similar messages Feb 18 00:26:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 00:26:56 fir-io1-s1 kernel: Lustre: Skipped 270 previous similar messages Feb 18 00:36:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 00:36:56 fir-io1-s1 kernel: Lustre: Skipped 86 previous similar messages Feb 18 00:41:08 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 06d02912-21c3-0083-002c-079c99b7291f (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985769984c00, cur 1550479268 expire 1550479118 last 1550479041 Feb 18 00:41:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 00:41:09 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 06d02912-21c3-0083-002c-079c99b7291f (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801545000, cur 1550479269 expire 1550479119 last 1550479042 Feb 18 00:41:15 fir-io1-s1 kernel: Lustre: 96252:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550479268/real 1550479268] req@ffff984302e80c00 x1624934297322816/t0(0) o104->fir-OST000a@10.8.1.29@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550479275 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 18 00:41:15 fir-io1-s1 kernel: Lustre: 96252:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 112 previous similar messages Feb 18 00:41:17 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 06d02912-21c3-0083-002c-079c99b7291f (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c85a800, cur 1550479277 expire 1550479127 last 1550479050 Feb 18 00:46:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 00:46:57 fir-io1-s1 kernel: Lustre: Skipped 176 previous similar messages Feb 18 00:54:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 44db3f97-0cb6-b10b-5a91-12663fdfbdcc (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867f8209000, cur 1550480088 expire 1550479938 last 1550479861 Feb 18 00:54:48 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 00:54:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 44db3f97-0cb6-b10b-5a91-12663fdfbdcc (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b9e3000, cur 1550480096 expire 1550479946 last 1550479869 Feb 18 00:54:56 fir-io1-s1 kernel: LustreError: 96514:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff985ca30ec500 x1624934303603840/t0(0) o104->fir-OST000a@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 18 00:54:56 fir-io1-s1 kernel: LustreError: 96514:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Feb 18 00:54:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 44db3f97-0cb6-b10b-5a91-12663fdfbdcc (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b9e5400, cur 1550480097 expire 1550479947 last 1550479870 Feb 18 00:54:57 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 00:56:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 866bfb2e-3715-8e7b-5fdd-befaff184f50 (at 10.8.17.19@o2ib6) Feb 18 00:56:57 fir-io1-s1 kernel: Lustre: Skipped 148 previous similar messages Feb 18 01:07:10 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8f86419d-3c7d-f8e0-fb5d-facc0f493f73 (at 10.8.27.31@o2ib6) Feb 18 01:07:10 fir-io1-s1 kernel: Lustre: Skipped 182 previous similar messages Feb 18 01:17:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7a4f2fb-7087-2da1-fe97-a6b3f3961662 (at 10.9.103.38@o2ib4) Feb 18 01:17:42 fir-io1-s1 kernel: Lustre: Skipped 120 previous similar messages Feb 18 01:28:04 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.8.9@o2ib6) Feb 18 01:28:04 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 18 01:38:13 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.8.2.22@o2ib6) Feb 18 01:38:13 fir-io1-s1 kernel: Lustre: Skipped 180 previous similar messages Feb 18 01:39:37 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550482770/real 1550482770] req@ffff9851e2f48600 x1624934318986128/t0(0) o104->fir-OST0008@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550482777 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 18 01:39:44 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550482777/real 1550482777] req@ffff9851e2f48600 x1624934318986128/t0(0) o104->fir-OST0008@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550482784 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 01:39:51 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550482784/real 1550482784] req@ffff9851e2f48600 x1624934318986128/t0(0) o104->fir-OST0008@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550482791 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 01:40:05 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550482798/real 1550482798] req@ffff9851e2f48600 x1624934318986128/t0(0) o104->fir-OST0008@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550482805 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 01:40:05 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 18 01:40:26 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550482819/real 1550482819] req@ffff9851e2f48600 x1624934318986128/t0(0) o104->fir-OST0008@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550482826 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 01:40:26 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 18 01:41:01 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550482854/real 1550482854] req@ffff9851e2f48600 x1624934318986128/t0(0) o104->fir-OST0008@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550482861 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 01:41:01 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 18 01:41:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 73cb712d-0461-b725-a230-b3e96b86f861 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868fd800, cur 1550482875 expire 1550482725 last 1550482648 Feb 18 01:41:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 73cb712d-0461-b725-a230-b3e96b86f861 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480f61d400, cur 1550482887 expire 1550482737 last 1550482660 Feb 18 01:41:27 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 18 01:48:22 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 50f28a0e-eb03-3ed4-df6b-96db06d3f42b (at 10.9.107.34@o2ib4) Feb 18 01:48:22 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Feb 18 01:58:36 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.9.102.48@o2ib4) Feb 18 01:58:36 fir-io1-s1 kernel: Lustre: Skipped 183 previous similar messages Feb 18 02:08:37 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to cf5e8e7d-f901-3743-4b81-bf1f286015d4 (at 10.9.107.44@o2ib4) Feb 18 02:08:37 fir-io1-s1 kernel: Lustre: Skipped 202 previous similar messages Feb 18 02:18:43 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 91ff7dbd-cf53-2593-4756-f4e069057d0e (at 10.9.105.72@o2ib4) Feb 18 02:18:43 fir-io1-s1 kernel: Lustre: Skipped 214 previous similar messages Feb 18 02:18:49 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550485122/real 1550485122] req@ffff984803b6bf00 x1624934331595376/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550485129 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 18 02:18:49 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 18 02:19:03 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550485136/real 1550485136] req@ffff984803b6bf00 x1624934331595376/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550485143 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 02:19:03 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 18 02:19:24 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550485157/real 1550485157] req@ffff984803b6bf00 x1624934331595376/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550485164 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 02:19:24 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 18 02:19:59 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550485192/real 1550485192] req@ffff984803b6bf00 x1624934331595376/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550485199 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 02:19:59 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 18 02:21:09 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550485262/real 1550485262] req@ffff984803b6bf00 x1624934331595376/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550485269 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 02:21:09 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 18 02:21:16 fir-io1-s1 kernel: LustreError: 96332:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.13.14@o2ib6) failed to reply to blocking AST (req@ffff984803b6bf00 x1624934331595376 status 0 rc -110), evict it ns: filter-fir-OST0004_UUID lock: ffff98574449a880/0x49e185e9908dfb6a lrc: 4/0,0 mode: PR/PR res: [0x8c0000400:0x38e85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.13.14@o2ib6 remote: 0x8e03aeded37a5c6f expref: 342 pid: 96524 timeout: 833170 lvb_type: 1 Feb 18 02:21:16 fir-io1-s1 kernel: LustreError: 96332:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Feb 18 02:21:16 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.13.14@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 18 02:21:16 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 18 02:21:16 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.13.14@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff98574449a880/0x49e185e9908dfb6a lrc: 3/0,0 mode: PR/PR res: [0x8c0000400:0x38e85:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.13.14@o2ib6 remote: 0x8e03aeded37a5c6f expref: 343 pid: 96524 timeout: 0 lvb_type: 1 Feb 18 02:21:16 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Feb 18 02:21:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 05ff7be8-35d7-c62c-efec-7141c5a32ead (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98707f69c000, cur 1550485299 expire 1550485149 last 1550485072 Feb 18 02:21:39 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 18 02:21:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 05ff7be8-35d7-c62c-efec-7141c5a32ead (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f4dc00, cur 1550485318 expire 1550485168 last 1550485091 Feb 18 02:21:58 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 18 02:29:05 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 74f9852d-189b-8596-eb4f-bcf617e42f7c (at 10.8.7.22@o2ib6) Feb 18 02:29:05 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 18 02:39:13 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.113.1@o2ib4) Feb 18 02:39:13 fir-io1-s1 kernel: Lustre: Skipped 111 previous similar messages Feb 18 02:49:15 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 49507785-4778-3df3-6428-f7e53034ffec (at 10.9.107.32@o2ib4) Feb 18 02:49:15 fir-io1-s1 kernel: Lustre: Skipped 96 previous similar messages Feb 18 02:53:45 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550487217/real 1550487217] req@ffff984d3059f800 x1624934341537184/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550487224 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 18 02:53:45 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 18 02:54:06 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550487239/real 1550487239] req@ffff984d3059f800 x1624934341537184/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550487246 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 02:54:06 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 18 02:54:41 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550487274/real 1550487274] req@ffff984d3059f800 x1624934341537184/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550487281 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 02:54:41 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 18 02:55:51 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550487344/real 1550487344] req@ffff984d3059f800 x1624934341537184/t0(0) o104->fir-OST0004@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550487351 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 18 02:55:51 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 18 02:56:12 fir-io1-s1 kernel: LustreError: 96583:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.13.14@o2ib6) failed to reply to blocking AST (req@ffff984d3059f800 x1624934341537184 status 0 rc -110), evict it ns: filter-fir-OST0004_UUID lock: ffff985fdc8f0000/0x49e185e990f1fb9e lrc: 4/0,0 mode: PR/PR res: [0x8c0000400:0x38cec:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.13.14@o2ib6 remote: 0x7cf7048c5415c954 expref: 455 pid: 96945 timeout: 835265 lvb_type: 1 Feb 18 02:56:12 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.13.14@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 18 02:56:12 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.8.13.14@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff985fdc8f0000/0x49e185e990f1fb9e lrc: 3/0,0 mode: PR/PR res: [0x8c0000400:0x38cec:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.13.14@o2ib6 remote: 0x7cf7048c5415c954 expref: 456 pid: 96945 timeout: 0 lvb_type: 1 Feb 18 02:56:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a1056e05-3e2d-082b-64c1-5a268e5e474c (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768558800, cur 1550487408 expire 1550487258 last 1550487181 Feb 18 02:56:48 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 18 02:56:48 fir-io1-s1 kernel: LustreError: 96275:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9862976e8000 x1624934342298368/t0(0) o104->fir-OST0000@10.8.13.14@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 18 02:59:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 39eb100c-0d25-d4de-9dbe-7a71ed238778 (at 10.8.2.28@o2ib6) Feb 18 02:59:16 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 18 03:09:23 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6f29d7ca-d9bc-eef1-1913-bbb7c0bca1a0 (at 10.8.1.5@o2ib6) Feb 18 03:09:23 fir-io1-s1 kernel: Lustre: Skipped 181 previous similar messages Feb 18 03:19:42 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 866bfb2e-3715-8e7b-5fdd-befaff184f50 (at 10.8.17.19@o2ib6) Feb 18 03:19:42 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Feb 18 03:30:06 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.9.106.61@o2ib4) Feb 18 03:30:06 fir-io1-s1 kernel: Lustre: Skipped 159 previous similar messages Feb 18 03:40:19 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 013cf202-b3d4-6a22-f4d9-6c984ce87e6f (at 10.8.7.10@o2ib6) Feb 18 03:40:19 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 18 03:50:26 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 8c0347c3-135a-e940-667b-27edd6a0ad7f (at 10.8.20.6@o2ib6) Feb 18 03:50:26 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 18 04:00:33 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9f1de1b3-7e56-c812-677b-e9a4e7cdbca5 (at 10.8.8.5@o2ib6) Feb 18 04:00:33 fir-io1-s1 kernel: Lustre: Skipped 138 previous similar messages Feb 18 04:11:25 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ed8a9d5a-3c64-88a2-4de9-5c7913d6ef08 (at 10.9.101.15@o2ib4) Feb 18 04:11:25 fir-io1-s1 kernel: Lustre: Skipped 181 previous similar messages Feb 18 04:16:10 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a085332d-10de-ed9d-29d6-61956e2ab6a7 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762a5f400, cur 1550492170 expire 1550492020 last 1550491943 Feb 18 04:16:10 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 18 04:21:26 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to dba9c610-260e-8121-cf3c-59fccbb7189a (at 10.8.26.20@o2ib6) Feb 18 04:21:26 fir-io1-s1 kernel: Lustre: Skipped 160 previous similar messages Feb 18 04:31:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0e25277b-1ad0-ec5f-2777-56d7cafdcd31 (at 10.8.17.6@o2ib6) Feb 18 04:31:29 fir-io1-s1 kernel: Lustre: Skipped 203 previous similar messages Feb 18 04:41:36 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9dd791fc-5e27-d5c0-d08d-b2cd561ae98d (at 10.8.30.34@o2ib6) Feb 18 04:41:36 fir-io1-s1 kernel: Lustre: Skipped 87 previous similar messages Feb 18 04:51:36 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 73ccef87-be96-5ca5-528b-f0b5192c7ff5 (at 10.8.13.8@o2ib6) Feb 18 04:51:36 fir-io1-s1 kernel: Lustre: Skipped 191 previous similar messages Feb 18 05:01:37 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 18 05:01:37 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 18 05:11:42 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 503151d3-e911-b92b-974a-493626aee137 (at 10.8.15.8@o2ib6) Feb 18 05:11:42 fir-io1-s1 kernel: Lustre: Skipped 172 previous similar messages Feb 18 05:21:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 05:21:56 fir-io1-s1 kernel: Lustre: Skipped 178 previous similar messages Feb 18 05:31:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 05:31:56 fir-io1-s1 kernel: Lustre: Skipped 84 previous similar messages Feb 18 05:41:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 05:41:56 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Feb 18 05:51:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.2.21@o2ib6) Feb 18 05:51:57 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Feb 18 06:02:17 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 5daab973-4b80-b2b7-c9e1-b8d5b6fc11e2 (at 10.9.105.2@o2ib4) Feb 18 06:02:17 fir-io1-s1 kernel: Lustre: Skipped 158 previous similar messages Feb 18 06:12:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ffcc7510-f875-7549-61d4-9f6248a33eef (at 10.9.105.7@o2ib4) Feb 18 06:12:22 fir-io1-s1 kernel: Lustre: Skipped 223 previous similar messages Feb 18 06:22:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c5f414d6-b086-f007-4ceb-21404d074992 (at 10.8.1.1@o2ib6) Feb 18 06:22:30 fir-io1-s1 kernel: Lustre: Skipped 316 previous similar messages Feb 18 06:32:31 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to dba9c610-260e-8121-cf3c-59fccbb7189a (at 10.8.26.20@o2ib6) Feb 18 06:32:31 fir-io1-s1 kernel: Lustre: Skipped 321 previous similar messages Feb 18 06:42:45 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 874d84ab-2918-f27e-a1fe-cdc3435eb5ad (at 10.8.2.18@o2ib6) Feb 18 06:42:45 fir-io1-s1 kernel: Lustre: Skipped 247 previous similar messages Feb 18 06:53:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 18 06:53:00 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 18 07:03:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0dab976e-a8e8-3b9f-2d0d-436920c3d0f0 (at 10.9.108.3@o2ib4) Feb 18 07:03:06 fir-io1-s1 kernel: Lustre: Skipped 218 previous similar messages Feb 18 07:13:14 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.9.102.48@o2ib4) Feb 18 07:13:14 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 18 07:23:18 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3e106753-f830-b044-4cf1-87da4380b4a5 (at 10.8.6.12@o2ib6) Feb 18 07:23:18 fir-io1-s1 kernel: Lustre: Skipped 129 previous similar messages Feb 18 07:30:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 397fa907-e102-3f53-aab2-0a8d498781b4 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ad1c000, cur 1550503820 expire 1550503670 last 1550503593 Feb 18 07:30:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 07:33:36 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 56f157dc-9f05-a349-697a-ac16ba31313e (at 10.9.104.59@o2ib4) Feb 18 07:33:36 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 18 07:43:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 57269b50-0465-38e8-cbda-995a4a22296e (at 10.8.8.1@o2ib6) Feb 18 07:43:49 fir-io1-s1 kernel: Lustre: Skipped 138 previous similar messages Feb 18 07:54:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.27.30@o2ib6) Feb 18 07:54:17 fir-io1-s1 kernel: Lustre: Skipped 159 previous similar messages Feb 18 08:04:24 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 18 08:04:24 fir-io1-s1 kernel: Lustre: Skipped 293 previous similar messages Feb 18 08:13:23 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8e1192c3-b894-0acf-715e-6e851bcc8291 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762604400, cur 1550506403 expire 1550506253 last 1550506176 Feb 18 08:13:23 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 08:14:29 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 16db5e03-e2d9-f103-4a0b-78f283c497a4 (at 10.8.3.2@o2ib6) Feb 18 08:14:29 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 18 08:24:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 1c47c083-1697-95bd-3469-0636ee21aa42 (at 10.8.2.32@o2ib6) Feb 18 08:24:30 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 18 08:34:34 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to bf952ca0-7fd9-96a6-ed3d-ea105cabe163 (at 10.9.103.34@o2ib4) Feb 18 08:34:34 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Feb 18 08:44:53 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c4f458ee-079e-1f6b-715d-4cc60d32c4b8 (at 10.8.11.4@o2ib6) Feb 18 08:44:53 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 18 08:54:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e6e1afb6-7acc-3808-2f05-02b79c99637e (at 10.8.23.5@o2ib6) Feb 18 08:54:58 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 18 08:57:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6ca5e9cd-72b7-aa8d-22bc-7c65d3c2b848 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786887c00, cur 1550509072 expire 1550508922 last 1550508845 Feb 18 08:57:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 09:04:59 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 156070c6-6b1a-c523-d65c-fc06e69c00b3 (at 10.9.103.39@o2ib4) Feb 18 09:04:59 fir-io1-s1 kernel: Lustre: Skipped 133 previous similar messages Feb 18 09:13:21 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client aa37b357-2f00-7704-92ab-62941662c1ec (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e52800, cur 1550510001 expire 1550509851 last 1550509774 Feb 18 09:15:27 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Feb 18 09:15:27 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 18 09:25:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 18 09:25:33 fir-io1-s1 kernel: Lustre: Skipped 172 previous similar messages Feb 18 09:27:17 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e17157c4-af6e-5f44-4373-ec8ee7f3de2c (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c211000, cur 1550510837 expire 1550510687 last 1550510610 Feb 18 09:31:23 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 722fc89e-4fb6-0519-04d0-92f8091a9aa0 (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848831e0000, cur 1550511083 expire 1550510933 last 1550510856 Feb 18 09:35:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f03c187f-2a55-47d4-d485-c19c17624703 (at 10.9.107.45@o2ib4) Feb 18 09:35:39 fir-io1-s1 kernel: Lustre: Skipped 339 previous similar messages Feb 18 09:41:29 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 989a7240-66b9-d636-b694-67f2b044a425 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98680476a000, cur 1550511689 expire 1550511539 last 1550511462 Feb 18 09:41:29 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 09:45:51 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 54df73a3-4915-b589-f8b2-dd262402c8c5 (at 10.9.107.65@o2ib4) Feb 18 09:45:51 fir-io1-s1 kernel: Lustre: Skipped 138 previous similar messages Feb 18 09:55:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 217aa13d-68cf-b5e7-ea61-382bfbba5454 (at 10.8.17.24@o2ib6) Feb 18 09:55:52 fir-io1-s1 kernel: Lustre: Skipped 204 previous similar messages Feb 18 10:05:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9a3cf601-be04-7391-f3a7-da644b04df16 (at 10.9.104.29@o2ib4) Feb 18 10:05:53 fir-io1-s1 kernel: Lustre: Skipped 128 previous similar messages Feb 18 10:12:14 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a2be75e1-b091-cb0f-962b-6346b6b94918 (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575f4dc400, cur 1550513534 expire 1550513384 last 1550513307 Feb 18 10:12:14 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 10:15:59 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 25860b57-d235-98fa-8b01-03b6e8b0ca4a (at 10.8.27.33@o2ib6) Feb 18 10:15:59 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 18 10:23:50 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 7a809938-3bc7-b231-f947-2330cd94f7f8 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583bbab800, cur 1550514230 expire 1550514080 last 1550514003 Feb 18 10:23:50 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 10:26:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 4a2d8744-adf9-f19c-6bc9-0fc47a0742fd (at 10.8.1.25@o2ib6) Feb 18 10:26:08 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 18 10:36:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 82554708-89ca-b987-84aa-d0535c262262 (at 10.8.12.28@o2ib6) Feb 18 10:36:26 fir-io1-s1 kernel: Lustre: Skipped 430 previous similar messages Feb 18 10:46:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) Feb 18 10:46:52 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Feb 18 10:55:45 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8503ff91-ab95-750b-fd2c-49a560dbee7b (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e799400, cur 1550516145 expire 1550515995 last 1550515918 Feb 18 10:55:45 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 18 10:56:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 10:56:56 fir-io1-s1 kernel: Lustre: Skipped 224 previous similar messages Feb 18 11:06:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 11:06:56 fir-io1-s1 kernel: Lustre: Skipped 103 previous similar messages Feb 18 11:16:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 11:16:56 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 18 11:26:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 11:26:56 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 18 11:36:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 11:36:56 fir-io1-s1 kernel: Lustre: Skipped 484 previous similar messages Feb 18 11:46:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 11:46:56 fir-io1-s1 kernel: Lustre: Skipped 151 previous similar messages Feb 18 11:48:02 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 626fb198-bef7-6e15-fc7a-12f682534806 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986cc985e800, cur 1550519282 expire 1550519132 last 1550519055 Feb 18 11:48:02 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 11:48:07 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 626fb198-bef7-6e15-fc7a-12f682534806 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677dc5fc00, cur 1550519287 expire 1550519137 last 1550519060 Feb 18 11:48:12 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 626fb198-bef7-6e15-fc7a-12f682534806 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848845fa400, cur 1550519292 expire 1550519142 last 1550519065 Feb 18 11:48:22 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 626fb198-bef7-6e15-fc7a-12f682534806 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849be95f400, cur 1550519302 expire 1550519152 last 1550519075 Feb 18 11:48:22 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 18 11:56:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 11:56:56 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 18 12:01:45 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client bff9307e-f75d-c4cf-03e8-280bb0efa4e9 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677c02fc00, cur 1550520105 expire 1550519955 last 1550519878 Feb 18 12:01:45 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 18 12:07:00 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to e18ec55f-27df-5b55-2c1d-0fb1ae5cad9b (at 10.8.27.28@o2ib6) Feb 18 12:07:00 fir-io1-s1 kernel: Lustre: Skipped 134 previous similar messages Feb 18 12:13:50 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 48bb6e92-ca77-76c3-38ff-061456175e56 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848000ee800, cur 1550520830 expire 1550520680 last 1550520603 Feb 18 12:13:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 12:17:20 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 82bcfbd3-d6e5-0967-d3f2-c921c94e988c (at 10.9.105.71@o2ib4) Feb 18 12:17:20 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 18 12:27:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 549e6cee-c337-f3e7-7014-19c01d9b5967 (at 10.9.105.17@o2ib4) Feb 18 12:27:31 fir-io1-s1 kernel: Lustre: Skipped 297 previous similar messages Feb 18 12:37:32 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 1220ec8d-e345-b247-0664-9ffbca04ef6f (at 10.9.104.9@o2ib4) Feb 18 12:37:32 fir-io1-s1 kernel: Lustre: Skipped 151 previous similar messages Feb 18 12:46:49 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 92ee5f17-5351-c939-5057-50fd96aee445 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d50c00, cur 1550522809 expire 1550522659 last 1550522582 Feb 18 12:46:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 12:47:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 1220ec8d-e345-b247-0664-9ffbca04ef6f (at 10.9.104.9@o2ib4) Feb 18 12:47:37 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 18 12:57:40 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dbe81c3f-e038-b02e-a6dc-aaba56293b77 (at 10.8.2.19@o2ib6) Feb 18 12:57:40 fir-io1-s1 kernel: Lustre: Skipped 179 previous similar messages Feb 18 13:06:25 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 77f4daca-40b6-d1b7-8e90-3c49cadd0e60 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583f51e400, cur 1550523985 expire 1550523835 last 1550523758 Feb 18 13:06:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 13:07:52 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 1bdafb8f-098e-e990-f183-dce8ce68db0c (at 10.9.102.5@o2ib4) Feb 18 13:07:52 fir-io1-s1 kernel: Lustre: Skipped 182 previous similar messages Feb 18 13:18:10 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) Feb 18 13:18:10 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Feb 18 13:24:42 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7313ace2-d579-fe65-9c57-4f7346f4773e (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575871d800, cur 1550525082 expire 1550524932 last 1550524855 Feb 18 13:24:42 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 18 13:28:27 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 7f26f4a5-b09c-90cc-57f5-181682b8827f (at 10.9.103.33@o2ib4) Feb 18 13:28:27 fir-io1-s1 kernel: Lustre: Skipped 98 previous similar messages Feb 18 13:35:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2fe81daa-f812-daa2-f891-c08a856aa606 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387bac00, cur 1550525737 expire 1550525587 last 1550525510 Feb 18 13:35:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 13:38:44 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.29.2@o2ib6) Feb 18 13:38:44 fir-io1-s1 kernel: Lustre: Skipped 108 previous similar messages Feb 18 13:48:45 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 18 13:48:45 fir-io1-s1 kernel: Lustre: Skipped 122 previous similar messages Feb 18 13:59:12 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f08c4e42-b3ea-c5e8-6917-a75e80e5af29 (at 10.8.8.36@o2ib6) Feb 18 13:59:12 fir-io1-s1 kernel: Lustre: Skipped 128 previous similar messages Feb 18 14:09:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to efc6b332-a736-88e8-194a-588aa3e05348 (at 10.8.21.36@o2ib6) Feb 18 14:09:16 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Feb 18 14:19:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b59a0d33-8556-89d2-09a5-3aa1a07e86fa (at 10.9.102.19@o2ib4) Feb 18 14:19:18 fir-io1-s1 kernel: Lustre: Skipped 221 previous similar messages Feb 18 14:29:21 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.29.2@o2ib6) Feb 18 14:29:21 fir-io1-s1 kernel: Lustre: Skipped 239 previous similar messages Feb 18 14:39:35 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.29.2@o2ib6) Feb 18 14:39:35 fir-io1-s1 kernel: Lustre: Skipped 154 previous similar messages Feb 18 14:49:41 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 85810729-2f82-b4f2-1241-3806d86f03d3 (at 10.8.30.6@o2ib6) Feb 18 14:49:41 fir-io1-s1 kernel: Lustre: Skipped 194 previous similar messages Feb 18 15:00:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 76e809db-e90a-7c07-20d4-e3130ed3be85 (at 10.9.104.30@o2ib4) Feb 18 15:00:07 fir-io1-s1 kernel: Lustre: Skipped 180 previous similar messages Feb 18 15:10:14 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 22505c78-b9c2-e28a-88c4-7dadc4be41e9 (at 10.9.101.28@o2ib4) Feb 18 15:10:14 fir-io1-s1 kernel: Lustre: Skipped 364 previous similar messages Feb 18 15:20:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Feb 18 15:20:18 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 18 15:27:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d569a0fc-511a-7770-d20b-31685f473b10 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848008f2c00, cur 1550532459 expire 1550532309 last 1550532232 Feb 18 15:27:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 15:30:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to baf8fd69-3c06-48ef-44c1-991a98b1784d (at 10.8.13.14@o2ib6) Feb 18 15:30:29 fir-io1-s1 kernel: Lustre: Skipped 146 previous similar messages Feb 18 15:40:33 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0b22cd50-4b3f-cc55-9158-e0958bde4beb (at 10.8.18.27@o2ib6) Feb 18 15:40:33 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Feb 18 15:50:42 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) Feb 18 15:50:42 fir-io1-s1 kernel: Lustre: Skipped 144 previous similar messages Feb 18 16:00:45 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.29.2@o2ib6) Feb 18 16:00:45 fir-io1-s1 kernel: Lustre: Skipped 236 previous similar messages Feb 18 16:10:46 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 18 16:10:46 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 18 16:20:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 86d44073-2a86-f18b-4a0f-e98051cdbb2e (at 10.9.105.51@o2ib4) Feb 18 16:20:47 fir-io1-s1 kernel: Lustre: Skipped 212 previous similar messages Feb 18 16:30:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) Feb 18 16:30:54 fir-io1-s1 kernel: Lustre: Skipped 360 previous similar messages Feb 18 16:41:01 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to c44e04df-54bc-73f6-f90b-f3a0ff5829c3 (at 10.9.105.48@o2ib4) Feb 18 16:41:01 fir-io1-s1 kernel: Lustre: Skipped 142 previous similar messages Feb 18 16:51:01 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 1bf63035-2382-2247-57ec-f4958613068d (at 10.8.24.11@o2ib6) Feb 18 16:51:01 fir-io1-s1 kernel: Lustre: Skipped 379 previous similar messages Feb 18 17:01:04 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 5d84d21d-4aed-2e66-e01b-378e9d302a9a (at 10.9.102.26@o2ib4) Feb 18 17:01:04 fir-io1-s1 kernel: Lustre: Skipped 265 previous similar messages Feb 18 17:11:11 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 974e5b94-1244-d95f-418b-7ea7a4073539 (at 10.8.26.34@o2ib6) Feb 18 17:11:11 fir-io1-s1 kernel: Lustre: Skipped 307 previous similar messages Feb 18 17:21:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8d3900cc-5d6e-76ce-7239-094cf5a8f78d (at 10.9.104.8@o2ib4) Feb 18 17:21:17 fir-io1-s1 kernel: Lustre: Skipped 249 previous similar messages Feb 18 17:31:49 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.29.2@o2ib6) Feb 18 17:31:49 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 18 17:41:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 17:41:56 fir-io1-s1 kernel: Lustre: Skipped 231 previous similar messages Feb 18 17:49:28 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c067c441-5b7b-97a6-3218-61e420e8f167 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e44800, cur 1550540968 expire 1550540818 last 1550540741 Feb 18 17:49:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 17:49:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c067c441-5b7b-97a6-3218-61e420e8f167 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767570000, cur 1550540978 expire 1550540828 last 1550540751 Feb 18 17:49:42 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c067c441-5b7b-97a6-3218-61e420e8f167 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e44c00, cur 1550540982 expire 1550540832 last 1550540755 Feb 18 17:49:48 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c067c441-5b7b-97a6-3218-61e420e8f167 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984804dbbc00, cur 1550540988 expire 1550540838 last 1550540761 Feb 18 17:49:48 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 18 17:51:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 17:51:56 fir-io1-s1 kernel: Lustre: Skipped 119 previous similar messages Feb 18 18:01:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 18:01:56 fir-io1-s1 kernel: Lustre: Skipped 214 previous similar messages Feb 18 18:11:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 18:11:56 fir-io1-s1 kernel: Lustre: Skipped 143 previous similar messages Feb 18 18:21:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 18:21:56 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 18 18:31:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 18:31:56 fir-io1-s1 kernel: Lustre: Skipped 221 previous similar messages Feb 18 18:41:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 18:41:56 fir-io1-s1 kernel: Lustre: Skipped 157 previous similar messages Feb 18 18:46:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client fa859b2d-ad59-cc43-b0d8-fb96b0b071f4 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b2f3400, cur 1550544366 expire 1550544216 last 1550544139 Feb 18 18:51:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 18:51:56 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 18 19:01:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 19:01:56 fir-io1-s1 kernel: Lustre: Skipped 198 previous similar messages Feb 18 19:11:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 19:11:56 fir-io1-s1 kernel: Lustre: Skipped 202 previous similar messages Feb 18 19:14:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 24f10cc3-d07d-b1e9-1db0-491a45d68140 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985815682c00, cur 1550546081 expire 1550545931 last 1550545854 Feb 18 19:14:41 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 19:21:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 19:21:56 fir-io1-s1 kernel: Lustre: Skipped 169 previous similar messages Feb 18 19:31:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 19:31:56 fir-io1-s1 kernel: Lustre: Skipped 169 previous similar messages Feb 18 19:41:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 19:41:56 fir-io1-s1 kernel: Lustre: Skipped 129 previous similar messages Feb 18 19:51:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 19:51:56 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 18 20:01:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 18 20:01:56 fir-io1-s1 kernel: Lustre: Skipped 144 previous similar messages Feb 18 20:12:41 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a908dcd0-9db1-9f70-5f22-f8c81c7a1077 (at 10.8.23.10@o2ib6) Feb 18 20:12:41 fir-io1-s1 kernel: Lustre: Skipped 188 previous similar messages Feb 18 20:22:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d99aa6c5-95ff-be26-f78a-b1cfe9fb5439 (at 10.9.101.70@o2ib4) Feb 18 20:22:54 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 18 20:32:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fe789eb3-1cd9-3594-b889-6606ba1b8e4a (at 10.9.113.2@o2ib4) Feb 18 20:32:55 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 18 20:43:04 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0e1cc7ee-ac14-2533-62de-8aa817b3cbc6 (at 10.8.4.26@o2ib6) Feb 18 20:43:04 fir-io1-s1 kernel: Lustre: Skipped 192 previous similar messages Feb 18 20:53:21 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45a162ef-0459-9991-8b7a-2377aa3c8022 (at 10.9.101.32@o2ib4) Feb 18 20:53:21 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 18 21:03:28 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to bd16151d-af0e-00df-69f0-bc73398a9c87 (at 10.8.4.5@o2ib6) Feb 18 21:03:28 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 18 21:13:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 103f9da3-989e-ad73-cfdb-75395d4c9148 (at 10.8.8.35@o2ib6) Feb 18 21:13:35 fir-io1-s1 kernel: Lustre: Skipped 271 previous similar messages Feb 18 21:15:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 7e21959a-c722-e1a1-fb7f-d0f3ff27bf08 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904996800, cur 1550553356 expire 1550553206 last 1550553129 Feb 18 21:15:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 21:23:35 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 8c0347c3-135a-e940-667b-27edd6a0ad7f (at 10.8.20.6@o2ib6) Feb 18 21:23:35 fir-io1-s1 kernel: Lustre: Skipped 240 previous similar messages Feb 18 21:33:36 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to b2753dc4-19d1-7b06-13e5-9ceb58fcc4d7 (at 10.9.113.4@o2ib4) Feb 18 21:33:36 fir-io1-s1 kernel: Lustre: Skipped 198 previous similar messages Feb 18 21:43:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6da5269e-e6e7-e930-ea8b-e990b1fd18b0 (at 10.9.101.72@o2ib4) Feb 18 21:43:53 fir-io1-s1 kernel: Lustre: Skipped 84 previous similar messages Feb 18 21:47:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fb3dce28-86d7-088f-2e99-03834df35e02 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f7ea000, cur 1550555255 expire 1550555105 last 1550555028 Feb 18 21:47:35 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 18 21:54:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f97f4802-8c0b-e9cd-e8d7-93692decf22a (at 10.9.102.8@o2ib4) Feb 18 21:54:26 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 18 21:59:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e45a36c3-638c-79d7-586c-fcf00dbc7d14 (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480f61d400, cur 1550555962 expire 1550555812 last 1550555735 Feb 18 21:59:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 18 22:04:48 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6f43d80a-d4ed-b27e-b29c-0d22cee6d831 (at 10.9.105.8@o2ib4) Feb 18 22:04:48 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 18 22:15:01 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 21ad58eb-b0eb-378b-d1d7-0646aa1b95cf (at 10.9.102.28@o2ib4) Feb 18 22:15:01 fir-io1-s1 kernel: Lustre: Skipped 200 previous similar messages Feb 18 22:25:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 18 22:25:04 fir-io1-s1 kernel: Lustre: Skipped 307 previous similar messages Feb 18 22:35:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0127092e-ab70-ddef-6a66-286028d84f5d (at 10.9.102.43@o2ib4) Feb 18 22:35:05 fir-io1-s1 kernel: Lustre: Skipped 178 previous similar messages Feb 18 22:45:41 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to e6f15ae7-1884-d676-5174-cffd0284b92d (at 10.8.1.30@o2ib6) Feb 18 22:45:41 fir-io1-s1 kernel: Lustre: Skipped 124 previous similar messages Feb 18 22:55:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 173446cc-39b1-333f-81fc-6684fb678e20 (at 10.8.3.19@o2ib6) Feb 18 22:55:47 fir-io1-s1 kernel: Lustre: Skipped 132 previous similar messages Feb 18 23:05:50 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fe8f9169-1b75-ba67-02e9-ac6ba53a9586 (at 10.8.24.32@o2ib6) Feb 18 23:05:50 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 18 23:15:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 18 23:15:58 fir-io1-s1 kernel: Lustre: Skipped 127 previous similar messages Feb 18 23:26:14 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to fed507e8-5435-f949-539f-6cb9d563cc12 (at 10.9.106.52@o2ib4) Feb 18 23:26:14 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 18 23:36:35 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to b2364f4b-9129-81e8-7e2f-15aa4210b663 (at 10.9.107.12@o2ib4) Feb 18 23:36:35 fir-io1-s1 kernel: Lustre: Skipped 138 previous similar messages Feb 18 23:46:37 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ed540f5a-5df6-5998-8f5c-40181564f690 (at 10.9.106.50@o2ib4) Feb 18 23:46:37 fir-io1-s1 kernel: Lustre: Skipped 181 previous similar messages Feb 18 23:56:38 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 173446cc-39b1-333f-81fc-6684fb678e20 (at 10.8.3.19@o2ib6) Feb 18 23:56:38 fir-io1-s1 kernel: Lustre: Skipped 236 previous similar messages Feb 19 00:06:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 00:06:56 fir-io1-s1 kernel: Lustre: Skipped 213 previous similar messages Feb 19 00:16:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 00:16:56 fir-io1-s1 kernel: Lustre: Skipped 203 previous similar messages Feb 19 00:26:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 00:26:56 fir-io1-s1 kernel: Lustre: Skipped 159 previous similar messages Feb 19 00:36:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 00:36:56 fir-io1-s1 kernel: Lustre: Skipped 182 previous similar messages Feb 19 00:47:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to f6f2dc0e-bc9f-2120-e971-29d8049b1247 (at 10.8.20.12@o2ib6) Feb 19 00:47:03 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 19 00:57:04 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3e2bfa45-013a-e48d-7bcc-c486bbeaa49b (at 10.9.108.9@o2ib4) Feb 19 00:57:04 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 19 01:07:19 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 40113a8d-c41f-508e-9772-7563fba01286 (at 10.9.105.5@o2ib4) Feb 19 01:07:19 fir-io1-s1 kernel: Lustre: Skipped 97 previous similar messages Feb 19 01:17:31 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.3.17@o2ib6) Feb 19 01:17:31 fir-io1-s1 kernel: Lustre: Skipped 160 previous similar messages Feb 19 01:27:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72ef71e7-ecde-a7f2-d85d-452a22011f5b (at 10.9.101.14@o2ib4) Feb 19 01:27:45 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Feb 19 01:37:54 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 6d1355d0-7b33-677d-c8cf-a270e3061917 (at 10.8.7.15@o2ib6) Feb 19 01:37:54 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Feb 19 01:48:34 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 19 01:48:34 fir-io1-s1 kernel: Lustre: Skipped 163 previous similar messages Feb 19 01:58:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fb273c41-c272-402d-98b5-3e5f91dba50e (at 10.9.114.15@o2ib4) Feb 19 01:58:38 fir-io1-s1 kernel: Lustre: Skipped 173 previous similar messages Feb 19 02:08:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.3.17@o2ib6) Feb 19 02:08:39 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 19 02:18:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 19 02:18:42 fir-io1-s1 kernel: Lustre: Skipped 181 previous similar messages Feb 19 02:28:48 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) Feb 19 02:28:48 fir-io1-s1 kernel: Lustre: Skipped 188 previous similar messages Feb 19 02:38:50 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 64d3667d-1c77-7867-b1f3-ec9c4a6035ad (at 10.9.104.56@o2ib4) Feb 19 02:38:50 fir-io1-s1 kernel: Lustre: Skipped 120 previous similar messages Feb 19 02:49:15 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 94c2dba8-b225-c91a-753d-91a7f0495a0f (at 10.9.101.8@o2ib4) Feb 19 02:49:15 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 19 02:59:28 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e033b2fb-58ee-ad20-dbe1-c069873ac977 (at 10.9.101.47@o2ib4) Feb 19 02:59:28 fir-io1-s1 kernel: Lustre: Skipped 105 previous similar messages Feb 19 03:08:38 fir-io1-s1 kernel: perf: interrupt took too long (3919 > 3915), lowering kernel.perf_event_max_sample_rate to 51000 Feb 19 03:09:33 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 69e04fcf-27a0-cb59-92a0-ef1d06a212ef (at 10.9.104.6@o2ib4) Feb 19 03:09:33 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 19 03:14:08 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0fb95963-5ce5-3ad8-9993-da404a3760f0 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e57000, cur 1550574848 expire 1550574698 last 1550574621 Feb 19 03:14:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 19 03:19:48 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to (at 10.8.3.27@o2ib6) Feb 19 03:19:48 fir-io1-s1 kernel: Lustre: Skipped 114 previous similar messages Feb 19 03:29:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 97988560-485b-3aac-0ec9-309ca48dcc10 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849be95ac00, cur 1550575788 expire 1550575638 last 1550575561 Feb 19 03:29:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 19 03:29:51 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to f854c2f0-b53f-8306-9638-bc37f75b2b94 (at 10.8.8.7@o2ib6) Feb 19 03:29:51 fir-io1-s1 kernel: Lustre: Skipped 112 previous similar messages Feb 19 03:35:15 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d2c000, cur 1550576115 expire 1550575965 last 1550575888 Feb 19 03:35:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 19 03:38:21 fir-io1-s1 kernel: Lustre: 96932:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550576294/real 1550576294] req@ffff9872b7a89b00 x1624934762307856/t0(0) o106->fir-OST0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550576301 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 19 03:38:21 fir-io1-s1 kernel: Lustre: 82278:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550576294/real 1550576294] req@ffff98695a527800 x1624934762307888/t0(0) o106->fir-OST0008@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550576301 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 19 03:38:21 fir-io1-s1 kernel: Lustre: 82278:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 19 03:38:42 fir-io1-s1 kernel: Lustre: 96356:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550576315/real 1550576315] req@ffff983a66feb600 x1624934762307904/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550576322 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 03:38:42 fir-io1-s1 kernel: Lustre: 96356:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 19 03:39:17 fir-io1-s1 kernel: Lustre: 96281:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550576350/real 1550576350] req@ffff984aa0617500 x1624934762307872/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550576357 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 03:39:17 fir-io1-s1 kernel: Lustre: 96281:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Feb 19 03:39:52 fir-io1-s1 kernel: LustreError: 82278:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) returned error from glimpse AST (req@ffff98695a527800 x1624934762307888 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff985f16800fc0/0x49e185e99df8ca67 lrc: 3/0,0 mode: PW/PW res: [0xf3ebf:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0x6898580bdc4d4e1f expref: 6 pid: 96939 timeout: 0 lvb_type: 0 Feb 19 03:39:52 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 19 03:39:52 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550576392s: evicting client at 10.8.9.8@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff984ad319f740/0x49e185e99df8ca59 lrc: 3/0,0 mode: PW/PW res: [0xf3d9c:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0x6898580bdc4d4de7 expref: 7 pid: 96939 timeout: 0 lvb_type: 0 Feb 19 03:39:52 fir-io1-s1 kernel: LustreError: 82278:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Feb 19 03:40:05 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9d90a6a6-e463-02e9-3fef-fe0fa60e4307 (at 10.9.114.13@o2ib4) Feb 19 03:40:05 fir-io1-s1 kernel: Lustre: Skipped 119 previous similar messages Feb 19 03:42:26 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ad109ec5-323b-a9c1-764f-376442efd82e (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848831e3000, cur 1550576546 expire 1550576396 last 1550576319 Feb 19 03:42:26 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 19 03:50:06 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b3fc52f5-cc19-f1e2-5d13-43190203fae8 (at 10.9.106.22@o2ib4) Feb 19 03:50:06 fir-io1-s1 kernel: Lustre: Skipped 143 previous similar messages Feb 19 04:00:25 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a9ad9001-d2d9-466a-37c8-3f54fd94183d (at 10.9.101.26@o2ib4) Feb 19 04:00:25 fir-io1-s1 kernel: Lustre: Skipped 105 previous similar messages Feb 19 04:10:53 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) Feb 19 04:10:53 fir-io1-s1 kernel: Lustre: Skipped 118 previous similar messages Feb 19 04:20:53 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to ad4e6ecf-07e4-50a5-1377-7c6668e4ff22 (at 10.8.24.15@o2ib6) Feb 19 04:20:53 fir-io1-s1 kernel: Lustre: Skipped 198 previous similar messages Feb 19 04:30:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 19 04:30:58 fir-io1-s1 kernel: Lustre: Skipped 161 previous similar messages Feb 19 04:41:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b2753dc4-19d1-7b06-13e5-9ceb58fcc4d7 (at 10.9.113.4@o2ib4) Feb 19 04:41:00 fir-io1-s1 kernel: Lustre: Skipped 114 previous similar messages Feb 19 04:51:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b3fc52f5-cc19-f1e2-5d13-43190203fae8 (at 10.9.106.22@o2ib4) Feb 19 04:51:06 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 19 05:01:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f03c187f-2a55-47d4-d485-c19c17624703 (at 10.9.107.45@o2ib4) Feb 19 05:01:12 fir-io1-s1 kernel: Lustre: Skipped 159 previous similar messages Feb 19 05:11:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 874d84ab-2918-f27e-a1fe-cdc3435eb5ad (at 10.8.2.18@o2ib6) Feb 19 05:11:41 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 19 05:21:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 05:21:56 fir-io1-s1 kernel: Lustre: Skipped 178 previous similar messages Feb 19 05:31:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 05:31:56 fir-io1-s1 kernel: Lustre: Skipped 120 previous similar messages Feb 19 05:41:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 05:41:56 fir-io1-s1 kernel: Lustre: Skipped 145 previous similar messages Feb 19 05:51:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 05:51:56 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 19 06:01:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 06:01:56 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Feb 19 06:11:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 06:11:56 fir-io1-s1 kernel: Lustre: Skipped 150 previous similar messages Feb 19 06:21:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 06:21:56 fir-io1-s1 kernel: Lustre: Skipped 124 previous similar messages Feb 19 06:32:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f88d3e4f-b8ad-7e3f-e052-b857e571de2a (at 10.9.107.13@o2ib4) Feb 19 06:32:12 fir-io1-s1 kernel: Lustre: Skipped 164 previous similar messages Feb 19 06:42:12 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2dfe635f-2c67-15af-7599-8846ef697285 (at 10.8.8.34@o2ib6) Feb 19 06:42:12 fir-io1-s1 kernel: Lustre: Skipped 171 previous similar messages Feb 19 06:52:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3f539eb7-bad7-deea-5d73-69ef13653da8 (at 10.8.26.15@o2ib6) Feb 19 06:52:17 fir-io1-s1 kernel: Lustre: Skipped 238 previous similar messages Feb 19 07:02:24 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1cac0206-7dc8-7985-dbe6-f16507ebcfe0 (at 10.8.1.19@o2ib6) Feb 19 07:02:24 fir-io1-s1 kernel: Lustre: Skipped 193 previous similar messages Feb 19 07:12:32 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a03158af-bffb-aadf-bfb0-07125d4dfb10 (at 10.9.105.60@o2ib4) Feb 19 07:12:32 fir-io1-s1 kernel: Lustre: Skipped 148 previous similar messages Feb 19 07:22:33 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2105faed-3f3f-f302-d9e7-f8bce33a4b72 (at 10.8.3.23@o2ib6) Feb 19 07:22:33 fir-io1-s1 kernel: Lustre: Skipped 144 previous similar messages Feb 19 07:32:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6fa95ecf-4554-3236-796b-9301a5a09ace (at 10.8.6.24@o2ib6) Feb 19 07:32:35 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 19 07:42:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 76e46ee9-2b41-bbbb-c588-f884c93ae793 (at 10.8.6.13@o2ib6) Feb 19 07:42:41 fir-io1-s1 kernel: Lustre: Skipped 118 previous similar messages Feb 19 07:52:58 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 54df73a3-4915-b589-f8b2-dd262402c8c5 (at 10.9.107.65@o2ib4) Feb 19 07:52:58 fir-io1-s1 kernel: Lustre: Skipped 141 previous similar messages Feb 19 08:03:03 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to cec68a9e-fc42-8b83-b21d-285fcd29817f (at 10.8.7.16@o2ib6) Feb 19 08:03:03 fir-io1-s1 kernel: Lustre: Skipped 190 previous similar messages Feb 19 08:13:27 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9cf6d7a8-4898-44eb-2590-b689cf0f2dd8 (at 10.8.6.11@o2ib6) Feb 19 08:13:27 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 19 08:15:34 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9a643424-2c87-97d1-5985-bee245b7674b (at 10.8.13.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d36a400, cur 1550592934 expire 1550592784 last 1550592707 Feb 19 08:15:34 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 19 08:24:10 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to e63b86ba-9595-7419-fca7-82e45f9f64cb (at 10.9.104.67@o2ib4) Feb 19 08:24:10 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 19 08:34:20 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 283673e9-8136-ebb7-35e9-2d12f60edf66 (at 10.9.105.23@o2ib4) Feb 19 08:34:20 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 19 08:44:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e63b86ba-9595-7419-fca7-82e45f9f64cb (at 10.9.104.67@o2ib4) Feb 19 08:44:30 fir-io1-s1 kernel: Lustre: Skipped 139 previous similar messages Feb 19 08:55:08 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) Feb 19 08:55:08 fir-io1-s1 kernel: Lustre: Skipped 218 previous similar messages Feb 19 09:05:16 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to c7981bef-8624-1b06-32b3-1f88bc1711f2 (at 10.8.8.16@o2ib6) Feb 19 09:05:16 fir-io1-s1 kernel: Lustre: Skipped 74 previous similar messages Feb 19 09:15:24 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d5a74680-e7af-ebfb-7dfd-72e2645d277b (at 10.9.101.51@o2ib4) Feb 19 09:15:24 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 19 09:25:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 13ccf343-7ccf-96f3-9354-06c5a49c0c5d (at 10.8.2.11@o2ib6) Feb 19 09:25:30 fir-io1-s1 kernel: Lustre: Skipped 147 previous similar messages Feb 19 09:35:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 58531951-dcfc-2dad-91c4-688aefd85811 (at 10.9.104.5@o2ib4) Feb 19 09:35:30 fir-io1-s1 kernel: Lustre: Skipped 164 previous similar messages Feb 19 09:39:59 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2b189825-d1f1-71d3-cfde-67bf3bf39cf4 (at 10.8.10.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581bfe8c00, cur 1550597999 expire 1550597849 last 1550597772 Feb 19 09:39:59 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 19 09:45:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fa13283d-cb82-377d-1a63-e89c95a71a0d (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596b8000, cur 1550598339 expire 1550598189 last 1550598112 Feb 19 09:45:39 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 19 09:45:50 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1c418863-78cc-8f23-893e-27e5ce2dfd94 (at 10.9.101.71@o2ib4) Feb 19 09:45:50 fir-io1-s1 kernel: Lustre: Skipped 169 previous similar messages Feb 19 09:45:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d1eafd0d-f5d5-63d1-f545-e28e22ce25f0 (at 10.9.106.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a6d000, cur 1550598356 expire 1550598206 last 1550598129 Feb 19 09:55:55 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) Feb 19 09:55:55 fir-io1-s1 kernel: Lustre: Skipped 266 previous similar messages Feb 19 10:05:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to bfe736a6-72da-534a-a0b9-aa8669f81433 (at 10.8.25.11@o2ib6) Feb 19 10:05:56 fir-io1-s1 kernel: Lustre: Skipped 267 previous similar messages Feb 19 10:15:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ceafb54-89ce-8961-d103-913efe379d81 (at 10.8.21.7@o2ib6) Feb 19 10:15:57 fir-io1-s1 kernel: Lustre: Skipped 367 previous similar messages Feb 19 10:26:04 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to fca022be-6585-be05-88a7-6e814634b560 (at 10.9.106.42@o2ib4) Feb 19 10:26:04 fir-io1-s1 kernel: Lustre: Skipped 249 previous similar messages Feb 19 10:36:06 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Feb 19 10:36:06 fir-io1-s1 kernel: Lustre: Skipped 209 previous similar messages Feb 19 10:46:09 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 19 10:46:09 fir-io1-s1 kernel: Lustre: Skipped 254 previous similar messages Feb 19 10:56:25 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 5874979c-2d42-d4b8-b0f7-48cd970c494d (at 10.8.4.28@o2ib6) Feb 19 10:56:25 fir-io1-s1 kernel: Lustre: Skipped 154 previous similar messages Feb 19 11:06:55 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 156070c6-6b1a-c523-d65c-fc06e69c00b3 (at 10.9.103.39@o2ib4) Feb 19 11:06:55 fir-io1-s1 kernel: Lustre: Skipped 134 previous similar messages Feb 19 11:16:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 11:16:56 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 19 11:26:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 11:26:56 fir-io1-s1 kernel: Lustre: Skipped 177 previous similar messages Feb 19 11:36:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 11:36:56 fir-io1-s1 kernel: Lustre: Skipped 179 previous similar messages Feb 19 11:46:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 11:46:56 fir-io1-s1 kernel: Lustre: Skipped 191 previous similar messages Feb 19 11:54:22 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0000: Connection to fir-MDT0003 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 19 11:54:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 19 11:55:12 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0003-lwp-OST000a: This client was evicted by fir-MDT0003; in progress operations using this service will fail. Feb 19 11:55:12 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 19 11:55:20 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:239888 to 0x6c0000401:240001 Feb 19 11:55:20 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:239824 to 0xc40000400:239937 Feb 19 11:55:20 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:239855 to 0x8c0000400:239905 Feb 19 11:55:20 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:240002 to 0xc80000400:240065 Feb 19 11:55:20 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:240124 to 0x5c0000401:240193 Feb 19 11:55:20 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:240180 to 0x580000401:240353 Feb 19 11:56:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 11:56:56 fir-io1-s1 kernel: Lustre: Skipped 249 previous similar messages Feb 19 11:59:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ea940d01-96cf-27ac-7561-ea49afae7a93 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b261400, cur 1550606341 expire 1550606191 last 1550606114 Feb 19 11:59:01 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 19 12:00:08 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550606401/real 1550606401] req@ffff9869f62aa700 x1624934976132272/t0(0) o104->fir-OST0006@10.9.0.63@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550606408 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 19 12:00:08 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 19 12:00:22 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550606415/real 1550606415] req@ffff9869f62aa700 x1624934976132272/t0(0) o104->fir-OST0006@10.9.0.63@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550606422 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 12:00:22 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 19 12:00:43 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550606436/real 1550606436] req@ffff9869f62aa700 x1624934976132272/t0(0) o104->fir-OST0006@10.9.0.63@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550606443 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 12:00:43 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 19 12:01:18 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550606471/real 1550606471] req@ffff9869f62aa700 x1624934976132272/t0(0) o104->fir-OST0006@10.9.0.63@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550606478 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 12:01:18 fir-io1-s1 kernel: Lustre: 96248:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 19 12:02:14 fir-io1-s1 kernel: LustreError: 96248:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.63@o2ib4) returned error from blocking AST (req@ffff9869f62aa700 x1624934976132272 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff985b5724b840/0x49e185e9a3f0183e lrc: 4/0,0 mode: PR/PR res: [0xc40000401:0x6a857:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.9.0.63@o2ib4 remote: 0xc057deb8b7962c8e expref: 139 pid: 96562 timeout: 954435 lvb_type: 1 Feb 19 12:02:14 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.9.0.63@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Feb 19 12:02:14 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 19 12:02:14 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 25s: evicting client at 10.9.0.63@o2ib4 ns: filter-fir-OST0008_UUID lock: ffff983e4aa22880/0x49e185e9a3f032a1 lrc: 3/0,0 mode: PR/PR res: [0xc80000400:0x3a81f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.9.0.63@o2ib4 remote: 0xc057deb8b797642a expref: 140 pid: 96332 timeout: 0 lvb_type: 1 Feb 19 12:02:14 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Feb 19 12:02:14 fir-io1-s1 kernel: LustreError: 96248:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 19 12:02:15 fir-io1-s1 kernel: LustreError: 96918:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.63@o2ib4) returned error from blocking AST (req@ffff985674410000 x1624934976282528 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff986f01440480/0x49e185e9a3f032a8 lrc: 4/0,0 mode: PR/PR res: [0x580000401:0x3a8d1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x60000400000020 nid: 10.9.0.63@o2ib4 remote: 0xc057deb8b7976462 expref: 146 pid: 94514 timeout: 954436 lvb_type: 1 Feb 19 12:02:15 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.9.0.63@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Feb 19 12:02:15 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 19 12:02:15 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.9.0.63@o2ib4 ns: filter-fir-OST000a_UUID lock: ffff986f01440480/0x49e185e9a3f032a8 lrc: 3/0,0 mode: PR/PR res: [0x580000401:0x3a8d1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x60000400000020 nid: 10.9.0.63@o2ib4 remote: 0xc057deb8b7976462 expref: 147 pid: 94514 timeout: 0 lvb_type: 1 Feb 19 12:02:15 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Feb 19 12:02:33 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ba1e1fbb-a72d-16b5-8ad9-93cddc13d3a5 (at 10.9.0.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b9dc00, cur 1550606553 expire 1550606403 last 1550606326 Feb 19 12:06:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 12:06:56 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 19 12:16:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 12:16:56 fir-io1-s1 kernel: Lustre: Skipped 215 previous similar messages Feb 19 12:26:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 12:26:56 fir-io1-s1 kernel: Lustre: Skipped 333 previous similar messages Feb 19 12:34:07 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550608440/real 1550608440] req@ffff98516dd95100 x1624934978445808/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550608447 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 19 12:34:07 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Feb 19 12:34:21 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550608454/real 1550608454] req@ffff986dd3a93000 x1624934978445824/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550608461 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 12:34:21 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 19 12:34:42 fir-io1-s1 kernel: Lustre: 96372:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550608475/real 1550608475] req@ffff985674413900 x1624934978445840/t0(0) o106->fir-OST0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550608482 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 12:34:42 fir-io1-s1 kernel: Lustre: 96372:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Feb 19 12:35:17 fir-io1-s1 kernel: Lustre: 96252:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550608510/real 1550608510] req@ffff983933079800 x1624934978445856/t0(0) o106->fir-OST0008@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550608517 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 12:35:17 fir-io1-s1 kernel: Lustre: 96252:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Feb 19 12:36:21 fir-io1-s1 kernel: Lustre: 96573:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550608574/real 1550608574] req@ffff984930c39200 x1624934978800240/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550608581 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 19 12:36:21 fir-io1-s1 kernel: Lustre: 96573:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 101 previous similar messages Feb 19 12:36:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 12:36:56 fir-io1-s1 kernel: Lustre: Skipped 267 previous similar messages Feb 19 12:37:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 44ec95d4-e3a4-c250-7076-b7a547019295 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef17400, cur 1550608626 expire 1550608476 last 1550608399 Feb 19 12:41:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 279e5f54-8f67-2ca6-b298-80b0d4a06fdf (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800e4fc00, cur 1550608879 expire 1550608729 last 1550608652 Feb 19 12:41:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 19 12:46:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 12:46:56 fir-io1-s1 kernel: Lustre: Skipped 287 previous similar messages Feb 19 12:56:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 19 12:56:56 fir-io1-s1 kernel: Lustre: Skipped 265 previous similar messages Feb 19 13:07:00 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6da5269e-e6e7-e930-ea8b-e990b1fd18b0 (at 10.9.101.72@o2ib4) Feb 19 13:07:00 fir-io1-s1 kernel: Lustre: Skipped 289 previous similar messages Feb 19 13:17:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 19 13:17:03 fir-io1-s1 kernel: Lustre: Skipped 450 previous similar messages Feb 19 13:27:06 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) Feb 19 13:27:06 fir-io1-s1 kernel: Lustre: Skipped 396 previous similar messages Feb 19 13:37:06 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 19 13:37:06 fir-io1-s1 kernel: Lustre: Skipped 278 previous similar messages Feb 19 13:47:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dd000554-0e1b-ac1d-70ac-6e23e66f928d (at 10.9.102.40@o2ib4) Feb 19 13:47:17 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 19 13:57:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 19 13:57:23 fir-io1-s1 kernel: Lustre: Skipped 265 previous similar messages Feb 19 14:07:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c694e053-04d0-ee79-c9a4-0ace9e2f2c9a (at 10.8.3.29@o2ib6) Feb 19 14:07:23 fir-io1-s1 kernel: Lustre: Skipped 252 previous similar messages Feb 19 14:17:23 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 19 14:17:23 fir-io1-s1 kernel: Lustre: Skipped 283 previous similar messages Feb 19 14:27:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9468a1d3-3abd-8063-5952-288cca0f1dec (at 10.8.27.35@o2ib6) Feb 19 14:27:28 fir-io1-s1 kernel: Lustre: Skipped 253 previous similar messages Feb 19 14:37:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 19 14:37:41 fir-io1-s1 kernel: Lustre: Skipped 372 previous similar messages Feb 19 14:45:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f700f996-a39c-4a99-7c8d-5ce2a4aa5064 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cb6c00, cur 1550616339 expire 1550616189 last 1550616112 Feb 19 14:47:48 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 2acb2116-5227-530a-f563-866a3449ba51 (at 10.9.106.13@o2ib4) Feb 19 14:47:48 fir-io1-s1 kernel: Lustre: Skipped 184 previous similar messages Feb 19 14:57:50 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ca01de82-8e25-cf1b-bcf0-8a49048dd46d (at 10.8.18.7@o2ib6) Feb 19 14:57:50 fir-io1-s1 kernel: Lustre: Skipped 210 previous similar messages Feb 19 15:07:58 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 2019e0f4-a199-9a37-e710-fbb1c8ccd2aa (at 10.9.107.60@o2ib4) Feb 19 15:07:58 fir-io1-s1 kernel: Lustre: Skipped 162 previous similar messages Feb 19 15:18:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c95ae3a3-cc0b-6e03-007e-3f43096cb7c1 (at 10.9.101.21@o2ib4) Feb 19 15:18:02 fir-io1-s1 kernel: Lustre: Skipped 176 previous similar messages Feb 19 15:28:25 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Feb 19 15:28:25 fir-io1-s1 kernel: Lustre: Skipped 202 previous similar messages Feb 19 15:38:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 890f9dc9-b9bc-0354-4c1a-b7392d8a9570 (at 10.8.19.5@o2ib6) Feb 19 15:38:30 fir-io1-s1 kernel: Lustre: Skipped 186 previous similar messages Feb 19 15:48:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.24.20@o2ib6) Feb 19 15:48:31 fir-io1-s1 kernel: Lustre: Skipped 191 previous similar messages Feb 19 15:50:34 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 37647a65-b7cb-14d4-b4fd-020bc3a7f211 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aab8400, cur 1550620234 expire 1550620084 last 1550620007 Feb 19 15:50:34 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 19 15:58:38 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 173446cc-39b1-333f-81fc-6684fb678e20 (at 10.8.3.19@o2ib6) Feb 19 15:58:38 fir-io1-s1 kernel: Lustre: Skipped 273 previous similar messages Feb 19 16:09:00 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ba7fcf94-2de7-e538-86a8-3e889630dfc7 (at 10.8.24.34@o2ib6) Feb 19 16:09:00 fir-io1-s1 kernel: Lustre: Skipped 214 previous similar messages Feb 19 16:19:00 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 495369c4-40aa-b2ab-a0e0-f943478581b7 (at 10.8.20.23@o2ib6) Feb 19 16:19:00 fir-io1-s1 kernel: Lustre: Skipped 204 previous similar messages Feb 19 16:29:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 76e46ee9-2b41-bbbb-c588-f884c93ae793 (at 10.8.6.13@o2ib6) Feb 19 16:29:13 fir-io1-s1 kernel: Lustre: Skipped 313 previous similar messages Feb 19 16:39:19 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 22d5bc36-ef0d-3a91-83bc-25150ae0af1e (at 10.8.2.7@o2ib6) Feb 19 16:39:19 fir-io1-s1 kernel: Lustre: Skipped 166 previous similar messages Feb 19 16:49:26 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d7f0cefd-f5dc-ae79-9fbe-8c42036c5092 (at 10.9.105.21@o2ib4) Feb 19 16:49:26 fir-io1-s1 kernel: Lustre: Skipped 298 previous similar messages Feb 19 16:59:35 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 08958bbc-0f90-1cbd-61ae-768cfa6c9459 (at 10.9.104.69@o2ib4) Feb 19 16:59:35 fir-io1-s1 kernel: Lustre: Skipped 340 previous similar messages Feb 19 17:09:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0935f868-2cb6-7be4-32e4-6f8243d37d7c (at 10.9.0.1@o2ib4) Feb 19 17:09:37 fir-io1-s1 kernel: Lustre: Skipped 289 previous similar messages Feb 19 17:19:53 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to f4b03aa2-d5b7-4f9a-0875-baaa698d022e (at 10.8.25.19@o2ib6) Feb 19 17:19:53 fir-io1-s1 kernel: Lustre: Skipped 337 previous similar messages Feb 19 17:29:53 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to cb67b941-504b-3226-9e75-e94440d73a8e (at 10.9.104.3@o2ib4) Feb 19 17:29:53 fir-io1-s1 kernel: Lustre: Skipped 476 previous similar messages Feb 19 17:35:14 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b9e5800, cur 1550626514 expire 1550626364 last 1550626287 Feb 19 17:35:32 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b9e1400, cur 1550626532 expire 1550626382 last 1550626305 Feb 19 17:39:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8c69e2c6-dc8f-f3ec-9050-fc304e24bc67 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b90a400, cur 1550626789 expire 1550626639 last 1550626562 Feb 19 17:39:49 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 19 17:39:59 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d1755442-892e-a805-5fa2-c61746c310b0 (at 10.9.113.7@o2ib4) Feb 19 17:39:59 fir-io1-s1 kernel: Lustre: Skipped 417 previous similar messages Feb 19 17:50:04 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 19 17:50:04 fir-io1-s1 kernel: Lustre: Skipped 195 previous similar messages Feb 19 18:00:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 38a104b5-26ce-5d2d-596d-9304083f888f (at 10.9.112.14@o2ib4) Feb 19 18:00:06 fir-io1-s1 kernel: Lustre: Skipped 278 previous similar messages Feb 19 18:10:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) Feb 19 18:10:08 fir-io1-s1 kernel: Lustre: Skipped 122 previous similar messages Feb 19 18:20:12 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 19 18:20:12 fir-io1-s1 kernel: Lustre: Skipped 151 previous similar messages Feb 19 18:22:24 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST000a: Connection to fir-MDT0003 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 19 18:22:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 19 18:22:49 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection to fir-MDT0001 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 19 18:22:49 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 19 18:23:14 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 19 18:23:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 19 18:23:21 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550629394/real 1550629394] req@ffff986e56a55100 x1624935103888864/t0(0) o400->fir-MDT0000-lwp-OST0006@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1550629401 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Feb 19 18:23:21 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST000a: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 19 18:23:21 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 19 18:23:21 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 75 previous similar messages Feb 19 18:24:37 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 2 seconds Feb 19 18:24:37 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message Feb 19 18:24:38 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 547 seconds Feb 19 18:24:38 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Feb 19 18:24:42 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Feb 19 18:24:42 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 19 18:25:02 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 45 seconds Feb 19 18:25:02 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages Feb 19 18:25:07 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds Feb 19 18:25:07 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 19 previous similar messages Feb 19 18:25:07 fir-io1-s1 kernel: Lustre: 91455:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550629501/real 1550629507] req@ffff98695742da00 x1624935103907488/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1550629508 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 19 18:25:07 fir-io1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail Feb 19 18:25:32 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Feb 19 18:25:32 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 19 18:25:57 fir-io1-s1 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x6bcd99f60a0d4c23 to 0xb7044c4b134c3dec Feb 19 18:25:57 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Feb 19 18:25:57 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1494740 to 0xc80000402:1494945 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1494694 to 0x8c0000402:1494881 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1495318 to 0x580000400:1495393 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1494979 to 0x5c0000400:1495105 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1495129 to 0x6c0000400:1495201 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1494855 to 0xc40000402:1495073 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1032780 to 0x0:1032897 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1032513 to 0x0:1032577 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1032278 to 0x0:1032321 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1032126 to 0x0:1032225 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1033070 to 0x0:1033121 Feb 19 18:26:11 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1032148 to 0x0:1032481 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:245631 to 0x6c0000401:245665 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:245570 to 0xc40000400:245601 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:245825 to 0x5c0000401:245857 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:245537 to 0x8c0000400:245569 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:245987 to 0x580000401:246017 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:245698 to 0xc80000400:245729 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:440388 to 0xc40000401:440481 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:440574 to 0x6c0000402:440609 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:441789 to 0x5c0000402:441889 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:441478 to 0x580000402:441665 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:440479 to 0x8c0000401:440577 Feb 19 18:26:12 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:440494 to 0xc80000401:440577 Feb 19 18:30:18 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7d6292c2-dc0a-0082-5273-c1ff8e6163ed (at 10.9.102.25@o2ib4) Feb 19 18:30:18 fir-io1-s1 kernel: Lustre: Skipped 367 previous similar messages Feb 19 18:40:28 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 318de70a-4b49-6572-6064-ea964a3568c4 (at 10.9.107.37@o2ib4) Feb 19 18:40:28 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 19 18:50:01 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3886283b-4b1b-8eec-fde1-73b9ce410739 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ad81a8400, cur 1550631001 expire 1550630851 last 1550630774 Feb 19 18:50:01 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 19 18:50:31 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to c92dc8f6-832a-e505-c370-d933825f236a (at 10.8.2.16@o2ib6) Feb 19 18:50:31 fir-io1-s1 kernel: Lustre: Skipped 259 previous similar messages Feb 19 19:00:35 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 45a12f91-6aa3-f0ae-a299-8aadf4c776a5 (at 10.8.17.5@o2ib6) Feb 19 19:00:35 fir-io1-s1 kernel: Lustre: Skipped 282 previous similar messages Feb 19 19:10:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9fdeb7e4-dbe7-36a2-5704-e3cbd89d3a9c (at 10.9.104.25@o2ib4) Feb 19 19:10:36 fir-io1-s1 kernel: Lustre: Skipped 222 previous similar messages Feb 19 19:20:43 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 88f11576-1898-ee78-29af-b93b7778dcb7 (at 10.8.13.26@o2ib6) Feb 19 19:20:43 fir-io1-s1 kernel: Lustre: Skipped 211 previous similar messages Feb 19 19:30:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 054c4743-619d-965f-f786-cc0afc52d348 (at 10.9.101.68@o2ib4) Feb 19 19:30:56 fir-io1-s1 kernel: Lustre: Skipped 199 previous similar messages Feb 19 19:41:00 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0b309d49-2542-6f98-a52a-d9c9ba202a4b (at 10.8.23.33@o2ib6) Feb 19 19:41:00 fir-io1-s1 kernel: Lustre: Skipped 211 previous similar messages Feb 19 19:51:00 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f6f2dc0e-bc9f-2120-e971-29d8049b1247 (at 10.8.20.12@o2ib6) Feb 19 19:51:00 fir-io1-s1 kernel: Lustre: Skipped 378 previous similar messages Feb 19 20:01:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3dd7b3c3-369e-14ba-c881-c252e5dc17a0 (at 10.8.8.27@o2ib6) Feb 19 20:01:03 fir-io1-s1 kernel: Lustre: Skipped 257 previous similar messages Feb 19 20:11:11 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 03d523d0-d33d-b8d0-52c8-1ff235ea28e5 (at 10.9.102.16@o2ib4) Feb 19 20:11:11 fir-io1-s1 kernel: Lustre: Skipped 292 previous similar messages Feb 19 20:21:15 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to (at 10.9.104.62@o2ib4) Feb 19 20:21:15 fir-io1-s1 kernel: Lustre: Skipped 182 previous similar messages Feb 19 20:31:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Feb 19 20:31:17 fir-io1-s1 kernel: Lustre: Skipped 238 previous similar messages Feb 19 20:41:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2c25d9b5-7087-1cbe-8ca1-be689c9839e7 (at 10.8.27.24@o2ib6) Feb 19 20:41:28 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 19 20:52:09 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 2c25d9b5-7087-1cbe-8ca1-be689c9839e7 (at 10.8.27.24@o2ib6) Feb 19 20:52:09 fir-io1-s1 kernel: Lustre: Skipped 165 previous similar messages Feb 19 21:02:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2c25d9b5-7087-1cbe-8ca1-be689c9839e7 (at 10.8.27.24@o2ib6) Feb 19 21:02:15 fir-io1-s1 kernel: Lustre: Skipped 161 previous similar messages Feb 19 21:12:18 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 19 21:12:18 fir-io1-s1 kernel: Lustre: Skipped 219 previous similar messages Feb 19 21:22:21 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9ae76257-9e30-ddcb-f15b-a8db6da186f5 (at 10.8.8.6@o2ib6) Feb 19 21:22:21 fir-io1-s1 kernel: Lustre: Skipped 164 previous similar messages Feb 19 21:32:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0b309d49-2542-6f98-a52a-d9c9ba202a4b (at 10.8.23.33@o2ib6) Feb 19 21:32:22 fir-io1-s1 kernel: Lustre: Skipped 250 previous similar messages Feb 19 21:42:23 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 6b4e7742-58c9-f909-d7eb-e1d6e2f8c34e (at 10.8.31.9@o2ib6) Feb 19 21:42:23 fir-io1-s1 kernel: Lustre: Skipped 236 previous similar messages Feb 19 21:52:27 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.8.8.9@o2ib6) Feb 19 21:52:27 fir-io1-s1 kernel: Lustre: Skipped 284 previous similar messages Feb 19 22:02:28 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to (at 10.8.8.9@o2ib6) Feb 19 22:02:28 fir-io1-s1 kernel: Lustre: Skipped 287 previous similar messages Feb 19 22:12:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a70355fa-62e2-5007-8b78-0a9448aecdda (at 10.9.102.59@o2ib4) Feb 19 22:12:35 fir-io1-s1 kernel: Lustre: Skipped 291 previous similar messages Feb 19 22:22:39 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 57bd9565-249c-08fa-b75e-115d9c0f2fee (at 10.9.104.26@o2ib4) Feb 19 22:22:39 fir-io1-s1 kernel: Lustre: Skipped 282 previous similar messages Feb 19 22:32:55 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to e033b2fb-58ee-ad20-dbe1-c069873ac977 (at 10.9.101.47@o2ib4) Feb 19 22:32:55 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Feb 19 22:42:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d4ede191-33ac-db3d-8e23-e76bd511a700 (at 10.8.28.3@o2ib6) Feb 19 22:42:57 fir-io1-s1 kernel: Lustre: Skipped 204 previous similar messages Feb 19 22:53:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1c418863-78cc-8f23-893e-27e5ce2dfd94 (at 10.9.101.71@o2ib4) Feb 19 22:53:06 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 19 23:03:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 19 23:03:19 fir-io1-s1 kernel: Lustre: Skipped 207 previous similar messages Feb 19 23:13:31 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 92747d3f-03b1-884a-8015-40ea9f51416f (at 10.8.2.26@o2ib6) Feb 19 23:13:31 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 19 23:23:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.24.20@o2ib6) Feb 19 23:23:35 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 19 23:33:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 19 23:33:38 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 19 23:43:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 19 23:43:38 fir-io1-s1 kernel: Lustre: Skipped 271 previous similar messages Feb 19 23:53:38 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 2c25d9b5-7087-1cbe-8ca1-be689c9839e7 (at 10.8.27.24@o2ib6) Feb 19 23:53:38 fir-io1-s1 kernel: Lustre: Skipped 286 previous similar messages Feb 20 00:03:39 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to bdfd7f2b-f57d-9f82-b295-bd88659acc70 (at 10.9.105.24@o2ib4) Feb 20 00:03:39 fir-io1-s1 kernel: Lustre: Skipped 333 previous similar messages Feb 20 00:13:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0b309d49-2542-6f98-a52a-d9c9ba202a4b (at 10.8.23.33@o2ib6) Feb 20 00:13:57 fir-io1-s1 kernel: Lustre: Skipped 286 previous similar messages Feb 20 00:24:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 63d43612-6a45-cc62-c015-db6d91359b53 (at 10.9.104.32@o2ib4) Feb 20 00:24:00 fir-io1-s1 kernel: Lustre: Skipped 215 previous similar messages Feb 20 00:34:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e391b7dc-837c-aad7-6ebd-5ea1c73131db (at 10.8.18.24@o2ib6) Feb 20 00:34:08 fir-io1-s1 kernel: Lustre: Skipped 311 previous similar messages Feb 20 00:44:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) Feb 20 00:44:16 fir-io1-s1 kernel: Lustre: Skipped 310 previous similar messages Feb 20 00:54:23 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 04bbddc7-37a9-9b79-7fa8-c451901e5d15 (at 10.9.101.67@o2ib4) Feb 20 00:54:23 fir-io1-s1 kernel: Lustre: Skipped 281 previous similar messages Feb 20 01:04:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 77a364b6-c2d7-052b-2332-0b2d1cfdace4 (at 10.9.106.54@o2ib4) Feb 20 01:04:35 fir-io1-s1 kernel: Lustre: Skipped 282 previous similar messages Feb 20 01:14:36 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to ec7c15c1-1122-48df-e09c-cacc05cb75a8 (at 10.8.1.15@o2ib6) Feb 20 01:14:36 fir-io1-s1 kernel: Lustre: Skipped 203 previous similar messages Feb 20 01:24:41 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to e265f84a-d19d-6fce-343c-d86c6eba2d5b (at 10.8.29.3@o2ib6) Feb 20 01:24:41 fir-io1-s1 kernel: Lustre: Skipped 190 previous similar messages Feb 20 01:34:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 194ae90b-dda5-aeec-e623-aa1c27f6c383 (at 10.8.17.21@o2ib6) Feb 20 01:34:41 fir-io1-s1 kernel: Lustre: Skipped 316 previous similar messages Feb 20 01:43:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 418725ea-30b2-5f58-ef6b-1a8d6c02ebe0 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987822e51800, cur 1550655789 expire 1550655639 last 1550655562 Feb 20 01:43:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 01:44:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 20 01:44:45 fir-io1-s1 kernel: Lustre: Skipped 245 previous similar messages Feb 20 01:54:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0405d36b-1dfe-417d-33da-88f65ca0bd9f (at 10.8.28.4@o2ib6) Feb 20 01:54:58 fir-io1-s1 kernel: Lustre: Skipped 208 previous similar messages Feb 20 02:04:59 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to b1c1324f-29ea-b1e6-d3ab-a5c4feafdaa0 (at 10.8.22.3@o2ib6) Feb 20 02:04:59 fir-io1-s1 kernel: Lustre: Skipped 330 previous similar messages Feb 20 02:14:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 14aae05e-d3ff-54ad-8b93-c5dd42954ce5 (at 10.8.23.31@o2ib6) Feb 20 02:14:59 fir-io1-s1 kernel: Lustre: Skipped 374 previous similar messages Feb 20 02:24:59 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b2364f4b-9129-81e8-7e2f-15aa4210b663 (at 10.9.107.12@o2ib4) Feb 20 02:24:59 fir-io1-s1 kernel: Lustre: Skipped 339 previous similar messages Feb 20 02:35:04 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9042fb3a-c0ab-6915-0268-4626f11a023e (at 10.9.106.45@o2ib4) Feb 20 02:35:04 fir-io1-s1 kernel: Lustre: Skipped 371 previous similar messages Feb 20 02:45:08 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 121431d1-d11c-6cf2-7fd0-eaf8c3ca6b3e (at 10.9.103.12@o2ib4) Feb 20 02:45:08 fir-io1-s1 kernel: Lustre: Skipped 312 previous similar messages Feb 20 02:55:08 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) Feb 20 02:55:08 fir-io1-s1 kernel: Lustre: Skipped 320 previous similar messages Feb 20 03:05:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 20 03:05:08 fir-io1-s1 kernel: Lustre: Skipped 356 previous similar messages Feb 20 03:15:14 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 0f04a117-405d-119b-b6bd-8127d1f76e2a (at 10.8.20.2@o2ib6) Feb 20 03:15:14 fir-io1-s1 kernel: Lustre: Skipped 354 previous similar messages Feb 20 03:25:15 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 154f3468-6cd2-c082-14ee-143b859d4abb (at 10.8.17.18@o2ib6) Feb 20 03:25:15 fir-io1-s1 kernel: Lustre: Skipped 431 previous similar messages Feb 20 03:35:16 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 5dafe1c8-3c93-e104-2f19-ffcaca2d90cd (at 10.8.16.5@o2ib6) Feb 20 03:35:16 fir-io1-s1 kernel: Lustre: Skipped 397 previous similar messages Feb 20 03:45:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 5dafe1c8-3c93-e104-2f19-ffcaca2d90cd (at 10.8.16.5@o2ib6) Feb 20 03:45:17 fir-io1-s1 kernel: Lustre: Skipped 273 previous similar messages Feb 20 03:55:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0dbe37ca-2471-ddda-9bbd-b589c5cc0a2b (at 10.8.22.11@o2ib6) Feb 20 03:55:18 fir-io1-s1 kernel: Lustre: Skipped 358 previous similar messages Feb 20 04:05:20 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 20 04:05:20 fir-io1-s1 kernel: Lustre: Skipped 405 previous similar messages Feb 20 04:15:21 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Feb 20 04:15:21 fir-io1-s1 kernel: Lustre: Skipped 311 previous similar messages Feb 20 04:25:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to d99aa6c5-95ff-be26-f78a-b1cfe9fb5439 (at 10.9.101.70@o2ib4) Feb 20 04:25:30 fir-io1-s1 kernel: Lustre: Skipped 256 previous similar messages Feb 20 04:35:30 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 1220ec8d-e345-b247-0664-9ffbca04ef6f (at 10.9.104.9@o2ib4) Feb 20 04:35:30 fir-io1-s1 kernel: Lustre: Skipped 343 previous similar messages Feb 20 04:45:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 995d8de4-2fcc-bd82-b265-a0369659d360 (at 10.8.23.6@o2ib6) Feb 20 04:45:35 fir-io1-s1 kernel: Lustre: Skipped 311 previous similar messages Feb 20 04:46:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8e2d5e99-b397-a377-5023-c35c0e7c8405 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c532400, cur 1550666776 expire 1550666626 last 1550666549 Feb 20 04:46:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 04:55:40 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to eff4e35a-40a8-7f40-f399-3a3a0d536dc6 (at 10.9.103.41@o2ib4) Feb 20 04:55:40 fir-io1-s1 kernel: Lustre: Skipped 292 previous similar messages Feb 20 05:05:41 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 7d78b5a7-dae3-eca3-5a98-d1b9fe987149 (at 10.8.17.22@o2ib6) Feb 20 05:05:41 fir-io1-s1 kernel: Lustre: Skipped 271 previous similar messages Feb 20 05:15:43 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to b59a0d33-8556-89d2-09a5-3aa1a07e86fa (at 10.9.102.19@o2ib4) Feb 20 05:15:43 fir-io1-s1 kernel: Lustre: Skipped 333 previous similar messages Feb 20 05:25:44 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Feb 20 05:25:44 fir-io1-s1 kernel: Lustre: Skipped 244 previous similar messages Feb 20 05:35:48 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9fdeb7e4-dbe7-36a2-5704-e3cbd89d3a9c (at 10.9.104.25@o2ib4) Feb 20 05:35:48 fir-io1-s1 kernel: Lustre: Skipped 255 previous similar messages Feb 20 05:42:11 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 38e37557-4ae8-2b15-496c-670c02253b85 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786881800, cur 1550670131 expire 1550669981 last 1550669904 Feb 20 05:42:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 05:45:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 20 05:45:54 fir-io1-s1 kernel: Lustre: Skipped 262 previous similar messages Feb 20 05:55:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 20 05:55:54 fir-io1-s1 kernel: Lustre: Skipped 302 previous similar messages Feb 20 06:05:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 20 06:05:56 fir-io1-s1 kernel: Lustre: Skipped 305 previous similar messages Feb 20 06:15:58 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 96dc3f28-35a7-d1a0-d554-ac4259066293 (at 10.8.25.15@o2ib6) Feb 20 06:15:58 fir-io1-s1 kernel: Lustre: Skipped 376 previous similar messages Feb 20 06:25:58 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a1a32c6e-a603-c7ad-bbfa-4137583a5bae (at 10.8.17.20@o2ib6) Feb 20 06:25:58 fir-io1-s1 kernel: Lustre: Skipped 280 previous similar messages Feb 20 06:35:58 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 0f04a117-405d-119b-b6bd-8127d1f76e2a (at 10.8.20.2@o2ib6) Feb 20 06:35:58 fir-io1-s1 kernel: Lustre: Skipped 312 previous similar messages Feb 20 06:46:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 20 06:46:08 fir-io1-s1 kernel: Lustre: Skipped 293 previous similar messages Feb 20 06:56:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 20 06:56:09 fir-io1-s1 kernel: Lustre: Skipped 331 previous similar messages Feb 20 07:06:16 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 83fafa06-153a-a1fd-dd02-4b2d3ef6c90e (at 10.9.114.2@o2ib4) Feb 20 07:06:16 fir-io1-s1 kernel: Lustre: Skipped 427 previous similar messages Feb 20 07:16:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 20 07:16:21 fir-io1-s1 kernel: Lustre: Skipped 390 previous similar messages Feb 20 07:22:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 88af1be2-2241-75bc-2ae8-74aa833ad663 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15a2400, cur 1550676167 expire 1550676017 last 1550675940 Feb 20 07:22:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 07:22:49 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 88af1be2-2241-75bc-2ae8-74aa833ad663 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480eeb1800, cur 1550676169 expire 1550676019 last 1550675942 Feb 20 07:26:21 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to f1a84f43-a319-5e01-ac8c-7daa09178d6b (at 10.8.21.15@o2ib6) Feb 20 07:26:21 fir-io1-s1 kernel: Lustre: Skipped 424 previous similar messages Feb 20 07:36:34 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Feb 20 07:36:34 fir-io1-s1 kernel: Lustre: Skipped 353 previous similar messages Feb 20 07:46:38 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 9dd791fc-5e27-d5c0-d08d-b2cd561ae98d (at 10.8.30.34@o2ib6) Feb 20 07:46:38 fir-io1-s1 kernel: Lustre: Skipped 321 previous similar messages Feb 20 07:56:39 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 92c5a2a2-e228-d051-8b1a-a4a6b8577967 (at 10.9.102.41@o2ib4) Feb 20 07:56:39 fir-io1-s1 kernel: Lustre: Skipped 287 previous similar messages Feb 20 08:06:43 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 7b159774-9739-84d0-f0ac-fcc62a72d585 (at 10.8.19.6@o2ib6) Feb 20 08:06:43 fir-io1-s1 kernel: Lustre: Skipped 252 previous similar messages Feb 20 08:16:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 96dc3f28-35a7-d1a0-d554-ac4259066293 (at 10.8.25.15@o2ib6) Feb 20 08:16:56 fir-io1-s1 kernel: Lustre: Skipped 242 previous similar messages Feb 20 08:26:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 4199656a-fc55-4ace-07e9-a689e8e8d80b (at 10.8.10.7@o2ib6) Feb 20 08:26:56 fir-io1-s1 kernel: Lustre: Skipped 287 previous similar messages Feb 20 08:37:14 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 89311c25-5811-466c-95ed-d7a183bd4753 (at 10.9.113.15@o2ib4) Feb 20 08:37:14 fir-io1-s1 kernel: Lustre: Skipped 223 previous similar messages Feb 20 08:47:18 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Feb 20 08:47:18 fir-io1-s1 kernel: Lustre: Skipped 226 previous similar messages Feb 20 08:57:21 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9959e7c4-f852-ddb1-c97a-0e5563751bfc (at 10.8.28.12@o2ib6) Feb 20 08:57:21 fir-io1-s1 kernel: Lustre: Skipped 266 previous similar messages Feb 20 09:07:24 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to f3c26b94-2261-c91e-b422-79918936510b (at 10.9.115.4@o2ib4) Feb 20 09:07:24 fir-io1-s1 kernel: Lustre: Skipped 293 previous similar messages Feb 20 09:13:30 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7f026092-c1eb-c59f-c39b-5e4767d59132 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867f8208000, cur 1550682810 expire 1550682660 last 1550682583 Feb 20 09:13:30 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 20 09:17:26 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 2a27f76c-78b6-7e1a-cff3-64717b5ae1ff (at 10.9.106.59@o2ib4) Feb 20 09:17:26 fir-io1-s1 kernel: Lustre: Skipped 277 previous similar messages Feb 20 09:27:30 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 890f9dc9-b9bc-0354-4c1a-b7392d8a9570 (at 10.8.19.5@o2ib6) Feb 20 09:27:30 fir-io1-s1 kernel: Lustre: Skipped 314 previous similar messages Feb 20 09:37:31 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 82ffae03-c02c-86e8-2dc8-ed4f97ac9c9d (at 10.8.25.9@o2ib6) Feb 20 09:37:31 fir-io1-s1 kernel: Lustre: Skipped 406 previous similar messages Feb 20 09:59:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 958dbe37-44f4-f3c8-99a0-45b8f975616f (at 10.9.101.60@o2ib4) Feb 20 09:59:50 fir-io1-s1 kernel: Lustre: Skipped 3724 previous similar messages Feb 20 10:51:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 20 10:51:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:01:59 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fd3e8a17-a540-ab0b-fc84-50ecec44025b (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98749bde9000, cur 1550689319 expire 1550689169 last 1550689092 Feb 20 11:01:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:02:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 20 11:02:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:09:56 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client df1db331-667f-674d-0d2f-0045b18fdfbc (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387bac00, cur 1550689796 expire 1550689646 last 1550689569 Feb 20 11:09:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:09:58 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client df1db331-667f-674d-0d2f-0045b18fdfbc (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801245c00, cur 1550689798 expire 1550689648 last 1550689571 Feb 20 11:09:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 20 11:10:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 20 11:10:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:16:22 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9c65ba9d-2b07-fa6a-de7a-3139912893e0 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872d8edfc00, cur 1550690182 expire 1550690032 last 1550689955 Feb 20 11:16:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 20 11:16:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:26:08 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5e1bc973-c5bd-9412-e91b-bacbe7d8161e (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fae79400, cur 1550690768 expire 1550690618 last 1550690541 Feb 20 11:26:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:29:56 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9e1a92b2-3516-6436-5e9c-334e833902d5 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848000ea000, cur 1550690996 expire 1550690846 last 1550690769 Feb 20 11:29:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:36:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 20 11:36:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:37:48 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3713f7a7-255a-d76e-e866-1a94b4308c52 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed02000, cur 1550691468 expire 1550691318 last 1550691241 Feb 20 11:37:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:38:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 20 11:38:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:44:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 20 11:44:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:52:59 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8bfc05a4-0174-119a-738b-c93e09d812ff (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868d9faa000, cur 1550692379 expire 1550692229 last 1550692152 Feb 20 11:52:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 11:53:12 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8bfc05a4-0174-119a-738b-c93e09d812ff (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f7f800, cur 1550692392 expire 1550692242 last 1550692165 Feb 20 11:53:12 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 20 12:01:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 958dbe37-44f4-f3c8-99a0-45b8f975616f (at 10.9.101.60@o2ib4) Feb 20 12:01:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 12:04:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 958dbe37-44f4-f3c8-99a0-45b8f975616f (at 10.9.101.60@o2ib4) Feb 20 12:04:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 12:14:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 958dbe37-44f4-f3c8-99a0-45b8f975616f (at 10.9.101.60@o2ib4) Feb 20 12:14:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 12:17:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 20 12:17:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 12:26:23 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550694376/real 1550694376] req@ffff986dbfb75a00 x1624935477125104/t0(0) o104->fir-OST000a@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550694383 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 20 12:26:23 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 20 12:26:30 fir-io1-s1 kernel: Lustre: 96250:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550694383/real 1550694383] req@ffff9874e242f200 x1624935477125072/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550694390 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 12:26:44 fir-io1-s1 kernel: Lustre: 96250:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550694397/real 1550694397] req@ffff9874e242f200 x1624935477125072/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550694404 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 12:26:44 fir-io1-s1 kernel: Lustre: 96250:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 20 12:27:05 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550694418/real 1550694418] req@ffff986dbfb75a00 x1624935477125104/t0(0) o104->fir-OST000a@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550694425 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 12:27:05 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 20 12:27:40 fir-io1-s1 kernel: Lustre: 96250:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550694453/real 1550694453] req@ffff9874e242f200 x1624935477125072/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550694460 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 12:27:40 fir-io1-s1 kernel: Lustre: 96250:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 20 12:28:50 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550694523/real 1550694523] req@ffff986dbfb75a00 x1624935477125104/t0(0) o104->fir-OST000a@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550694530 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 12:28:50 fir-io1-s1 kernel: LustreError: 96250:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.114.3@o2ib4) failed to reply to blocking AST (req@ffff9874e242f200 x1624935477125072 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff984ec5d4ad00/0x49e185e9bdd9f0d4 lrc: 4/0,0 mode: PW/PW res: [0x6c0000400:0x172c67:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.9.114.3@o2ib4 remote: 0x95da51f3243c51a9 expref: 9 pid: 96929 timeout: 1042424 lvb_type: 0 Feb 20 12:28:50 fir-io1-s1 kernel: LustreError: 96250:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Feb 20 12:28:50 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.9.114.3@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Feb 20 12:28:50 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 20 12:28:50 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.9.114.3@o2ib4 ns: filter-fir-OST0000_UUID lock: ffff984ec5d4ad00/0x49e185e9bdd9f0d4 lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x172c67:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.9.114.3@o2ib4 remote: 0x95da51f3243c51a9 expref: 10 pid: 96929 timeout: 0 lvb_type: 0 Feb 20 12:28:50 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Feb 20 12:28:50 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Feb 20 12:29:58 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b78dd3c0-9fb2-b77d-d068-950356aeea71 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bc6dc00, cur 1550694598 expire 1550694448 last 1550694371 Feb 20 12:29:58 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 20 12:30:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b78dd3c0-9fb2-b77d-d068-950356aeea71 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987021d06400, cur 1550694601 expire 1550694451 last 1550694374 Feb 20 12:37:59 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 064df9b0-4227-5b19-c99f-3ad8e8b9cca5 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d2be43000, cur 1550695079 expire 1550694929 last 1550694852 Feb 20 12:37:59 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 20 12:42:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 20 12:42:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 12:52:44 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3e85655b-122d-661a-cdd4-e5163468b09a (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849be958400, cur 1550695964 expire 1550695814 last 1550695737 Feb 20 12:52:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 12:54:00 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) in 225 seconds. I think it's dead, and I am evicting it. exp ffff98677f472800, cur 1550696040 expire 1550695890 last 1550695815 Feb 20 12:54:00 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 20 13:17:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dcd406ad-ffdd-a7c9-489f-309957a1236e (at 10.8.15.7@o2ib6) Feb 20 13:17:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:17:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca1af5b2-4b74-b03d-4a2b-13a823b2dc8f (at 10.8.15.10@o2ib6) Feb 20 13:17:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:17:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7cee7bf7-9aa1-cc50-5aed-b23b669bf632 (at 10.8.15.2@o2ib6) Feb 20 13:17:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:19:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d2da3ac-e307-036e-f8a3-99f8b3ab4ed7 (at 10.8.15.5@o2ib6) Feb 20 13:19:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:19:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd6b0907-bbf0-754e-ba62-411999a5fe50 (at 10.8.15.1@o2ib6) Feb 20 13:19:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:20:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0cda1ca9-b849-4b59-7ce7-48abe2de3c2e (at 10.8.15.6@o2ib6) Feb 20 13:20:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:21:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 142a6e04-872a-1c7a-e7fe-9168d1c1b90d (at 10.8.9.5@o2ib6) Feb 20 13:21:36 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 20 13:22:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d7ed545-667f-2ef8-6bba-6c20aaec9c9f (at 10.8.14.9@o2ib6) Feb 20 13:22:38 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 20 13:24:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3dbbb061-d93c-c7e8-88c8-e262ff513397 (at 10.8.14.6@o2ib6) Feb 20 13:24:27 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 20 13:26:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 20 13:26:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:31:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9cc869db-86b5-83f4-3d1d-91c30da3920c (at 10.8.14.1@o2ib6) Feb 20 13:31:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:37:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 665b3c07-f583-a6a2-a031-36b81104a696 (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3af400, cur 1550698669 expire 1550698519 last 1550698442 Feb 20 13:37:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:37:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 665b3c07-f583-a6a2-a031-36b81104a696 (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c0e800, cur 1550698677 expire 1550698527 last 1550698450 Feb 20 13:37:57 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 20 13:43:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7b7e02d1-56ce-1646-fdf2-fbf074562774 (at 10.8.17.29@o2ib6) Feb 20 13:43:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:48:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3253f743-c741-9d7e-206f-a11dc62cfe9e (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784bbc800, cur 1550699284 expire 1550699134 last 1550699057 Feb 20 13:48:04 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 20 13:53:04 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3dead388-794f-90e5-9820-d4ab88063b1a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9873d22ebc00, cur 1550699584 expire 1550699434 last 1550699357 Feb 20 13:53:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:57:08 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0000: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 20 13:57:08 fir-io1-s1 kernel: Lustre: Skipped 25 previous similar messages Feb 20 13:57:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fb567489-c55d-642a-6271-722e1d06c9c7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9873d22ea400, cur 1550699857 expire 1550699707 last 1550699630 Feb 20 13:57:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 13:57:59 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 35 seconds Feb 20 13:57:59 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Feb 20 13:58:03 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Feb 20 13:58:03 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Feb 20 13:58:24 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 15 seconds Feb 20 13:58:24 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 20 13:58:29 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Feb 20 13:58:29 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Feb 20 13:58:35 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Feb 20 13:58:35 fir-io1-s1 kernel: Lustre: Skipped 13 previous similar messages Feb 20 13:58:49 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST000a: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Feb 20 13:58:49 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Feb 20 14:01:19 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0003-lwp-OST0000: This client was evicted by fir-MDT0003; in progress operations using this service will fail. Feb 20 14:01:19 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 20 14:01:45 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2aa82e99-1b3b-ccc9-52a7-4260b6507ba1 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9875017cb400, cur 1550700105 expire 1550699955 last 1550699878 Feb 20 14:01:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 14:02:00 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1520350 to 0xc80000402:1520385 Feb 20 14:02:00 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1520611 to 0x6c0000400:1520673 Feb 20 14:02:00 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1520257 to 0x8c0000402:1520289 Feb 20 14:02:00 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1520497 to 0x5c0000400:1520513 Feb 20 14:02:00 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1520446 to 0xc40000402:1520481 Feb 20 14:02:00 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1520778 to 0x580000400:1520801 Feb 20 14:03:55 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1043759 to 0x0:1043777 Feb 20 14:03:55 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1043509 to 0x0:1043553 Feb 20 14:03:55 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1044300 to 0x0:1044321 Feb 20 14:03:55 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1043394 to 0x0:1043425 Feb 20 14:03:55 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1044077 to 0x0:1044097 Feb 20 14:03:55 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1043663 to 0x0:1043681 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:260727 to 0x6c0000401:260769 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:260614 to 0x8c0000400:260641 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:260908 to 0x5c0000401:260929 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:260673 to 0xc40000400:260705 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:260803 to 0xc80000400:260833 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:261084 to 0x580000401:261121 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:457244 to 0x6c0000402:457281 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:457100 to 0xc40000401:457121 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:458514 to 0x5c0000402:458529 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:457211 to 0x8c0000401:457249 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:457194 to 0xc80000401:457217 Feb 20 14:07:02 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:458295 to 0x580000402:458337 Feb 20 14:36:31 fir-io1-s1 kernel: LustreError: 96444:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1483110(3580262) req@ffff984832b4b050 x1626024111114928/t0(0) o4->bfc9bb61-2244-2013-4f41-81b58e7a13ed@10.8.15.1@o2ib6:150/0 lens 488/448 e 1 to 0 dl 1550702200 ref 1 fl Interpret:/0/0 rc 0/0 Feb 20 14:36:31 fir-io1-s1 kernel: Lustre: fir-OST000a: Bulk IO write error with bfc9bb61-2244-2013-4f41-81b58e7a13ed (at 10.8.15.1@o2ib6), client will retry: rc = -110 Feb 20 14:38:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bfc9bb61-2244-2013-4f41-81b58e7a13ed (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98748726cc00, cur 1550702333 expire 1550702183 last 1550702106 Feb 20 14:38:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 14:38:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bfc9bb61-2244-2013-4f41-81b58e7a13ed (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811fd5800, cur 1550702336 expire 1550702186 last 1550702109 Feb 20 14:38:56 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 20 14:39:06 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client bfc9bb61-2244-2013-4f41-81b58e7a13ed (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98748726a400, cur 1550702346 expire 1550702196 last 1550702119 Feb 20 14:39:17 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client bfc9bb61-2244-2013-4f41-81b58e7a13ed (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98748726c800, cur 1550702357 expire 1550702207 last 1550702130 Feb 20 14:43:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 4cba41b5-7f88-73cd-f04a-6f2c37ad6f8c (at 10.9.101.60@o2ib4) reconnecting Feb 20 14:43:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 958dbe37-44f4-f3c8-99a0-45b8f975616f (at 10.9.101.60@o2ib4) Feb 20 14:43:16 fir-io1-s1 kernel: Lustre: Skipped 58 previous similar messages Feb 20 14:43:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 20 15:21:50 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550704903/real 1550704903] req@ffff984e6ca58600 x1624935552447280/t0(0) o106->fir-OST0000@10.8.9.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550704910 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 20 15:21:50 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 20 15:22:11 fir-io1-s1 kernel: Lustre: 96404:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550704924/real 1550704924] req@ffff9853bd3b4800 x1624935552447312/t0(0) o106->fir-OST0006@10.8.9.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550704931 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 15:22:11 fir-io1-s1 kernel: Lustre: 96404:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 20 15:22:46 fir-io1-s1 kernel: Lustre: 96570:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550704959/real 1550704959] req@ffff984fcd4ef200 x1624935552447296/t0(0) o106->fir-OST0004@10.8.9.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550704966 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 15:22:46 fir-io1-s1 kernel: Lustre: 96570:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 20 15:22:55 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client aff71b74-7050-1a79-ef86-3b2a0fea26d1 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984830f6b400, cur 1550704975 expire 1550704825 last 1550704748 Feb 20 15:46:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5511cb54-6709-809b-e9b2-444936d394ba (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bd0b400, cur 1550706383 expire 1550706233 last 1550706156 Feb 20 15:46:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 15:46:32 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5511cb54-6709-809b-e9b2-444936d394ba (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bd08400, cur 1550706392 expire 1550706242 last 1550706165 Feb 20 15:46:32 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 20 15:47:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 20 15:47:24 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 20 16:45:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Feb 20 16:45:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 16:46:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d9016de5-f918-5f84-9c31-6a1d3f97f1df (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c348dc800, cur 1550709987 expire 1550709837 last 1550709760 Feb 20 16:48:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d7ed545-667f-2ef8-6bba-6c20aaec9c9f (at 10.8.14.9@o2ib6) Feb 20 16:48:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 16:49:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 722fc89e-4fb6-0519-04d0-92f8091a9aa0 (at 10.8.14.5@o2ib6) Feb 20 16:49:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 16:50:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7b7e02d1-56ce-1646-fdf2-fbf074562774 (at 10.8.17.29@o2ib6) Feb 20 16:50:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 17:20:05 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 76037383-80fb-843b-ef8b-f5c8469efbde (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9870d327d400, cur 1550712005 expire 1550711855 last 1550711778 Feb 20 17:20:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 17:20:06 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 76037383-80fb-843b-ef8b-f5c8469efbde (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986cb8e12400, cur 1550712006 expire 1550711856 last 1550711779 Feb 20 17:20:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 76037383-80fb-843b-ef8b-f5c8469efbde (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9870d327a800, cur 1550712016 expire 1550711866 last 1550711789 Feb 20 17:20:16 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 20 17:20:21 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 76037383-80fb-843b-ef8b-f5c8469efbde (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9870d327e000, cur 1550712021 expire 1550711871 last 1550711794 Feb 20 17:33:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 387d59f0-6bba-bf1b-3af3-8003b959c70f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986db3306c00, cur 1550712831 expire 1550712681 last 1550712604 Feb 20 17:34:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 387d59f0-6bba-bf1b-3af3-8003b959c70f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986db3302400, cur 1550712845 expire 1550712695 last 1550712618 Feb 20 17:51:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 20 17:51:49 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 20 18:17:30 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 031d42d0-894f-b37f-ebbd-45aa637dedc9 (at 10.9.107.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987836514000, cur 1550715450 expire 1550715300 last 1550715223 Feb 20 18:17:30 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 20 18:27:02 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client fb1af9ac-8ab3-6e5f-bd29-0417933be014 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867869bec00, cur 1550716022 expire 1550715872 last 1550715795 Feb 20 18:27:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 18:50:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 031d42d0-894f-b37f-ebbd-45aa637dedc9 (at 10.9.107.33@o2ib4) Feb 20 18:50:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 20:04:15 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 142a6e04-872a-1c7a-e7fe-9168d1c1b90d (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a0a51000, cur 1550721855 expire 1550721705 last 1550721628 Feb 20 20:04:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 20:29:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 20 20:29:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 20:51:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Feb 20 20:51:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 20:52:12 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5654893d-4ac2-81f3-b87a-711dbf8073b8 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986025046c00, cur 1550724732 expire 1550724582 last 1550724505 Feb 20 20:52:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 20:52:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5654893d-4ac2-81f3-b87a-711dbf8073b8 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a0055c000, cur 1550724739 expire 1550724589 last 1550724512 Feb 20 20:52:19 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 20 21:17:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fda90133-1582-04d3-63c7-2c4e16be8b89 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986ee5fb9000, cur 1550726228 expire 1550726078 last 1550726001 Feb 20 21:17:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 20 21:17:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 21:19:41 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f4e92421-7a6d-ba38-fabf-70f33fa90cf2 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575883b400, cur 1550726381 expire 1550726231 last 1550726154 Feb 20 21:19:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 21:19:42 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f4e92421-7a6d-ba38-fabf-70f33fa90cf2 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582bba7c00, cur 1550726382 expire 1550726232 last 1550726155 Feb 20 21:19:42 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 20 22:16:19 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 77ca8972-8331-2cde-23bd-493fe3f03a97 (at 10.9.102.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576299e800, cur 1550729779 expire 1550729629 last 1550729552 Feb 20 22:16:19 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 20 23:04:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 20 23:04:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:05:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd262000, cur 1550732701 expire 1550732551 last 1550732474 Feb 20 23:05:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:24:04 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550733837/real 1550733837] req@ffff986e66b81200 x1624935769374704/t0(0) o106->fir-OST0004@10.8.18.35@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550733844 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 20 23:24:04 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 20 23:24:17 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3553a44c-9805-d124-ae0d-78e632c32008 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986025044800, cur 1550733857 expire 1550733707 last 1550733630 Feb 20 23:24:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:24:18 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550733851/real 1550733851] req@ffff986e66b81200 x1624935769374704/t0(0) o106->fir-OST0004@10.8.18.35@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550733858 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 20 23:24:18 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 20 23:24:32 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3553a44c-9805-d124-ae0d-78e632c32008 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e3f4ba400, cur 1550733872 expire 1550733722 last 1550733645 Feb 20 23:24:32 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 20 23:25:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Client a5728575-a4f7-6b0a-f0a9-44d0ea52ed96 (at 10.9.101.59@o2ib4) reconnecting Feb 20 23:25:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a5728575-a4f7-6b0a-f0a9-44d0ea52ed96 (at 10.9.101.59@o2ib4) Feb 20 23:25:42 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 20 23:35:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 20 23:35:10 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 20 23:39:34 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 09e96f09-88f3-17cc-ba26-88dee3b61d1c (at 10.8.7.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bf76c00, cur 1550734774 expire 1550734624 last 1550734547 Feb 20 23:41:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 09e96f09-88f3-17cc-ba26-88dee3b61d1c (at 10.8.7.12@o2ib6) Feb 20 23:41:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:42:58 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 914b784a-f82f-816e-bae7-a31bec3f7d6e (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fae7f400, cur 1550734978 expire 1550734828 last 1550734751 Feb 20 23:42:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:44:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 20 23:44:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:50:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 20 23:50:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:52:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d2da3ac-e307-036e-f8a3-99f8b3ab4ed7 (at 10.8.15.5@o2ib6) Feb 20 23:52:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:53:12 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 800fbfbd-9c33-798d-fb23-c4bc100ac7d6 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9870d3279400, cur 1550735592 expire 1550735442 last 1550735365 Feb 20 23:53:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 20 23:53:19 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 800fbfbd-9c33-798d-fb23-c4bc100ac7d6 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9875017c8c00, cur 1550735599 expire 1550735449 last 1550735372 Feb 21 00:05:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7cee7bf7-9aa1-cc50-5aed-b23b669bf632 (at 10.8.15.2@o2ib6) Feb 21 00:05:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 00:08:03 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 92e52d6e-0f08-d7ee-f73b-8bcd137777a3 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986eb3656400, cur 1550736483 expire 1550736333 last 1550736256 Feb 21 00:08:03 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 00:08:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 92e52d6e-0f08-d7ee-f73b-8bcd137777a3 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986eb3651000, cur 1550736484 expire 1550736334 last 1550736257 Feb 21 00:12:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7d772057-b33e-cbfd-263f-3fc1563e2cc1 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9863d7956400, cur 1550736760 expire 1550736610 last 1550736533 Feb 21 00:12:40 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 00:13:56 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1ca532b7-eecd-d2f1-a5f7-7049bc97bfdf (at 10.8.15.2@o2ib6) in 203 seconds. I think it's dead, and I am evicting it. exp ffff98692abdb000, cur 1550736836 expire 1550736686 last 1550736633 Feb 21 00:13:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 00:19:03 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f6fb8f82-2642-b787-3aab-1e0d1a47ae41 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986c95f97400, cur 1550737143 expire 1550736993 last 1550736916 Feb 21 00:19:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 00:19:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 21 00:19:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 00:31:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 00:31:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 00:55:34 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 7581ace3-cddd-3af2-5012-429ebd16e5ec (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98692abdc000, cur 1550739334 expire 1550739184 last 1550739107 Feb 21 00:55:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 01:25:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 01:25:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 01:25:44 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0007_UUID: not available for connect from 10.8.3.11@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Feb 21 01:32:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c78a40f2-4226-71b2-feff-43fa99ec3ad6 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0df3000, cur 1550741571 expire 1550741421 last 1550741344 Feb 21 01:32:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 02:13:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 02:13:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 02:30:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 21 02:30:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 02:38:03 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4d43b2c0-bbb9-b61a-e2e8-00470b2e66bb (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848324a5000, cur 1550745483 expire 1550745333 last 1550745256 Feb 21 02:38:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 02:38:04 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4d43b2c0-bbb9-b61a-e2e8-00470b2e66bb (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987131f56c00, cur 1550745484 expire 1550745334 last 1550745257 Feb 21 02:38:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 02:57:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 21 02:57:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 03:28:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4db9b02e-6d4b-af15-85f8-029390d826b5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98692abd9400, cur 1550748528 expire 1550748378 last 1550748301 Feb 21 03:29:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 21 03:29:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 03:48:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0402b0ad-eee7-b57a-176e-56f933780ea1 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986c92205c00, cur 1550749698 expire 1550749548 last 1550749471 Feb 21 03:48:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 03:48:32 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0402b0ad-eee7-b57a-176e-56f933780ea1 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9869cde29000, cur 1550749712 expire 1550749562 last 1550749485 Feb 21 03:48:32 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 21 03:48:33 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0402b0ad-eee7-b57a-176e-56f933780ea1 (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9869cde28c00, cur 1550749713 expire 1550749563 last 1550749486 Feb 21 03:48:33 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 21 03:49:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 21 03:49:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 03:54:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 21 03:54:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 04:03:13 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550750586/real 1550750586] req@ffff986e351dad00 x1624935857695360/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550750593 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 04:03:13 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 04:03:20 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550750593/real 1550750593] req@ffff986e351dad00 x1624935857695360/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550750600 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 04:03:27 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550750600/real 1550750600] req@ffff986e351dad00 x1624935857695360/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550750607 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 04:03:41 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550750614/real 1550750614] req@ffff986e351dad00 x1624935857695360/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550750621 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 04:03:41 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 04:04:02 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550750635/real 1550750635] req@ffff986e351dad00 x1624935857695360/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550750642 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 04:04:02 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 21 04:04:37 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550750670/real 1550750670] req@ffff986e351dad00 x1624935857695360/t0(0) o104->fir-OST0000@10.9.114.3@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1550750677 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 04:04:37 fir-io1-s1 kernel: Lustre: 96279:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 21 04:05:41 fir-io1-s1 kernel: LustreError: 96279:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.114.3@o2ib4) failed to reply to blocking AST (req@ffff986e351dad00 x1624935857695360 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff9853a72de540/0x49e185e9d4611d8b lrc: 4/0,0 mode: PW/PW res: [0x6c0000400:0x18410a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.9.114.3@o2ib4 remote: 0x95c0b80af628714b expref: 157 pid: 96922 timeout: 1098634 lvb_type: 0 Feb 21 04:05:41 fir-io1-s1 kernel: LustreError: 96279:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 21 04:05:41 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.9.114.3@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Feb 21 04:05:41 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 21 04:05:41 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 155s: evicting client at 10.9.114.3@o2ib4 ns: filter-fir-OST0000_UUID lock: ffff9853a72de540/0x49e185e9d4611d8b lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x18410a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.9.114.3@o2ib4 remote: 0x95c0b80af628714b expref: 158 pid: 96922 timeout: 0 lvb_type: 0 Feb 21 04:05:41 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Feb 21 04:06:30 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3e1f68b0-5fdc-0720-c864-d9264dd3e3fa (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985acab6b800, cur 1550750790 expire 1550750640 last 1550750563 Feb 21 04:06:30 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 21 04:06:38 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3e1f68b0-5fdc-0720-c864-d9264dd3e3fa (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9862a52b4c00, cur 1550750798 expire 1550750648 last 1550750571 Feb 21 04:22:53 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 77a364b6-c2d7-052b-2332-0b2d1cfdace4 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00ee800, cur 1550751773 expire 1550751623 last 1550751546 Feb 21 04:22:53 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 04:22:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 77a364b6-c2d7-052b-2332-0b2d1cfdace4 (at 10.9.106.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a5d400, cur 1550751775 expire 1550751625 last 1550751548 Feb 21 04:25:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 77a364b6-c2d7-052b-2332-0b2d1cfdace4 (at 10.9.106.54@o2ib4) Feb 21 04:25:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 05:14:36 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3235e068-73c7-8b5b-3b93-ef27294ce41e (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986995ad3400, cur 1550754876 expire 1550754726 last 1550754649 Feb 21 05:14:36 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 05:14:55 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3235e068-73c7-8b5b-3b93-ef27294ce41e (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9863d7951c00, cur 1550754895 expire 1550754745 last 1550754668 Feb 21 05:14:55 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 21 05:15:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 05:15:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 05:24:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5a422c02-4043-2fc0-d019-94ad240ed756 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986112874000, cur 1550755464 expire 1550755314 last 1550755237 Feb 21 05:24:24 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 05:24:33 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5a422c02-4043-2fc0-d019-94ad240ed756 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985acab68c00, cur 1550755473 expire 1550755323 last 1550755246 Feb 21 06:03:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 06:03:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 06:12:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0e10cfa2-0a0c-f553-2597-0a0934dbe8b1 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009fe000, cur 1550758324 expire 1550758174 last 1550758097 Feb 21 06:12:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 06:12:05 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0e10cfa2-0a0c-f553-2597-0a0934dbe8b1 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865f0ecac00, cur 1550758325 expire 1550758175 last 1550758098 Feb 21 06:12:05 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 06:12:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0e10cfa2-0a0c-f553-2597-0a0934dbe8b1 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986aa0ab1000, cur 1550758343 expire 1550758193 last 1550758116 Feb 21 06:14:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 06:14:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 06:20:00 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6a03ee68-75aa-ae17-47a5-d0022f59673a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f60f80c00, cur 1550758800 expire 1550758650 last 1550758573 Feb 21 06:21:16 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a791c090-23de-3a42-6b27-309dd21341a6 (at 10.8.3.11@o2ib6) in 192 seconds. I think it's dead, and I am evicting it. exp ffff9848324a1c00, cur 1550758876 expire 1550758726 last 1550758684 Feb 21 06:21:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 06:21:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a791c090-23de-3a42-6b27-309dd21341a6 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986bc4a2ac00, cur 1550758897 expire 1550758747 last 1550758670 Feb 21 06:21:37 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 21 06:22:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 06:22:50 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 21 06:25:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 21 06:25:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 06:30:57 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a8d686d9-6499-00df-544b-0a95c5d2451e (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987264ffe000, cur 1550759457 expire 1550759307 last 1550759230 Feb 21 06:30:57 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 06:33:14 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 21 06:33:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 07:25:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9587535e-c171-b5d2-73a1-572d4528f02f (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986be1f89c00, cur 1550762716 expire 1550762566 last 1550762489 Feb 21 07:25:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 07:25:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 07:25:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 08:15:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 38748444-40b6-6835-7b4e-70f11820c260 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986986a00000, cur 1550765714 expire 1550765564 last 1550765487 Feb 21 08:15:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 08:15:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 08:15:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 08:31:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Feb 21 08:31:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 08:43:45 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3a5f59d3-00cc-4b9f-9cbe-3f9597d8adb2 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984996392800, cur 1550767425 expire 1550767275 last 1550767198 Feb 21 08:43:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 08:45:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 21 08:45:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 08:46:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 08:46:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 08:49:11 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 664b2324-0e06-55b7-c477-2a04c4427134 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9862a032c800, cur 1550767751 expire 1550767601 last 1550767524 Feb 21 08:49:11 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 09:45:01 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8d7447da-a916-3b69-a4a6-96dac754d087 (at 10.9.0.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00ea000, cur 1550771101 expire 1550770951 last 1550770874 Feb 21 09:45:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 09:48:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ba1e1fbb-a72d-16b5-8ad9-93cddc13d3a5 (at 10.9.0.63@o2ib4) Feb 21 09:48:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 09:49:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a6ac00, cur 1550771386 expire 1550771236 last 1550771159 Feb 21 09:49:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 09:51:02 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 088f04e7-ef11-eb35-8a4d-c6daa7c5fcdc (at 10.8.0.65@o2ib6) in 217 seconds. I think it's dead, and I am evicting it. exp ffff986785d7d800, cur 1550771462 expire 1550771312 last 1550771245 Feb 21 09:51:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 09:51:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 088f04e7-ef11-eb35-8a4d-c6daa7c5fcdc (at 10.8.0.65@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9878332c9800, cur 1550771472 expire 1550771322 last 1550771245 Feb 21 09:51:12 fir-io1-s1 kernel: Lustre: Skipped 14 previous similar messages Feb 21 10:11:39 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c2c7af35-528a-d5b1-7a14-ba23d739ebf6 (at 10.9.0.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98725f9c4000, cur 1550772699 expire 1550772549 last 1550772472 Feb 21 10:11:39 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 21 10:25:08 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 40711f2a-43e9-3ae3-a82f-765d3df901bc (at 10.8.6.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762c73800, cur 1550773508 expire 1550773358 last 1550773281 Feb 21 10:25:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 10:29:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Feb 21 10:29:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 10:33:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c62ed1f1-3793-7a2f-05b0-df79888e04df (at 10.8.0.65@o2ib6) Feb 21 10:33:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 10:35:59 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 70fba524-1486-a60e-6bc9-6cdcc41e09a1 (at 10.9.113.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a61400, cur 1550774159 expire 1550774009 last 1550773932 Feb 21 10:35:59 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 10:36:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Feb 21 10:36:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 10:41:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 67525666-a063-cbc2-2269-3d7cda07ffe3 (at 10.9.0.81@o2ib4) Feb 21 10:41:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 10:55:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 85cfcf77-29ac-d755-b385-af543ebdafc6 (at 10.9.101.31@o2ib4) Feb 21 10:55:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:00:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4aaa1c3-b0e3-e4ce-f076-edf6ec6bd632 (at 10.8.4.34@o2ib6) Feb 21 11:00:25 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 11:00:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 9ca88d6a-bbf2-01c1-11ca-c3f6715dc691 (at 10.8.2.12@o2ib6) Feb 21 11:00:57 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 11:02:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 40711f2a-43e9-3ae3-a82f-765d3df901bc (at 10.8.6.17@o2ib6) Feb 21 11:02:19 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 21 11:09:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 70fba524-1486-a60e-6bc9-6cdcc41e09a1 (at 10.9.113.13@o2ib4) Feb 21 11:09:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:28:25 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 05d8ca20-c344-bd96-2627-67efcff39463 (at 10.9.101.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987833e91800, cur 1550777305 expire 1550777155 last 1550777078 Feb 21 11:28:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:34:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fc1bc7ce-5d0d-3dd1-97c3-5b7a2adca326 (at 10.9.0.2@o2ib4) Feb 21 11:34:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:35:21 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 71af6a2a-6813-06f2-0088-f01c2d7ce339 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986fb4e55000, cur 1550777721 expire 1550777571 last 1550777494 Feb 21 11:35:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:36:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0935f868-2cb6-7be4-32e4-6f8243d37d7c (at 10.9.0.1@o2ib4) Feb 21 11:36:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:38:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 45a162ef-0459-9991-8b7a-2377aa3c8022 (at 10.9.101.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b16800, cur 1550777931 expire 1550777781 last 1550777704 Feb 21 11:38:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:39:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 45a162ef-0459-9991-8b7a-2377aa3c8022 (at 10.9.101.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c533800, cur 1550777941 expire 1550777791 last 1550777714 Feb 21 11:39:01 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 11:45:52 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5c0efcec-5002-c530-2a40-b31a9d26affb (at 10.8.3.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762497400, cur 1550778352 expire 1550778202 last 1550778125 Feb 21 11:48:33 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client af97372a-4a25-ee38-8463-1889e32712ed (at 10.9.112.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576184c800, cur 1550778513 expire 1550778363 last 1550778286 Feb 21 11:48:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 11:59:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.7@o2ib4) Feb 21 11:59:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:00:36 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bf617e6a-e2c4-2972-fed8-58b2cb638da2 (at 10.9.101.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784ac4400, cur 1550779236 expire 1550779086 last 1550779009 Feb 21 12:00:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:00:39 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bf617e6a-e2c4-2972-fed8-58b2cb638da2 (at 10.9.101.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784ac6000, cur 1550779239 expire 1550779089 last 1550779012 Feb 21 12:00:39 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 21 12:01:02 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client bf617e6a-e2c4-2972-fed8-58b2cb638da2 (at 10.9.101.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd0f2c00, cur 1550779262 expire 1550779112 last 1550779035 Feb 21 12:01:02 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 21 12:01:52 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c85fef69-2d23-ee70-7307-437e48666801 (at 10.8.3.1@o2ib6) in 199 seconds. I think it's dead, and I am evicting it. exp ffff98575eaa7800, cur 1550779312 expire 1550779162 last 1550779113 Feb 21 12:02:05 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550779318/real 1550779318] req@ffff987256babf00 x1624935936078768/t0(0) o106->fir-OST000a@10.8.3.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550779325 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 12:02:05 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 21 12:02:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c85fef69-2d23-ee70-7307-437e48666801 (at 10.8.3.1@o2ib6) in 225 seconds. I think it's dead, and I am evicting it. exp ffff986816f3d800, cur 1550779338 expire 1550779188 last 1550779113 Feb 21 12:02:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 12:09:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45a162ef-0459-9991-8b7a-2377aa3c8022 (at 10.9.101.32@o2ib4) Feb 21 12:09:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:18:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5c0efcec-5002-c530-2a40-b31a9d26affb (at 10.8.3.8@o2ib6) Feb 21 12:18:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:19:20 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 804b7596-eb7f-ca8b-a586-917ca597cb67 (at 10.8.3.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848831e4c00, cur 1550780360 expire 1550780210 last 1550780133 Feb 21 12:22:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.112.5@o2ib4) Feb 21 12:22:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:24:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 657c4193-574d-7ac0-e927-0684806e81d2 (at 10.9.101.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985769988c00, cur 1550780688 expire 1550780538 last 1550780461 Feb 21 12:24:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:32:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 12:32:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:35:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bf617e6a-e2c4-2972-fed8-58b2cb638da2 (at 10.9.101.25@o2ib4) Feb 21 12:35:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:38:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 31e62a46-be09-fbbd-f18a-19afc191851d (at 10.8.3.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786970c00, cur 1550781516 expire 1550781366 last 1550781289 Feb 21 12:38:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:46:09 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 69023f0b-28b9-1d08-eb1b-f3a097e42672 (at 10.8.3.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784bbac00, cur 1550781969 expire 1550781819 last 1550781742 Feb 21 12:46:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:46:15 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 69023f0b-28b9-1d08-eb1b-f3a097e42672 (at 10.8.3.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678685e000, cur 1550781975 expire 1550781825 last 1550781748 Feb 21 12:46:15 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 21 12:49:25 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 83aae8f5-2878-9c05-a916-30575aca7f13 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801511400, cur 1550782165 expire 1550782015 last 1550781938 Feb 21 12:49:25 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 12:49:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 21 12:49:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:51:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 804b7596-eb7f-ca8b-a586-917ca597cb67 (at 10.8.3.21@o2ib6) Feb 21 12:51:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 12:56:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 657c4193-574d-7ac0-e927-0684806e81d2 (at 10.9.101.42@o2ib4) Feb 21 12:56:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:07:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 13ccf343-7ccf-96f3-9354-06c5a49c0c5d (at 10.8.2.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da0c00, cur 1550783242 expire 1550783092 last 1550783015 Feb 21 13:07:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:07:27 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 13ccf343-7ccf-96f3-9354-06c5a49c0c5d (at 10.8.2.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e52800, cur 1550783247 expire 1550783097 last 1550783020 Feb 21 13:07:32 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 13ccf343-7ccf-96f3-9354-06c5a49c0c5d (at 10.8.2.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e51c00, cur 1550783252 expire 1550783102 last 1550783025 Feb 21 13:08:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 834ac188-c073-46e6-5e30-84e7d37c751c (at 10.9.101.16@o2ib4) in 153 seconds. I think it's dead, and I am evicting it. exp ffff98480073b000, cur 1550783318 expire 1550783168 last 1550783165 Feb 21 13:08:38 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 13:08:48 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 834ac188-c073-46e6-5e30-84e7d37c751c (at 10.9.101.16@o2ib4) in 170 seconds. I think it's dead, and I am evicting it. exp ffff98480073c000, cur 1550783328 expire 1550783178 last 1550783158 Feb 21 13:08:48 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 21 13:10:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.3.27@o2ib6) Feb 21 13:10:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:11:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a1e9342e-47a1-556e-4e94-1554686a12b0 (at 10.8.4.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4c2000, cur 1550783499 expire 1550783349 last 1550783272 Feb 21 13:11:39 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 13:17:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 69023f0b-28b9-1d08-eb1b-f3a097e42672 (at 10.8.3.14@o2ib6) Feb 21 13:17:39 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 13:18:50 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6da7219f-139c-94b7-44d5-3e81a85a248c (at 10.8.6.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877c7d56400, cur 1550783930 expire 1550783780 last 1550783703 Feb 21 13:18:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:21:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2470bb85-005a-2d15-0ba1-2b7c597370c1 (at 10.9.101.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b3c400, cur 1550784097 expire 1550783947 last 1550783870 Feb 21 13:21:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:24:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ecc5b628-5efd-fbfd-7392-a1abe17de407 (at 10.9.101.36@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f0ca2b400, cur 1550784280 expire 1550784130 last 1550784053 Feb 21 13:24:40 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 13:27:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 21 13:27:28 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 21 13:28:59 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fb0e351b-5ab8-b43d-813c-60db20cd78c1 (at 10.9.101.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf29000, cur 1550784539 expire 1550784389 last 1550784312 Feb 21 13:28:59 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 13:38:24 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client fec00937-b402-85ef-c96f-4ad570d39702 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98716550bc00, cur 1550785104 expire 1550784954 last 1550784877 Feb 21 13:38:24 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 13:41:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 13ccf343-7ccf-96f3-9354-06c5a49c0c5d (at 10.8.2.11@o2ib6) Feb 21 13:41:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:42:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 21 13:42:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:42:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 834ac188-c073-46e6-5e30-84e7d37c751c (at 10.9.101.16@o2ib4) Feb 21 13:42:34 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 13:45:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a1e9342e-47a1-556e-4e94-1554686a12b0 (at 10.8.4.21@o2ib6) Feb 21 13:45:21 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 21 13:48:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6da7219f-139c-94b7-44d5-3e81a85a248c (at 10.8.6.21@o2ib6) Feb 21 13:48:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:52:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 47ce4289-7e25-8c66-9590-6b36cdee8e22 (at 10.9.101.1@o2ib4) Feb 21 13:52:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:55:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ecc5b628-5efd-fbfd-7392-a1abe17de407 (at 10.9.101.36@o2ib4) Feb 21 13:55:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 13:58:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8b9d39cd-5973-2d32-c0c6-445ad6e2af9d (at 10.8.6.15@o2ib6) Feb 21 13:58:31 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 14:00:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fb0e351b-5ab8-b43d-813c-60db20cd78c1 (at 10.9.101.45@o2ib4) Feb 21 14:00:34 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 14:03:22 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 405a922c-acd7-226c-f53f-840190be85ca (at 10.9.115.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b266c00, cur 1550786602 expire 1550786452 last 1550786375 Feb 21 14:03:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 14:03:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.44@o2ib4) Feb 21 14:03:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 14:18:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5c820789-8e24-ad89-a0df-b1759dd671b0 (at 10.9.101.56@o2ib4) Feb 21 14:18:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 14:18:32 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bf62c7df-fe55-c26d-63f8-adbf89ed0ecb (at 10.8.3.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849be95ec00, cur 1550787512 expire 1550787362 last 1550787285 Feb 21 14:18:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 14:22:28 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867869e0400, cur 1550787748 expire 1550787598 last 1550787521 Feb 21 14:22:28 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 14:23:03 fir-io1-s1 kernel: Lustre: 94629:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787776/real 1550787776] req@ffff983e28ba5400 x1624935945840608/t0(0) o106->fir-OST0002@10.8.7.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787783 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 14:23:03 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787776/real 1550787776] req@ffff98428689fb00 x1624935945840592/t0(0) o106->fir-OST000a@10.8.7.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787783 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 14:23:03 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 14:23:10 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787783/real 1550787783] req@ffff98428689fb00 x1624935945840592/t0(0) o106->fir-OST000a@10.8.7.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787790 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 14:23:17 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787790/real 1550787790] req@ffff98428689fb00 x1624935945840592/t0(0) o106->fir-OST000a@10.8.7.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787797 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 14:23:17 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 14:23:24 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787797/real 1550787797] req@ffff98428689fb00 x1624935945840592/t0(0) o106->fir-OST000a@10.8.7.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787804 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 14:23:24 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 21 14:23:38 fir-io1-s1 kernel: Lustre: 94629:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787811/real 1550787811] req@ffff983e28ba5400 x1624935945840608/t0(0) o106->fir-OST0002@10.8.7.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787818 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 14:23:38 fir-io1-s1 kernel: Lustre: 94629:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 21 14:23:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 37b4854b-e93e-85a5-e644-9d0c6be8cc09 (at 10.8.2.29@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff98576f3ae400, cur 1550787824 expire 1550787674 last 1550787598 Feb 21 14:23:44 fir-io1-s1 kernel: Lustre: Skipped 152 previous similar messages Feb 21 14:26:25 fir-io1-s1 kernel: Lustre: 94629:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787978/real 1550787978] req@ffff98428689ec00 x1624935945847760/t0(0) o106->fir-OST0006@10.8.18.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787985 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 14:26:25 fir-io1-s1 kernel: Lustre: 96270:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550787978/real 1550787978] req@ffff983c468cc800 x1624935945847728/t0(0) o106->fir-OST0004@10.8.18.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550787985 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 14:26:25 fir-io1-s1 kernel: Lustre: 96270:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 21 14:26:25 fir-io1-s1 kernel: Lustre: 94629:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 14:26:36 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 531debd8-6dde-8101-c0c5-b86120a894b1 (at 10.8.18.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987832655c00, cur 1550787996 expire 1550787846 last 1550787769 Feb 21 14:26:36 fir-io1-s1 kernel: Lustre: Skipped 710 previous similar messages Feb 21 14:27:00 fir-io1-s1 kernel: Lustre: 96270:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550788013/real 1550788013] req@ffff983c468cc800 x1624935945847728/t0(0) o106->fir-OST0004@10.8.18.2@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550788020 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 14:27:00 fir-io1-s1 kernel: Lustre: 96270:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Feb 21 14:28:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 54b0700c-697e-0244-02c1-dbdca5773fee (at 10.8.27.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d64c400, cur 1550788088 expire 1550787938 last 1550787861 Feb 21 14:28:08 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 14:30:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 405a922c-acd7-226c-f53f-840190be85ca (at 10.9.115.9@o2ib4) Feb 21 14:30:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 14:31:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fa88db94-cc25-b94a-c1ba-db4210d375b6 (at 10.8.3.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483ed4a400, cur 1550788304 expire 1550788154 last 1550788077 Feb 21 14:31:44 fir-io1-s1 kernel: Lustre: Skipped 533 previous similar messages Feb 21 14:38:09 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1b2318e4-eaea-3698-adbd-cbdb84cddc5c (at 10.8.24.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857ac1d2800, cur 1550788689 expire 1550788539 last 1550788462 Feb 21 14:38:09 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 14:52:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25ed9ae1-b634-f924-a1f1-218edef29ff0 (at 10.9.101.52@o2ib4) Feb 21 14:52:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 14:55:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0afc86ec-aba3-e28e-ab47-9d17266465ab (at 10.8.21.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877c7f78000, cur 1550789749 expire 1550789599 last 1550789522 Feb 21 14:55:49 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 15:06:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dba0ac03-f756-7e65-44de-ab2d15018d1d (at 10.9.101.41@o2ib4) Feb 21 15:06:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 15:10:12 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2286c65a-f3fa-726e-64b2-7afa3a454d38 (at 10.8.3.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e44800, cur 1550790612 expire 1550790462 last 1550790385 Feb 21 15:10:12 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 15:27:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5da38a2b-1f85-f985-e647-807ffb38b26c (at 10.8.30.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c858c00, cur 1550791629 expire 1550791479 last 1550791402 Feb 21 15:27:09 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 15:33:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e033b2fb-58ee-ad20-dbe1-c069873ac977 (at 10.9.101.47@o2ib4) Feb 21 15:33:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 15:37:23 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0edeac5b-ee1e-024f-de97-9e0fc3efb1af (at 10.8.6.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283d9800, cur 1550792243 expire 1550792093 last 1550792016 Feb 21 15:37:23 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 15:47:03 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550792816/real 1550792816] req@ffff986e56a56f00 x1624935950722192/t0(0) o104->fir-OST0008@10.8.3.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550792823 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 15:47:03 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Feb 21 15:47:17 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550792830/real 1550792830] req@ffff986e56a56f00 x1624935950722192/t0(0) o104->fir-OST0008@10.8.3.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550792837 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 15:47:17 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 15:47:38 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550792851/real 1550792851] req@ffff986e56a56f00 x1624935950722192/t0(0) o104->fir-OST0008@10.8.3.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550792858 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 15:47:38 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 21 15:48:13 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550792886/real 1550792886] req@ffff986e56a56f00 x1624935950722192/t0(0) o104->fir-OST0008@10.8.3.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550792893 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 15:48:13 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 21 15:49:23 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550792956/real 1550792956] req@ffff986e56a56f00 x1624935950722192/t0(0) o104->fir-OST0008@10.8.3.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550792963 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 15:49:23 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 21 15:49:30 fir-io1-s1 kernel: LustreError: 96493:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.3@o2ib6) failed to reply to blocking AST (req@ffff986e56a56f00 x1624935950722192 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff984897aff080/0x49e185e9dc72e28e lrc: 4/0,0 mode: PW/PW res: [0xc80000402:0x189bae:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400010020 nid: 10.8.3.3@o2ib6 remote: 0xb12b797491038adf expref: 6 pid: 96524 timeout: 1140864 lvb_type: 0 Feb 21 15:49:30 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.3.3@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 21 15:49:30 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.3.3@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff984897aff080/0x49e185e9dc72e28e lrc: 3/0,0 mode: PW/PW res: [0xc80000402:0x189bae:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400010020 nid: 10.8.3.3@o2ib6 remote: 0xb12b797491038adf expref: 7 pid: 96524 timeout: 0 lvb_type: 0 Feb 21 15:49:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client cf9d17c1-1ffb-39b7-f814-1f005dbcb1a0 (at 10.9.101.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe8f1c00, cur 1550792996 expire 1550792846 last 1550792769 Feb 21 15:49:56 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 21 15:58:20 fir-io1-s1 kernel: Lustre: 96905:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550793493/real 1550793493] req@ffff983c468ce600 x1624935951438416/t0(0) o106->fir-OST0006@10.8.18.24@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550793500 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 15:58:20 fir-io1-s1 kernel: Lustre: 96905:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 21 16:01:12 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 400e2c70-3670-eb05-66c0-e754ea5cd280 (at 10.8.29.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e45800, cur 1550793672 expire 1550793522 last 1550793445 Feb 21 16:01:12 fir-io1-s1 kernel: Lustre: Skipped 58 previous similar messages Feb 21 16:02:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fda7a4af-47c0-0068-cddf-309c3a9c784c (at 10.9.101.13@o2ib4) Feb 21 16:02:45 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 21 16:02:56 fir-io1-s1 kernel: Lustre: 96249:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550793769/real 1550793769] req@ffff985185b23900 x1624935951746192/t0(0) o106->fir-OST0006@10.8.22.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550793776 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 16:02:56 fir-io1-s1 kernel: Lustre: 96364:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550793769/real 1550793769] req@ffff9855c7996300 x1624935951746208/t0(0) o106->fir-OST0008@10.8.22.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550793776 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 16:02:56 fir-io1-s1 kernel: Lustre: 96364:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Feb 21 16:04:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 21 16:04:25 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 16:04:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to efc6b332-a736-88e8-194a-588aa3e05348 (at 10.8.21.36@o2ib6) Feb 21 16:04:42 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 21 16:04:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6179b8f9-2b15-63f8-671c-eb8d9ecd9187 (at 10.8.16.3@o2ib6) Feb 21 16:04:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 16:04:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b9de1fd1-ccbe-721f-e4ab-c6e06447a81c (at 10.8.15.4@o2ib6) Feb 21 16:04:58 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Feb 21 16:05:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a397e427-2a18-94f0-0f06-4c6a9e455efa (at 10.8.18.8@o2ib6) Feb 21 16:05:08 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 21 16:05:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7d78b5a7-dae3-eca3-5a98-d1b9fe987149 (at 10.8.17.22@o2ib6) Feb 21 16:05:50 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 16:06:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ed72929c-d604-56ff-ec0a-5b1f9da0af84 (at 10.8.17.9@o2ib6) Feb 21 16:06:40 fir-io1-s1 kernel: Lustre: Skipped 101 previous similar messages Feb 21 16:08:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 103f9da3-989e-ad73-cfdb-75395d4c9148 (at 10.8.8.35@o2ib6) Feb 21 16:08:05 fir-io1-s1 kernel: Lustre: Skipped 123 previous similar messages Feb 21 16:10:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.17.26@o2ib6) Feb 21 16:10:21 fir-io1-s1 kernel: Lustre: Skipped 225 previous similar messages Feb 21 16:13:57 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e2c4a480-f1d1-2209-d809-5088ddc9ced3 (at 10.8.4.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987822e50800, cur 1550794437 expire 1550794287 last 1550794210 Feb 21 16:13:57 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 16:14:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9ad9001-d2d9-466a-37c8-3f54fd94183d (at 10.9.101.26@o2ib4) Feb 21 16:14:44 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Feb 21 16:23:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 23d2bcaf-f181-5f6e-6636-b07b46e525e0 (at 10.8.3.3@o2ib6) Feb 21 16:23:16 fir-io1-s1 kernel: Lustre: Skipped 407 previous similar messages Feb 21 16:31:07 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a253e634-6f73-cada-6654-0ab67e1de2bb (at 10.8.3.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a69400, cur 1550795467 expire 1550795317 last 1550795240 Feb 21 16:31:07 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 16:44:50 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f00399f0-46cf-c80f-b0b6-1b044a6fb9c6 (at 10.9.101.40@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801bff400, cur 1550796290 expire 1550796140 last 1550796063 Feb 21 16:44:50 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 16:46:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 64c2e10f-595e-bb8e-efbd-0544992f523c (at 10.9.101.49@o2ib4) Feb 21 16:46:33 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 21 16:55:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 74990111-11b6-f5c4-552b-3761c975ef1a (at 10.8.13.15@o2ib6) Feb 21 16:55:38 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 16:57:14 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0e1cc7ee-ac14-2533-62de-8aa817b3cbc6 (at 10.8.4.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857ac1d4000, cur 1550797034 expire 1550796884 last 1550796807 Feb 21 16:57:14 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 17:03:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e2c4a480-f1d1-2209-d809-5088ddc9ced3 (at 10.8.4.24@o2ib6) Feb 21 17:03:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 17:12:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1f01bd1a-f9b0-ca9d-a06a-d294a56e03aa (at 10.8.3.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857606b4800, cur 1550797946 expire 1550797796 last 1550797719 Feb 21 17:12:26 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 21 17:18:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f00399f0-46cf-c80f-b0b6-1b044a6fb9c6 (at 10.9.101.40@o2ib4) Feb 21 17:18:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 17:24:02 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c77ad9a7-9999-ae72-c251-ba6defdb48ac (at 10.8.24.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984f5e9d0800, cur 1550798642 expire 1550798492 last 1550798415 Feb 21 17:24:02 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 17:36:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b0f92329-0f7e-f3d9-c31b-8d79c2778f5e (at 10.8.30.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596b8400, cur 1550799415 expire 1550799265 last 1550799188 Feb 21 17:36:55 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Feb 21 17:38:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Feb 21 17:38:14 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 17:48:53 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5dee1cfe-a09b-9e59-8000-88c395f6096c (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984838c6e800, cur 1550800133 expire 1550799983 last 1550799906 Feb 21 17:48:53 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 21 17:49:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 17:49:42 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 21 18:00:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.29@o2ib4) Feb 21 18:00:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 18:04:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c5c1b991-8670-7a1c-e657-202610a30605 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ad1f800, cur 1550801063 expire 1550800913 last 1550800836 Feb 21 18:04:23 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 21 18:18:54 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ed2f9b2c-50f5-3c91-9334-1d613b5f5aeb (at 10.8.24.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678685fc00, cur 1550801934 expire 1550801784 last 1550801707 Feb 21 18:18:54 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 18:32:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c0b0af12-3020-f664-817f-9f5c8301ffef (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868adc00, cur 1550802731 expire 1550802581 last 1550802504 Feb 21 18:32:11 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 21 18:33:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 18:33:13 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 18:35:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 21 18:35:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 18:43:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client e9560223-f857-8af8-8e66-18924c1e4b0e (at 10.8.3.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987191c6c400, cur 1550803391 expire 1550803241 last 1550803164 Feb 21 18:43:11 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 18:47:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4944a2ce-d92c-784e-9536-c95b01530191 (at 10.9.101.48@o2ib4) Feb 21 18:47:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 18:52:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a253e634-6f73-cada-6654-0ab67e1de2bb (at 10.8.3.5@o2ib6) Feb 21 18:52:32 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 21 18:57:22 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f3bddf34-379e-90f9-b8a3-37dd2323157d (at 10.8.3.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c673c00, cur 1550804242 expire 1550804092 last 1550804015 Feb 21 18:57:22 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 19:09:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ced38ea1-bc1c-667b-da28-176debc6acc3 (at 10.8.3.36@o2ib6) Feb 21 19:09:29 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 21 19:13:17 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3f539eb7-bad7-deea-5d73-69ef13653da8 (at 10.8.26.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984810a28000, cur 1550805197 expire 1550805047 last 1550804970 Feb 21 19:13:17 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 19:30:49 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 11b17224-2f07-8a37-a442-a2272ea47d98 (at 10.8.6.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b88ac00, cur 1550806249 expire 1550806099 last 1550806022 Feb 21 19:30:49 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 19:33:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 173446cc-39b1-333f-81fc-6684fb678e20 (at 10.8.3.19@o2ib6) Feb 21 19:33:00 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 19:38:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cbc3c983-083d-dd6f-e278-5d22e24e864d (at 10.8.20.11@o2ib6) Feb 21 19:38:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 19:41:02 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 85810729-2f82-b4f2-1241-3806d86f03d3 (at 10.8.30.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756580800, cur 1550806862 expire 1550806712 last 1550806635 Feb 21 19:41:02 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 19:42:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 70c3879e-e355-8a86-c7f6-86725077b527 (at 10.8.3.28@o2ib6) Feb 21 19:42:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 19:48:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1eb50bff-8205-25b7-6e2a-3278212917f5 (at 10.8.3.26@o2ib6) Feb 21 19:48:17 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Feb 21 19:55:43 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3394282b-acd2-5166-a3e3-f942df4fe27c (at 10.8.12.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825fd000, cur 1550807743 expire 1550807593 last 1550807516 Feb 21 19:55:43 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 19:59:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 285f267a-e8b2-ef8e-69b5-0024d8da45d8 (at 10.8.12.14@o2ib6) Feb 21 19:59:54 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Feb 21 20:10:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 85810729-2f82-b4f2-1241-3806d86f03d3 (at 10.8.30.6@o2ib6) Feb 21 20:10:27 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 20:16:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 553f7c09-7b2f-bfaa-2ec4-3819c2429915 (at 10.8.13.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762a59400, cur 1550808960 expire 1550808810 last 1550808733 Feb 21 20:16:00 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 21 20:20:32 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 364c8566-8b44-8def-9ccb-d070a0bfcffe (at 10.8.26.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ea41800, cur 1550809232 expire 1550809082 last 1550809005 Feb 21 20:20:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 20:27:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0cda1ca9-b849-4b59-7ce7-48abe2de3c2e (at 10.8.15.6@o2ib6) Feb 21 20:27:03 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 21 20:35:07 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 903503fd-dde5-802d-335c-2751477db8c4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986781f74800, cur 1550810107 expire 1550809957 last 1550809880 Feb 21 20:35:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 20:41:04 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1cac0206-7dc8-7985-dbe6-f16507ebcfe0 (at 10.8.1.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f2400, cur 1550810464 expire 1550810314 last 1550810237 Feb 21 20:41:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 20:42:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 485a9b04-eba2-1548-66ce-7dbb0247de17 (at 10.8.20.15@o2ib6) in 207 seconds. I think it's dead, and I am evicting it. exp ffff985756f5f800, cur 1550810540 expire 1550810390 last 1550810333 Feb 21 20:42:20 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Feb 21 20:45:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 31f9f1e5-0053-c9bc-655f-d68cfd64847e (at 10.8.21.32@o2ib6) Feb 21 20:45:55 fir-io1-s1 kernel: Lustre: Skipped 46 previous similar messages Feb 21 20:55:47 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 71ef200a-3224-e9f5-baae-7d7ceae57d55 (at 10.8.6.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d507800, cur 1550811347 expire 1550811197 last 1550811120 Feb 21 20:55:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:00:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4cdb4c1f-c631-73f5-0cc6-576f1959d1eb (at 10.8.17.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762b64800, cur 1550811619 expire 1550811469 last 1550811392 Feb 21 21:00:19 fir-io1-s1 kernel: Lustre: Skipped 167 previous similar messages Feb 21 21:08:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5874979c-2d42-d4b8-b0f7-48cd970c494d (at 10.8.4.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867818aa800, cur 1550812093 expire 1550811943 last 1550811866 Feb 21 21:08:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:13:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45a12f91-6aa3-f0ae-a299-8aadf4c776a5 (at 10.8.17.5@o2ib6) Feb 21 21:13:12 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 21:16:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bab702f8-b44a-da13-8f71-e38d2f6bf022 (at 10.8.1.13@o2ib6) Feb 21 21:16:30 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Feb 21 21:21:31 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ce7c2edb-40f8-3a94-8d08-7f35c7e4a9ee (at 10.9.101.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786886000, cur 1550812891 expire 1550812741 last 1550812664 Feb 21 21:21:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:26:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 22505c78-b9c2-e28a-88c4-7dadc4be41e9 (at 10.9.101.28@o2ib4) Feb 21 21:26:50 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Feb 21 21:28:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client cd2f196e-9e42-81f8-3052-f3c348cb8b16 (at 10.8.20.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800f52000, cur 1550813328 expire 1550813178 last 1550813101 Feb 21 21:28:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:32:25 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bdf3d0f5-851d-26cd-ba90-9355de313856 (at 10.8.6.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762862000, cur 1550813545 expire 1550813395 last 1550813318 Feb 21 21:32:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:37:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client e70b2e25-d2b0-7198-e1d2-7be1db612168 (at 10.8.22.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678445b400, cur 1550813875 expire 1550813725 last 1550813648 Feb 21 21:37:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:37:56 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 763f631c-7b84-895c-764f-d88426b5fe26 (at 10.8.1.3@o2ib6) Feb 21 21:37:56 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 21:42:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 414dcc40-1a1f-dafe-b9b7-84383e8013e5 (at 10.8.4.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aabf800, cur 1550814126 expire 1550813976 last 1550813899 Feb 21 21:42:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:45:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f6844397-c58e-1716-47c7-98dc229eec16 (at 10.8.21.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e798c00, cur 1550814339 expire 1550814189 last 1550814112 Feb 21 21:45:39 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 21 21:48:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6f29d7ca-d9bc-eef1-1913-bbb7c0bca1a0 (at 10.8.1.5@o2ib6) Feb 21 21:48:04 fir-io1-s1 kernel: Lustre: Skipped 168 previous similar messages Feb 21 21:50:03 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 07910fca-ff29-cdaf-c7fe-7ea9a99e216e (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a5f800, cur 1550814603 expire 1550814453 last 1550814376 Feb 21 21:50:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 21:56:38 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 78f38211-beb8-aca3-b985-9281f7d5f62c (at 10.8.29.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00e8000, cur 1550814998 expire 1550814848 last 1550814771 Feb 21 21:56:38 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 21 21:58:41 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550815114/real 1550815114] req@ffff987578000f00 x1624936210341568/t0(0) o106->fir-OST0008@10.8.29.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550815121 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 21:58:41 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 114 previous similar messages Feb 21 21:59:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 664f5f86-7e37-c3dd-9009-3eec77c4bd45 (at 10.8.11.1@o2ib6) Feb 21 21:59:07 fir-io1-s1 kernel: Lustre: Skipped 380 previous similar messages Feb 21 21:59:51 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550815184/real 1550815184] req@ffff983ba41de900 x1624936210341584/t0(0) o106->fir-OST0002@10.8.29.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550815191 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 21:59:51 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 27 previous similar messages Feb 21 22:11:27 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e265f84a-d19d-6fce-343c-d86c6eba2d5b (at 10.8.29.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c85c000, cur 1550815887 expire 1550815737 last 1550815660 Feb 21 22:11:27 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 21 22:11:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fc60883d-f9c1-82aa-8312-f53a10d6b6ff (at 10.8.9.1@o2ib6) Feb 21 22:11:59 fir-io1-s1 kernel: Lustre: Skipped 215 previous similar messages Feb 21 22:13:18 fir-io1-s1 kernel: Lustre: 96266:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550815991/real 1550815991] req@ffff9872059a0600 x1624936238641040/t0(0) o106->fir-OST000a@10.8.29.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550815998 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 22:13:18 fir-io1-s1 kernel: Lustre: 96266:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 38 previous similar messages Feb 21 22:13:39 fir-io1-s1 kernel: Lustre: 97128:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550816012/real 1550816012] req@ffff9869f5ba3600 x1624936238640992/t0(0) o106->fir-OST0004@10.8.29.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550816019 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 22:13:39 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550816012/real 1550816012] req@ffff9876d3254200 x1624936238641024/t0(0) o106->fir-OST0008@10.8.29.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550816019 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 22:13:39 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 21 22:13:39 fir-io1-s1 kernel: Lustre: 97128:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 21 22:14:14 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550816047/real 1550816047] req@ffff9876d3254200 x1624936238641024/t0(0) o106->fir-OST0008@10.8.29.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550816054 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 22:14:14 fir-io1-s1 kernel: Lustre: 96504:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 21 22:22:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 884d46bb-c96d-5856-481d-822f553b82b2 (at 10.8.3.6@o2ib6) Feb 21 22:22:49 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Feb 21 22:29:52 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550816985/real 1550816985] req@ffff987542165a00 x1624936256108016/t0(0) o104->fir-OST0006@10.8.26.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550816992 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 21 22:29:52 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Feb 21 22:30:06 fir-io1-s1 kernel: Lustre: 94539:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550816999/real 1550816999] req@ffff984e6ca5cb00 x1624936256109024/t0(0) o104->fir-OST0000@10.8.26.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550817006 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 22:30:06 fir-io1-s1 kernel: Lustre: 94539:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 21 22:30:27 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550817020/real 1550817020] req@ffff987542161200 x1624936256109280/t0(0) o104->fir-OST0000@10.8.26.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550817027 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 22:30:27 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 21 22:31:02 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550817055/real 1550817055] req@ffff987542166000 x1624936256109568/t0(0) o104->fir-OST0008@10.8.26.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550817062 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 22:31:02 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 21 22:31:30 fir-io1-s1 kernel: LustreError: 94539:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.26.33@o2ib6) failed to reply to blocking AST (req@ffff984e6ca5cb00 x1624936256109024 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff98383db54a40/0x49e185e9e684943a lrc: 4/0,0 mode: PW/PW res: [0x6c0000402:0x903c4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.26.33@o2ib6 remote: 0x6b06eaa8ff1a328d expref: 14 pid: 96368 timeout: 1164934 lvb_type: 0 Feb 21 22:31:30 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.26.33@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 21 22:31:30 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.26.33@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff9840828a1f80/0x49e185e9e68717f4 lrc: 3/0,0 mode: PW/PW res: [0x6c0000402:0x903d5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.8.26.33@o2ib6 remote: 0x6b06eaa8ff1a5e93 expref: 15 pid: 96329 timeout: 0 lvb_type: 0 Feb 21 22:31:30 fir-io1-s1 kernel: LustreError: 94539:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 21 22:32:12 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550817125/real 1550817125] req@ffff987542165a00 x1624936256108016/t0(0) o104->fir-OST0006@10.8.26.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550817132 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 21 22:32:12 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 27 previous similar messages Feb 21 22:32:19 fir-io1-s1 kernel: LustreError: 96254:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.26.33@o2ib6) failed to reply to blocking AST (req@ffff987542166000 x1624936256109568 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff984154f61f80/0x49e185e9e6842cb4 lrc: 4/0,0 mode: PW/PW res: [0xc80000401:0x9037c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.26.33@o2ib6 remote: 0x6b06eaa8ff1a2f29 expref: 13 pid: 96253 timeout: 1165033 lvb_type: 0 Feb 21 22:32:19 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.26.33@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 21 22:32:19 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 21 22:32:19 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.26.33@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff98380ef0f080/0x49e185e9e682475a lrc: 3/0,0 mode: PW/PW res: [0xc40000401:0x90313:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.26.33@o2ib6 remote: 0x6b06eaa8ff1a19ea expref: 10 pid: 96583 timeout: 0 lvb_type: 0 Feb 21 22:32:19 fir-io1-s1 kernel: LustreError: 96254:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 21 22:33:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d12493fe-7ac4-1be6-bc85-09a57d42ebeb (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d41fc00, cur 1550817195 expire 1550817045 last 1550816968 Feb 21 22:33:15 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 22:34:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 22:34:00 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 22:34:31 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 23eb2845-7211-395f-8dcc-3218241cf5c6 (at 10.8.20.15@o2ib6) in 184 seconds. I think it's dead, and I am evicting it. exp ffff98581bfeac00, cur 1550817271 expire 1550817121 last 1550817087 Feb 21 22:34:31 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 21 22:45:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9065faec-fdd8-46fc-db53-5ff36e99d790 (at 10.8.26.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762a5e000, cur 1550817942 expire 1550817792 last 1550817715 Feb 21 22:45:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 21 22:47:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 21 22:47:19 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 21 22:53:02 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 77244e83-b250-397d-f179-1d8851db1705 (at 10.8.12.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d206400, cur 1550818382 expire 1550818232 last 1550818155 Feb 21 22:53:02 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 21 22:59:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 21 22:59:12 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 21 23:03:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d48251b2-974f-9e84-8e65-f892c6ead1ec (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785784000, cur 1550819031 expire 1550818881 last 1550818804 Feb 21 23:03:51 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Feb 21 23:10:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 21 23:10:21 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 21 23:15:09 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 70103d89-ec34-ce17-2e6c-cc604ff9ae8a (at 10.8.27.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848346a2000, cur 1550819709 expire 1550819559 last 1550819482 Feb 21 23:15:09 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 21 23:22:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 21 23:22:27 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 23:25:33 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 97ef30b3-4ac9-b614-cd55-284b5ff037ed (at 10.8.27.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b5d685400, cur 1550820333 expire 1550820183 last 1550820106 Feb 21 23:25:33 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 21 23:33:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8f86419d-3c7d-f8e0-fb5d-facc0f493f73 (at 10.8.27.31@o2ib6) Feb 21 23:33:10 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Feb 21 23:37:17 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d7ec95d2-bb29-cf12-9cec-599a5cfd1fc9 (at 10.8.23.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480ee88400, cur 1550821037 expire 1550820887 last 1550820810 Feb 21 23:37:17 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 21 23:44:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 70103d89-ec34-ce17-2e6c-cc604ff9ae8a (at 10.8.27.29@o2ib6) Feb 21 23:44:39 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 21 23:51:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f1a84f43-a319-5e01-ac8c-7daa09178d6b (at 10.8.21.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98681de46000, cur 1550821893 expire 1550821743 last 1550821666 Feb 21 23:51:33 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 21 23:54:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.27.30@o2ib6) Feb 21 23:54:55 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 22 00:02:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9be70e11-7c19-2eb9-f15c-fc93676b48c5 (at 10.8.20.15@o2ib6) in 210 seconds. I think it's dead, and I am evicting it. exp ffff98483b4e6400, cur 1550822560 expire 1550822410 last 1550822350 Feb 22 00:02:40 fir-io1-s1 kernel: Lustre: Skipped 77 previous similar messages Feb 22 00:06:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d7ec95d2-bb29-cf12-9cec-599a5cfd1fc9 (at 10.8.23.24@o2ib6) Feb 22 00:06:30 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 00:14:46 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e9f481fe-1440-9638-a19c-db89b2be27c3 (at 10.9.101.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800f56400, cur 1550823286 expire 1550823136 last 1550823059 Feb 22 00:14:46 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Feb 22 00:20:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f1a84f43-a319-5e01-ac8c-7daa09178d6b (at 10.8.21.15@o2ib6) Feb 22 00:20:33 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 00:27:19 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 94c2dba8-b225-c91a-753d-91a7f0495a0f (at 10.9.101.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987764851400, cur 1550824039 expire 1550823889 last 1550823812 Feb 22 00:27:19 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 00:31:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 430053da-1990-00e9-bf49-61b924573c3b (at 10.8.22.33@o2ib6) Feb 22 00:31:06 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 00:37:52 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e2d946d5-12d9-285b-e4ca-825370f3d1df (at 10.9.101.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985837fe4400, cur 1550824672 expire 1550824522 last 1550824445 Feb 22 00:37:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 00:45:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e9f481fe-1440-9638-a19c-db89b2be27c3 (at 10.9.101.17@o2ib4) Feb 22 00:45:45 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Feb 22 00:49:03 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 84f0fb31-db8b-0d43-a1cb-f5ecfec599af (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf2ec00, cur 1550825343 expire 1550825193 last 1550825116 Feb 22 00:49:03 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 00:58:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 94c2dba8-b225-c91a-753d-91a7f0495a0f (at 10.9.101.8@o2ib4) Feb 22 00:58:29 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Feb 22 00:59:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3d5ac7cc-5cb1-6aef-d2d6-2af538ce79c6 (at 10.8.25.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984f5a258c00, cur 1550825988 expire 1550825838 last 1550825761 Feb 22 00:59:48 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 01:08:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5291d87c-c332-89fe-37a2-5aad94038a93 (at 10.8.24.16@o2ib6) Feb 22 01:08:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 01:15:31 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 18f10ddf-d44e-cfab-f9e8-f3ef5580a393 (at 10.8.26.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857654f4000, cur 1550826931 expire 1550826781 last 1550826704 Feb 22 01:15:31 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Feb 22 01:18:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5b8aac1d-db3a-957a-49fa-fd5d736eb555 (at 10.8.27.13@o2ib6) Feb 22 01:18:58 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 01:26:38 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 81ece3a3-4ce6-c10f-0f1a-dfedb115d731 (at 10.8.21.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d505800, cur 1550827598 expire 1550827448 last 1550827371 Feb 22 01:26:38 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 01:29:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d5ac7cc-5cb1-6aef-d2d6-2af538ce79c6 (at 10.8.25.22@o2ib6) Feb 22 01:29:02 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Feb 22 01:36:46 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8e580007-1439-26bf-2cad-74288a308748 (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583baf0800, cur 1550828206 expire 1550828056 last 1550827979 Feb 22 01:36:46 fir-io1-s1 kernel: Lustre: Skipped 54 previous similar messages Feb 22 01:39:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f7d79af9-3424-b0d8-6dc6-f23e0df4e16a (at 10.8.6.32@o2ib6) Feb 22 01:39:53 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Feb 22 01:47:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3f2e0558-152b-3ca0-be2b-5080fc19b5c2 (at 10.8.21.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480f61a400, cur 1550828830 expire 1550828680 last 1550828603 Feb 22 01:47:10 fir-io1-s1 kernel: Lustre: Skipped 46 previous similar messages Feb 22 01:53:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.24.2@o2ib6) Feb 22 01:53:41 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 22 02:01:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6f460c24-f6dc-99fb-dece-05ba714311b0 (at 10.8.27.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a0598400, cur 1550829661 expire 1550829511 last 1550829434 Feb 22 02:01:01 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 02:05:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3364cbf9-a01c-3bb2-78b9-5e8955b36f20 (at 10.9.101.30@o2ib4) Feb 22 02:05:24 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 02:11:29 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 33893189-0abb-4d15-5624-4b6aa533cb69 (at 10.8.8.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848008f7000, cur 1550830289 expire 1550830139 last 1550830062 Feb 22 02:11:29 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 02:15:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3f2e0558-152b-3ca0-be2b-5080fc19b5c2 (at 10.8.21.14@o2ib6) Feb 22 02:15:56 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Feb 22 02:22:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c4f458ee-079e-1f6b-715d-4cc60d32c4b8 (at 10.8.11.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811215c00, cur 1550830933 expire 1550830783 last 1550830706 Feb 22 02:22:13 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 22 02:31:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6f460c24-f6dc-99fb-dece-05ba714311b0 (at 10.8.27.15@o2ib6) Feb 22 02:31:30 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 22 02:33:13 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 57269b50-0465-38e8-cbda-995a4a22296e (at 10.8.8.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768455c00, cur 1550831593 expire 1550831443 last 1550831366 Feb 22 02:33:13 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 02:42:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 33893189-0abb-4d15-5624-4b6aa533cb69 (at 10.8.8.4@o2ib6) Feb 22 02:42:37 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 02:44:08 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 609a75ae-bcd8-62a8-78c3-eceb01368f0a (at 10.8.22.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678677a000, cur 1550832248 expire 1550832098 last 1550832021 Feb 22 02:44:08 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 02:52:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7e080438-c60b-bb7f-7851-337ce8c3d22a (at 10.8.30.16@o2ib6) Feb 22 02:52:48 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 03:05:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 57269b50-0465-38e8-cbda-995a4a22296e (at 10.8.8.1@o2ib6) Feb 22 03:05:54 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 03:06:30 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 72ef71e7-ecde-a7f2-d85d-452a22011f5b (at 10.9.101.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877c6d87400, cur 1550833590 expire 1550833440 last 1550833363 Feb 22 03:06:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:10:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b29100f0-7f9b-f4f6-2c85-7505f2641dbf (at 10.8.6.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762b61000, cur 1550833823 expire 1550833673 last 1550833596 Feb 22 03:10:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:24:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 85dd464d-e86b-f56a-846e-fdb51d5f1d4c (at 10.8.21.13@o2ib6) Feb 22 03:24:04 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 03:34:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 18690fc5-fbdf-b1ea-b451-9b4b1123dbab (at 10.8.3.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784647400, cur 1550835250 expire 1550835100 last 1550835023 Feb 22 03:34:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:35:26 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client af14ffc8-eb6d-feed-8cd5-18f27d720d95 (at 10.8.13.7@o2ib6) in 178 seconds. I think it's dead, and I am evicting it. exp ffff984838275400, cur 1550835326 expire 1550835176 last 1550835148 Feb 22 03:35:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:37:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72ef71e7-ecde-a7f2-d85d-452a22011f5b (at 10.9.101.14@o2ib4) Feb 22 03:37:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:48:27 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c27b95f5-b24a-0587-3803-2e633f8c27fe (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576184e800, cur 1550836107 expire 1550835957 last 1550835880 Feb 22 03:48:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:50:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 03:50:35 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 22 03:51:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 7723713e-1b81-462e-47bc-22d65939e693 (at 10.8.21.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762495800, cur 1550836293 expire 1550836143 last 1550836066 Feb 22 03:51:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:52:49 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 23f9a8de-c6a0-76ea-9378-57e47966c02c (at 10.8.21.23@o2ib6) in 173 seconds. I think it's dead, and I am evicting it. exp ffff9867811fa800, cur 1550836369 expire 1550836219 last 1550836196 Feb 22 03:52:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 03:53:43 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 23f9a8de-c6a0-76ea-9378-57e47966c02c (at 10.8.21.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780866c00, cur 1550836423 expire 1550836273 last 1550836196 Feb 22 04:06:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 18690fc5-fbdf-b1ea-b451-9b4b1123dbab (at 10.8.3.35@o2ib6) Feb 22 04:06:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 04:15:31 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d19ea69a-0a10-0e0b-8ba2-f4e7c0778c3d (at 10.8.30.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848845ff800, cur 1550837731 expire 1550837581 last 1550837504 Feb 22 04:15:31 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 22 04:15:37 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d19ea69a-0a10-0e0b-8ba2-f4e7c0778c3d (at 10.8.30.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da2400, cur 1550837737 expire 1550837587 last 1550837510 Feb 22 04:15:37 fir-io1-s1 kernel: Lustre: Skipped 14 previous similar messages Feb 22 04:18:29 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ea827f53-1edf-c98c-6637-0c731c4e9044 (at 10.9.101.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043fcc00, cur 1550837909 expire 1550837759 last 1550837682 Feb 22 04:18:29 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 22 04:18:55 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ea827f53-1edf-c98c-6637-0c731c4e9044 (at 10.9.101.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c3a800, cur 1550837935 expire 1550837785 last 1550837708 Feb 22 04:18:55 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 22 04:20:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7723713e-1b81-462e-47bc-22d65939e693 (at 10.8.21.22@o2ib6) Feb 22 04:20:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 04:26:04 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client aa3b9181-076e-9d38-c41a-e93a0915658f (at 10.8.22.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480073fc00, cur 1550838364 expire 1550838214 last 1550838137 Feb 22 04:26:04 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 22 04:30:21 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e0872b2a-fa00-24f1-33af-f06a57058e2d (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785781800, cur 1550838621 expire 1550838471 last 1550838394 Feb 22 04:30:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 04:31:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 04:31:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 04:36:00 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683a53a000, cur 1550838960 expire 1550838810 last 1550838733 Feb 22 04:36:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 04:41:59 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 67f5c0a5-d6e0-715a-6a48-d2b2401623ab (at 10.9.101.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3abc00, cur 1550839319 expire 1550839169 last 1550839092 Feb 22 04:41:59 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 04:44:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d19ea69a-0a10-0e0b-8ba2-f4e7c0778c3d (at 10.8.30.3@o2ib6) Feb 22 04:44:31 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Feb 22 04:55:15 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bc4a40c8-15db-3b39-b53d-1c1fe1d7ad06 (at 10.8.30.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de4000, cur 1550840115 expire 1550839965 last 1550839888 Feb 22 04:55:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 04:55:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aa3b9181-076e-9d38-c41a-e93a0915658f (at 10.8.22.16@o2ib6) Feb 22 04:55:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 05:05:43 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c64427f6-8a09-6654-0dfd-900cb8162098 (at 10.8.25.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984bb4fe5c00, cur 1550840743 expire 1550840593 last 1550840516 Feb 22 05:05:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 05:05:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 05:05:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 05:23:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bc4a40c8-15db-3b39-b53d-1c1fe1d7ad06 (at 10.8.30.21@o2ib6) Feb 22 05:23:59 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 05:26:10 fir-io1-s1 kernel: Lustre: 96619:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550841963/real 1550841963] req@ffff984aa0615a00 x1624936467578368/t0(0) o104->fir-OST0000@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550841970 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 05:26:10 fir-io1-s1 kernel: Lustre: 96619:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 22 05:26:31 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550841984/real 1550841984] req@ffff9855c7991e00 x1624936467578608/t0(0) o104->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550841991 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 05:26:31 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 22 05:27:06 fir-io1-s1 kernel: Lustre: 96619:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550842019/real 1550842019] req@ffff984aa0615a00 x1624936467578368/t0(0) o104->fir-OST0000@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550842026 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 05:27:06 fir-io1-s1 kernel: Lustre: 96619:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 22 05:28:16 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550842089/real 1550842089] req@ffff9855c7991e00 x1624936467578608/t0(0) o104->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1550842096 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 05:28:16 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Feb 22 05:28:37 fir-io1-s1 kernel: LustreError: 96619:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff984aa0615a00 x1624936467578368 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff98573abf3840/0x49e185e9ecf22a58 lrc: 4/0,0 mode: PW/PW res: [0x6c0000402:0x97af1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.3.11@o2ib6 remote: 0xbef4d394a5dad8bb expref: 310 pid: 96904 timeout: 1190011 lvb_type: 0 Feb 22 05:28:37 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Feb 22 05:28:37 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 22 05:28:37 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff983804cdfbc0/0x49e185e9ecf21e26 lrc: 3/0,0 mode: PW/PW res: [0x8c0000401:0x97ae8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400010020 nid: 10.8.3.11@o2ib6 remote: 0xbef4d394a5dad684 expref: 292 pid: 49827 timeout: 0 lvb_type: 0 Feb 22 05:28:37 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Feb 22 05:28:37 fir-io1-s1 kernel: LustreError: 96619:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 22 05:29:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client dcdcb27f-727a-f68b-d987-dd889041dbd3 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480052d000, cur 1550842176 expire 1550842026 last 1550841949 Feb 22 05:29:36 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 22 05:34:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f1ecbb01-7b7c-667a-3a9f-343625d79564 (at 10.8.22.13@o2ib6) Feb 22 05:34:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 05:42:32 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 811fd087-e95e-3790-3b1f-499e971702dc (at 10.8.12.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499a65f800, cur 1550842952 expire 1550842802 last 1550842725 Feb 22 05:42:32 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 22 05:45:12 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550843105/real 1550843105] req@ffff9840e2d01200 x1624936473053920/t0(0) o106->fir-OST0004@10.8.15.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550843112 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 05:45:12 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 22 05:45:33 fir-io1-s1 kernel: Lustre: 96269:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550843126/real 1550843126] req@ffff986889423900 x1624936473053936/t0(0) o106->fir-OST0006@10.8.15.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550843133 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 05:45:33 fir-io1-s1 kernel: Lustre: 96269:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Feb 22 05:46:08 fir-io1-s1 kernel: Lustre: 96562:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550843161/real 1550843161] req@ffff985b64804b00 x1624936473053968/t0(0) o106->fir-OST000a@10.8.15.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550843168 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 05:46:08 fir-io1-s1 kernel: Lustre: 96333:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550843161/real 1550843161] req@ffff986e351df800 x1624936473053952/t0(0) o106->fir-OST0008@10.8.15.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550843168 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 05:46:08 fir-io1-s1 kernel: Lustre: 96333:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 58 previous similar messages Feb 22 05:46:08 fir-io1-s1 kernel: Lustre: 96562:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 22 05:47:44 fir-io1-s1 kernel: LustreError: 96570:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.15.1@o2ib6) returned error from glimpse AST (req@ffff987542167b00 x1624936473054528 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff9839ccb75340/0x49e185e9ed20edc4 lrc: 3/0,0 mode: PW/PW res: [0xa58c8:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000020000 nid: 10.8.15.1@o2ib6 remote: 0xa208989f5cefc2b4 expref: 10 pid: 96265 timeout: 0 lvb_type: 0 Feb 22 05:47:44 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.15.1@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 22 05:47:44 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 22 05:47:44 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550843264s: evicting client at 10.8.15.1@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff9839ccb72880/0x49e185e9ed20ede7 lrc: 3/0,0 mode: PW/PW res: [0xa58f2:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.15.1@o2ib6 remote: 0xa208989f5cefc35c expref: 11 pid: 96265 timeout: 0 lvb_type: 0 Feb 22 05:47:44 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Feb 22 05:47:44 fir-io1-s1 kernel: LustreError: 96570:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 6 previous similar messages Feb 22 05:47:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd6b0907-bbf0-754e-ba62-411999a5fe50 (at 10.8.15.1@o2ib6) Feb 22 05:47:45 fir-io1-s1 kernel: Lustre: Skipped 44 previous similar messages Feb 22 05:48:25 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 696ec4e4-4f79-f247-b0f6-742daa25e948 (at 10.8.11.8@o2ib6) in 195 seconds. I think it's dead, and I am evicting it. exp ffff984bb4fe0c00, cur 1550843305 expire 1550843155 last 1550843110 Feb 22 05:48:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 05:48:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 696ec4e4-4f79-f247-b0f6-742daa25e948 (at 10.8.11.8@o2ib6) in 214 seconds. I think it's dead, and I am evicting it. exp ffff98575a0adc00, cur 1550843324 expire 1550843174 last 1550843110 Feb 22 05:48:44 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 22 05:53:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0c5bfd47-78a4-9fe9-81b5-fa2e264592ad (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9862a52b7400, cur 1550843624 expire 1550843474 last 1550843397 Feb 22 05:53:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 22 05:58:45 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a0ea9325-c185-42fb-8d0b-47d5a0b124af (at 10.8.26.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d200400, cur 1550843925 expire 1550843775 last 1550843698 Feb 22 05:58:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 06:04:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 65ad79ee-001c-5939-0b4a-f0cbeb92b2c0 (at 10.8.22.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef13400, cur 1550844263 expire 1550844113 last 1550844036 Feb 22 06:04:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 06:13:08 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4c5b69ea-d1f1-0261-ea03-15f22270fb92 (at 10.9.101.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986add0d4400, cur 1550844788 expire 1550844638 last 1550844561 Feb 22 06:13:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 06:14:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 811fd087-e95e-3790-3b1f-499e971702dc (at 10.8.12.21@o2ib6) Feb 22 06:14:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 06:18:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 06:18:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 06:23:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1e63f594-4caa-53ef-0c47-c05ca8852eb7 (at 10.8.13.25@o2ib6) Feb 22 06:23:20 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 06:27:01 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9aacc486-953f-6d3b-6b58-f6063066bbe2 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a059f800, cur 1550845621 expire 1550845471 last 1550845394 Feb 22 06:27:01 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 06:29:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 06:29:03 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 22 06:44:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4c5b69ea-d1f1-0261-ea03-15f22270fb92 (at 10.9.101.2@o2ib4) Feb 22 06:44:42 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 06:59:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 39b764af-0b2c-a261-a716-56fdad49854a (at 10.8.21.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480ee8ac00, cur 1550847552 expire 1550847402 last 1550847325 Feb 22 06:59:12 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 07:02:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Feb 22 07:02:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 07:03:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7974f169-1fb9-f080-42cd-427870aa20d4 (at 10.8.11.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758dbd000, cur 1550847823 expire 1550847673 last 1550847596 Feb 22 07:03:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 07:23:28 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 437d1579-92b6-f525-c35c-e4b5e2296f59 (at 10.8.3.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fbda1000, cur 1550849008 expire 1550848858 last 1550848781 Feb 22 07:23:28 fir-io1-s1 kernel: Lustre: Skipped 101 previous similar messages Feb 22 07:23:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 437d1579-92b6-f525-c35c-e4b5e2296f59 (at 10.8.3.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984998f86400, cur 1550849028 expire 1550848878 last 1550848801 Feb 22 07:23:48 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 22 07:29:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 39b764af-0b2c-a261-a716-56fdad49854a (at 10.8.21.12@o2ib6) Feb 22 07:29:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 07:30:46 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4b161527-b7d4-cb96-22b6-3891827325ad (at 10.9.112.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f4a6a0000, cur 1550849446 expire 1550849296 last 1550849219 Feb 22 07:33:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7648c697-029a-1f3f-4734-502e908d693b (at 10.8.20.30@o2ib6) Feb 22 07:33:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 07:36:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7cc66746-6c86-3dd0-f0f5-12b62fae5f96 (at 10.8.11.33@o2ib6) Feb 22 07:36:13 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 07:40:53 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 873e399e-4db8-d9cc-8b77-26669f609a7b (at 10.8.10.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857630f8400, cur 1550850053 expire 1550849903 last 1550849826 Feb 22 07:40:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 07:46:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c34ae119-43a8-1066-416d-873c10713275 (at 10.8.6.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762a58000, cur 1550850376 expire 1550850226 last 1550850149 Feb 22 07:46:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 07:53:54 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e2149003-749e-c138-5dd7-5ff91f76a584 (at 10.8.10.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583bbaf800, cur 1550850834 expire 1550850684 last 1550850607 Feb 22 07:53:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 07:55:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 437d1579-92b6-f525-c35c-e4b5e2296f59 (at 10.8.3.13@o2ib6) Feb 22 07:55:11 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Feb 22 08:04:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4b161527-b7d4-cb96-22b6-3891827325ad (at 10.9.112.2@o2ib4) Feb 22 08:04:01 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 22 08:12:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 873e399e-4db8-d9cc-8b77-26669f609a7b (at 10.8.10.21@o2ib6) Feb 22 08:12:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 08:16:19 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e0ef9edb-a610-ebb1-ffff-144d0d8928d5 (at 10.8.26.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780867800, cur 1550852179 expire 1550852029 last 1550851952 Feb 22 08:16:19 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 08:17:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 973e2f47-6cd2-2b42-a9c1-a77390ce7f35 (at 10.8.26.1@o2ib6) Feb 22 08:17:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 08:22:20 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a09cba77-577d-9f9b-7dd7-1eb9d4d6a7c4 (at 10.8.27.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4c3000, cur 1550852540 expire 1550852390 last 1550852313 Feb 22 08:22:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 08:25:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 08:25:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 08:33:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b8172c4b-510b-79e1-9295-536191f9f544 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848008f5000, cur 1550853235 expire 1550853085 last 1550853008 Feb 22 08:33:55 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 08:36:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 08:36:08 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Feb 22 08:41:34 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5bf2654c-0d76-778b-ddb2-a76ce86f79a7 (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9871f0f58c00, cur 1550853694 expire 1550853544 last 1550853467 Feb 22 08:41:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 08:51:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a09cba77-577d-9f9b-7dd7-1eb9d4d6a7c4 (at 10.8.27.19@o2ib6) Feb 22 08:51:14 fir-io1-s1 kernel: Lustre: Skipped 24 previous similar messages Feb 22 08:53:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2197d288-c10a-4ccf-5895-2dbedfb9c38c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d200800, cur 1550854434 expire 1550854284 last 1550854207 Feb 22 08:53:54 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 09:06:15 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4528b246-5066-1725-9266-362913fa3a4e (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986816f39800, cur 1550855175 expire 1550855025 last 1550854948 Feb 22 09:06:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 09:06:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 09:06:32 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 09:09:51 fir-io1-s1 kernel: Lustre: 96782:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550855384/real 1550855384] req@ffff985421e7d400 x1624936528886480/t0(0) o106->fir-OST0002@10.9.113.12@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550855391 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 09:09:51 fir-io1-s1 kernel: Lustre: 96782:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 194 previous similar messages Feb 22 09:10:12 fir-io1-s1 kernel: Lustre: 96755:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550855405/real 1550855405] req@ffff986e56a54e00 x1624936528886512/t0(0) o106->fir-OST0004@10.9.113.12@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550855412 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 09:10:12 fir-io1-s1 kernel: Lustre: 96755:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 22 09:10:47 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550855440/real 1550855440] req@ffff985661ddad00 x1624936528886528/t0(0) o106->fir-OST0006@10.9.113.12@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550855447 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 09:10:47 fir-io1-s1 kernel: Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 22 09:11:57 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550855510/real 1550855510] req@ffff9869f5ba4800 x1624936528886496/t0(0) o106->fir-OST0000@10.9.113.12@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550855517 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 09:11:57 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Feb 22 09:30:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d3acdff5-12e0-1cdf-9a27-8dcc902cb72a (at 10.8.6.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c348dd000, cur 1550856633 expire 1550856483 last 1550856406 Feb 22 09:30:33 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 09:31:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 76e46ee9-2b41-bbbb-c588-f884c93ae793 (at 10.8.6.13@o2ib6) in 188 seconds. I think it's dead, and I am evicting it. exp ffff986785cd8c00, cur 1550856709 expire 1550856559 last 1550856521 Feb 22 09:31:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 09:34:41 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 82ffae03-c02c-86e8-2dc8-ed4f97ac9c9d (at 10.8.25.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857ac1d4400, cur 1550856881 expire 1550856731 last 1550856654 Feb 22 09:34:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 09:38:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6fa95ecf-4554-3236-796b-9301a5a09ace (at 10.8.6.24@o2ib6) Feb 22 09:38:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 09:41:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b15137cf-311a-3865-c282-8f1cad0a5e07 (at 10.8.30.14@o2ib6) Feb 22 09:41:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 09:44:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25860b57-d235-98fa-8b01-03b6e8b0ca4a (at 10.8.27.33@o2ib6) Feb 22 09:44:34 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 09:48:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f25e95fd-ca39-a936-c3ea-af6c0e743e71 (at 10.8.31.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fa00cc00, cur 1550857729 expire 1550857579 last 1550857502 Feb 22 09:48:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:00:55 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5533cdb3-4930-9d2d-eaf3-02c95876b779 (at 10.8.11.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986813a73c00, cur 1550858455 expire 1550858305 last 1550858228 Feb 22 10:00:55 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 10:02:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d3acdff5-12e0-1cdf-9a27-8dcc902cb72a (at 10.8.6.28@o2ib6) Feb 22 10:02:01 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 22 10:03:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 82ffae03-c02c-86e8-2dc8-ed4f97ac9c9d (at 10.8.25.9@o2ib6) Feb 22 10:03:47 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 22 10:16:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f25e95fd-ca39-a936-c3ea-af6c0e743e71 (at 10.8.31.6@o2ib6) Feb 22 10:16:41 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 10:20:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 19e7b7d0-d3c0-ac2c-8667-586d5e8148d1 (at 10.8.30.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480461a800, cur 1550859644 expire 1550859494 last 1550859417 Feb 22 10:20:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 10:21:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45131b93-0481-3e36-c438-de6a4ced0b9e (at 10.8.24.7@o2ib6) Feb 22 10:21:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:27:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cd77514a-c92d-dc94-95c2-44bf91c41d35 (at 10.8.11.32@o2ib6) Feb 22 10:27:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:32:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5533cdb3-4930-9d2d-eaf3-02c95876b779 (at 10.8.11.16@o2ib6) Feb 22 10:32:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:41:11 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 19be48ce-590d-e165-1f0e-bcb4d8683288 (at 10.8.4.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780d58000, cur 1550860871 expire 1550860721 last 1550860644 Feb 22 10:41:11 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 10:47:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7d64396e-7a86-2e01-38b5-8f4fd2cfeb04 (at 10.8.19.4@o2ib6) Feb 22 10:47:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 10:48:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 19e7b7d0-d3c0-ac2c-8667-586d5e8148d1 (at 10.8.30.19@o2ib6) Feb 22 10:48:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:55:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d64c000, cur 1550861735 expire 1550861585 last 1550861508 Feb 22 10:55:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:56:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cbb26482-d44a-a426-021d-f7356b628c78 (at 10.8.22.9@o2ib6) Feb 22 10:56:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:57:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 16fe1f06-91d4-6364-b5d9-1d6caad6f915 (at 10.8.22.22@o2ib6) Feb 22 10:57:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 10:58:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 466e0697-4f87-b89b-339f-1a2ef924a810 (at 10.8.6.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aab0c00, cur 1550861916 expire 1550861766 last 1550861689 Feb 22 10:58:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 10:59:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ef869551-e461-6e8e-7c21-160fa48ea76c (at 10.8.20.20@o2ib6) Feb 22 10:59:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:09:09 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5f3e0f70-44ba-54a0-e2fb-b96f70c934c0 (at 10.8.23.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f0000, cur 1550862549 expire 1550862399 last 1550862322 Feb 22 11:09:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:13:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.4.32@o2ib6) Feb 22 11:13:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:14:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4de04c00-036a-d169-5318-daba6e51697c (at 10.9.102.17@o2ib4) Feb 22 11:14:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:15:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ba1e1fbb-a72d-16b5-8ad9-93cddc13d3a5 (at 10.9.0.63@o2ib4) Feb 22 11:15:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:24:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) Feb 22 11:24:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:38:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5f3e0f70-44ba-54a0-e2fb-b96f70c934c0 (at 10.8.23.32@o2ib6) Feb 22 11:38:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:40:23 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3ddb4ef0-6f70-bfd2-5871-ec7ff140e21d (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98680476d800, cur 1550864423 expire 1550864273 last 1550864196 Feb 22 11:40:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 11:40:24 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3ddb4ef0-6f70-bfd2-5871-ec7ff140e21d (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0df4c00, cur 1550864424 expire 1550864274 last 1550864197 Feb 22 11:40:24 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 22 11:40:25 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3ddb4ef0-6f70-bfd2-5871-ec7ff140e21d (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98680476d000, cur 1550864425 expire 1550864275 last 1550864198 Feb 22 11:40:25 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 22 11:41:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e63b86ba-9595-7419-fca7-82e45f9f64cb (at 10.9.104.67@o2ib4) in 169 seconds. I think it's dead, and I am evicting it. exp ffff98683bb5b400, cur 1550864499 expire 1550864349 last 1550864330 Feb 22 11:42:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e63b86ba-9595-7419-fca7-82e45f9f64cb (at 10.9.104.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a62c00, cur 1550864557 expire 1550864407 last 1550864330 Feb 22 11:42:37 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 22 11:44:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd6b0907-bbf0-754e-ba62-411999a5fe50 (at 10.8.15.1@o2ib6) Feb 22 11:44:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:02:22 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 30dcc583-cbdf-cec2-f55a-7e64ec810b38 (at 10.8.26.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e10400, cur 1550865742 expire 1550865592 last 1550865515 Feb 22 12:03:38 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client bd187aad-1a83-4fe4-e778-47cba8e6f409 (at 10.8.25.26@o2ib6) in 178 seconds. I think it's dead, and I am evicting it. exp ffff98575a285400, cur 1550865818 expire 1550865668 last 1550865640 Feb 22 12:03:38 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Feb 22 12:08:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 282817f5-6ae5-e49a-959f-04d16934f700 (at 10.9.101.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8c400, cur 1550866081 expire 1550865931 last 1550865854 Feb 22 12:08:01 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 12:09:17 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client eb304df2-55f5-4c00-5490-ecefc431af89 (at 10.8.12.18@o2ib6) in 220 seconds. I think it's dead, and I am evicting it. exp ffff98683a538800, cur 1550866157 expire 1550866007 last 1550865937 Feb 22 12:09:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:09:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Feb 22 12:09:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:14:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c85fef69-2d23-ee70-7307-437e48666801 (at 10.8.3.1@o2ib6) Feb 22 12:14:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:16:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e63b86ba-9595-7419-fca7-82e45f9f64cb (at 10.9.104.67@o2ib4) Feb 22 12:16:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:18:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1cac0206-7dc8-7985-dbe6-f16507ebcfe0 (at 10.8.1.19@o2ib6) Feb 22 12:18:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:26:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3edecda9-b6fa-154c-0a0a-8f2902018cc8 (at 10.8.17.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfa400, cur 1550867217 expire 1550867067 last 1550866990 Feb 22 12:26:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:27:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3edecda9-b6fa-154c-0a0a-8f2902018cc8 (at 10.8.17.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833e2dc00, cur 1550867229 expire 1550867079 last 1550867002 Feb 22 12:27:09 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 22 12:27:14 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3edecda9-b6fa-154c-0a0a-8f2902018cc8 (at 10.8.17.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a285000, cur 1550867234 expire 1550867084 last 1550867007 Feb 22 12:27:14 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 22 12:27:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 890f9dc9-b9bc-0354-4c1a-b7392d8a9570 (at 10.8.19.5@o2ib6) Feb 22 12:27:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:30:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 824f0587-167c-fe30-e5f5-4a8a8b3eb359 (at 10.8.26.17@o2ib6) Feb 22 12:30:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:33:04 fir-io1-s1 kernel: Lustre: 96260:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550867577/real 1550867577] req@ffff98431fb8d400 x1624936586299680/t0(0) o106->fir-OST0002@10.8.11.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550867584 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 12:33:04 fir-io1-s1 kernel: Lustre: 96260:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 22 12:33:25 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550867598/real 1550867598] req@ffff98393307e900 x1624936586299664/t0(0) o106->fir-OST000a@10.8.11.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550867605 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 12:33:25 fir-io1-s1 kernel: Lustre: 49832:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550867598/real 1550867598] req@ffff983e28ba0000 x1624936586299648/t0(0) o106->fir-OST0008@10.8.11.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550867605 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 12:33:25 fir-io1-s1 kernel: Lustre: 49832:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 22 12:33:25 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 22 12:33:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7dac1966-a349-fc4a-4911-f2d61efbd1f7 (at 10.8.11.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987633a46400, cur 1550867614 expire 1550867464 last 1550867387 Feb 22 12:33:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 30dcc583-cbdf-cec2-f55a-7e64ec810b38 (at 10.8.26.24@o2ib6) Feb 22 12:33:45 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 12:39:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to eb304df2-55f5-4c00-5490-ecefc431af89 (at 10.8.12.18@o2ib6) Feb 22 12:39:57 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Feb 22 12:46:58 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 25fddf26-1140-8712-6eb9-db80a7fa48a5 (at 10.8.21.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678448a800, cur 1550868418 expire 1550868268 last 1550868191 Feb 22 12:46:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 12:55:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3edecda9-b6fa-154c-0a0a-8f2902018cc8 (at 10.8.17.2@o2ib6) Feb 22 12:55:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 13:06:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7dac1966-a349-fc4a-4911-f2d61efbd1f7 (at 10.8.11.21@o2ib6) Feb 22 13:06:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 13:17:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ba196fcc-f372-6c13-d1d6-766c67a1554e (at 10.8.12.31@o2ib6) Feb 22 13:17:55 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 13:18:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 850f5687-1489-7b0d-073f-8431f2aebf8a (at 10.8.20.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ad81af800, cur 1550870301 expire 1550870151 last 1550870074 Feb 22 13:18:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 13:24:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1ee05b7c-884c-a9ef-aeb5-480002f038ca (at 10.8.30.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833e77c00, cur 1550870660 expire 1550870510 last 1550870433 Feb 22 13:24:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 13:27:15 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a10633db-3389-9c5a-c26d-af02296e5868 (at 10.8.25.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984810a28400, cur 1550870835 expire 1550870685 last 1550870608 Feb 22 13:27:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 13:37:22 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8cdc893b-daba-c297-1e8a-2ab3a53eaf14 (at 10.8.11.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848008f0c00, cur 1550871442 expire 1550871292 last 1550871215 Feb 22 13:37:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 13:37:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 22 13:37:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 13:42:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 948ddf77-30e8-6f7f-29c3-d9d6fe8d8435 (at 10.9.101.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8c000, cur 1550871755 expire 1550871605 last 1550871528 Feb 22 13:42:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 13:54:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1ee05b7c-884c-a9ef-aeb5-480002f038ca (at 10.8.30.29@o2ib6) Feb 22 13:54:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 13:55:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 43919f92-312a-6c66-0c6f-fd2b1239d673 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c6dd800, cur 1550872502 expire 1550872352 last 1550872275 Feb 22 13:55:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 14:02:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bd16151d-af0e-00df-69f0-bc73398a9c87 (at 10.8.4.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483462b000, cur 1550872977 expire 1550872827 last 1550872750 Feb 22 14:02:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 14:09:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ab5bcaa-8d5e-27d9-5913-f9d8f76ca855 (at 10.8.11.17@o2ib6) Feb 22 14:09:56 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 14:14:11 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4982a077-3e10-aea0-ab4e-56dc6ba7fd04 (at 10.9.108.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e47400, cur 1550873651 expire 1550873501 last 1550873424 Feb 22 14:14:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 14:15:27 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 1bf63035-2382-2247-57ec-f4958613068d (at 10.8.24.11@o2ib6) in 200 seconds. I think it's dead, and I am evicting it. exp ffff98480315b400, cur 1550873727 expire 1550873577 last 1550873527 Feb 22 14:15:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 14:15:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1bf63035-2382-2247-57ec-f4958613068d (at 10.8.24.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984803159800, cur 1550873754 expire 1550873604 last 1550873527 Feb 22 14:15:54 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 22 14:19:18 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b4cb7f16-034a-89d6-e75a-b942f49f0dd5 (at 10.9.107.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867860bc400, cur 1550873958 expire 1550873808 last 1550873731 Feb 22 14:19:18 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 22 14:25:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 75c103c5-8c22-70ed-cfb0-bd07e014990e (at 10.8.11.36@o2ib6) Feb 22 14:25:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 14:31:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575e61a800, cur 1550874708 expire 1550874558 last 1550874481 Feb 22 14:31:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 14:31:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985765736c00, cur 1550874709 expire 1550874559 last 1550874482 Feb 22 14:31:49 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 22 14:37:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa3988d8-312e-baa0-298b-1666a8960425 (at 10.8.14.2@o2ib6) Feb 22 14:37:34 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 14:41:53 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 35991a00-931a-59c7-56b4-d69cc30c2b4f (at 10.8.26.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b2f0000, cur 1550875313 expire 1550875163 last 1550875086 Feb 22 14:41:53 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 22 14:43:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client d0d2544f-519b-c61a-26eb-fe835b950c7f (at 10.8.13.11@o2ib6) in 196 seconds. I think it's dead, and I am evicting it. exp ffff984835488800, cur 1550875389 expire 1550875239 last 1550875193 Feb 22 14:43:09 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 14:44:25 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 261aa682-a08b-ea8d-1927-fbda32010351 (at 10.9.107.41@o2ib4) in 170 seconds. I think it's dead, and I am evicting it. exp ffff986785d2f000, cur 1550875465 expire 1550875315 last 1550875295 Feb 22 14:44:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 14:46:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 869f43f2-8b19-4a37-4693-17e9606f8339 (at 10.8.25.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d207000, cur 1550875618 expire 1550875468 last 1550875391 Feb 22 14:46:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 14:51:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 323d9e9b-f4db-fb48-a1bd-689a69067782 (at 10.8.25.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d29800, cur 1550875886 expire 1550875736 last 1550875659 Feb 22 14:51:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 14:59:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 20496aa3-f5dd-45dd-445e-4072b7526be4 (at 10.8.17.3@o2ib6) Feb 22 14:59:27 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Feb 22 15:07:25 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client fe59dcfb-3bbe-5505-5f06-837da604cf7f (at 10.9.101.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe8f7000, cur 1550876845 expire 1550876695 last 1550876618 Feb 22 15:07:25 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Feb 22 15:10:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1ba5aef6-2be1-c2db-6ba0-6e3a31e32627 (at 10.9.107.41@o2ib4) Feb 22 15:10:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 15:12:41 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 110f398c-860f-9c94-fae8-88dd42a6445a (at 10.8.24.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a5e800, cur 1550877161 expire 1550877011 last 1550876934 Feb 22 15:12:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 15:13:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7b159774-9739-84d0-f0ac-fcc62a72d585 (at 10.8.19.6@o2ib6) Feb 22 15:13:02 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 15:13:23 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550877196/real 1550877196] req@ffff98633db74b00 x1624936619534912/t0(0) o106->fir-OST0006@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550877203 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 15:13:23 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 22 15:13:30 fir-io1-s1 kernel: Lustre: 96888:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550877203/real 1550877203] req@ffff98659cf0f500 x1624936619534928/t0(0) o106->fir-OST0008@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550877210 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 15:13:30 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550877203/real 1550877203] req@ffff984e0da39800 x1624936619534944/t0(0) o106->fir-OST000a@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550877210 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 15:13:30 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 22 15:13:30 fir-io1-s1 kernel: Lustre: 96888:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 22 15:13:44 fir-io1-s1 kernel: Lustre: 96267:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550877217/real 1550877217] req@ffff98498f31b900 x1624936619534896/t0(0) o106->fir-OST0004@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550877224 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 15:13:44 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550877217/real 1550877217] req@ffff984e0da39800 x1624936619534944/t0(0) o106->fir-OST000a@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550877224 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 15:13:44 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Feb 22 15:13:44 fir-io1-s1 kernel: Lustre: 96267:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 22 15:13:57 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 55a14b83-51dc-ab7e-3863-f08de7e82105 (at 10.8.15.5@o2ib6) in 189 seconds. I think it's dead, and I am evicting it. exp ffff985ef8cf9800, cur 1550877237 expire 1550877087 last 1550877048 Feb 22 15:13:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 15:14:19 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550877252/real 1550877252] req@ffff984e0da39800 x1624936619534944/t0(0) o106->fir-OST000a@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550877259 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 15:14:19 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Feb 22 15:14:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 55a14b83-51dc-ab7e-3863-f08de7e82105 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b88e800, cur 1550877275 expire 1550877125 last 1550877048 Feb 22 15:14:35 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 22 15:15:13 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 29c1c7e9-03be-2d5d-2626-14b5e6b7deaa (at 10.8.27.23@o2ib6) in 170 seconds. I think it's dead, and I am evicting it. exp ffff986780d5a400, cur 1550877313 expire 1550877163 last 1550877143 Feb 22 15:15:27 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550877320/real 1550877320] req@ffff986e49c36300 x1624936619747376/t0(0) o106->fir-OST0004@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550877327 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 15:15:27 fir-io1-s1 kernel: Lustre: 96493:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 105 previous similar messages Feb 22 15:16:10 fir-io1-s1 kernel: LustreError: 96888:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff98659cf0bc00 x1624936619851776/t0(0) o106->fir-OST0004@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Feb 22 15:18:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Feb 22 15:18:04 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 15:30:28 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e4d9d6b9-1b66-c9bd-6587-2eb2edd6c048 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835770000, cur 1550878228 expire 1550878078 last 1550878001 Feb 22 15:30:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 15:34:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fdeab83c-a7a1-7e5b-a1b6-bc622d510300 (at 10.9.108.62@o2ib4) Feb 22 15:34:20 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Feb 22 15:40:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client cb3c2cc0-ee78-b6ab-377d-c21804f516a4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0fc00, cur 1550878834 expire 1550878684 last 1550878607 Feb 22 15:40:34 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 15:45:47 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 15dad9fc-3eb8-7474-9754-6cd76df1d1c0 (at 10.8.4.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f4e400, cur 1550879147 expire 1550878997 last 1550878920 Feb 22 15:45:47 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 15:50:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e7f79d5a-c457-b231-b489-ce4d49914e20 (at 10.9.102.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f58fd6400, cur 1550879444 expire 1550879294 last 1550879217 Feb 22 15:50:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 15:52:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8c8c33b7-9214-13d6-3684-562e6494935b (at 10.9.108.37@o2ib4) Feb 22 15:52:00 fir-io1-s1 kernel: Lustre: Skipped 197 previous similar messages Feb 22 15:53:54 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 14c211dc-def9-30b3-8703-6d8fa95aeff3 (at 10.8.10.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868fc400, cur 1550879634 expire 1550879484 last 1550879407 Feb 22 15:53:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 16:15:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 893ecbfa-42aa-6ec6-c224-ad15b6f22ba2 (at 10.8.25.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f4000, cur 1550880948 expire 1550880798 last 1550880721 Feb 22 16:15:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:16:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15dad9fc-3eb8-7474-9754-6cd76df1d1c0 (at 10.8.4.22@o2ib6) Feb 22 16:16:54 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 16:28:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 14c211dc-def9-30b3-8703-6d8fa95aeff3 (at 10.8.10.22@o2ib6) Feb 22 16:28:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:28:54 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 179e0b48-b58d-d9b1-7e3f-f996ca06f525 (at 10.8.10.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984d4f610400, cur 1550881734 expire 1550881584 last 1550881507 Feb 22 16:28:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:34:02 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 27e955dd-7637-ab7e-c263-efbff6b8418b (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784bb8800, cur 1550882042 expire 1550881892 last 1550881815 Feb 22 16:34:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:36:09 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 22 16:36:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:44:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 893ecbfa-42aa-6ec6-c224-ad15b6f22ba2 (at 10.8.25.12@o2ib6) Feb 22 16:44:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:45:36 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b26cca2d-0a74-4355-69dd-41d80334653e (at 10.8.11.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de2400, cur 1550882736 expire 1550882586 last 1550882509 Feb 22 16:45:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:49:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 45b092b1-6c8e-ab91-8268-dcaea9394686 (at 10.8.13.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985815682400, cur 1550882950 expire 1550882800 last 1550882723 Feb 22 16:49:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:50:26 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4dcaa581-be64-fe6b-fa97-73c2b004579c (at 10.8.13.13@o2ib6) in 206 seconds. I think it's dead, and I am evicting it. exp ffff98575c531000, cur 1550883026 expire 1550882876 last 1550882820 Feb 22 16:50:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 16:50:47 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4dcaa581-be64-fe6b-fa97-73c2b004579c (at 10.8.13.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581bfef400, cur 1550883047 expire 1550882897 last 1550882820 Feb 22 16:50:47 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 22 16:54:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 62b8e745-55d6-06c6-f56c-66b6a8e58cb8 (at 10.8.20.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834685800, cur 1550883291 expire 1550883141 last 1550883064 Feb 22 16:54:51 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 22 16:56:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 65ce365b-9919-9006-b389-aa58d742b616 (at 10.8.18.35@o2ib6) in 201 seconds. I think it's dead, and I am evicting it. exp ffff98483e4ec800, cur 1550883367 expire 1550883217 last 1550883166 Feb 22 16:56:07 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 16:56:33 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 65ce365b-9919-9006-b389-aa58d742b616 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868053ec000, cur 1550883393 expire 1550883243 last 1550883166 Feb 22 16:56:33 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 22 16:57:49 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ee4f43b6-eeda-2bf1-5676-b3cc4b94f3db (at 10.8.12.24@o2ib6) in 176 seconds. I think it's dead, and I am evicting it. exp ffff98677c23e800, cur 1550883469 expire 1550883319 last 1550883293 Feb 22 16:57:49 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 22 17:00:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 60886aef-e9ee-9e0a-827b-479433a754b9 (at 10.8.11.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15f2000, cur 1550883622 expire 1550883472 last 1550883395 Feb 22 17:00:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 17:02:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 179e0b48-b58d-d9b1-7e3f-f996ca06f525 (at 10.8.10.6@o2ib6) Feb 22 17:02:46 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 17:13:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 22 17:13:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 17:17:29 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5c93f7b4-bdb4-332b-9d7b-c8f1dda3f8c6 (at 10.8.12.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98783ba1b400, cur 1550884649 expire 1550884499 last 1550884422 Feb 22 17:17:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 17:24:03 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 6b0eec94-8c31-6197-c049-1b0d583b7567 (at 10.8.11.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825ffc00, cur 1550885043 expire 1550884893 last 1550884816 Feb 22 17:24:03 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 17:26:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to baf8fd69-3c06-48ef-44c1-991a98b1784d (at 10.8.13.14@o2ib6) Feb 22 17:26:05 fir-io1-s1 kernel: Lustre: Skipped 20 previous similar messages Feb 22 17:36:03 fir-io1-s1 kernel: Lustre: 96908:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550885756/real 1550885756] req@ffff983ba41df800 x1624936668339264/t0(0) o106->fir-OST000a@10.9.107.25@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550885763 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 17:36:03 fir-io1-s1 kernel: Lustre: 96908:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 224 previous similar messages Feb 22 17:36:24 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550885777/real 1550885777] req@ffff984286898000 x1624936668339248/t0(0) o106->fir-OST0008@10.9.107.25@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550885784 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 17:36:24 fir-io1-s1 kernel: Lustre: 96762:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 22 17:36:59 fir-io1-s1 kernel: Lustre: 96918:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550885812/real 1550885812] req@ffff984e0da38300 x1624936668339280/t0(0) o106->fir-OST0002@10.9.107.25@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550885819 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 17:36:59 fir-io1-s1 kernel: Lustre: 96918:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Feb 22 17:38:08 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550885880/real 1550885880] req@ffff9850d5fd5d00 x1624936668424592/t0(0) o106->fir-OST0002@10.9.105.45@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550885887 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 17:38:08 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 62 previous similar messages Feb 22 17:38:45 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a5728575-a4f7-6b0a-f0a9-44d0ea52ed96 (at 10.9.101.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786880c00, cur 1550885925 expire 1550885775 last 1550885698 Feb 22 17:38:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 17:40:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4bd8642d-f0ee-a784-85c9-bd31540eadc6 (at 10.9.104.12@o2ib4) in 223 seconds. I think it's dead, and I am evicting it. exp ffff98680476dc00, cur 1550886001 expire 1550885851 last 1550885778 Feb 22 17:40:01 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Feb 22 17:40:16 fir-io1-s1 kernel: Lustre: 96612:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550886009/real 1550886009] req@ffff986889422a00 x1624936668498352/t0(0) o106->fir-OST000a@10.9.114.6@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550886016 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 17:40:16 fir-io1-s1 kernel: Lustre: 96612:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 339 previous similar messages Feb 22 17:41:17 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c19e0773-e0c3-86d8-b980-e6720ddb58b0 (at 10.9.107.29@o2ib4) in 226 seconds. I think it's dead, and I am evicting it. exp ffff986785da6c00, cur 1550886077 expire 1550885927 last 1550885851 Feb 22 17:41:17 fir-io1-s1 kernel: Lustre: Skipped 583 previous similar messages Feb 22 17:44:37 fir-io1-s1 kernel: Lustre: 96918:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550886270/real 1550886270] req@ffff98633db77b00 x1624936668627776/t0(0) o106->fir-OST0004@10.9.105.64@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550886277 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 17:44:37 fir-io1-s1 kernel: Lustre: 96918:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 230 previous similar messages Feb 22 17:44:45 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client afd89a08-6b67-04f7-2303-d504e382d3cd (at 10.9.105.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872e8fc7800, cur 1550886285 expire 1550886135 last 1550886058 Feb 22 17:44:45 fir-io1-s1 kernel: Lustre: Skipped 153 previous similar messages Feb 22 17:46:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 54f5e187-1e98-ac4c-4ac5-25b2f08cd5e0 (at 10.8.12.33@o2ib6) in 190 seconds. I think it's dead, and I am evicting it. exp ffff98583c214000, cur 1550886361 expire 1550886211 last 1550886171 Feb 22 17:46:01 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 17:47:17 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3825d956-b97e-6239-e799-7bf6492ff2c9 (at 10.9.105.56@o2ib4) in 186 seconds. I think it's dead, and I am evicting it. exp ffff986784e69400, cur 1550886437 expire 1550886287 last 1550886251 Feb 22 17:47:17 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 17:48:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5c93f7b4-bdb4-332b-9d7b-c8f1dda3f8c6 (at 10.8.12.11@o2ib6) Feb 22 17:48:30 fir-io1-s1 kernel: Lustre: Skipped 24 previous similar messages Feb 22 17:49:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a22b77da-2d58-b631-91e1-c081daeb2699 (at 10.9.105.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786927800, cur 1550886595 expire 1550886445 last 1550886368 Feb 22 17:49:55 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 17:53:20 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550886793/real 1550886793] req@ffff98633db75400 x1624936672567808/t0(0) o106->fir-OST000a@10.9.105.72@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550886800 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 17:53:20 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 25 previous similar messages Feb 22 17:55:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6b0eec94-8c31-6197-c049-1b0d583b7567 (at 10.8.11.12@o2ib6) Feb 22 17:55:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 17:55:25 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 33db19ea-8b3f-bd56-986a-188195b40494 (at 10.9.104.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784415000, cur 1550886925 expire 1550886775 last 1550886698 Feb 22 17:55:25 fir-io1-s1 kernel: Lustre: Skipped 63 previous similar messages Feb 22 18:05:18 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b9d000, cur 1550887518 expire 1550887368 last 1550887291 Feb 22 18:05:18 fir-io1-s1 kernel: Lustre: Skipped 43 previous similar messages Feb 22 18:09:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 22 18:09:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 18:14:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9cf6d7a8-4898-44eb-2590-b689cf0f2dd8 (at 10.8.6.11@o2ib6) Feb 22 18:14:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 18:18:35 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 676b9462-d0c7-96e9-ddb9-5790c315c2e9 (at 10.9.103.26@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043fbc00, cur 1550888315 expire 1550888165 last 1550888088 Feb 22 18:18:35 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Feb 22 18:20:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 22 18:20:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 18:29:34 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 26c5771f-6bff-b79c-941d-a328fa48123c (at 10.9.102.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d79800, cur 1550888974 expire 1550888824 last 1550888747 Feb 22 18:29:34 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 18:38:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 22ee14eb-5a96-ad04-6e5f-188b7aec897d (at 10.8.12.33@o2ib6) Feb 22 18:38:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 18:40:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1e5f7f4c-78fa-5eb7-a0ea-e8f04fabf57f (at 10.8.30.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834b12000, cur 1550889655 expire 1550889505 last 1550889428 Feb 22 18:40:55 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 18:51:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6b3fefc8-73fb-ce51-5a1b-5e43dc04a3e8 (at 10.8.18.11@o2ib6) Feb 22 18:51:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 18:53:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 62a94e57-5eb9-1e28-4a37-9d6b953b2a83 (at 10.8.23.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984835c96800, cur 1550890396 expire 1550890246 last 1550890169 Feb 22 18:53:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 19:04:32 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client eb0df602-8348-c011-6e89-eb251abd235e (at 10.9.102.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b8bac00, cur 1550891072 expire 1550890922 last 1550890845 Feb 22 19:04:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 19:10:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 801c5583-df50-ef54-ebf8-d76e7be7922a (at 10.8.21.18@o2ib6) Feb 22 19:10:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 19:24:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1ca40c95-5615-186b-162f-92f0324c3c09 (at 10.8.26.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481df01800, cur 1550892263 expire 1550892113 last 1550892036 Feb 22 19:24:23 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 19:36:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6e567ec2-4246-8031-905b-2345d1c162bb (at 10.9.106.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984d4f616800, cur 1550893009 expire 1550892859 last 1550892782 Feb 22 19:36:49 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 19:37:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6f23ee85-f1bd-65e1-fb10-c877f871546c (at 10.8.23.16@o2ib6) Feb 22 19:37:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 22 19:47:01 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9527a536-8600-5a1e-674b-7df5f0b5a516 (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f1b2ef400, cur 1550893621 expire 1550893471 last 1550893394 Feb 22 19:47:01 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 19:51:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 22 19:51:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 20:08:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1e5f7f4c-78fa-5eb7-a0ea-e8f04fabf57f (at 10.8.30.32@o2ib6) Feb 22 20:08:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 20:13:35 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a51ee541-bf68-f6d3-6ffe-0b75e0908e4b (at 10.9.106.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762864c00, cur 1550895215 expire 1550895065 last 1550894988 Feb 22 20:13:35 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 20:21:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9fdeb7e4-dbe7-36a2-5704-e3cbd89d3a9c (at 10.9.104.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987472674000, cur 1550895718 expire 1550895568 last 1550895491 Feb 22 20:21:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 20:27:29 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550896042/real 1550896042] req@ffff98706744aa00 x1624936720599264/t0(0) o106->fir-OST0006@10.9.112.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550896049 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 20:27:29 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 79 previous similar messages Feb 22 20:28:46 fir-io1-s1 kernel: Lustre: 94512:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550896119/real 1550896119] req@ffff985e4a857800 x1624936720599248/t0(0) o106->fir-OST0004@10.9.112.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550896126 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 20:28:46 fir-io1-s1 kernel: Lustre: 94512:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 59 previous similar messages Feb 22 20:30:08 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client cb8f269b-ec20-d0c0-ae72-4c4d4201a6f1 (at 10.9.112.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986817884c00, cur 1550896208 expire 1550896058 last 1550895981 Feb 22 20:30:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 20:30:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4b161527-b7d4-cb96-22b6-3891827325ad (at 10.9.112.2@o2ib4) Feb 22 20:30:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 20:36:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 62a94e57-5eb9-1e28-4a37-9d6b953b2a83 (at 10.8.23.26@o2ib6) Feb 22 20:36:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 20:42:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a4f2c4c1-03c5-819a-6852-1875c7d76a33 (at 10.8.19.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283df800, cur 1550896946 expire 1550896796 last 1550896719 Feb 22 20:42:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 20:56:12 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2c23db06-cdad-eba0-53fa-6796532a23fe (at 10.8.13.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780d5fc00, cur 1550897772 expire 1550897622 last 1550897545 Feb 22 20:56:12 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 21:06:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dcd406ad-ffdd-a7c9-489f-309957a1236e (at 10.8.15.7@o2ib6) Feb 22 21:06:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 21:06:28 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9bd541c8-5e18-2470-8262-fd1a455e43c1 (at 10.9.102.35@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986806759800, cur 1550898388 expire 1550898238 last 1550898161 Feb 22 21:06:28 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 21:09:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:09:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 21:13:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:13:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 21:16:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:16:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 21:16:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client fdb2a59a-7a9e-9263-2b82-30ce49f54eeb (at 10.8.23.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857606b0400, cur 1550899017 expire 1550898867 last 1550898790 Feb 22 21:16:57 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 21:19:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:19:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 21:23:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:23:28 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 22 21:27:01 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:27:01 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 22 21:27:15 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f15d13db-282d-708f-1efb-3545e2bcdb20 (at 10.8.23.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784aeb000, cur 1550899635 expire 1550899485 last 1550899408 Feb 22 21:27:15 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 21:30:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:30:40 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 22 21:32:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 926fa24d-f3ab-7ad6-dbc7-f8a15bdf8c5a (at 10.8.19.8@o2ib6) Feb 22 21:32:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 21:38:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:38:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 21:41:53 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 53415c92-9adc-eb55-0060-92152b47b5e2 (at 10.8.23.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de4c00, cur 1550900513 expire 1550900363 last 1550900286 Feb 22 21:41:53 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 21:45:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e92ed96-3125-f479-d091-1906f658d58f (at 10.8.23.3@o2ib6) Feb 22 21:45:51 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 22 21:53:14 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c811d668-97b3-896b-c549-c82ed0ba01e9 (at 10.8.23.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8adc00, cur 1550901194 expire 1550901044 last 1550900967 Feb 22 21:53:14 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 21:56:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a4f2c4c1-03c5-819a-6852-1875c7d76a33 (at 10.8.19.7@o2ib6) Feb 22 21:56:48 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 22:04:04 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c590ade0-bbc8-3f9f-b092-b2d1d33d9d83 (at 10.9.108.19@o2ib4) in 180 seconds. I think it's dead, and I am evicting it. exp ffff9877a15f7800, cur 1550901844 expire 1550901694 last 1550901664 Feb 22 22:04:04 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 22 22:06:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fc0522a2-e86f-7812-92f9-18c8c5b33bdc (at 10.9.105.45@o2ib4) Feb 22 22:06:49 fir-io1-s1 kernel: Lustre: Skipped 1242 previous similar messages Feb 22 22:17:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9c6f080c-fb48-8d2c-8874-d1b40b096cfb (at 10.9.104.27@o2ib4) Feb 22 22:17:02 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Feb 22 22:17:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 139dfd6f-09c7-a451-4566-439377d243d5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985815685800, cur 1550902647 expire 1550902497 last 1550902420 Feb 22 22:17:27 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 22:31:55 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 93050781-6631-de96-974b-50ec6e56fef4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985838d88800, cur 1550903515 expire 1550903365 last 1550903288 Feb 22 22:31:55 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 22:32:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 22 22:32:52 fir-io1-s1 kernel: Lustre: Skipped 54 previous similar messages Feb 22 22:40:43 fir-io1-s1 kernel: Lustre: 96268:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904036/real 1550904036] req@ffff9854f9599500 x1624936734917712/t0(0) o106->fir-OST0004@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904043 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 22:40:43 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904036/real 1550904036] req@ffff9853ce37ef00 x1624936734917728/t0(0) o106->fir-OST0006@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904043 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 22:40:43 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84 previous similar messages Feb 22 22:40:50 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904043/real 1550904043] req@ffff9853ce37ef00 x1624936734917728/t0(0) o106->fir-OST0006@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904050 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 22:40:50 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904043/real 1550904043] req@ffff98633db75700 x1624936734917744/t0(0) o106->fir-OST0008@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904050 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 22:40:50 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 22 22:40:50 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 22 22:40:57 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904050/real 1550904050] req@ffff98633db75700 x1624936734917744/t0(0) o106->fir-OST0008@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904057 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 22:40:57 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 22 22:41:11 fir-io1-s1 kernel: Lustre: 96284:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904064/real 1550904064] req@ffff986e351db900 x1624936734917696/t0(0) o106->fir-OST0000@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904071 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 22:41:11 fir-io1-s1 kernel: Lustre: 96284:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 22 22:41:32 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904085/real 1550904085] req@ffff98633db75700 x1624936734917744/t0(0) o106->fir-OST0008@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904092 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 22:41:32 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Feb 22 22:42:14 fir-io1-s1 kernel: Lustre: 96284:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550904127/real 1550904127] req@ffff986e351db900 x1624936734917696/t0(0) o106->fir-OST0000@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550904134 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 22:42:14 fir-io1-s1 kernel: Lustre: 96284:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 22 22:42:49 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 135b49e4-9bbb-43fc-66e9-1f7ec8c75a96 (at 10.9.113.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98746d96a800, cur 1550904169 expire 1550904019 last 1550903942 Feb 22 22:42:49 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 22:44:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f581dc0e-77f4-4b88-6617-ee8e5aeb1f31 (at 10.9.105.30@o2ib4) Feb 22 22:44:34 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Feb 22 22:53:21 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 18759a8b-1d8a-beb8-df84-89689c8aa9e2 (at 10.9.113.11@o2ib4) in 199 seconds. I think it's dead, and I am evicting it. exp ffff9877a1461800, cur 1550904801 expire 1550904651 last 1550904602 Feb 22 22:53:21 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 22 22:55:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 22 22:55:23 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 23:06:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client dbe64986-3522-e2d0-d57e-b8c002fb5170 (at 10.9.106.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985757baf800, cur 1550905597 expire 1550905447 last 1550905370 Feb 22 23:06:37 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 22 23:09:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 135b49e4-9bbb-43fc-66e9-1f7ec8c75a96 (at 10.9.113.3@o2ib4) Feb 22 23:09:42 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 22 23:17:38 fir-io1-s1 kernel: Lustre: 94240:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550906251/real 1550906251] req@ffff98498b653900 x1624936747502448/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550906258 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 22 23:17:38 fir-io1-s1 kernel: Lustre: 94240:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Feb 22 23:17:52 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550906265/real 1550906265] req@ffff98633db76600 x1624936747502496/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550906272 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 23:17:52 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 22 23:18:12 fir-io1-s1 kernel: Lustre: 96277:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550906285/real 1550906285] req@ffff984abb017500 x1624936747607488/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550906292 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 23:18:12 fir-io1-s1 kernel: Lustre: 96277:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 22 23:18:25 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 72de669b-43e4-0190-ae39-127b27902f92 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848807f4c00, cur 1550906305 expire 1550906155 last 1550906078 Feb 22 23:18:25 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 22 23:18:54 fir-io1-s1 kernel: Lustre: 96281:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550906327/real 1550906327] req@ffff9850d5fd4500 x1624936747607520/t0(0) o106->fir-OST0008@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550906334 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 23:18:54 fir-io1-s1 kernel: Lustre: 96281:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Feb 22 23:20:11 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550906404/real 1550906404] req@ffff9853ce37a700 x1624936747607504/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550906411 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 22 23:20:11 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84 previous similar messages Feb 22 23:20:52 fir-io1-s1 kernel: LNet: Service thread pid 96574 was inactive for 200.26s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 22 23:20:52 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Feb 22 23:20:52 fir-io1-s1 kernel: Pid: 96574, comm: ll_ost02_040 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 22 23:20:52 fir-io1-s1 kernel: Call Trace: Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 22 23:20:52 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 22 23:20:52 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550906452.96574 Feb 22 23:20:53 fir-io1-s1 kernel: LNet: Service thread pid 94240 was inactive for 201.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 22 23:20:53 fir-io1-s1 kernel: Pid: 94240, comm: ll_ost01_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 22 23:20:53 fir-io1-s1 kernel: Call Trace: Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 22 23:20:53 fir-io1-s1 kernel: Pid: 96254, comm: ll_ost01_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 22 23:20:53 fir-io1-s1 kernel: Call Trace: Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 22 23:20:53 fir-io1-s1 kernel: Pid: 96266, comm: ll_ost03_005 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 22 23:20:53 fir-io1-s1 kernel: Call Trace: Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 22 23:20:53 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 22 23:21:01 fir-io1-s1 kernel: LNet: Service thread pid 96574 completed after 209.24s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 22 23:21:01 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Feb 22 23:21:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 22 23:21:27 fir-io1-s1 kernel: Lustre: Skipped 46 previous similar messages Feb 22 23:33:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 87ec8f50-ded4-4054-973f-07f9352c58cb (at 10.9.107.55@o2ib4) Feb 22 23:33:22 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 22 23:35:32 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9ac3440e-d549-d9f0-dcb7-0e2523b76b7a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f1b2eb000, cur 1550907332 expire 1550907182 last 1550907105 Feb 22 23:35:32 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 22 23:44:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 22 23:44:23 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 22 23:45:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fc10d79a-3f6b-019c-a0b4-c0c26e4270e7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c6f000, cur 1550907941 expire 1550907791 last 1550907714 Feb 22 23:45:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 22 23:54:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ab76fb7a-d531-3276-4580-9b28d21debea (at 10.8.24.35@o2ib6) Feb 22 23:54:45 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 00:00:43 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client eb806d00-883e-3db1-b7e2-902541ec3652 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480332a400, cur 1550908843 expire 1550908693 last 1550908616 Feb 23 00:00:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 00:08:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 00:08:55 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 00:15:57 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f4f4f0e5-321d-f874-7419-92109bafab0f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d558800, cur 1550909757 expire 1550909607 last 1550909530 Feb 23 00:15:57 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 00:23:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 00:23:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 00:30:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 920b5308-5d11-3fe2-3720-2c0e8657fe59 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985ef8cfbc00, cur 1550910626 expire 1550910476 last 1550910399 Feb 23 00:30:26 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 00:34:12 fir-io1-s1 kernel: Lustre: 96277:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550910845/real 1550910845] req@ffff9853884a2700 x1624936779891968/t0(0) o106->fir-OST000a@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550910852 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 00:34:12 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550910845/real 1550910845] req@ffff984e0da3f500 x1624936779892000/t0(0) o106->fir-OST0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550910852 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 00:34:12 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 56 previous similar messages Feb 23 00:34:33 fir-io1-s1 kernel: Lustre: 96573:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550910866/real 1550910866] req@ffff98633db76600 x1624936779892016/t0(0) o106->fir-OST0004@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550910873 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 00:34:33 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550910866/real 1550910866] req@ffff984e0da3f500 x1624936779892000/t0(0) o106->fir-OST0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550910873 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 00:34:33 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 23 00:35:15 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550910908/real 1550910908] req@ffff984e0da3f500 x1624936779892000/t0(0) o106->fir-OST0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550910915 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 00:35:15 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Feb 23 00:36:32 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550910985/real 1550910985] req@ffff984e0da3f500 x1624936779892000/t0(0) o106->fir-OST0000@10.9.112.13@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1550910992 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 00:36:32 fir-io1-s1 kernel: Lustre: 96916:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Feb 23 00:39:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 00:39:24 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 00:46:59 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b0c7f62f-6b37-0a24-03f3-a1393c8b0db2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583bbaec00, cur 1550911619 expire 1550911469 last 1550911392 Feb 23 00:46:59 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 00:53:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 00:53:16 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 23 00:59:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b3334779-eb11-a366-1847-c7faa427af7b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfd000, cur 1550912350 expire 1550912200 last 1550912123 Feb 23 00:59:10 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 23 01:03:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Feb 23 01:03:23 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 01:19:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c75dbc91-80df-2d9d-d9e8-6fbfb61c6654 (at 10.8.13.2@o2ib6) Feb 23 01:19:42 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 01:26:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7cb3ac75-c4ba-852b-da0c-65f673c65326 (at 10.9.102.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15a6800, cur 1550913978 expire 1550913828 last 1550913751 Feb 23 01:26:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 01:30:55 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 23 01:30:55 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.204@o2ib7 (0): c: 0, oc: 0, rc: 8 Feb 23 01:30:55 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9849865c3200 Feb 23 01:30:55 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9849865c3200 Feb 23 01:30:55 fir-io1-s1 kernel: Lustre: 97133:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550914248/real 1550914248] req@ffff9876b1631800 x1624937672290048/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550914255 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 01:30:55 fir-io1-s1 kernel: Lustre: 97133:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Feb 23 01:30:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Client ac16def5-1a59-80e5-2e16-45b58fcd0330 (at 10.8.2.8@o2ib6) reconnecting Feb 23 01:30:56 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to ac16def5-1a59-80e5-2e16-45b58fcd0330 (at 10.8.2.8@o2ib6) Feb 23 01:30:56 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 01:30:59 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 987e399f-1b58-3586-c868-9447d08ddb0a (at 10.8.30.16@o2ib6) reconnecting Feb 23 01:30:59 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 23 01:31:02 fir-io1-s1 kernel: Lustre: fir-OST000a: Client c3cc7cbd-51cc-593c-6d4c-bea491c2ec9c (at 10.8.14.2@o2ib6) reconnecting Feb 23 01:31:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8e6d3868-74d9-72fe-fb96-d3986c757b41 (at 10.9.104.71@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677be30800, cur 1550914268 expire 1550914118 last 1550914041 Feb 23 01:31:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 01:31:15 fir-io1-s1 kernel: Lustre: 49824:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550914268/real 1550914268] req@ffff983c1d5d3000 x1624937672501840/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550914275 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 01:31:15 fir-io1-s1 kernel: Lustre: 49827:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550914268/real 1550914268] req@ffff983ba41dd700 x1624937672501824/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550914275 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 01:31:15 fir-io1-s1 kernel: Lustre: 49827:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 214 previous similar messages Feb 23 01:31:38 fir-io1-s1 kernel: Lustre: fir-OST0002: Client ac16def5-1a59-80e5-2e16-45b58fcd0330 (at 10.8.2.8@o2ib6) reconnecting Feb 23 01:31:38 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 23 01:31:56 fir-io1-s1 kernel: LustreError: 96625:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffff9860b8af8050 x1626125563593952/t0(0) o4->f7039615-ade8-7d11-ebf2-59e1717925cd@10.8.17.24@o2ib6:127/0 lens 488/448 e 3 to 0 dl 1550914332 ref 1 fl Interpret:/0/0 rc 0/0 Feb 23 01:31:56 fir-io1-s1 kernel: Lustre: fir-OST0006: Bulk IO write error with f7039615-ade8-7d11-ebf2-59e1717925cd (at 10.8.17.24@o2ib6), client will retry: rc = -110 Feb 23 01:32:30 fir-io1-s1 kernel: Lustre: 96479:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550914343/real 1550914343] req@ffff98498f31bc00 x1624937672478560/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550914350 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 01:32:30 fir-io1-s1 kernel: Lustre: 96479:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1245 previous similar messages Feb 23 01:33:59 fir-io1-s1 kernel: Lustre: fir-OST0006: Client f7039615-ade8-7d11-ebf2-59e1717925cd (at 10.8.17.24@o2ib6) reconnecting Feb 23 01:34:08 fir-io1-s1 kernel: LNet: Service thread pid 96758 was inactive for 200.31s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 23 01:34:08 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Feb 23 01:34:08 fir-io1-s1 kernel: Pid: 96758, comm: ll_ost02_048 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:34:08 fir-io1-s1 kernel: Call Trace: Feb 23 01:34:08 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:34:09 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914449.96758 Feb 23 01:34:09 fir-io1-s1 kernel: Pid: 96768, comm: ll_ost00_042 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:34:09 fir-io1-s1 kernel: Call Trace: Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:34:09 fir-io1-s1 kernel: Pid: 96272, comm: ll_ost02_016 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:34:09 fir-io1-s1 kernel: Call Trace: Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:34:09 fir-io1-s1 kernel: LNet: Service thread pid 96347 was inactive for 200.47s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 23 01:34:09 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Feb 23 01:34:09 fir-io1-s1 kernel: Pid: 96347, comm: ll_ost03_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:34:09 fir-io1-s1 kernel: Call Trace: Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:34:09 fir-io1-s1 kernel: Pid: 96265, comm: ll_ost00_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:34:09 fir-io1-s1 kernel: Call Trace: Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:34:09 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:34:09 fir-io1-s1 kernel: LNet: Service thread pid 96332 was inactive for 200.69s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:34:10 fir-io1-s1 kernel: LNet: Service thread pid 96789 was inactive for 200.50s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:34:10 fir-io1-s1 kernel: LNet: Skipped 28 previous similar messages Feb 23 01:34:10 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914450.96789 Feb 23 01:34:15 fir-io1-s1 kernel: LNet: Service thread pid 49830 was inactive for 200.76s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:34:15 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 23 01:34:15 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914455.49830 Feb 23 01:34:16 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914456.96762 Feb 23 01:34:17 fir-io1-s1 kernel: LNet: Service thread pid 96760 was inactive for 200.11s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:34:17 fir-io1-s1 kernel: LNet: Skipped 27 previous similar messages Feb 23 01:34:17 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914457.96760 Feb 23 01:34:18 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914458.97135 Feb 23 01:34:19 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914459.96560 Feb 23 01:34:20 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914460.96757 Feb 23 01:34:23 fir-io1-s1 kernel: LNet: Service thread pid 96357 was inactive for 200.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:34:23 fir-io1-s1 kernel: LNet: Skipped 27 previous similar messages Feb 23 01:34:23 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914463.96357 Feb 23 01:34:25 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914465.96507 Feb 23 01:34:26 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914466.96615 Feb 23 01:34:28 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914468.49824 Feb 23 01:34:31 fir-io1-s1 kernel: LNet: Service thread pid 96904 was inactive for 200.42s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:34:31 fir-io1-s1 kernel: LNet: Skipped 23 previous similar messages Feb 23 01:34:31 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550914471.96904 Feb 23 01:34:34 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c6674ff7-9e4a-0356-e72b-984a5d22f233 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe8f1400, cur 1550914474 expire 1550914324 last 1550914247 Feb 23 01:34:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 01:34:34 fir-io1-s1 kernel: LNet: Service thread pid 96360 completed after 215.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 23 01:34:34 fir-io1-s1 kernel: LNet: Skipped 86 previous similar messages Feb 23 01:34:35 fir-io1-s1 kernel: LNet: Service thread pid 94245 completed after 216.92s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 23 01:34:35 fir-io1-s1 kernel: LNet: Skipped 30 previous similar messages Feb 23 01:41:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 01:41:34 fir-io1-s1 kernel: Lustre: Skipped 25 previous similar messages Feb 23 01:48:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 310af0d6-5044-a088-888a-b2aa74403b22 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c859c00, cur 1550915298 expire 1550915148 last 1550915071 Feb 23 01:48:18 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 01:53:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 01:53:48 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 01:55:15 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 23 01:55:15 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (0): c: 0, oc: 1, rc: 8 Feb 23 01:55:15 fir-io1-s1 kernel: Lustre: 96250:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1550915709/real 1550915715] req@ffff9876b6495a00 x1624938143828496/t0(0) o106->fir-OST0008@10.8.8.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550915716 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Feb 23 01:55:15 fir-io1-s1 kernel: Lustre: 96250:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2075 previous similar messages Feb 23 01:55:16 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 0a2f91be-c616-b43f-8c90-25ce951913de (at 10.8.24.20@o2ib6) reconnecting Feb 23 01:55:17 fir-io1-s1 kernel: Lustre: fir-OST0006: Client b82e0f9a-ca1d-6e83-6c3e-40c5f3d8794d (at 10.8.23.3@o2ib6) reconnecting Feb 23 01:55:17 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 01:55:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 6e758192-e0ca-b2a5-05dc-17da210aee86 (at 10.8.13.19@o2ib6) reconnecting Feb 23 01:55:25 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 23 01:55:34 fir-io1-s1 kernel: Lustre: 96574:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550915727/real 1550915727] req@ffff98724033aa00 x1624938143936400/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550915734 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 01:55:34 fir-io1-s1 kernel: Lustre: 96574:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 235 previous similar messages Feb 23 01:56:12 fir-io1-s1 kernel: Lustre: 96364:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550915765/real 1550915765] req@ffff984aa0612100 x1624938143851568/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550915772 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 01:56:12 fir-io1-s1 kernel: Lustre: 96364:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 662 previous similar messages Feb 23 01:57:27 fir-io1-s1 kernel: Lustre: 96481:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550915840/real 1550915840] req@ffff9872ad77c200 x1624938143942176/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550915847 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 01:57:27 fir-io1-s1 kernel: Lustre: 96481:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1372 previous similar messages Feb 23 01:58:30 fir-io1-s1 kernel: LNet: Service thread pid 96932 was inactive for 200.60s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 23 01:58:30 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Feb 23 01:58:30 fir-io1-s1 kernel: Pid: 96932, comm: ll_ost02_056 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:58:30 fir-io1-s1 kernel: Call Trace: Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:58:30 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915910.96932 Feb 23 01:58:30 fir-io1-s1 kernel: Pid: 96948, comm: ll_ost02_065 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:58:30 fir-io1-s1 kernel: Call Trace: Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:58:30 fir-io1-s1 kernel: Pid: 96935, comm: ll_ost02_058 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:58:30 fir-io1-s1 kernel: Call Trace: Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:58:30 fir-io1-s1 kernel: Pid: 82098, comm: ll_ost02_066 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:58:30 fir-io1-s1 kernel: Call Trace: Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:58:30 fir-io1-s1 kernel: Pid: 96333, comm: ll_ost02_023 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 01:58:30 fir-io1-s1 kernel: Call Trace: Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 01:58:30 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 01:58:30 fir-io1-s1 kernel: LNet: Service thread pid 96788 was inactive for 201.18s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:58:30 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 23 01:58:36 fir-io1-s1 kernel: LNet: Service thread pid 96360 was inactive for 200.03s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:58:36 fir-io1-s1 kernel: LNet: Skipped 35 previous similar messages Feb 23 01:58:36 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915916.96360 Feb 23 01:58:38 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915918.96906 Feb 23 01:58:40 fir-io1-s1 kernel: LNet: Service thread pid 96504 was inactive for 200.70s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:58:40 fir-io1-s1 kernel: LNet: Skipped 50 previous similar messages Feb 23 01:58:40 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915920.96504 Feb 23 01:58:42 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915922.96345 Feb 23 01:58:45 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915925.938 Feb 23 01:58:47 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915927.96385 Feb 23 01:58:48 fir-io1-s1 kernel: LNet: Service thread pid 96481 was inactive for 200.18s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 01:58:48 fir-io1-s1 kernel: LNet: Skipped 27 previous similar messages Feb 23 01:58:48 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915928.96481 Feb 23 01:58:51 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915931.49819 Feb 23 01:58:52 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550915932.96573 Feb 23 01:58:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 20715fce-7dee-6e53-3ebd-9c5acf0f4b53 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483b553800, cur 1550915935 expire 1550915785 last 1550915708 Feb 23 01:58:55 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 01:58:55 fir-io1-s1 kernel: LNet: Service thread pid 96903 completed after 218.72s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 23 01:58:55 fir-io1-s1 kernel: LNet: Skipped 98 previous similar messages Feb 23 01:58:56 fir-io1-s1 kernel: LNet: Service thread pid 946 completed after 208.76s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 23 01:58:56 fir-io1-s1 kernel: LNet: Skipped 25 previous similar messages Feb 23 02:07:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8e6d3868-74d9-72fe-fb96-d3986c757b41 (at 10.9.104.71@o2ib4) Feb 23 02:07:29 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 23 02:17:36 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client dbadd88c-a785-0470-3598-8647cc79bf6f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98567fe62000, cur 1550917056 expire 1550916906 last 1550916829 Feb 23 02:17:36 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 23 02:22:49 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 02:24:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 02:24:41 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 02:31:04 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f74f6780-500b-a99c-769a-05932d2be074 (at 10.9.102.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c6d9000, cur 1550917864 expire 1550917714 last 1550917637 Feb 23 02:31:04 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 02:34:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 02:34:55 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 02:43:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 033127f9-d684-a263-9e10-535f772c4f1a (at 10.9.106.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985763310000, cur 1550918588 expire 1550918438 last 1550918361 Feb 23 02:43:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 02:46:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 02:46:57 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 23 02:54:01 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ac16def5-1a59-80e5-2e16-45b58fcd0330 (at 10.8.2.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d41cc00, cur 1550919241 expire 1550919091 last 1550919014 Feb 23 02:54:01 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 03:04:58 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d3c39393-4e7c-8aa1-9ad3-618b87d86071 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835777800, cur 1550919898 expire 1550919748 last 1550919671 Feb 23 03:04:58 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 03:06:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f74f6780-500b-a99c-769a-05932d2be074 (at 10.9.102.10@o2ib4) Feb 23 03:06:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 03:17:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 033127f9-d684-a263-9e10-535f772c4f1a (at 10.9.106.25@o2ib4) Feb 23 03:17:30 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 03:18:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client dea91ada-1bda-2712-7a12-372475f95fb7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b94400, cur 1550920703 expire 1550920553 last 1550920476 Feb 23 03:18:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 03:35:33 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bdfd7f2b-f57d-9f82-b295-bd88659acc70 (at 10.9.105.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786886800, cur 1550921733 expire 1550921583 last 1550921506 Feb 23 03:35:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 03:41:56 fir-io1-s1 kernel: LNetError: 91393:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 03:48:07 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 735f20ae-b55f-c0ce-fab7-dbc0e70cae54 (at 10.9.107.70@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480332ac00, cur 1550922487 expire 1550922337 last 1550922260 Feb 23 03:48:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 03:52:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 03:52:02 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 23 04:05:41 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 62faecbe-6e06-a70f-dde9-8810f97cf707 (at 10.9.103.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e3c285800, cur 1550923541 expire 1550923391 last 1550923314 Feb 23 04:05:41 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 04:10:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bdfd7f2b-f57d-9f82-b295-bd88659acc70 (at 10.9.105.24@o2ib4) Feb 23 04:10:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 04:15:21 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 735f20ae-b55f-c0ce-fab7-dbc0e70cae54 (at 10.9.107.70@o2ib4) Feb 23 04:15:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 04:24:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c95418d8-abb0-d4fc-c763-439a35e76a6c (at 10.9.102.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a8a800, cur 1550924662 expire 1550924512 last 1550924435 Feb 23 04:24:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 04:35:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 23 04:35:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 04:39:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1c97183d-5ee9-5d13-a850-76ae3fca9eb1 (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f4a6a3000, cur 1550925584 expire 1550925434 last 1550925357 Feb 23 04:39:44 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 23 04:41:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.103.17@o2ib4) Feb 23 04:41:25 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 04:43:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 23 04:43:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 04:50:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 23 04:50:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 05:00:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6999d791-63e6-9f77-9076-8032dae4068f (at 10.9.102.66@o2ib4) Feb 23 05:00:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 05:11:19 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 584fee0b-77b5-bf9b-2e1d-78e423ba7c74 (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582bba4400, cur 1550927479 expire 1550927329 last 1550927252 Feb 23 05:11:19 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Feb 23 05:14:37 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 05:16:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 420eb17f-e3a0-1fd9-bcf2-389dfdfba340 (at 10.8.20.5@o2ib6) Feb 23 05:16:14 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 23 05:16:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0fd824ce-5c22-acbb-f997-3c3af1c05035 (at 10.8.24.1@o2ib6) Feb 23 05:16:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 05:16:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4c4f9b6e-ffa8-1fee-3df9-3f645a83c731 (at 10.8.24.6@o2ib6) Feb 23 05:16:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 05:17:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cc4ba4bd-a4c2-aaab-d4c9-cbd3275e0887 (at 10.8.20.13@o2ib6) Feb 23 05:17:01 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 05:32:00 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 05:47:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Client 06a0640d-de7f-9715-947b-5ac203d15e9f (at 10.0.10.3@o2ib7) reconnecting Feb 23 05:47:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 23 05:47:30 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 23 05:50:50 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 05:57:53 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c01b1480-3980-c5f3-2ed5-b0bfc9b513b5 (at 10.8.4.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c004800, cur 1550930273 expire 1550930123 last 1550930046 Feb 23 05:57:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:00:34 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8c0347c3-135a-e940-667b-27edd6a0ad7f (at 10.8.20.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b16400, cur 1550930434 expire 1550930284 last 1550930207 Feb 23 06:00:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:01:40 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 06:04:47 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8f66044b-a9e8-7100-700d-8cee83a0d250 (at 10.8.21.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762495000, cur 1550930687 expire 1550930537 last 1550930460 Feb 23 06:04:47 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 23 06:07:41 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 60c991c9-bbb6-b45f-4869-dff10ed664d7 (at 10.8.3.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677bc9a000, cur 1550930861 expire 1550930711 last 1550930634 Feb 23 06:07:41 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 23 06:13:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 23 06:13:32 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 06:17:47 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 06:20:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 44c34e5e-d358-e5f1-f032-e5118620e81b (at 10.8.24.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575eaa4800, cur 1550931624 expire 1550931474 last 1550931397 Feb 23 06:20:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:30:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8c0347c3-135a-e940-667b-27edd6a0ad7f (at 10.8.20.6@o2ib6) Feb 23 06:30:29 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 06:30:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 83533265-92ee-9e02-bccc-2318c2ff57de (at 10.8.20.7@o2ib6) Feb 23 06:30:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:30:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 924a230f-7a9e-90ca-0838-4eb0790eb9f6 (at 10.8.20.16@o2ib6) Feb 23 06:30:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:31:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 33c84328-02bf-7749-b411-57d8a76b2873 (at 10.8.20.14@o2ib6) Feb 23 06:31:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:33:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 43935648-ed38-77f7-e746-028bb2b23f4a (at 10.8.21.9@o2ib6) Feb 23 06:33:44 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 23 06:34:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8f66044b-a9e8-7100-700d-8cee83a0d250 (at 10.8.21.1@o2ib6) Feb 23 06:34:00 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 06:34:03 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867cbf8f000, cur 1550932443 expire 1550932293 last 1550932216 Feb 23 06:34:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:34:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d457dd0b-bcd3-cc31-d257-0e8a7a8274a4 (at 10.8.21.10@o2ib6) Feb 23 06:34:23 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 06:40:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 60c991c9-bbb6-b45f-4869-dff10ed664d7 (at 10.8.3.25@o2ib6) Feb 23 06:40:38 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 06:45:07 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ca78cddb-ea39-0254-4028-7b0b6c7c780d (at 10.8.19.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857618e1800, cur 1550933107 expire 1550932957 last 1550932880 Feb 23 06:45:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:45:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ca78cddb-ea39-0254-4028-7b0b6c7c780d (at 10.8.19.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857618e4400, cur 1550933109 expire 1550932959 last 1550932882 Feb 23 06:48:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 44c34e5e-d358-e5f1-f032-e5118620e81b (at 10.8.24.9@o2ib6) Feb 23 06:48:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 06:52:44 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 06:58:24 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 07:01:34 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b7eb786c-4da9-9431-a597-6f5f4ba4c9ed (at 10.9.114.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848001fa800, cur 1550934094 expire 1550933944 last 1550933867 Feb 23 07:01:34 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 07:06:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca78cddb-ea39-0254-4028-7b0b6c7c780d (at 10.8.19.2@o2ib6) Feb 23 07:06:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:08:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ccd0dfc-b23e-3ec2-23e3-efd7656777a0 (at 10.9.106.48@o2ib4) Feb 23 07:08:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:11:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ac36ccbf-474d-9877-36e4-9824199ab544 (at 10.9.105.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984cd57abc00, cur 1550934673 expire 1550934523 last 1550934446 Feb 23 07:11:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:12:29 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 04bbddc7-37a9-9b79-7fa8-c451901e5d15 (at 10.9.101.67@o2ib4) in 225 seconds. I think it's dead, and I am evicting it. exp ffff984803329000, cur 1550934749 expire 1550934599 last 1550934524 Feb 23 07:12:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:23:07 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 07:28:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7eb786c-4da9-9431-a597-6f5f4ba4c9ed (at 10.9.114.12@o2ib4) Feb 23 07:28:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:43:49 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 07:44:53 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fed507e8-5435-f949-539f-6cb9d563cc12 (at 10.9.106.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985764d61800, cur 1550936693 expire 1550936543 last 1550936466 Feb 23 07:44:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:46:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ac36ccbf-474d-9877-36e4-9824199ab544 (at 10.9.105.42@o2ib4) Feb 23 07:46:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:46:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 04bbddc7-37a9-9b79-7fa8-c451901e5d15 (at 10.9.101.67@o2ib4) Feb 23 07:46:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 07:55:25 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 06a0640d-de7f-9715-947b-5ac203d15e9f (at 10.0.10.3@o2ib7) reconnecting Feb 23 07:55:25 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 23 07:55:25 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 08:01:43 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client cbf2c542-50a0-0087-7c10-0da00c4b573b (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a08000, cur 1550937703 expire 1550937553 last 1550937476 Feb 23 08:01:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:08:21 fir-io1-s1 kernel: LNetError: 91393:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 08:19:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fed507e8-5435-f949-539f-6cb9d563cc12 (at 10.9.106.52@o2ib4) Feb 23 08:19:10 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 23 08:24:25 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 334bb23d-7a49-1e3e-532b-59b5c5becc5c (at 10.9.106.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835770800, cur 1550939065 expire 1550938915 last 1550938838 Feb 23 08:24:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 08:27:05 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5d333fe8-2d2e-778b-34c9-702cc9f2963f (at 10.9.106.57@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd443400, cur 1550939225 expire 1550939075 last 1550938998 Feb 23 08:27:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:28:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 23 08:28:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:34:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 946fc237-8ed2-da96-a706-c3ec5bd87de8 (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a67800, cur 1550939645 expire 1550939495 last 1550939418 Feb 23 08:34:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:35:16 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 08:35:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 23 08:35:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:37:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 16214717-e51a-197b-25e8-75bbe23d8ed2 (at 10.9.101.63@o2ib4) Feb 23 08:37:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:42:57 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 08:47:26 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c8bbd681-6d42-3196-83fc-d269a276781d (at 10.9.105.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ea44000, cur 1550940446 expire 1550940296 last 1550940219 Feb 23 08:47:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:48:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ab01728a-7f0a-5d0b-1dba-f4bfe5ee2a67 (at 10.9.105.67@o2ib4) in 189 seconds. I think it's dead, and I am evicting it. exp ffff984836940800, cur 1550940522 expire 1550940372 last 1550940333 Feb 23 08:48:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 08:59:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 334bb23d-7a49-1e3e-532b-59b5c5becc5c (at 10.9.106.43@o2ib4) Feb 23 08:59:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 09:06:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5d333fe8-2d2e-778b-34c9-702cc9f2963f (at 10.9.106.57@o2ib4) Feb 23 09:06:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 09:21:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c8bbd681-6d42-3196-83fc-d269a276781d (at 10.9.105.55@o2ib4) Feb 23 09:21:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 09:26:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ab01728a-7f0a-5d0b-1dba-f4bfe5ee2a67 (at 10.9.105.67@o2ib4) Feb 23 09:26:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 09:35:13 fir-io1-s1 kernel: LNetError: 91391:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 09:40:12 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 090b1219-2d44-d095-6331-57a66bc20e45 (at 10.9.105.65@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c122800, cur 1550943612 expire 1550943462 last 1550943385 Feb 23 09:40:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 09:59:14 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986ee851e400, cur 1550944754 expire 1550944604 last 1550944527 Feb 23 09:59:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:04:20 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5d1e4dc6-641b-8057-25cd-d6a06c5ac7ad (at 10.9.104.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c127c00, cur 1550945060 expire 1550944910 last 1550944833 Feb 23 10:04:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:10:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 449d2561-bd9d-471a-d03c-7cc4c311a708 (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801541400, cur 1550945444 expire 1550945294 last 1550945217 Feb 23 10:10:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 10:16:47 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 090b1219-2d44-d095-6331-57a66bc20e45 (at 10.9.105.65@o2ib4) Feb 23 10:16:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:18:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5a2e8619-a490-b680-1aa3-dbcfdcdd1ec9 (at 10.9.106.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e3c283400, cur 1550945938 expire 1550945788 last 1550945711 Feb 23 10:18:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:21:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 994c77ca-5a3e-accc-3ddc-a08f18403cd1 (at 10.8.17.23@o2ib6) Feb 23 10:21:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:23:12 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 386cceaf-b887-f800-5980-0888a79dd601 (at 10.9.105.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00ba000, cur 1550946192 expire 1550946042 last 1550945965 Feb 23 10:23:12 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 10:23:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) Feb 23 10:23:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:24:28 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 73ccef87-be96-5ca5-528b-f0b5192c7ff5 (at 10.8.13.8@o2ib6) in 181 seconds. I think it's dead, and I am evicting it. exp ffff984904996800, cur 1550946268 expire 1550946118 last 1550946087 Feb 23 10:24:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:27:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 82ac53d4-e935-ad2f-ac24-800763287c87 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984998f85800, cur 1550946450 expire 1550946300 last 1550946223 Feb 23 10:27:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:30:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 23 10:30:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:32:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 20038beb-5afb-dfb0-8b3f-2e2b6f7195f5 (at 10.9.106.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480eeb3800, cur 1550946742 expire 1550946592 last 1550946515 Feb 23 10:32:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:38:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5d1e4dc6-641b-8057-25cd-d6a06c5ac7ad (at 10.9.104.2@o2ib4) Feb 23 10:38:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:39:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5188c60b-8e1f-a67c-e214-04fbb301fd99 (at 10.9.106.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800529c00, cur 1550947141 expire 1550946991 last 1550946914 Feb 23 10:39:01 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 23 10:39:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 23 10:39:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:47:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5a2e8619-a490-b680-1aa3-dbcfdcdd1ec9 (at 10.9.106.17@o2ib4) Feb 23 10:47:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:49:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f6a811d2-f77e-6c94-690d-cc60be6676e4 (at 10.9.108.7@o2ib4) Feb 23 10:49:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:51:33 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 10:56:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 73ccef87-be96-5ca5-528b-f0b5192c7ff5 (at 10.8.13.8@o2ib6) Feb 23 10:56:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 10:57:17 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 10:59:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 386cceaf-b887-f800-5980-0888a79dd601 (at 10.9.105.63@o2ib4) Feb 23 10:59:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 11:02:02 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 166c4791-1a84-8206-fe53-c61fa08583c2 (at 10.9.103.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872e8fc6800, cur 1550948522 expire 1550948372 last 1550948295 Feb 23 11:02:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 11:02:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 166c4791-1a84-8206-fe53-c61fa08583c2 (at 10.9.103.19@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98763803b400, cur 1550948543 expire 1550948393 last 1550948316 Feb 23 11:02:23 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 11:05:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 37aa7d65-aee3-56c5-fbe1-78e40cec7bbc (at 10.9.101.66@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a37f6800, cur 1550948718 expire 1550948568 last 1550948491 Feb 23 11:06:34 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 054c4743-619d-965f-f786-cc0afc52d348 (at 10.9.101.68@o2ib4) in 173 seconds. I think it's dead, and I am evicting it. exp ffff98677dc5f000, cur 1550948794 expire 1550948644 last 1550948621 Feb 23 11:06:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 11:07:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5188c60b-8e1f-a67c-e214-04fbb301fd99 (at 10.9.106.14@o2ib4) Feb 23 11:07:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 11:16:45 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5bd1312d-d4d9-79f2-e6c0-fb1a0e35eed8 (at 10.8.12.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c348da800, cur 1550949405 expire 1550949255 last 1550949178 Feb 23 11:16:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 11:31:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 88ddc3f9-1f21-4f81-e1f1-3f396b007308 (at 10.9.101.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986add0d7000, cur 1550950283 expire 1550950133 last 1550950056 Feb 23 11:31:23 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 11:35:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e43da944-d239-923f-8f68-10646264727b (at 10.8.21.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481df07400, cur 1550950502 expire 1550950352 last 1550950275 Feb 23 11:35:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 11:37:44 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 166c4791-1a84-8206-fe53-c61fa08583c2 (at 10.9.103.19@o2ib4) Feb 23 11:37:44 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Feb 23 11:39:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 37aa7d65-aee3-56c5-fbe1-78e40cec7bbc (at 10.9.101.66@o2ib4) Feb 23 11:39:54 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 11:42:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 054c4743-619d-965f-f786-cc0afc52d348 (at 10.9.101.68@o2ib4) Feb 23 11:42:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 11:47:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5bd1312d-d4d9-79f2-e6c0-fb1a0e35eed8 (at 10.8.12.17@o2ib6) Feb 23 11:47:09 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 11:47:11 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2cc850be-dc6d-45a9-7198-5a65d8b1618d (at 10.9.104.39@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576184f400, cur 1550951231 expire 1550951081 last 1550951004 Feb 23 11:47:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 12:04:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e43da944-d239-923f-8f68-10646264727b (at 10.8.21.20@o2ib6) Feb 23 12:04:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 12:08:21 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1cc72755-0b18-692a-013c-e5abb0ad9b59 (at 10.9.106.44@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00ec800, cur 1550952501 expire 1550952351 last 1550952274 Feb 23 12:08:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 12:14:01 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 12:24:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2cc850be-dc6d-45a9-7198-5a65d8b1618d (at 10.9.104.39@o2ib4) Feb 23 12:24:15 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 12:40:43 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 12:42:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1cc72755-0b18-692a-013c-e5abb0ad9b59 (at 10.9.106.44@o2ib4) Feb 23 12:42:41 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 12:57:23 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 889a8bd9-6c23-c824-f979-e28b7cf41d1f (at 10.9.114.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868e447cc00, cur 1550955443 expire 1550955293 last 1550955216 Feb 23 12:57:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 13:06:03 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 13:12:54 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4f44a399-702a-8d9a-166c-3e54590e6073 (at 10.8.10.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483860c400, cur 1550956374 expire 1550956224 last 1550956147 Feb 23 13:12:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 13:22:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 889a8bd9-6c23-c824-f979-e28b7cf41d1f (at 10.9.114.10@o2ib4) Feb 23 13:22:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 13:26:21 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 13:36:29 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2cfcad01-8df5-2887-8f5c-a2aec6d77cee (at 10.9.107.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801bfc400, cur 1550957789 expire 1550957639 last 1550957562 Feb 23 13:36:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 13:44:13 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 4f44a399-702a-8d9a-166c-3e54590e6073 (at 10.8.10.16@o2ib6) Feb 23 13:44:13 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 13:50:06 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 13:55:24 fir-io1-s1 kernel: Lustre: 96755:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550958917/real 1550958917] req@ffff9874256f4e00 x1624977995712256/t0(0) o106->fir-OST0006@10.8.17.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550958924 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 13:55:24 fir-io1-s1 kernel: Lustre: 96755:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1903 previous similar messages Feb 23 13:55:45 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550958938/real 1550958938] req@ffff9868c1506900 x1624977995714928/t0(0) o106->fir-OST000a@10.8.17.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550958945 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 13:55:45 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 23 13:56:06 fir-io1-s1 kernel: LustreError: 96622:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(2430992) req@ffff985758d2f850 x1626200909521968/t0(0) o4->0a7da039-2609-d0e5-96ae-ab5338f1b782@10.8.9.2@o2ib6:225/0 lens 488/448 e 3 to 0 dl 1550958975 ref 1 fl Interpret:/0/0 rc 0/0 Feb 23 13:56:06 fir-io1-s1 kernel: Lustre: fir-OST0006: Bulk IO write error with 0a7da039-2609-d0e5-96ae-ab5338f1b782 (at 10.8.9.2@o2ib6), client will retry: rc = -110 Feb 23 13:56:27 fir-io1-s1 kernel: Lustre: 96788:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550958980/real 1550958980] req@ffff986c57ad9500 x1624977995714912/t0(0) o106->fir-OST0008@10.8.17.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550958987 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 13:56:27 fir-io1-s1 kernel: Lustre: 96788:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Feb 23 13:56:38 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ea7e114e-3d27-0438-c912-7927f0cdf6fc (at 10.8.17.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98783ba1f000, cur 1550958998 expire 1550958848 last 1550958771 Feb 23 13:56:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 13:57:54 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0a7da039-2609-d0e5-96ae-ab5338f1b782 (at 10.8.9.2@o2ib6) in 193 seconds. I think it's dead, and I am evicting it. exp ffff98480073c400, cur 1550959074 expire 1550958924 last 1550958881 Feb 23 13:57:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 13:58:52 fir-io1-s1 kernel: Lustre: 96890:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550959125/real 1550959125] req@ffff9855b98b4e00 x1624978214203104/t0(0) o106->fir-OST000a@10.8.9.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550959132 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 13:58:52 fir-io1-s1 kernel: Lustre: 96890:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 23 14:01:26 fir-io1-s1 kernel: Lustre: 96479:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550959279/real 1550959279] req@ffff9857a7992400 x1624978214203056/t0(0) o106->fir-OST0002@10.8.9.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550959286 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 14:01:26 fir-io1-s1 kernel: Lustre: 96479:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 84 previous similar messages Feb 23 14:01:46 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d1de27e8-6cf3-dd3c-72a3-f5c65dcce9a1 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987728a1c400, cur 1550959306 expire 1550959156 last 1550959079 Feb 23 14:01:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:02:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 06a0640d-de7f-9715-947b-5ac203d15e9f (at 10.0.10.3@o2ib7) reconnecting Feb 23 14:02:29 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 23 14:02:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Feb 23 14:02:29 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 23 14:10:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2cfcad01-8df5-2887-8f5c-a2aec6d77cee (at 10.9.107.6@o2ib4) Feb 23 14:10:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:17:58 fir-io1-s1 kernel: LNetError: 91391:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 14:23:41 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 935e7eb3-1ff6-7dda-ab9a-d14a4b5f1855 (at 10.9.103.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f4a6a0c00, cur 1550960621 expire 1550960471 last 1550960394 Feb 23 14:23:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:26:04 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 14:26:23 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 14:27:52 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 28529dc2-a4f3-77ae-28a6-713b2825ee6a (at 10.9.106.27@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987191c69000, cur 1550960872 expire 1550960722 last 1550960645 Feb 23 14:27:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:40:22 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e527b419-86b9-ff93-48e0-b15e55994667 (at 10.9.106.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d00000, cur 1550961622 expire 1550961472 last 1550961395 Feb 23 14:40:22 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 23 14:41:13 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 14:46:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7936d72a-098c-6dfd-6893-32b658b77868 (at 10.9.103.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868ceff3800, cur 1550961971 expire 1550961821 last 1550961744 Feb 23 14:46:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:46:21 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 7936d72a-098c-6dfd-6893-32b658b77868 (at 10.9.103.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868ceff3c00, cur 1550961981 expire 1550961831 last 1550961754 Feb 23 14:46:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 7936d72a-098c-6dfd-6893-32b658b77868 (at 10.9.103.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868ceff0c00, cur 1550961984 expire 1550961834 last 1550961757 Feb 23 14:47:18 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 14:47:27 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 66de0a73-cb41-c788-e30e-7505e7f80015 (at 10.9.106.41@o2ib4) in 199 seconds. I think it's dead, and I am evicting it. exp ffff986786a34c00, cur 1550962047 expire 1550961897 last 1550961848 Feb 23 14:47:27 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 23 14:47:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 66de0a73-cb41-c788-e30e-7505e7f80015 (at 10.9.106.41@o2ib4) in 209 seconds. I think it's dead, and I am evicting it. exp ffff9848346a5000, cur 1550962057 expire 1550961907 last 1550961848 Feb 23 14:47:37 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 23 14:47:51 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 14:50:36 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5fd68af5-0c2b-1947-5fc1-6504b55b60fb (at 10.9.103.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15f0400, cur 1550962236 expire 1550962086 last 1550962009 Feb 23 14:50:36 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 23 14:55:15 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 37ee44d5-114c-01d9-35af-baaa0cdcbaff (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984bb4fe4000, cur 1550962515 expire 1550962365 last 1550962288 Feb 23 14:55:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:56:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ea7e114e-3d27-0438-c912-7927f0cdf6fc (at 10.8.17.12@o2ib6) Feb 23 14:56:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:57:04 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0343f8c1-f803-943e-238c-e83a0eb1a3ba (at 10.9.106.34@o2ib4) Feb 23 14:57:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:58:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 14:58:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 14:59:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3b2e850b-830c-c045-0e53-e91a4da0ae80 (at 10.9.108.6@o2ib4) Feb 23 14:59:29 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 14:59:30 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 15:00:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 935e7eb3-1ff6-7dda-ab9a-d14a4b5f1855 (at 10.9.103.32@o2ib4) Feb 23 15:00:54 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 15:03:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f9ba6989-1b37-0945-75ce-916db80d0755 (at 10.9.106.29@o2ib4) Feb 23 15:03:34 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 15:03:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 28529dc2-a4f3-77ae-28a6-713b2825ee6a (at 10.9.106.27@o2ib4) Feb 23 15:03:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 15:08:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e527b419-86b9-ff93-48e0-b15e55994667 (at 10.9.106.28@o2ib4) Feb 23 15:08:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 15:09:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15977d36-cdb8-43c9-109d-47180b552ba3 (at 10.9.106.36@o2ib4) Feb 23 15:09:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 15:13:49 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 15:14:35 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 15:17:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9dd791fc-5e27-d5c0-d08d-b2cd561ae98d (at 10.8.30.34@o2ib6) Feb 23 15:17:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 15:22:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 66de0a73-cb41-c788-e30e-7505e7f80015 (at 10.9.106.41@o2ib4) Feb 23 15:22:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 15:31:00 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 15:57:05 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 977fe910-a1af-e6c8-e791-05648df9545c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985ef8cff800, cur 1550966225 expire 1550966075 last 1550965998 Feb 23 15:57:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 15:59:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 15:59:30 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 16:18:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a8212744-dcf7-9fd1-3ca7-bcd9aa4460d5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868fa000, cur 1550967533 expire 1550967383 last 1550967306 Feb 23 16:18:53 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 16:19:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a8212744-dcf7-9fd1-3ca7-bcd9aa4460d5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780867c00, cur 1550967552 expire 1550967402 last 1550967325 Feb 23 16:19:19 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a8212744-dcf7-9fd1-3ca7-bcd9aa4460d5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e7da29c00, cur 1550967559 expire 1550967409 last 1550967332 Feb 23 16:22:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 16:22:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 16:24:43 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 16:25:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3dbbb061-d93c-c7e8-88c8-e262ff513397 (at 10.8.14.6@o2ib6) Feb 23 16:25:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 16:28:50 fir-io1-s1 kernel: Lustre: 96399:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550968123/real 1550968123] req@ffff9849149cb000 x1624987622827840/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550968130 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 16:28:50 fir-io1-s1 kernel: Lustre: 96399:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 23 16:29:28 fir-io1-s1 kernel: Lustre: 94628:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550968161/real 1550968161] req@ffff985b6b3ce600 x1624987625339424/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550968168 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 16:29:28 fir-io1-s1 kernel: Lustre: 94628:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 87 previous similar messages Feb 23 16:30:43 fir-io1-s1 kernel: Lustre: 96356:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550968236/real 1550968236] req@ffff984b0eed8f00 x1624987623369184/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550968243 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 16:30:43 fir-io1-s1 kernel: Lustre: 96356:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 173 previous similar messages Feb 23 16:32:04 fir-io1-s1 kernel: LNet: Service thread pid 96399 was inactive for 200.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 23 16:32:04 fir-io1-s1 kernel: LNet: Skipped 4 previous similar messages Feb 23 16:32:04 fir-io1-s1 kernel: Pid: 96399, comm: ll_ost01_052 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 16:32:04 fir-io1-s1 kernel: Call Trace: Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 16:32:04 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 16:32:04 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550968324.96399 Feb 23 16:32:05 fir-io1-s1 kernel: LNet: Service thread pid 96355 was inactive for 200.63s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 23 16:32:05 fir-io1-s1 kernel: Pid: 96355, comm: ll_ost01_033 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 16:32:05 fir-io1-s1 kernel: Call Trace: Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 16:32:05 fir-io1-s1 kernel: Pid: 96929, comm: ll_ost01_103 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 16:32:05 fir-io1-s1 kernel: Call Trace: Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 16:32:05 fir-io1-s1 kernel: Pid: 96524, comm: ll_ost01_062 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 16:32:05 fir-io1-s1 kernel: Call Trace: Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 16:32:05 fir-io1-s1 kernel: Pid: 96356, comm: ll_ost01_034 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 16:32:05 fir-io1-s1 kernel: Call Trace: Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 16:32:05 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 16:32:05 fir-io1-s1 kernel: LNet: Service thread pid 96269 was inactive for 200.39s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 16:32:05 fir-io1-s1 kernel: LNet: Skipped 11 previous similar messages Feb 23 16:32:06 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550968326.96281 Feb 23 16:32:10 fir-io1-s1 kernel: LNet: Service thread pid 96242 was inactive for 200.31s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 23 16:32:10 fir-io1-s1 kernel: LNet: Skipped 8 previous similar messages Feb 23 16:32:10 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550968330.96242 Feb 23 16:32:20 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 16:32:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8935155c-923e-0e1a-614d-043a8b7360f6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480eeb5400, cur 1550968342 expire 1550968192 last 1550968115 Feb 23 16:32:22 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 23 16:32:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8935155c-923e-0e1a-614d-043a8b7360f6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480461f800, cur 1550968347 expire 1550968197 last 1550968120 Feb 23 16:32:27 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 23 16:32:28 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8935155c-923e-0e1a-614d-043a8b7360f6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986813a77800, cur 1550968348 expire 1550968198 last 1550968121 Feb 23 16:32:28 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 23 16:32:28 fir-io1-s1 kernel: LNet: Service thread pid 96356 completed after 223.75s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 23 16:32:28 fir-io1-s1 kernel: LNet: Skipped 14 previous similar messages Feb 23 16:35:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 16:35:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 16:36:14 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7f5beeff-0d6a-4733-4195-4c7221570e39 (at 10.9.102.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043f9800, cur 1550968574 expire 1550968424 last 1550968347 Feb 23 16:36:14 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 23 16:36:29 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a70355fa-62e2-5007-8b78-0a9448aecdda (at 10.9.102.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9874d460c800, cur 1550968589 expire 1550968439 last 1550968362 Feb 23 16:36:29 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 23 16:40:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1815ed5c-ff36-8b6d-f1d6-dad784199dec (at 10.9.107.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825fc800, cur 1550968814 expire 1550968664 last 1550968587 Feb 23 16:40:14 fir-io1-s1 kernel: Lustre: Skipped 19 previous similar messages Feb 23 16:40:33 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 16:50:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 649d9284-d440-0c7f-1ed6-43d836dcd91b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481df00800, cur 1550969422 expire 1550969272 last 1550969195 Feb 23 16:50:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 16:51:38 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 79d8a0e0-7e23-2184-06c2-32c9e0b34341 (at 10.9.106.24@o2ib4) in 155 seconds. I think it's dead, and I am evicting it. exp ffff984ad81ad800, cur 1550969498 expire 1550969348 last 1550969343 Feb 23 16:51:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 16:51:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 16:51:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 16:53:04 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 16:55:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 16:55:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 16:56:15 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 87c0c300-07f4-086b-b0d2-a53fd4dfdbea (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a0c000, cur 1550969775 expire 1550969625 last 1550969548 Feb 23 16:56:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:03:45 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client cd890cd0-1d77-3613-bdc4-492a35d0a4fe (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e2e1b3c00, cur 1550970225 expire 1550970075 last 1550969998 Feb 23 17:03:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:04:38 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 17:06:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d99aa6c5-95ff-be26-f78a-b1cfe9fb5439 (at 10.9.101.70@o2ib4) Feb 23 17:06:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:06:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 17:06:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:10:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1815ed5c-ff36-8b6d-f1d6-dad784199dec (at 10.9.107.72@o2ib4) Feb 23 17:10:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:10:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bcdbcaa2-b5a0-6ff6-1390-a90accf35015 (at 10.9.106.20@o2ib4) Feb 23 17:10:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:12:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to c79ca333-f9c5-5051-a3db-11ab3a354438 (at 10.9.102.65@o2ib4) Feb 23 17:12:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:14:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a70355fa-62e2-5007-8b78-0a9448aecdda (at 10.9.102.59@o2ib4) Feb 23 17:14:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:15:11 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 779500f5-04a5-6db6-eecf-3ec5aecdf9bc (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857f2762000, cur 1550970911 expire 1550970761 last 1550970684 Feb 23 17:15:11 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 17:18:06 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 17:20:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 17:20:17 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 23 17:27:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 17:27:29 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 17:27:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5b322ed5-1f83-3d4a-541c-ca479ed4d108 (at 10.8.18.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867860be000, cur 1550971671 expire 1550971521 last 1550971444 Feb 23 17:27:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 17:36:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b3c5bc63-aa36-b5eb-1ad9-6c8f48fdb4c3 (at 10.9.106.68@o2ib4) Feb 23 17:36:51 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 17:47:47 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 17:56:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5b322ed5-1f83-3d4a-541c-ca479ed4d108 (at 10.8.18.18@o2ib6) Feb 23 17:56:39 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 17:56:39 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 18:01:33 fir-io1-s1 kernel: Lustre: 96782:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550973686/real 1550973686] req@ffff9854adb39800 x1624993261809360/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550973693 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 18:01:33 fir-io1-s1 kernel: Lustre: 96782:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 257 previous similar messages Feb 23 18:01:55 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550973707/real 1550973707] req@ffff984c9b57d400 x1624993261809296/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550973714 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 18:01:55 fir-io1-s1 kernel: Lustre: 94358:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550973707/real 1550973707] req@ffff985485cefb00 x1624993261809440/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550973714 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 18:01:55 fir-io1-s1 kernel: Lustre: 94358:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 23 18:01:55 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 23 18:02:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 89311c25-5811-466c-95ed-d7a183bd4753 (at 10.9.113.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848807f3c00, cur 1550973736 expire 1550973586 last 1550973509 Feb 23 18:02:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 18:02:36 fir-io1-s1 kernel: Lustre: 94358:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550973749/real 1550973749] req@ffff985485cefb00 x1624993261809440/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550973756 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 18:02:36 fir-io1-s1 kernel: Lustre: 94358:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 23 18:03:52 fir-io1-s1 kernel: Lustre: 82278:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550973825/real 1550973825] req@ffff985d4a24a400 x1624993390458976/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550973832 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 18:03:52 fir-io1-s1 kernel: Lustre: 82278:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 62 previous similar messages Feb 23 18:04:47 fir-io1-s1 kernel: LNet: Service thread pid 94237 was inactive for 200.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 23 18:04:47 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 23 18:04:47 fir-io1-s1 kernel: Pid: 94237, comm: ll_ost00_002 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 18:04:47 fir-io1-s1 kernel: Call Trace: Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 18:04:47 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1550973887.94237 Feb 23 18:04:47 fir-io1-s1 kernel: Pid: 96895, comm: ll_ost01_084 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 18:04:47 fir-io1-s1 kernel: Call Trace: Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 18:04:47 fir-io1-s1 kernel: Pid: 94358, comm: ll_ost01_004 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 18:04:47 fir-io1-s1 kernel: Call Trace: Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 18:04:47 fir-io1-s1 kernel: Pid: 96782, comm: ll_ost01_075 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 23 18:04:47 fir-io1-s1 kernel: Call Trace: Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 23 18:04:47 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 23 18:05:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e53f2add-358d-9226-610c-379e20464883 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858b60bb400, cur 1550973907 expire 1550973757 last 1550973680 Feb 23 18:05:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 18:05:07 fir-io1-s1 kernel: LNet: Service thread pid 96895 completed after 220.10s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 23 18:05:07 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Feb 23 18:05:09 fir-io1-s1 kernel: LNet: Service thread pid 96782 completed after 222.07s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 23 18:05:09 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Feb 23 18:11:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 18:11:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 18:26:14 fir-io1-s1 kernel: Lustre: 94931:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550975167/real 1550975167] req@ffff987690d2a400 x1624994613457472/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550975174 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 23 18:26:14 fir-io1-s1 kernel: Lustre: 94931:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 168 previous similar messages Feb 23 18:26:35 fir-io1-s1 kernel: Lustre: 96781:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550975188/real 1550975188] req@ffff9849d8fd1800 x1624994613457808/t0(0) o106->fir-OST0008@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550975195 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 23 18:26:35 fir-io1-s1 kernel: Lustre: 96781:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 23 18:26:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 195f15d8-3dd1-29a7-37b0-04fae019fdda (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762745000, cur 1550975216 expire 1550975066 last 1550974989 Feb 23 18:26:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 18:26:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 89311c25-5811-466c-95ed-d7a183bd4753 (at 10.9.113.15@o2ib4) Feb 23 18:26:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 18:27:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 23 18:27:43 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 18:55:14 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 19:30:50 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client da5f786b-e264-8f88-0ed4-a578a7fe601a (at 10.8.23.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a0a400, cur 1550979050 expire 1550978900 last 1550978823 Feb 23 19:30:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 19:31:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 19:31:20 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 19:34:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 19:34:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 19:35:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 277a4ec7-3ac0-e45e-087f-65fdd568c699 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d2bc00, cur 1550979333 expire 1550979183 last 1550979106 Feb 23 19:35:33 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 19:40:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 19:40:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 19:40:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b04ede16-f617-a354-5e6b-afb6c0d8e2be (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984fcccf8000, cur 1550979657 expire 1550979507 last 1550979430 Feb 23 19:40:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 19:43:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 19:43:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 19:43:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1e30c562-725d-f9e2-396f-80fbfff6f7eb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811fec00, cur 1550979837 expire 1550979687 last 1550979610 Feb 23 19:43:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 19:44:47 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 19:49:26 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 20:00:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to da5f786b-e264-8f88-0ed4-a578a7fe601a (at 10.8.23.4@o2ib6) Feb 23 20:00:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:00:41 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ed540f5a-5df6-5998-8f5c-40181564f690 (at 10.9.106.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd0f7c00, cur 1550980841 expire 1550980691 last 1550980614 Feb 23 20:00:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:01:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 20:01:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:01:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 43049bb3-d88e-8796-1399-ea6ec6e5b847 (at 10.8.20.15@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff98565d8dec00, cur 1550980917 expire 1550980767 last 1550980695 Feb 23 20:01:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:12:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 20:12:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:14:01 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b79cd924-2ec2-718f-1c46-4394a6d2125f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a2000, cur 1550981641 expire 1550981491 last 1550981414 Feb 23 20:14:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:23:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 20:23:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:24:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2c492eab-1ab7-fa5b-2363-46ab117eeeeb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d4400, cur 1550982262 expire 1550982112 last 1550982035 Feb 23 20:24:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 20:33:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 20:33:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:34:32 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d8e6a916-5b4d-3836-2a5b-60df1017573e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480073ec00, cur 1550982872 expire 1550982722 last 1550982645 Feb 23 20:34:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:35:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ed540f5a-5df6-5998-8f5c-40181564f690 (at 10.9.106.50@o2ib4) Feb 23 20:35:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:36:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 20:36:44 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 20:37:41 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 84df42b9-7ba8-ae4b-dc70-c300392f3d7f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483ed48800, cur 1550983061 expire 1550982911 last 1550982834 Feb 23 20:37:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:42:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c3a20931-5dd5-dc38-eb14-c9d6c948f9a6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867851e3000, cur 1550983333 expire 1550983183 last 1550983106 Feb 23 20:42:13 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 20:42:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 20:42:27 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 23 20:47:25 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client aaae15f5-40c7-36c9-4422-d6a04c2bbf73 (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387b8800, cur 1550983645 expire 1550983495 last 1550983418 Feb 23 20:47:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:47:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 23 20:47:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:50:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9cc869db-86b5-83f4-3d1d-91c30da3920c (at 10.8.14.1@o2ib6) Feb 23 20:50:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:51:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 704d0c5c-d43d-2b11-670d-103d1e46872c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4e9000, cur 1550983899 expire 1550983749 last 1550983672 Feb 23 20:51:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:54:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 23 20:54:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 20:57:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 20:57:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 20:58:14 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4e9aeae9-1e65-db2b-5528-4eab52bc8f4b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a29800, cur 1550984294 expire 1550984144 last 1550984067 Feb 23 20:58:14 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 20:59:30 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 67239ca2-2a0c-e30b-bcc0-50c3477fbd39 (at 10.8.18.35@o2ib6) in 187 seconds. I think it's dead, and I am evicting it. exp ffff9849c8262800, cur 1550984370 expire 1550984220 last 1550984183 Feb 23 20:59:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 21:03:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 23 21:03:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 21:04:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0ea29eaf-f381-711e-01fc-797c696ea7f1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987832ebac00, cur 1550984663 expire 1550984513 last 1550984436 Feb 23 21:04:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 21:06:00 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 21:06:00 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 23 21:08:31 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bf2986a3-2b30-ff41-fb48-c2c1537498c9 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985836977000, cur 1550984911 expire 1550984761 last 1550984684 Feb 23 21:08:31 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 21:11:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 760698fb-b23d-f0aa-e1bc-00ba189aa748 (at 10.9.105.12@o2ib4) Feb 23 21:11:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 21:16:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 592dee67-1b91-10cf-db31-4cb0d227025d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984835c94800, cur 1550985380 expire 1550985230 last 1550985153 Feb 23 21:16:20 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 21:20:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 21:20:16 fir-io1-s1 kernel: Lustre: Skipped 24 previous similar messages Feb 23 21:26:10 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 87ceed2e-4992-6545-afaf-061798c4f899 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8267400, cur 1550985970 expire 1550985820 last 1550985743 Feb 23 21:26:10 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 21:34:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 21:34:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 21:38:24 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9b1bd111-c33a-7337-38d4-3e7f51605276 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8260400, cur 1550986704 expire 1550986554 last 1550986477 Feb 23 21:38:24 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 21:41:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 06a0640d-de7f-9715-947b-5ac203d15e9f (at 10.0.10.3@o2ib7) reconnecting Feb 23 21:44:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 23 21:44:47 fir-io1-s1 kernel: Lustre: Skipped 30 previous similar messages Feb 23 21:49:50 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e1c91a31-b7db-4840-fe86-9e44f79ca97a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c9f87f000, cur 1550987390 expire 1550987240 last 1550987163 Feb 23 21:49:50 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 23 21:51:49 fir-io1-s1 kernel: LustreError: 96375:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff984911fc2a00 x1625004488959440 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff983853999d40/0x49e185f24c7b33cc lrc: 3/0,0 mode: PW/PW res: [0xc40000402:0x1a4cf4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000080020000 nid: 10.8.20.15@o2ib6 remote: 0x833932dd65cab8d1 expref: 19 pid: 96328 timeout: 0 lvb_type: 0 Feb 23 21:51:49 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 23 21:51:49 fir-io1-s1 kernel: LustreError: Skipped 7 previous similar messages Feb 23 21:51:49 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1550987508s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff98651bbf86c0/0x49e185f24c7b29f4 lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x1a4d94:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000080020000 nid: 10.8.20.15@o2ib6 remote: 0x833932dd65cab861 expref: 30 pid: 96275 timeout: 0 lvb_type: 0 Feb 23 21:51:49 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages Feb 23 21:51:49 fir-io1-s1 kernel: LustreError: 96375:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages Feb 23 21:56:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d0459d48-5519-5735-755e-c95d121b2e6f (at 10.8.18.13@o2ib6) Feb 23 21:56:26 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 22:02:39 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5c1bee41-ad82-62af-493b-3a9e7abfcb5f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98776adfe800, cur 1550988159 expire 1550988009 last 1550987932 Feb 23 22:02:39 fir-io1-s1 kernel: Lustre: Skipped 13 previous similar messages Feb 23 22:08:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6663feee-c2d9-0be2-ebe4-112c26106cc2 (at 10.8.31.10@o2ib6) Feb 23 22:08:40 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 22:17:01 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 92df3c77-8f2b-0671-f9cd-494ab136ccd1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857630fb000, cur 1550989021 expire 1550988871 last 1550988794 Feb 23 22:17:01 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 22:18:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 23 22:18:58 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 22:29:34 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9c0a137b-d6f4-5152-5701-23443126b888 (at 10.9.102.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480332a000, cur 1550989774 expire 1550989624 last 1550989547 Feb 23 22:29:34 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Feb 23 22:32:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 22:32:51 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 22:43:43 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fab7f039-47ba-e6b2-d393-5785434d6f03 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583baf4000, cur 1550990623 expire 1550990473 last 1550990396 Feb 23 22:43:43 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Feb 23 22:44:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 23 22:44:34 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 23 22:53:29 fir-io1-s1 kernel: LNetError: 91392:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 22:54:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 75ddd0cd-f952-bbd8-4bd6-23cded8d16e6 (at 10.8.20.15@o2ib6) in 181 seconds. I think it's dead, and I am evicting it. exp ffff986785d2d000, cur 1550991283 expire 1550991133 last 1550991102 Feb 23 22:54:43 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Feb 23 22:55:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 92f34919-614d-c857-b0e6-2bbe68fc85f2 (at 10.8.23.28@o2ib6) Feb 23 22:55:28 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 23 23:05:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2b3e309f-8d18-a534-0df1-3ea431690a2a (at 10.9.102.68@o2ib4) Feb 23 23:05:57 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Feb 23 23:06:08 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 23:08:22 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 35bf3b4b-d36a-9779-1037-51ca0465fbd8 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857f2760000, cur 1550992102 expire 1550991952 last 1550991875 Feb 23 23:08:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 23:10:24 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 23 23:16:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e256f342-dd4a-0733-a0fa-e78c997bbd1d (at 10.8.23.27@o2ib6) Feb 23 23:16:02 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 23 23:25:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8883ece8-d067-6391-2453-e8030817f4d2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98622f65cc00, cur 1550993153 expire 1550993003 last 1550992926 Feb 23 23:25:53 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 23 23:27:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8085461b-ece8-9665-6f8f-29041438e681 (at 10.9.102.56@o2ib4) Feb 23 23:27:07 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 23 23:37:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 23 23:37:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 23 23:41:27 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1220edde-79f3-dcff-7608-2ebfef192d48 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833d01400, cur 1550994087 expire 1550993937 last 1550993860 Feb 23 23:41:27 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 23:52:08 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c7cfc7ed-9703-9da0-a322-82b34c36c571 (at 10.8.18.35@o2ib6) in 220 seconds. I think it's dead, and I am evicting it. exp ffff98622f65c400, cur 1550994728 expire 1550994578 last 1550994508 Feb 23 23:52:08 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 23 23:52:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 23 23:52:59 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 00:03:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client aade1654-229d-9ac6-4031-1b9b6c0e5046 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867814dd400, cur 1550995414 expire 1550995264 last 1550995187 Feb 24 00:03:34 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 00:04:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2b64d06d-0a33-7dc1-7c60-2608607acb48 (at 10.9.104.50@o2ib4) Feb 24 00:04:57 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 00:14:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6166ed6f-63c6-e485-6789-20e1472f97bc (at 10.8.18.31@o2ib6) in 171 seconds. I think it's dead, and I am evicting it. exp ffff984e2e1b1400, cur 1550996082 expire 1550995932 last 1550995911 Feb 24 00:14:42 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 00:16:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 00:16:12 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 00:19:24 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 24 00:24:11 fir-io1-s1 kernel: Lustre: 96574:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996644/real 1550996644] req@ffff986745327500 x1625011361742288/t0(0) o106->fir-OST0008@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996651 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 24 00:24:11 fir-io1-s1 kernel: Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996644/real 1550996644] req@ffff9866e779d700 x1625011361742384/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996651 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 24 00:24:11 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996644/real 1550996644] req@ffff9864a7a00c00 x1625011361742368/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996651 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 24 00:24:11 fir-io1-s1 kernel: Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 24 00:24:11 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 24 00:24:11 fir-io1-s1 kernel: Lustre: 96574:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 24 00:24:18 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996651/real 1550996651] req@ffff9864a7a00c00 x1625011361742368/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996658 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:24:18 fir-io1-s1 kernel: Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996651/real 1550996651] req@ffff9866e779d700 x1625011361742384/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996658 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:24:18 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 24 00:24:25 fir-io1-s1 kernel: Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996658/real 1550996658] req@ffff9866e779d700 x1625011361742384/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996665 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:24:25 fir-io1-s1 kernel: Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 24 00:24:32 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996665/real 1550996665] req@ffff9866e779e000 x1625011361742320/t0(0) o106->fir-OST000a@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996672 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:24:32 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 24 00:24:39 fir-io1-s1 kernel: Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996672/real 1550996672] req@ffff9866e779d700 x1625011361742384/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996679 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:24:39 fir-io1-s1 kernel: Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 24 00:24:49 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 66ecac58-2486-7d36-5254-f921b2319821 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852b0f4a400, cur 1550996689 expire 1550996539 last 1550996462 Feb 24 00:24:49 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Feb 24 00:24:53 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996686/real 1550996686] req@ffff9866e779e000 x1625011361742320/t0(0) o106->fir-OST000a@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996693 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:24:53 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 24 00:25:14 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996707/real 1550996707] req@ffff9864a7a00c00 x1625011361742368/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996714 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:25:14 fir-io1-s1 kernel: Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Feb 24 00:25:56 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1550996749/real 1550996749] req@ffff9866e779e000 x1625011361742320/t0(0) o106->fir-OST000a@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1550996756 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 00:25:56 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Feb 24 00:26:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d6335190-14e9-cc30-2081-663fdf52e20a (at 10.9.102.12@o2ib4) Feb 24 00:26:15 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 00:37:07 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client daf49c50-4f68-b584-a597-25cfd06150f7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfe800, cur 1550997427 expire 1550997277 last 1550997200 Feb 24 00:37:07 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Feb 24 00:38:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 00:38:18 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 00:48:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client bae97137-cfec-cfc4-5042-7fc1a249b9e7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832bdb400, cur 1550998136 expire 1550997986 last 1550997909 Feb 24 00:48:56 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 00:49:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 00:49:28 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 00:59:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f9dfa309-1fc7-d45c-2310-5b9b7771f941 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b3cc00, cur 1550998773 expire 1550998623 last 1550998546 Feb 24 00:59:33 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 24 01:01:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 01:01:51 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 01:11:32 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ccc0a221-5bfd-ad41-7305-73f08d2ba036 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83a800, cur 1550999492 expire 1550999342 last 1550999265 Feb 24 01:11:32 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 01:13:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 01:13:17 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 01:22:17 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 06e0ff62-3246-8df2-805f-35b887eaa9db (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852b0f4a400, cur 1551000137 expire 1550999987 last 1550999910 Feb 24 01:22:17 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 01:25:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a045669f-27d9-1372-2514-d5211db1ecd9 (at 10.9.104.68@o2ib4) Feb 24 01:25:30 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Feb 24 01:33:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client da2bb3fc-3dba-6cc2-6262-cb1153831ce2 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fbda1400, cur 1551000813 expire 1551000663 last 1551000586 Feb 24 01:33:33 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 01:35:02 fir-io1-s1 kernel: Lustre: 2371:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551000858/real 1551000858] req@ffff986b8975ce00 x1625013297068864/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551000902 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 24 01:35:02 fir-io1-s1 kernel: Lustre: 2371:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Feb 24 01:36:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 01:36:20 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 24 01:45:01 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683c83a400, cur 1551001501 expire 1551001351 last 1551001274 Feb 24 01:45:01 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 24 01:54:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 01:54:44 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 02:03:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 35966e1d-e10a-8746-c5a0-1d92cdb81f9a (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83f400, cur 1551002633 expire 1551002483 last 1551002406 Feb 24 02:03:53 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 02:09:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9ab75554-a5d2-ed3d-4577-f298de01252b (at 10.9.106.40@o2ib4) Feb 24 02:09:45 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 02:13:54 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8df1b755-5df0-9888-4bd0-b629235744b4 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855e70dfc00, cur 1551003234 expire 1551003084 last 1551003007 Feb 24 02:13:54 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 24 02:20:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 111255c5-0b7f-306e-408c-6abf6623385a (at 10.9.104.46@o2ib4) Feb 24 02:20:17 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 02:28:02 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client fca022be-6585-be05-88a7-6e814634b560 (at 10.9.106.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786949800, cur 1551004082 expire 1551003932 last 1551003855 Feb 24 02:28:02 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 24 02:33:20 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 02:33:20 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 02:38:24 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c1bd70a3-eca4-5675-8d48-7d20105a8bfe (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984836f2b400, cur 1551004704 expire 1551004554 last 1551004477 Feb 24 02:38:24 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 02:46:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 02:46:28 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 24 02:51:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7dad0ca1-419f-4d61-9748-c16359bd1bcf (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984fcccf9400, cur 1551005516 expire 1551005366 last 1551005289 Feb 24 02:51:56 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 02:58:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 02:58:50 fir-io1-s1 kernel: Lustre: Skipped 24 previous similar messages Feb 24 03:04:12 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 806117d8-bc82-a34c-615c-987857a7f22a (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98749c33d800, cur 1551006252 expire 1551006102 last 1551006025 Feb 24 03:04:12 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 03:11:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e7658bee-b529-b857-21ef-217c5e9fe7b7 (at 10.9.113.9@o2ib4) Feb 24 03:11:01 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Feb 24 03:14:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 40b5deb6-81a6-e2a2-f183-f2a3cc21a383 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dffcc00, cur 1551006889 expire 1551006739 last 1551006662 Feb 24 03:14:49 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 03:22:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to edc5d3f2-dae7-69fe-e7fb-9c4cf59a4b4c (at 10.8.31.8@o2ib6) Feb 24 03:22:12 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 03:26:31 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3d8f46f2-918b-28cb-993c-a13124f540a4 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833d79800, cur 1551007591 expire 1551007441 last 1551007364 Feb 24 03:26:31 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 03:49:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 87721f3b-4f03-c138-ffa3-cffa8a052df0 (at 10.8.26.5@o2ib6) Feb 24 03:49:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 04:00:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 6fccb913-f8e6-2056-033a-4c02e0e89d4f (at 10.9.103.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d64a000, cur 1551009601 expire 1551009451 last 1551009374 Feb 24 04:00:01 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 04:09:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1815d69a-de12-0777-af99-1fa63af02a98 (at 10.9.106.31@o2ib4) Feb 24 04:09:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 04:09:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7e9161e5-27d8-4cac-a415-4c23ea14bc0e (at 10.9.106.46@o2ib4) Feb 24 04:09:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 04:36:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6fccb913-f8e6-2056-033a-4c02e0e89d4f (at 10.9.103.6@o2ib4) Feb 24 04:36:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 04:47:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 04:47:29 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 24 04:47:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 24303766-1de3-4948-4974-2d2d6e46198d (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e7da2bc00, cur 1551012473 expire 1551012323 last 1551012246 Feb 24 04:47:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 04:52:33 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5f797900-3e70-d5b1-6f40-9a3fbbdfbe67 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480332f000, cur 1551012753 expire 1551012603 last 1551012526 Feb 24 04:52:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 04:52:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 04:52:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:04:48 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ed2d79f5-7b8f-9ba3-ea86-1acdd4ac5241 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a0d400, cur 1551013488 expire 1551013338 last 1551013261 Feb 24 05:04:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:05:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ed2d79f5-7b8f-9ba3-ea86-1acdd4ac5241 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483a2e9800, cur 1551013500 expire 1551013350 last 1551013273 Feb 24 05:05:00 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 24 05:05:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 05:05:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:06:04 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client bde58792-3602-962e-df58-c34b9dbd9136 (at 10.9.101.64@o2ib4) in 182 seconds. I think it's dead, and I am evicting it. exp ffff98729d774000, cur 1551013564 expire 1551013414 last 1551013382 Feb 24 05:06:16 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bde58792-3602-962e-df58-c34b9dbd9136 (at 10.9.101.64@o2ib4) in 194 seconds. I think it's dead, and I am evicting it. exp ffff985763312c00, cur 1551013576 expire 1551013426 last 1551013382 Feb 24 05:06:16 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 05:06:49 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client bde58792-3602-962e-df58-c34b9dbd9136 (at 10.9.101.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857618e0800, cur 1551013609 expire 1551013459 last 1551013382 Feb 24 05:15:04 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4c6307e0-e5e3-3295-9e75-ac7ce5c5822c (at 10.9.104.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857606b5c00, cur 1551014104 expire 1551013954 last 1551013877 Feb 24 05:15:04 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 24 05:20:42 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b3923a77-9e45-3833-8aa3-c4d76a48c186 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c46c00, cur 1551014442 expire 1551014292 last 1551014215 Feb 24 05:20:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:22:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 05:22:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:26:19 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3372d8ae-4a24-9eb1-fcd7-7dd890b14dc0 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d6c00, cur 1551014779 expire 1551014629 last 1551014552 Feb 24 05:26:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:28:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 05:28:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:29:05 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551014938/real 1551014938] req@ffff9851a6875a00 x1625013379765952/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551014945 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 24 05:29:05 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 24 05:29:12 fir-io1-s1 kernel: Lustre: 96359:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551014945/real 1551014945] req@ffff985187054800 x1625013379766000/t0(0) o106->fir-OST0008@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551014952 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 05:29:12 fir-io1-s1 kernel: Lustre: 96359:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 24 05:29:19 fir-io1-s1 kernel: Lustre: 96773:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551014952/real 1551014952] req@ffff986bb5127200 x1625013379765968/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551014959 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 05:29:19 fir-io1-s1 kernel: Lustre: 96773:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 24 05:29:26 fir-io1-s1 kernel: Lustre: 96243:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551014959/real 1551014959] req@ffff984f2aa62d00 x1625013379765984/t0(0) o106->fir-OST0006@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551014966 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 05:29:26 fir-io1-s1 kernel: Lustre: 96243:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 24 05:29:40 fir-io1-s1 kernel: Lustre: 96779:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551014973/real 1551014973] req@ffff9844c2ef4b00 x1625013379766144/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551014980 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 05:29:40 fir-io1-s1 kernel: Lustre: 96779:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Feb 24 05:30:01 fir-io1-s1 kernel: Lustre: 96275:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551014994/real 1551014994] req@ffff986740023000 x1625013379766160/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551015001 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 05:30:01 fir-io1-s1 kernel: Lustre: 96275:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Feb 24 05:30:43 fir-io1-s1 kernel: Lustre: 96367:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551015036/real 1551015036] req@ffff986232e6d700 x1625013379766192/t0(0) o106->fir-OST0008@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551015043 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 05:30:43 fir-io1-s1 kernel: Lustre: 96367:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 85 previous similar messages Feb 24 05:31:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 88d745c2-8792-e718-77da-e1b0f185d9b0 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987717fbf800, cur 1551015114 expire 1551014964 last 1551014887 Feb 24 05:31:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:32:00 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551015113/real 1551015113] req@ffff9851a6875a00 x1625013379765952/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551015120 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 24 05:32:00 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 157 previous similar messages Feb 24 05:32:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 05:32:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:36:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 05:36:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:41:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.64@o2ib4) Feb 24 05:41:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:44:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 05:44:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:44:59 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 80174adc-e835-6e66-d8ba-fd461f223dfc (at 10.9.101.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cb0800, cur 1551015899 expire 1551015749 last 1551015672 Feb 24 05:44:59 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 05:47:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 05:47:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:50:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.104.62@o2ib4) Feb 24 05:50:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:52:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 05:52:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 05:55:00 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6880ca8c-0ab1-e4af-8e15-a4cb158645a5 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98680476dc00, cur 1551016500 expire 1551016350 last 1551016273 Feb 24 05:55:00 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 05:56:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 05:56:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 06:02:31 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 06:02:31 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 24 06:05:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 39f91ffc-f969-2d2d-6e9d-ce4a2e82af12 (at 10.9.103.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677dc5c400, cur 1551017140 expire 1551016990 last 1551016913 Feb 24 06:05:40 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 24 06:16:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 24 06:16:28 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 24 06:20:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9f8248b9-38c3-92e5-4b28-b959529f9fe4 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c9f879400, cur 1551018015 expire 1551017865 last 1551017788 Feb 24 06:20:15 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Feb 24 06:27:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce2a5a1a-545c-760b-44a7-8c19aadb7a36 (at 10.9.107.71@o2ib4) Feb 24 06:27:59 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 06:37:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f8c56e48-2583-434a-b50d-30254178caf9 (at 10.9.103.20@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786880400, cur 1551019062 expire 1551018912 last 1551018835 Feb 24 06:37:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 06:44:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f88d3e4f-b8ad-7e3f-e052-b857e571de2a (at 10.9.107.13@o2ib4) Feb 24 06:44:37 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 07:08:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 276945c0-0d00-01d6-84a8-1f1c209c1b8f (at 10.9.103.36@o2ib4) Feb 24 07:08:27 fir-io1-s1 kernel: Lustre: Skipped 46 previous similar messages Feb 24 07:13:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f8c56e48-2583-434a-b50d-30254178caf9 (at 10.9.103.20@o2ib4) Feb 24 07:13:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 07:26:45 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client eb0b17ca-746e-7622-abd4-371c493253d0 (at 10.9.103.24@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f67ec00, cur 1551022005 expire 1551021855 last 1551021778 Feb 24 07:26:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 07:37:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf0ac00, cur 1551022627 expire 1551022477 last 1551022400 Feb 24 07:37:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 07:57:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 04c67229-21fb-0235-15ed-cccc9063a531 (at 10.8.27.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c0b800, cur 1551023862 expire 1551023712 last 1551023635 Feb 24 07:57:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:02:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to eb0b17ca-746e-7622-abd4-371c493253d0 (at 10.9.103.24@o2ib4) Feb 24 08:02:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:11:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) Feb 24 08:11:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:15:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7fe2e6bd-17ee-5658-a394-750649bff28a (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d53c71400, cur 1551024901 expire 1551024751 last 1551024674 Feb 24 08:15:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:22:46 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7f26f4a5-b09c-90cc-57f5-181682b8827f (at 10.9.103.33@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848343dc400, cur 1551025366 expire 1551025216 last 1551025139 Feb 24 08:22:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:26:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 04c67229-21fb-0235-15ed-cccc9063a531 (at 10.8.27.18@o2ib6) Feb 24 08:26:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:29:37 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2196625d-9992-1b8a-5a12-40751a9cdd4e (at 10.9.107.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a0fc00, cur 1551025777 expire 1551025627 last 1551025550 Feb 24 08:29:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:42:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 488cfd93-1121-504d-019d-485c13be114d (at 10.8.14.4@o2ib6) Feb 24 08:42:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:53:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7f26f4a5-b09c-90cc-57f5-181682b8827f (at 10.9.103.33@o2ib4) Feb 24 08:53:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 08:57:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2196625d-9992-1b8a-5a12-40751a9cdd4e (at 10.9.107.2@o2ib4) Feb 24 08:57:29 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 24 09:57:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 77963223-bc75-0922-f3f9-87c125865623 (at 10.8.31.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768559400, cur 1551031057 expire 1551030907 last 1551030830 Feb 24 09:57:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 10:18:19 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 967250d6-c1de-8fd6-c33c-9b0bc69f4cab (at 10.9.103.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762603c00, cur 1551032299 expire 1551032149 last 1551032072 Feb 24 10:18:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 10:23:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 6da5269e-e6e7-e930-ea8b-e990b1fd18b0 (at 10.9.101.72@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a9ec00, cur 1551032621 expire 1551032471 last 1551032394 Feb 24 10:23:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 10:26:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 77963223-bc75-0922-f3f9-87c125865623 (at 10.8.31.5@o2ib6) Feb 24 10:26:26 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 24 10:48:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 92ae2964-22f0-d9af-2db7-23fcbd1fe55b (at 10.9.102.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1414400, cur 1551034127 expire 1551033977 last 1551033900 Feb 24 10:48:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 10:52:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811fc000, cur 1551034364 expire 1551034214 last 1551034137 Feb 24 10:52:44 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 10:52:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768558000, cur 1551034369 expire 1551034219 last 1551034142 Feb 24 10:52:49 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 24 10:54:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 967250d6-c1de-8fd6-c33c-9b0bc69f4cab (at 10.9.103.5@o2ib4) Feb 24 10:54:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 10:57:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6da5269e-e6e7-e930-ea8b-e990b1fd18b0 (at 10.9.101.72@o2ib4) Feb 24 10:57:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 10:58:41 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ddd1b310-42a4-1c93-bc8a-ea4ff10c3b50 (at 10.8.10.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857606b6800, cur 1551034721 expire 1551034571 last 1551034494 Feb 24 10:58:41 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 11:07:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 96fb107e-6354-4a71-2925-e1f8a9a58d15 (at 10.9.103.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581ef42800, cur 1551035229 expire 1551035079 last 1551035002 Feb 24 11:07:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:24:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a28c293f-ee5d-ad54-991b-4f6f191450b5 (at 10.9.102.57@o2ib4) Feb 24 11:24:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:25:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 92ae2964-22f0-d9af-2db7-23fcbd1fe55b (at 10.9.102.69@o2ib4) Feb 24 11:25:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:25:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 855a0732-43f8-e86e-9a9d-26067a3c54be (at 10.9.102.64@o2ib4) Feb 24 11:25:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:26:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f5fe3def-7261-239e-1d83-4ac81d4cfaa5 (at 10.9.102.61@o2ib4) Feb 24 11:26:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:30:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dad15dc4-4b75-8f93-57ae-ea1cf5361955 (at 10.9.105.16@o2ib4) Feb 24 11:30:32 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 24 11:31:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ddd1b310-42a4-1c93-bc8a-ea4ff10c3b50 (at 10.8.10.11@o2ib6) Feb 24 11:31:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:42:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.103.21@o2ib4) Feb 24 11:42:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:42:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 829cb29b-c33b-daf1-5d36-1b68d0eb41a6 (at 10.9.107.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848000eb000, cur 1551037334 expire 1551037184 last 1551037107 Feb 24 11:42:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 11:42:26 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 829cb29b-c33b-daf1-5d36-1b68d0eb41a6 (at 10.9.107.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98783b701800, cur 1551037346 expire 1551037196 last 1551037119 Feb 24 11:42:26 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 11:57:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 14d1e2af-bfe3-6fa6-de44-b510e2b94a1a (at 10.9.101.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986782308c00, cur 1551038225 expire 1551038075 last 1551037998 Feb 24 11:57:05 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 24 12:07:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) Feb 24 12:07:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 12:31:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 14d1e2af-bfe3-6fa6-de44-b510e2b94a1a (at 10.9.101.62@o2ib4) Feb 24 12:31:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 12:44:23 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1ee3037c-52dd-207d-3196-b589ce5ac006 (at 10.9.114.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835775800, cur 1551041063 expire 1551040913 last 1551040836 Feb 24 12:44:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:01:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3e2bfa45-013a-e48d-7bcc-c486bbeaa49b (at 10.9.108.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786921000, cur 1551042118 expire 1551041968 last 1551041891 Feb 24 13:01:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:04:59 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1b1c689b-37b8-b4bb-d7e3-6c60b9889af8 (at 10.9.106.21@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15a7400, cur 1551042299 expire 1551042149 last 1551042072 Feb 24 13:04:59 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Feb 24 13:09:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1ee3037c-52dd-207d-3196-b589ce5ac006 (at 10.9.114.14@o2ib4) Feb 24 13:09:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:17:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b3fc52f5-cc19-f1e2-5d13-43190203fae8 (at 10.9.106.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b616c00, cur 1551043046 expire 1551042896 last 1551042819 Feb 24 13:17:26 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 13:39:34 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 54df73a3-4915-b589-f8b2-dd262402c8c5 (at 10.9.107.65@o2ib4) Feb 24 13:39:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:39:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0dab976e-a8e8-3b9f-2d0d-436920c3d0f0 (at 10.9.108.3@o2ib4) Feb 24 13:39:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:39:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3e2bfa45-013a-e48d-7bcc-c486bbeaa49b (at 10.9.108.9@o2ib4) Feb 24 13:39:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:40:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to be5b5c3f-91d7-81de-8f26-0a39412de9ac (at 10.9.108.1@o2ib4) Feb 24 13:40:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:41:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1ba76412-2df3-2c43-7ffc-b423548b8d30 (at 10.9.108.15@o2ib4) Feb 24 13:41:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:41:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 46d03983-7b94-1e28-0f8d-5e7a68fcc2fa (at 10.9.108.13@o2ib4) Feb 24 13:41:44 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 24 13:42:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2acb2116-5227-530a-f563-866a3449ba51 (at 10.9.106.13@o2ib4) Feb 24 13:42:14 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 24 13:42:34 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1e0dbe81-97a8-a9d0-3976-d5a8c6b1ba02 (at 10.9.108.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986dfabbb800, cur 1551044554 expire 1551044404 last 1551044327 Feb 24 13:42:34 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 13:47:06 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.106.19@o2ib4) Feb 24 13:47:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 13:48:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d1eafd0d-f5d5-63d1-f545-e28e22ce25f0 (at 10.9.106.21@o2ib4) Feb 24 13:48:38 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 14:10:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Feb 24 14:10:22 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 24 14:13:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c8d3469c-ec1f-9a7d-c4a5-37f7678112b1 (at 10.9.108.5@o2ib4) Feb 24 14:13:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 14:14:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e517df84-1255-2b59-31b3-99ff3b5db2cd (at 10.9.108.8@o2ib4) Feb 24 14:14:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 14:27:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9042fb3a-c0ab-6915-0268-4626f11a023e (at 10.9.106.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575e61d800, cur 1551047230 expire 1551047080 last 1551047003 Feb 24 14:27:10 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 14:52:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 275e2730-4d3f-dc89-2b66-e7a8cc62e3d6 (at 10.8.25.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a9c386400, cur 1551048766 expire 1551048616 last 1551048539 Feb 24 14:52:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:00:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9042fb3a-c0ab-6915-0268-4626f11a023e (at 10.9.106.45@o2ib4) Feb 24 15:00:52 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 15:21:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 275e2730-4d3f-dc89-2b66-e7a8cc62e3d6 (at 10.8.25.28@o2ib6) Feb 24 15:21:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:24:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5d079231-9ed4-0730-7be9-e123819c7379 (at 10.8.13.22@o2ib6) Feb 24 15:24:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:24:31 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 36c47535-7f19-9692-c5d3-687f789af19d (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848831e2000, cur 1551050671 expire 1551050521 last 1551050444 Feb 24 15:24:31 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 15:26:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 24 15:26:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:28:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d205800, cur 1551050886 expire 1551050736 last 1551050659 Feb 24 15:28:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:37:59 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f00f7e2a-8ac6-86cf-8aa7-d26ad9d6b9e7 (at 10.9.106.47@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009fac00, cur 1551051479 expire 1551051329 last 1551051252 Feb 24 15:37:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:44:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8f440af3-cd92-379d-a078-f053f705469f (at 10.9.106.58@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df1400, cur 1551051888 expire 1551051738 last 1551051661 Feb 24 15:44:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:47:28 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client cd13369b-5f5c-de37-391a-25d067b062d5 (at 10.9.106.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984f5a25a800, cur 1551052048 expire 1551051898 last 1551051821 Feb 24 15:47:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 15:58:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 15:58:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:05:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 24 16:05:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:07:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client db209fb3-e752-728f-7a78-57920189bb31 (at 10.9.106.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd0f2400, cur 1551053255 expire 1551053105 last 1551053028 Feb 24 16:07:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:08:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d9f51967-967f-e724-b4d7-c6424894c591 (at 10.8.30.2@o2ib6) in 223 seconds. I think it's dead, and I am evicting it. exp ffff985756586c00, cur 1551053331 expire 1551053181 last 1551053108 Feb 24 16:08:51 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 24 16:08:55 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d9f51967-967f-e724-b4d7-c6424894c591 (at 10.8.30.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583baf6400, cur 1551053335 expire 1551053185 last 1551053108 Feb 24 16:08:55 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 24 16:10:07 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 032aae30-e439-a130-1f18-efe924baca21 (at 10.9.106.70@o2ib4) in 194 seconds. I think it's dead, and I am evicting it. exp ffff987624cf5c00, cur 1551053407 expire 1551053257 last 1551053213 Feb 24 16:10:07 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 24 16:10:40 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 032aae30-e439-a130-1f18-efe924baca21 (at 10.9.106.70@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857620a6c00, cur 1551053440 expire 1551053290 last 1551053213 Feb 24 16:10:40 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 24 16:12:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f00f7e2a-8ac6-86cf-8aa7-d26ad9d6b9e7 (at 10.9.106.47@o2ib4) Feb 24 16:12:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:13:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8f440af3-cd92-379d-a078-f053f705469f (at 10.9.106.58@o2ib4) Feb 24 16:13:10 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 24 16:21:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cd13369b-5f5c-de37-391a-25d067b062d5 (at 10.9.106.64@o2ib4) Feb 24 16:21:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:35:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c50e9e63-bc69-ffb4-d9c5-0a1d77a8b849 (at 10.9.106.60@o2ib4) Feb 24 16:35:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:38:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ee98715-4bcf-b4e8-27bc-89f9f237369b (at 10.8.30.5@o2ib6) Feb 24 16:38:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:38:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 01bf31bc-51a2-4b95-74f5-d8893ea0150c (at 10.8.30.4@o2ib6) Feb 24 16:38:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:39:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e5be9ff2-873f-0542-6c5f-13af50413057 (at 10.8.30.1@o2ib6) Feb 24 16:39:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:42:27 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8766aca3-0b1a-78de-082b-08cc790415f9 (at 10.8.30.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857f2760800, cur 1551055347 expire 1551055197 last 1551055120 Feb 24 16:42:27 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 16:42:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 01bf31bc-51a2-4b95-74f5-d8893ea0150c (at 10.8.30.4@o2ib6) Feb 24 16:42:39 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 16:43:43 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 56d4d565-fe73-cd79-8098-610edfde7d3b (at 10.9.107.51@o2ib4) in 205 seconds. I think it's dead, and I am evicting it. exp ffff98575f4df400, cur 1551055423 expire 1551055273 last 1551055218 Feb 24 16:43:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:44:05 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 56d4d565-fe73-cd79-8098-610edfde7d3b (at 10.9.107.51@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5dc00, cur 1551055445 expire 1551055295 last 1551055218 Feb 24 16:44:05 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 16:44:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 032aae30-e439-a130-1f18-efe924baca21 (at 10.9.106.70@o2ib4) Feb 24 16:44:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 16:44:59 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client de4782a5-e692-37bf-ba9c-6501c38ac58a (at 10.9.107.49@o2ib4) in 155 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c7800, cur 1551055499 expire 1551055349 last 1551055344 Feb 24 16:44:59 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 24 16:45:21 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client de4782a5-e692-37bf-ba9c-6501c38ac58a (at 10.9.107.49@o2ib4) in 176 seconds. I think it's dead, and I am evicting it. exp ffff986785d50400, cur 1551055521 expire 1551055371 last 1551055345 Feb 24 16:45:21 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 16:46:11 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client de4782a5-e692-37bf-ba9c-6501c38ac58a (at 10.9.107.49@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b88d000, cur 1551055571 expire 1551055421 last 1551055344 Feb 24 16:46:11 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 17:08:37 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 861509dc-4523-55e8-f09a-16b6fca3f713 (at 10.9.107.51@o2ib4) Feb 24 17:08:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 17:10:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 58a24522-e5c5-5b7c-258b-500f9e3166ed (at 10.9.107.49@o2ib4) Feb 24 17:10:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 17:53:49 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a6d2f42c-4a04-9c4a-cc81-70ba500cf671 (at 10.9.106.38@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767c4fc00, cur 1551059629 expire 1551059479 last 1551059402 Feb 24 17:53:49 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 24 18:28:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a6d2f42c-4a04-9c4a-cc81-70ba500cf671 (at 10.9.106.38@o2ib4) Feb 24 18:28:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 19:02:14 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b312cb06-2b52-b7e1-8133-1c9687ba2033 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984aa3fb6000, cur 1551063734 expire 1551063584 last 1551063507 Feb 24 19:02:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 19:13:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 19:13:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 19:23:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1ecf1ab2-d481-bbed-3893-941aef9b4486 (at 10.8.11.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f4c00, cur 1551065015 expire 1551064865 last 1551064788 Feb 24 19:23:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 19:39:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 86b1a898-94cc-a41f-85e1-12abf7418dc9 (at 10.8.7.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c89800, cur 1551065988 expire 1551065838 last 1551065761 Feb 24 19:39:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 19:39:52 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5ba2142c-879f-ac28-9d4a-a3788afebea0 (at 10.8.7.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a9c387800, cur 1551065992 expire 1551065842 last 1551065765 Feb 24 19:39:52 fir-io1-s1 kernel: Lustre: Skipped 31 previous similar messages Feb 24 19:55:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1ecf1ab2-d481-bbed-3893-941aef9b4486 (at 10.8.11.23@o2ib6) Feb 24 19:55:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:01:52 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 680b2f8b-c457-9b65-c1bc-58d27b9ae2fe (at 10.8.27.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b5d687000, cur 1551067312 expire 1551067162 last 1551067085 Feb 24 20:01:52 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Feb 24 20:04:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 20:04:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:05:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c3535875-d3c1-a4f1-4d7f-21c04068d08d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801510c00, cur 1551067514 expire 1551067364 last 1551067287 Feb 24 20:05:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:08:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd525d3e-a429-d051-4dce-9b431ca4655f (at 10.8.7.33@o2ib6) Feb 24 20:08:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:09:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 686d431b-2171-7d61-28d2-83eee0a9bf4f (at 10.8.27.3@o2ib6) Feb 24 20:09:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:09:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ba2142c-879f-ac28-9d4a-a3788afebea0 (at 10.8.7.35@o2ib6) Feb 24 20:09:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:09:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7122cf14-0523-fe12-768f-cd0ed99220da (at 10.8.27.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857654f7800, cur 1551067775 expire 1551067625 last 1551067548 Feb 24 20:09:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 20:09:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to db45ffdf-edf3-91b4-c0ad-abe30f1ea215 (at 10.8.27.6@o2ib6) Feb 24 20:09:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 24 20:10:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 66e4ac0c-6519-785c-aa30-3457dbc9eea1 (at 10.8.27.8@o2ib6) Feb 24 20:10:18 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Feb 24 20:12:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 20:12:34 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 20:13:30 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 720160b3-b510-fd4a-7aef-667fe71d4b1d (at 10.8.27.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef12c00, cur 1551068010 expire 1551067860 last 1551067783 Feb 24 20:13:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:20:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 20:20:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:20:59 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 95a7bee7-b4e2-04fd-3dcf-8552f5380f6c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dffa800, cur 1551068459 expire 1551068309 last 1551068232 Feb 24 20:20:59 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 20:23:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 24 20:23:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:28:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 36fe8706-0099-adb4-4124-68e1ccc43c8e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985ef8cfac00, cur 1551068917 expire 1551068767 last 1551068690 Feb 24 20:28:37 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 20:32:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 680b2f8b-c457-9b65-c1bc-58d27b9ae2fe (at 10.8.27.14@o2ib6) Feb 24 20:32:03 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 24 20:35:24 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client dd23da3a-484d-0eb4-af89-89a069ff0621 (at 10.9.104.49@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784416000, cur 1551069324 expire 1551069174 last 1551069097 Feb 24 20:35:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:42:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2e6063a2-ca9b-ae5d-abce-e3daf4d673e4 (at 10.9.103.23@o2ib4) Feb 24 20:42:05 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Feb 24 20:43:18 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 248ffa45-ee9e-3f32-a526-c435dd0ee693 (at 10.8.17.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786948000, cur 1551069798 expire 1551069648 last 1551069571 Feb 24 20:43:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 20:51:21 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 00cd381d-5246-e5cd-af5e-792229d3fea2 (at 10.9.104.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98681467d000, cur 1551070281 expire 1551070131 last 1551070054 Feb 24 20:51:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 20:55:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 20609818-b83c-bf65-0dd2-090d3c6e2314 (at 10.9.108.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1647c00, cur 1551070549 expire 1551070399 last 1551070322 Feb 24 20:55:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 20:56:06 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8f273097-d34e-d604-e427-2da4f99ca32a (at 10.9.106.26@o2ib4) Feb 24 20:56:06 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Feb 24 21:07:47 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9342c549-8cbe-27f5-a8b8-f7759a7fb2aa (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984fcccf8c00, cur 1551071267 expire 1551071117 last 1551071040 Feb 24 21:07:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 21:09:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dd23da3a-484d-0eb4-af89-89a069ff0621 (at 10.9.104.49@o2ib4) Feb 24 21:09:42 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 24 21:17:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 23d559de-b43f-67f3-df68-2b8665f1ed2d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a08000, cur 1551071853 expire 1551071703 last 1551071626 Feb 24 21:17:33 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 21:21:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 21:21:42 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 21:28:51 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a824339b-2fdb-efc1-74f9-bf599e2dfda5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985573c61c00, cur 1551072531 expire 1551072381 last 1551072304 Feb 24 21:28:51 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 21:34:24 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 24 21:34:24 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 21:42:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8e2f5957-6dae-32b0-e35e-e535b38ca109 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834056c00, cur 1551073364 expire 1551073214 last 1551073137 Feb 24 21:42:44 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 24 21:45:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 91f6fe25-4c34-a621-dd26-00e6ccf4cbba (at 10.9.106.32@o2ib4) Feb 24 21:45:53 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 24 21:58:44 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 6d2e6aad-846f-aaaf-84ce-b90c242e3b34 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801541c00, cur 1551074324 expire 1551074174 last 1551074097 Feb 24 21:58:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 22:00:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d7132a19-51b8-e098-d0d8-a2755039375a (at 10.8.25.20@o2ib6) Feb 24 22:00:02 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 22:12:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 22:12:34 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 24 22:13:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bfea0b09-6468-5c05-7b35-593c7149e9db (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984aa3fb6000, cur 1551075185 expire 1551075035 last 1551074958 Feb 24 22:13:05 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 24 22:23:43 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e4b147b8-d2c2-9991-2497-8ddc687d5f8c (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c9f87f800, cur 1551075823 expire 1551075673 last 1551075596 Feb 24 22:23:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 22:25:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 22:25:00 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 22:35:01 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 65449291-0a20-4dcc-a5b6-a53ab778bafd (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984810a2f800, cur 1551076501 expire 1551076351 last 1551076274 Feb 24 22:35:01 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 24 22:36:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1587929a-7ff1-3f16-b1de-766c210b95a9 (at 10.8.20.33@o2ib6) Feb 24 22:36:24 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 24 22:47:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 22:47:16 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 22:47:52 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c96f8680-8899-c6f6-b9d8-0338b01efeba (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762602800, cur 1551077272 expire 1551077122 last 1551077045 Feb 24 22:47:52 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 24 23:04:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ceafb54-89ce-8961-d103-913efe379d81 (at 10.8.21.7@o2ib6) Feb 24 23:04:15 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 23:11:06 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e4a48af5-173b-69ab-3bac-2bcc464bdd13 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762740c00, cur 1551078666 expire 1551078516 last 1551078439 Feb 24 23:11:06 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 23:15:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d3b133e8-4ec8-ebe3-7fc5-79aa16e59c0b (at 10.9.106.35@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834683000, cur 1551078937 expire 1551078787 last 1551078710 Feb 24 23:15:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 23:21:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 23:21:36 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 24 23:22:06 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8b348f66-3fc4-430a-d8c7-359d09e3c590 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848000eb800, cur 1551079326 expire 1551079176 last 1551079099 Feb 24 23:22:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 24 23:31:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 24 23:31:51 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Feb 24 23:32:06 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bd9b9589-11d3-79b8-1ee1-3b3f95c56163 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b2f6800, cur 1551079926 expire 1551079776 last 1551079699 Feb 24 23:32:06 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 23:42:17 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 23:42:17 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 24 23:42:50 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 995ad94b-fea2-b418-fe5b-a795bf308d23 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848001fe800, cur 1551080570 expire 1551080420 last 1551080343 Feb 24 23:42:50 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 24 23:52:53 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 24f54dae-dc87-c0e2-6af2-2fff5088159d (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834054000, cur 1551081173 expire 1551081023 last 1551080946 Feb 24 23:52:53 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 24 23:53:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 24 23:53:52 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 00:04:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d8669901-936d-4342-8b67-d6fc1b6a851d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b5d687000, cur 1551081882 expire 1551081732 last 1551081655 Feb 25 00:04:42 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 00:07:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 00:07:28 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 00:16:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a41d4047-8ba0-22f5-caad-627b07cca9c4 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a6000, cur 1551082570 expire 1551082420 last 1551082343 Feb 25 00:16:10 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 25 00:28:35 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7ca862d9-fb36-f06e-d642-e42cfd5c8c83 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984810a29c00, cur 1551083315 expire 1551083165 last 1551083088 Feb 25 00:28:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 00:29:28 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 00:29:28 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 00:36:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 00:36:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 00:42:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fd01c7fd-7bec-8a77-ad04-40f1cfa2b200 (at 10.8.25.10@o2ib6) Feb 25 00:42:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 00:43:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fc07cbaa-b975-254a-ff30-137392ee3f44 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985757bab400, cur 1551084188 expire 1551084038 last 1551083961 Feb 25 00:43:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 00:54:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 00:54:00 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 25 01:00:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1f2b0e26-473c-92cc-1fc9-3f880eb97666 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d206000, cur 1551085220 expire 1551085070 last 1551084993 Feb 25 01:00:20 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 01:05:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 01:05:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 01:11:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7845f72c-b8b2-58a5-a96c-1e234dd4860e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857630f8400, cur 1551085870 expire 1551085720 last 1551085643 Feb 25 01:11:10 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 25 01:17:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 01:17:34 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 25 01:22:13 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 12daae0d-e8cd-642c-33c5-6f85068519db (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986bc6d6fc00, cur 1551086533 expire 1551086383 last 1551086306 Feb 25 01:22:13 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 25 01:29:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6f23ad32-0dd1-26f7-1bbe-7fefdeb50a2a (at 10.8.19.1@o2ib6) Feb 25 01:29:11 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 01:40:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 01:40:53 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 25 01:41:10 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 152568df-f8f3-5680-1eb9-4bfb2c89211f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d559400, cur 1551087670 expire 1551087520 last 1551087443 Feb 25 01:41:10 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Feb 25 01:57:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 01:57:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 01:57:52 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 37249470-7203-2d46-2985-30a67dbbc6fd (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984bb4fe2c00, cur 1551088672 expire 1551088522 last 1551088445 Feb 25 01:57:52 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 02:09:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aead70f5-f5fc-a972-7a47-29ab30efeef2 (at 10.9.104.23@o2ib4) Feb 25 02:09:11 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 02:17:20 fir-io1-s1 kernel: LustreError: 96574:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.18.31@o2ib6) returned error from glimpse AST (req@ffff9864a7a01500 x1625013710580448 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff9857d11de9c0/0x49e185f363f80cb5 lrc: 3/0,0 mode: PW/PW res: [0x580000402:0xc3a58:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x42bde76791968cc6 expref: 6 pid: 96243 timeout: 0 lvb_type: 0 Feb 25 02:17:20 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.18.31@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 25 02:17:20 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Feb 25 02:17:20 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551089840s: evicting client at 10.8.18.31@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff9857d11d8b40/0x49e185f363f80cca lrc: 3/0,0 mode: PW/PW res: [0x8c0000401:0xc35fb:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x42bde76791968d67 expref: 6 pid: 96243 timeout: 0 lvb_type: 0 Feb 25 02:17:20 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Feb 25 02:17:20 fir-io1-s1 kernel: LustreError: 96574:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages Feb 25 02:17:50 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e458a867-76f2-e121-ece5-a1940bbac4f5 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b3a400, cur 1551089870 expire 1551089720 last 1551089643 Feb 25 02:17:50 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 02:20:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 02:20:30 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 25 02:32:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1cd5416c-4828-ac30-6538-cc2847b45533 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98622f65b800, cur 1551090740 expire 1551090590 last 1551090513 Feb 25 02:32:20 fir-io1-s1 kernel: Lustre: Skipped 19 previous similar messages Feb 25 02:32:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 02:32:37 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 02:43:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 02:43:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 02:44:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 08958bbc-0f90-1cbd-61ae-768cfa6c9459 (at 10.9.104.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985765732400, cur 1551091491 expire 1551091341 last 1551091264 Feb 25 02:44:51 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 02:56:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 02:56:29 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 03:21:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 08958bbc-0f90-1cbd-61ae-768cfa6c9459 (at 10.9.104.69@o2ib4) Feb 25 03:21:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 03:37:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client dea6ff96-022f-bdf6-cc03-2d525b850638 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678685e800, cur 1551094671 expire 1551094521 last 1551094444 Feb 25 03:37:51 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 03:45:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 03:45:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:07:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3bc55b46-df6c-46c6-b66d-183e8631b835 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c672000, cur 1551096461 expire 1551096311 last 1551096234 Feb 25 04:07:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:12:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 04:12:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:14:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 25 04:14:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:18:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 04:18:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 04:18:49 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 559fe8f7-5377-a9fc-6a46-b3096519348c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a9b400, cur 1551097129 expire 1551096979 last 1551096902 Feb 25 04:18:49 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 04:21:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 04:21:16 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 25 04:22:05 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3cd4b19e-f214-f55a-3c2a-78687885e70c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985769987000, cur 1551097325 expire 1551097175 last 1551097098 Feb 25 04:22:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:23:21 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1b0a86b3-0fd0-cda1-1107-0c6301aea8f2 (at 10.8.11.10@o2ib6) in 170 seconds. I think it's dead, and I am evicting it. exp ffff98480332b800, cur 1551097401 expire 1551097251 last 1551097231 Feb 25 04:23:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:24:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1b0a86b3-0fd0-cda1-1107-0c6301aea8f2 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aab6400, cur 1551097458 expire 1551097308 last 1551097231 Feb 25 04:24:18 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 25 04:26:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 25 04:26:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:33:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a75a6a3c-cb83-2a7b-82b2-3df2cdacd1c6 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825f8400, cur 1551098015 expire 1551097865 last 1551097788 Feb 25 04:33:35 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 25 04:34:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 229460c3-4182-2dec-2c0d-9ca77ec4a979 (at 10.8.18.31@o2ib6) in 158 seconds. I think it's dead, and I am evicting it. exp ffff986784b92400, cur 1551098091 expire 1551097941 last 1551097933 Feb 25 04:34:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:35:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 25 04:35:57 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 04:36:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 229460c3-4182-2dec-2c0d-9ca77ec4a979 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b93800, cur 1551098160 expire 1551098010 last 1551097933 Feb 25 04:36:00 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 25 04:37:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 04:37:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:39:22 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d12e7805-0195-6cbb-9c53-6eb86c3dcff7 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780864000, cur 1551098362 expire 1551098212 last 1551098135 Feb 25 04:39:22 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 25 04:40:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 04:40:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:43:02 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 66075cf2-9785-a74c-455c-df4c6c6263fd (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15ce400, cur 1551098582 expire 1551098432 last 1551098355 Feb 25 04:43:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 04:47:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 04:47:02 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 04:50:21 fir-io1-s1 kernel: Lustre: 96944:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551099014/real 1551099014] req@ffff986232e6b300 x1625013725936976/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551099021 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 25 04:50:21 fir-io1-s1 kernel: Lustre: 96944:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 25 04:50:42 fir-io1-s1 kernel: Lustre: 96920:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551099035/real 1551099035] req@ffff984d2e67ec00 x1625013725937024/t0(0) o106->fir-OST0006@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551099042 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 04:50:42 fir-io1-s1 kernel: Lustre: 96920:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 25 04:51:24 fir-io1-s1 kernel: Lustre: 96251:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551099077/real 1551099077] req@ffff984ed7c71b00 x1625013725937008/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551099084 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 04:51:24 fir-io1-s1 kernel: Lustre: 96251:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Feb 25 04:52:39 fir-io1-s1 kernel: Lustre: 49831:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551099152/real 1551099152] req@ffff984605960900 x1625013726104576/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551099159 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 04:52:39 fir-io1-s1 kernel: Lustre: 96887:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551099152/real 1551099152] req@ffff983ea1966300 x1625013726104592/t0(0) o106->fir-OST0006@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551099159 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 04:52:39 fir-io1-s1 kernel: Lustre: 96887:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 98 previous similar messages Feb 25 04:53:21 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 69bc773c-93b7-5ebc-266d-a59bf91f8620 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596bd800, cur 1551099201 expire 1551099051 last 1551098974 Feb 25 04:53:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 04:53:28 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 04:53:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 05:03:39 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 13cda212-3c3a-2354-2af2-efa6674c6d4b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98622f659800, cur 1551099819 expire 1551099669 last 1551099592 Feb 25 05:03:39 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 05:05:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 05:05:43 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 05:06:00 fir-io1-s1 kernel: LustreError: 96946:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.18.31@o2ib6) returned error from glimpse AST (req@ffff985bcdafd700 x1625013727558688 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff985335d04140/0x49e185f364885c5b lrc: 3/0,0 mode: PW/PW res: [0x5c0000402:0xc3b50:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x10393d607f24d348 expref: 5 pid: 96620 timeout: 0 lvb_type: 0 Feb 25 05:06:00 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.18.31@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 25 05:06:00 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Feb 25 05:06:00 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551099960s: evicting client at 10.8.18.31@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff985335d006c0/0x49e185f364885c62 lrc: 3/0,0 mode: PW/PW res: [0x6c0000402:0xc35d3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x10393d607f24d380 expref: 6 pid: 96620 timeout: 0 lvb_type: 0 Feb 25 05:06:00 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Feb 25 05:06:00 fir-io1-s1 kernel: LustreError: 96946:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages Feb 25 05:16:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 20f8fe85-8425-8176-626e-350b50e6f45c (at 10.9.106.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581ef41800, cur 1551100560 expire 1551100410 last 1551100333 Feb 25 05:16:00 fir-io1-s1 kernel: Lustre: Skipped 25 previous similar messages Feb 25 05:23:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 05:23:07 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 05:26:54 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1a568aba-7495-1729-9eb2-6abe67e812c3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a2a000, cur 1551101214 expire 1551101064 last 1551100987 Feb 25 05:26:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 05:34:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 05:34:24 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 05:41:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client aaf7e9a0-069b-4052-0d9e-308c5e219529 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a5bc00, cur 1551102068 expire 1551101918 last 1551101841 Feb 25 05:41:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 05:48:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 05:48:01 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 05:51:48 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2cdcb9af-fc07-023b-0abc-9cd1511f8bd2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a2ac00, cur 1551102708 expire 1551102558 last 1551102481 Feb 25 05:51:48 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 06:19:14 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c6708fe1-bd2e-9567-4c18-7af0628e1eeb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833e70800, cur 1551104354 expire 1551104204 last 1551104127 Feb 25 06:19:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 06:22:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 06:22:46 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 06:28:41 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 997214e0-1aa8-bfb7-6d54-7e553060abad (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c677000, cur 1551104921 expire 1551104771 last 1551104694 Feb 25 06:28:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 06:31:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 06:31:12 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 06:36:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5b8b8274-3329-70cb-e2f9-7c5876fb0ee5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf2a000, cur 1551105401 expire 1551105251 last 1551105174 Feb 25 06:36:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 06:41:33 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 06:41:33 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 06:47:53 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 65d1370e-308d-58b6-f0c9-7ef4f0b85028 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767c4a400, cur 1551106073 expire 1551105923 last 1551105846 Feb 25 06:47:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 06:48:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 06:48:02 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 25 06:58:38 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ee2183cf-f67e-8663-d79a-58c5f102bdc3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758dbd400, cur 1551106718 expire 1551106568 last 1551106491 Feb 25 06:58:38 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 07:01:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 07:01:57 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 07:10:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 39b74f02-0c4c-fd51-e621-4bd6eb7173c0 (at 10.9.103.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e3c283000, cur 1551107409 expire 1551107259 last 1551107182 Feb 25 07:10:09 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 07:15:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 07:15:47 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 07:44:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 07:44:31 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 07:45:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4801c009-ea79-0e16-92b9-52267e516fc9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758db9c00, cur 1551109500 expire 1551109350 last 1551109273 Feb 25 07:45:00 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 07:45:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 39b74f02-0c4c-fd51-e621-4bd6eb7173c0 (at 10.9.103.18@o2ib4) Feb 25 07:45:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 07:48:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0e375274-c7da-0a3b-487e-3d8131620a2e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5dc00, cur 1551109698 expire 1551109548 last 1551109471 Feb 25 07:48:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 07:54:14 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 975f7ad7-f03d-15c7-83b1-221b445246ae (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483513c800, cur 1551110054 expire 1551109904 last 1551109827 Feb 25 07:54:14 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 07:58:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 07:58:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 08:04:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 08:04:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 08:04:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 47cbbf7a-086f-0227-f3f0-b1bf06fab51b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857618e5000, cur 1551110682 expire 1551110532 last 1551110455 Feb 25 08:04:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 08:10:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 08:10:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 08:15:14 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 08:15:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 08:15:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 76379dee-0ebd-960e-9130-f64a5595d718 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d7400, cur 1551111354 expire 1551111204 last 1551111127 Feb 25 08:15:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 08:18:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 08:18:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 08:25:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 97bc8e0c-1614-4de0-a593-98b585b7fd0b (at 10.9.103.30@o2ib4) Feb 25 08:25:57 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 25 08:29:28 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 874bae97-c73c-de86-53e9-317637b9cc61 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834ae9400, cur 1551112168 expire 1551112018 last 1551111941 Feb 25 08:29:28 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 08:48:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 08:48:41 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 25 08:49:31 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client bcfbd687-44f1-0d4d-4d16-407b08e2f759 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfb000, cur 1551113371 expire 1551113221 last 1551113144 Feb 25 08:49:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 08:57:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 08:57:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 08:59:58 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 471d4d1d-7684-41a1-beb7-bd2d4f0908de (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858388a6400, cur 1551113998 expire 1551113848 last 1551113771 Feb 25 08:59:58 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 09:01:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 09:01:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 09:08:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 09:08:03 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 09:13:03 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 32b6d3dd-e6ba-d67f-b889-fe8b65a0fcd6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833d05800, cur 1551114783 expire 1551114633 last 1551114556 Feb 25 09:13:03 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 09:18:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 09:18:12 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 09:23:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c3f3c991-ff25-174d-bb62-0b7226e99f39 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c47000, cur 1551115429 expire 1551115279 last 1551115202 Feb 25 09:23:49 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 09:30:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 09:30:45 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 25 09:43:20 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b9f4ca05-0fa9-649e-b1de-383d9bc47492 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cd8000, cur 1551116600 expire 1551116450 last 1551116373 Feb 25 09:43:20 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 25 09:48:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 09:48:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 09:55:24 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 6be83b83-a986-085c-687c-9542da46466e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834050c00, cur 1551117324 expire 1551117174 last 1551117097 Feb 25 09:55:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 10:01:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9e7dc1a5-746c-8e56-5ad9-e239237ff7d7 (at 10.8.24.22@o2ib6) Feb 25 10:01:40 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Feb 25 10:07:32 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 484257d4-5c98-71e3-6c72-be1a22245e31 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868053ee800, cur 1551118052 expire 1551117902 last 1551117825 Feb 25 10:07:32 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 10:15:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 10:15:29 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 10:24:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d4dd6b0c-9843-b272-918e-a7e2aa547f92 (at 10.8.17.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838a9800, cur 1551119042 expire 1551118892 last 1551118815 Feb 25 10:24:02 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 10:34:31 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a18587f4-6669-8a89-c311-224538b5a6f2 (at 10.8.27.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9874eeb8a000, cur 1551119671 expire 1551119521 last 1551119444 Feb 25 10:34:31 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 25 10:34:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 10:34:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 10:45:34 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0bf5624d-e79b-6fc1-1e06-62562c85448b (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848845fe000, cur 1551120334 expire 1551120184 last 1551120107 Feb 25 10:45:34 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 10:46:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d703ee38-a263-111a-74d3-dae3e1465c1f (at 10.9.108.22@o2ib4) Feb 25 10:46:36 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 25 10:55:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client cf27932c-5cfb-509a-c7ce-6753e8ed5f45 (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b741000, cur 1551120954 expire 1551120804 last 1551120727 Feb 25 10:55:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 10:58:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c5f65607-5ecb-3fae-104c-865ac2acdc12 (at 10.9.104.53@o2ib4) Feb 25 10:58:10 fir-io1-s1 kernel: Lustre: Skipped 45 previous similar messages Feb 25 11:12:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ff6dc77a-4656-6fd1-f0d6-fbaf7b10161c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834057c00, cur 1551121961 expire 1551121811 last 1551121734 Feb 25 11:12:41 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 25 11:15:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 11:15:11 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Feb 25 11:32:07 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 844c642b-70a6-ed76-9d2b-d135d03f8b90 (at 10.9.105.66@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987624cf6800, cur 1551123127 expire 1551122977 last 1551122900 Feb 25 11:32:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 11:32:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a26bda41-9e3a-f8bb-aa63-ba992cd69aad (at 10.9.0.62@o2ib4) Feb 25 11:32:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 11:43:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Feb 25 11:43:41 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 25 12:04:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 844c642b-70a6-ed76-9d2b-d135d03f8b90 (at 10.9.105.66@o2ib4) Feb 25 12:04:26 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 12:05:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a78ab922-10b1-e1a7-89e3-f90306408594 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811210800, cur 1551125120 expire 1551124970 last 1551124893 Feb 25 12:05:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:06:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 12:06:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:18:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848019c6000, cur 1551125924 expire 1551125774 last 1551125697 Feb 25 12:18:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:36:46 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 6c31a5cc-998a-3064-83b3-9f96e026df9d (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987228054000, cur 1551127006 expire 1551126856 last 1551126779 Feb 25 12:36:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:37:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 12:37:58 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 12:42:43 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4840388f-4bf3-516e-4c43-00080a4a9c17 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985765733000, cur 1551127363 expire 1551127213 last 1551127136 Feb 25 12:42:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:43:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 12:43:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 12:47:47 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cc6e114f-a671-c1d1-8519-730567c2dff7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834052000, cur 1551127667 expire 1551127517 last 1551127440 Feb 25 12:47:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:48:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 12:48:16 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 25 12:49:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ce6d475-6724-c117-8c44-da8378e50030 (at 10.9.101.69@o2ib4) Feb 25 12:49:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:52:59 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 697196e1-a7bb-ad7e-77cc-c256dae7fa68 (at 10.8.26.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c39800, cur 1551127979 expire 1551127829 last 1551127752 Feb 25 12:52:59 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 12:54:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 12:54:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 12:57:52 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 206dd526-56ca-7f0f-cc93-b55e80ec3979 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904996000, cur 1551128272 expire 1551128122 last 1551128045 Feb 25 12:57:52 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 13:05:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 67f0d42c-d76b-f97b-26ce-7d40fa9782ff (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9874eeb8fc00, cur 1551128700 expire 1551128550 last 1551128473 Feb 25 13:05:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 13:06:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 13:06:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 13:14:24 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9bb2e785-62db-f92a-cba1-7c1565dca222 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581bfed000, cur 1551129264 expire 1551129114 last 1551129037 Feb 25 13:14:24 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 13:20:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c313559f-a84d-1bc9-e226-9f1e30bc5add (at 10.8.26.31@o2ib6) Feb 25 13:20:46 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 13:21:19 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3aa13807-7e41-ad0d-47ec-3103159f30b6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801512c00, cur 1551129679 expire 1551129529 last 1551129452 Feb 25 13:21:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 13:25:19 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c9ef5dca-0fa4-a528-638c-49801ea07410 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480315dc00, cur 1551129919 expire 1551129769 last 1551129692 Feb 25 13:25:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 13:30:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 13:30:49 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Feb 25 13:31:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3d8d12b1-b418-73f2-579f-c2d5d13e51d5 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868147b1400, cur 1551130284 expire 1551130134 last 1551130057 Feb 25 13:31:24 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 13:42:32 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 436f530a-f45e-a33c-6a0d-a3414701ff7b (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985e28b3bc00, cur 1551130952 expire 1551130802 last 1551130725 Feb 25 13:42:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 13:43:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 13:43:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 13:54:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0560ee03-cb0d-3ab6-6564-7a223fe93704 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9874eeb8bc00, cur 1551131649 expire 1551131499 last 1551131422 Feb 25 13:54:09 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 25 13:54:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 25 13:54:23 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 14:05:14 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client feac05a4-716f-c34a-fd9d-1220a521af0c (at 10.9.107.69@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98783c2e0400, cur 1551132314 expire 1551132164 last 1551132087 Feb 25 14:05:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 14:10:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fabd17e2-268b-b1ba-b568-7f6550034520 (at 10.9.106.16@o2ib4) Feb 25 14:10:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 14:27:02 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 7c04f965-01e9-b008-bb36-681b1688b172 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c39400, cur 1551133622 expire 1551133472 last 1551133395 Feb 25 14:27:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 14:28:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to feac05a4-716f-c34a-fd9d-1220a521af0c (at 10.9.107.69@o2ib4) Feb 25 14:28:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 14:40:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 294204a3-86f8-ea6d-5140-898a94747d5f (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e7da2ec00, cur 1551134423 expire 1551134273 last 1551134196 Feb 25 14:40:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 14:41:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 14:41:14 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 14:45:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f272eff7-c7bc-edf0-a0c0-52f99d872519 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cb1000, cur 1551134701 expire 1551134551 last 1551134474 Feb 25 14:45:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 14:46:17 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6763c657-fada-25a2-6788-620374ff78bc (at 10.8.18.34@o2ib6) in 160 seconds. I think it's dead, and I am evicting it. exp ffff985761849400, cur 1551134777 expire 1551134627 last 1551134617 Feb 25 14:46:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 14:57:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ae6b1dd5-45e1-41b8-a174-9deffa1f0fce (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984818e9f800, cur 1551135471 expire 1551135321 last 1551135244 Feb 25 14:57:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 15:00:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 15:00:34 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 15:04:21 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d3f39939-667f-d802-1be4-b6955dfc9e5f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bd74000, cur 1551135861 expire 1551135711 last 1551135634 Feb 25 15:04:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 15:34:15 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a7b8045b-d476-b776-aa76-f8bea0000bff (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4b800, cur 1551137655 expire 1551137505 last 1551137428 Feb 25 15:34:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 15:34:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a7b8045b-d476-b776-aa76-f8bea0000bff (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4b000, cur 1551137666 expire 1551137516 last 1551137439 Feb 25 15:34:26 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 25 15:34:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 25 15:34:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 15:54:30 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8402838d-f75d-af53-0850-df7f3dd913ea (at 10.9.104.58@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a37dcc00, cur 1551138870 expire 1551138720 last 1551138643 Feb 25 15:54:30 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 16:02:51 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 47193d0c-f58e-8284-2388-8ad14259dbd5 (at 10.9.103.40@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832b4fc00, cur 1551139371 expire 1551139221 last 1551139144 Feb 25 16:02:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:03:01 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4712ded9-305f-102d-4761-daabc1364d31 (at 10.9.104.45@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c21c00, cur 1551139381 expire 1551139231 last 1551139154 Feb 25 16:03:01 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 25 16:14:19 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9a3e8168-75c8-46f5-0e45-82fbc55de66d (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768456800, cur 1551140059 expire 1551139909 last 1551139832 Feb 25 16:14:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:15:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 16:15:18 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 16:15:35 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d2cd3f73-7609-68cf-6396-48644edd975d (at 10.8.18.35@o2ib6) in 205 seconds. I think it's dead, and I am evicting it. exp ffff985573c66c00, cur 1551140135 expire 1551139985 last 1551139930 Feb 25 16:15:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:23:40 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 19a5af77-32db-6a2c-514e-095bb62b31d6 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768558000, cur 1551140620 expire 1551140470 last 1551140393 Feb 25 16:23:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:24:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 16:24:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 16:24:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 220b49be-4c22-e196-0372-93fb51bb5b15 (at 10.8.18.34@o2ib6) in 223 seconds. I think it's dead, and I am evicting it. exp ffff986784b15400, cur 1551140696 expire 1551140546 last 1551140473 Feb 25 16:24:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:25:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 62f38112-c51d-b467-8668-293d1a60bffb (at 10.9.103.40@o2ib4) Feb 25 16:25:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:27:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ec0647d1-f1b9-85b5-d320-be89cdc060c1 (at 10.9.103.42@o2ib4) Feb 25 16:27:50 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 16:34:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9d7a5689-b5b6-ae58-cc33-12cb33a7ed48 (at 10.9.104.44@o2ib4) Feb 25 16:34:27 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 16:37:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ec385dc1-10a6-ea22-c636-9ff43910b33d (at 10.8.14.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986b5a45e400, cur 1551141440 expire 1551141290 last 1551141213 Feb 25 16:37:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:54:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5e4c7a11-f131-2f99-ca39-7ac53f68733a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483513e800, cur 1551142477 expire 1551142327 last 1551142250 Feb 25 16:54:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 16:55:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 25 16:55:55 fir-io1-s1 kernel: Lustre: Skipped 24 previous similar messages Feb 25 17:04:19 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 06f9d9f1-7bd0-5264-d60d-bdb6943859a0 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857620a1800, cur 1551143059 expire 1551142909 last 1551142832 Feb 25 17:04:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 17:05:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 25 17:05:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 17:36:07 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 59f57a83-3792-f137-abcf-4d866e4efc34 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848807f1c00, cur 1551144967 expire 1551144817 last 1551144740 Feb 25 17:36:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 17:36:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 25 17:36:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 17:42:40 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8a46a015-1bb5-c1bc-442c-693f3f87856f (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c212000, cur 1551145360 expire 1551145210 last 1551145133 Feb 25 17:42:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 17:44:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 25 17:44:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 17:52:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8f8203d6-652b-3662-9526-9e3d8f796fde (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480315dc00, cur 1551145932 expire 1551145782 last 1551145705 Feb 25 17:52:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 17:53:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 25 17:53:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:01:38 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c94b9ef7-897f-4d35-11e0-46cac7178c3a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9876d835d400, cur 1551146498 expire 1551146348 last 1551146271 Feb 25 18:01:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:02:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 25 18:02:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:11:05 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client d94d3be2-2e52-9c9e-d516-b2f07c1624ad (at 10.9.104.42@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9869719a3000, cur 1551147065 expire 1551146915 last 1551146838 Feb 25 18:11:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:12:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 25 18:12:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:13:55 fir-io1-s1 kernel: Lustre: 96413:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551147228/real 1551147228] req@ffff984f2aa66f00 x1625014295040432/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551147235 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 25 18:13:55 fir-io1-s1 kernel: Lustre: 96413:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 109 previous similar messages Feb 25 18:14:37 fir-io1-s1 kernel: Lustre: 96302:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551147270/real 1551147270] req@ffff984f2aa61b00 x1625014295040400/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551147277 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 18:14:37 fir-io1-s1 kernel: Lustre: 96302:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Feb 25 18:15:54 fir-io1-s1 kernel: Lustre: 94526:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551147347/real 1551147347] req@ffff9857c6578000 x1625014295040448/t0(0) o106->fir-OST0002@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551147354 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 18:15:54 fir-io1-s1 kernel: Lustre: 94526:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 39 previous similar messages Feb 25 18:17:09 fir-io1-s1 kernel: LNet: Service thread pid 94526 was inactive for 200.48s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 25 18:17:09 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 25 18:17:09 fir-io1-s1 kernel: Pid: 94526, comm: ll_ost01_006 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 25 18:17:09 fir-io1-s1 kernel: Call Trace: Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 25 18:17:09 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 25 18:17:09 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551147429.94526 Feb 25 18:17:10 fir-io1-s1 kernel: LNet: Service thread pid 96288 was inactive for 201.95s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 25 18:17:10 fir-io1-s1 kernel: Pid: 96288, comm: ll_ost01_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 25 18:17:10 fir-io1-s1 kernel: Call Trace: Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 25 18:17:10 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 25 18:17:11 fir-io1-s1 kernel: Pid: 96413, comm: ll_ost01_056 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 25 18:17:11 fir-io1-s1 kernel: Call Trace: Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 25 18:17:11 fir-io1-s1 kernel: Pid: 96302, comm: ll_ost01_026 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 25 18:17:11 fir-io1-s1 kernel: Call Trace: Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 25 18:17:11 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 25 18:17:24 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5b625c3e-d246-904c-4ffd-0aab047575ee (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848019c4800, cur 1551147444 expire 1551147294 last 1551147217 Feb 25 18:17:24 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 25 18:17:24 fir-io1-s1 kernel: LNet: Service thread pid 96413 completed after 215.07s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 25 18:17:24 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 25 18:20:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a17b147f-c4f5-4427-2b99-c1baf15f83bd (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8260400, cur 1551147640 expire 1551147490 last 1551147413 Feb 25 18:20:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:21:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 18:21:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:33:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7a4f2fb-7087-2da1-fe97-a6b3f3961662 (at 10.9.103.38@o2ib4) Feb 25 18:33:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 18:41:31 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5a24845a-4df3-d113-b27e-e379baa435be (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8266000, cur 1551148891 expire 1551148741 last 1551148664 Feb 25 18:41:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 18:41:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5a24845a-4df3-d113-b27e-e379baa435be (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8261800, cur 1551148907 expire 1551148757 last 1551148680 Feb 25 18:41:47 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 18:42:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce970d8c-cf83-fbcf-a0f2-5b0ed3cb9f89 (at 10.9.104.43@o2ib4) Feb 25 18:42:33 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 18:45:38 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9da9cc6a-d27d-cf84-c5da-3812acf847ad (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801512c00, cur 1551149138 expire 1551148988 last 1551148911 Feb 25 18:45:38 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 25 18:45:59 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9da9cc6a-d27d-cf84-c5da-3812acf847ad (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762747000, cur 1551149159 expire 1551149009 last 1551148932 Feb 25 18:45:59 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 18:48:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 819a2974-4f03-e3b0-601d-2d97317e6637 (at 10.9.103.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987836514400, cur 1551149322 expire 1551149172 last 1551149095 Feb 25 19:00:37 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client eb3b9013-3ee1-3a96-8e91-182000608f99 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b913800, cur 1551150037 expire 1551149887 last 1551149810 Feb 25 19:00:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 19:06:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 19:06:53 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Feb 25 19:21:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 819a2974-4f03-e3b0-601d-2d97317e6637 (at 10.9.103.3@o2ib4) Feb 25 19:21:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 19:27:24 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b3fce559-01af-1741-be02-c46bc4b5ebb8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483ed4c800, cur 1551151644 expire 1551151494 last 1551151417 Feb 25 19:27:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 19:31:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 19:31:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 19:31:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2c96ed5b-7b98-2819-9f2b-c8d6f7172439 (at 10.9.101.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c1800, cur 1551151916 expire 1551151766 last 1551151689 Feb 25 19:31:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 19:32:00 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2c96ed5b-7b98-2819-9f2b-c8d6f7172439 (at 10.9.101.43@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b743400, cur 1551151920 expire 1551151770 last 1551151693 Feb 25 19:32:00 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 25 19:44:29 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8d831dd0-3dd8-79f5-f509-c084d8b78f7a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a0b000, cur 1551152669 expire 1551152519 last 1551152442 Feb 25 19:44:29 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 19:48:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 19:48:09 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 25 20:01:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 6ad65c1d-1c56-46ff-89db-441d3f64285e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576998ec00, cur 1551153695 expire 1551153545 last 1551153468 Feb 25 20:01:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 20:03:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2c96ed5b-7b98-2819-9f2b-c8d6f7172439 (at 10.9.101.43@o2ib4) Feb 25 20:03:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 20:11:20 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 20:11:20 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 20:15:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 044d49a5-61d5-0c66-16c0-c4250e19a31c (at 10.8.13.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c0000, cur 1551154513 expire 1551154363 last 1551154286 Feb 25 20:15:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 20:37:40 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f7811602-a35e-c9f6-69af-1aa2c35d6293 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed05000, cur 1551155860 expire 1551155710 last 1551155633 Feb 25 20:37:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 20:39:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 20:39:26 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 25 20:47:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 044d49a5-61d5-0c66-16c0-c4250e19a31c (at 10.8.13.16@o2ib6) Feb 25 20:47:00 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 21:01:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2686cabc-4957-43b2-fd06-e63bbe689c41 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b2f3000, cur 1551157286 expire 1551157136 last 1551157059 Feb 25 21:01:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 21:03:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 25 21:03:52 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 25 21:13:29 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f411b7e5-c3ac-92fa-a719-7c8c471e0ce5 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499a65f000, cur 1551158009 expire 1551157859 last 1551157782 Feb 25 21:13:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 21:13:45 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f411b7e5-c3ac-92fa-a719-7c8c471e0ce5 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f9490b400, cur 1551158025 expire 1551157875 last 1551157798 Feb 25 21:13:45 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 21:14:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 25 21:14:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 21:24:54 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2a5bd73b-b039-4236-618f-951a1ce35476 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596be000, cur 1551158694 expire 1551158544 last 1551158467 Feb 25 21:24:54 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 25 21:25:03 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2a5bd73b-b039-4236-618f-951a1ce35476 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596bcc00, cur 1551158703 expire 1551158553 last 1551158476 Feb 25 21:25:03 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 21:26:10 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4d8bcdc8-d35e-7acb-6372-5603c5ac3d2b (at 10.8.18.35@o2ib6) in 185 seconds. I think it's dead, and I am evicting it. exp ffff985758c24c00, cur 1551158770 expire 1551158620 last 1551158585 Feb 25 21:26:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 25 21:26:15 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 21:26:19 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4d8bcdc8-d35e-7acb-6372-5603c5ac3d2b (at 10.8.18.35@o2ib6) in 194 seconds. I think it's dead, and I am evicting it. exp ffff985758c27400, cur 1551158779 expire 1551158629 last 1551158585 Feb 25 21:26:19 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 21:26:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 21:26:20 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 25 21:33:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 67078a83-293a-3a96-655b-e1d1456aff5d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839351400, cur 1551159206 expire 1551159056 last 1551158979 Feb 25 21:33:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 21:33:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 21:34:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f6596a8c-3f77-8521-5252-5a76feead9f0 (at 10.8.3.11@o2ib6) in 204 seconds. I think it's dead, and I am evicting it. exp ffff9848845fb800, cur 1551159282 expire 1551159132 last 1551159078 Feb 25 21:34:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 21:35:05 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f6596a8c-3f77-8521-5252-5a76feead9f0 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985763314000, cur 1551159305 expire 1551159155 last 1551159078 Feb 25 21:35:05 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 25 21:37:08 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 25 21:37:08 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 21:46:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7a9f52f3-fad7-d644-ae70-019d1a25b459 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9838570c0000, cur 1551160007 expire 1551159857 last 1551159780 Feb 25 21:47:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 25 21:47:46 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 25 21:48:03 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e04a8aec-dcb2-5c8c-13d5-b4c039e3f592 (at 10.8.18.35@o2ib6) in 209 seconds. I think it's dead, and I am evicting it. exp ffff98654faaec00, cur 1551160083 expire 1551159933 last 1551159874 Feb 25 21:48:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 21:49:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 25 21:49:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 21:57:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0ce06e46-72c7-b07f-f910-fb248deafcb9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834056c00, cur 1551160635 expire 1551160485 last 1551160408 Feb 25 21:57:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:00:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 22:00:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:00:55 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b6bd4729-6340-f90c-f09c-89c915400705 (at 10.8.30.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867839e6800, cur 1551160855 expire 1551160705 last 1551160628 Feb 25 22:00:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:24:20 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client da953acd-2596-440d-58c4-8374a6a21470 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984803328800, cur 1551162260 expire 1551162110 last 1551162033 Feb 25 22:24:20 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 25 22:24:39 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client da953acd-2596-440d-58c4-8374a6a21470 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987068889c00, cur 1551162279 expire 1551162129 last 1551162052 Feb 25 22:24:39 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 22:24:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 22:24:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:29:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 863a9890-2a07-a756-283f-3f5a461acf11 (at 10.8.24.30@o2ib6) Feb 25 22:29:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:30:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 16b26052-afa3-f913-3f13-3b653d2521a8 (at 10.8.30.23@o2ib6) Feb 25 22:30:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:30:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd33c9b1-bb5e-c24e-0675-22654fcc67c5 (at 10.8.24.26@o2ib6) Feb 25 22:30:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:30:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b6bd4729-6340-f90c-f09c-89c915400705 (at 10.8.30.9@o2ib6) Feb 25 22:30:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 25 22:31:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 764f4d18-0114-7074-fa62-ebd0123856ba (at 10.8.30.10@o2ib6) Feb 25 22:31:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 22:34:32 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ecc6aef0-dc0a-3dfa-b81e-4b6bace22b92 (at 10.8.30.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e7da2f000, cur 1551162872 expire 1551162722 last 1551162645 Feb 25 22:34:32 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 25 22:34:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b6bd4729-6340-f90c-f09c-89c915400705 (at 10.8.30.9@o2ib6) Feb 25 22:34:40 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 25 23:12:09 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551165122/real 1551165122] req@ffff985e7a404b00 x1625015845468128/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551165129 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 25 23:12:09 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 51 previous similar messages Feb 25 23:12:30 fir-io1-s1 kernel: Lustre: 96329:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551165143/real 1551165143] req@ffff9842bc6cbf00 x1625015845468144/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551165150 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 23:12:30 fir-io1-s1 kernel: Lustre: 96329:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 25 23:13:12 fir-io1-s1 kernel: Lustre: 97133:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551165185/real 1551165185] req@ffff987169c7b600 x1625015845468112/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551165192 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 23:13:12 fir-io1-s1 kernel: Lustre: 1163:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551165185/real 1551165185] req@ffff98761328d700 x1625015845468096/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551165192 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 23:13:12 fir-io1-s1 kernel: Lustre: 1163:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 25 23:13:12 fir-io1-s1 kernel: Lustre: 97133:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 25 23:14:29 fir-io1-s1 kernel: Lustre: 1163:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551165262/real 1551165262] req@ffff98761328d700 x1625015845468096/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551165269 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 25 23:14:29 fir-io1-s1 kernel: Lustre: 1163:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 40 previous similar messages Feb 25 23:15:22 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 62b86656-e49b-4aad-513b-c6fdd686a279 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904994000, cur 1551165322 expire 1551165172 last 1551165095 Feb 25 23:15:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 23:15:22 fir-io1-s1 kernel: LNet: Service thread pid 1163 was inactive for 200.40s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 25 23:15:22 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Feb 25 23:15:22 fir-io1-s1 kernel: Pid: 1163, comm: ll_ost03_039 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 25 23:15:22 fir-io1-s1 kernel: Call Trace: Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 25 23:15:22 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 25 23:15:22 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551165322.1163 Feb 25 23:15:23 fir-io1-s1 kernel: LNet: Service thread pid 97133 was inactive for 201.22s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 25 23:15:23 fir-io1-s1 kernel: Pid: 97133, comm: ll_ost03_030 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 25 23:15:23 fir-io1-s1 kernel: Call Trace: Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 25 23:15:23 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 25 23:15:25 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 62b86656-e49b-4aad-513b-c6fdd686a279 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b5d681800, cur 1551165325 expire 1551165175 last 1551165098 Feb 25 23:15:25 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 25 23:15:25 fir-io1-s1 kernel: LNet: Service thread pid 97133 completed after 202.62s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 25 23:15:25 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Feb 25 23:16:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 23:16:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 23:20:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 23:20:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 23:21:24 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 857874a0-6c65-87fc-b21e-d57cfb925c3f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe8f6000, cur 1551165684 expire 1551165534 last 1551165457 Feb 25 23:21:24 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 25 23:30:49 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cffdbbde-fe7e-e65c-96dd-0c657ff57df3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480073dc00, cur 1551166249 expire 1551166099 last 1551166022 Feb 25 23:30:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 23:39:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 23:39:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 23:51:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 581597c4-fc7f-ab16-a755-4b98fb05a39e (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762999c00, cur 1551167517 expire 1551167367 last 1551167290 Feb 25 23:51:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 23:56:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 25 23:56:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 25 23:57:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 25 23:57:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 00:00:43 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c88800, cur 1551168043 expire 1551167893 last 1551167816 Feb 26 00:00:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 00:34:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) Feb 26 00:34:51 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 00:52:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ab18c7b8-19fe-0a85-20d5-e8ebe1d8b280 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a6000, cur 1551171147 expire 1551170997 last 1551170920 Feb 26 00:52:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 00:52:40 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ab18c7b8-19fe-0a85-20d5-e8ebe1d8b280 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683bb5f000, cur 1551171160 expire 1551171010 last 1551170933 Feb 26 00:52:40 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 26 00:52:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ab18c7b8-19fe-0a85-20d5-e8ebe1d8b280 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575f4d8000, cur 1551171162 expire 1551171012 last 1551170935 Feb 26 00:57:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 00:57:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 01:58:07 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 42e3d8b1-2fbe-8c39-9aa5-ba3ebc735d9f (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e16400, cur 1551175087 expire 1551174937 last 1551174860 Feb 26 01:58:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 42e3d8b1-2fbe-8c39-9aa5-ba3ebc735d9f (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483a2ef800, cur 1551175092 expire 1551174942 last 1551174865 Feb 26 01:58:13 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 42e3d8b1-2fbe-8c39-9aa5-ba3ebc735d9f (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483a2ee000, cur 1551175093 expire 1551174943 last 1551174866 Feb 26 02:14:12 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 02:47:08 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 03:26:48 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 03:28:04 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 03:31:05 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 03:33:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0909ba82-9930-4742-92e1-1f27f6cb6323 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d55fc00, cur 1551180801 expire 1551180651 last 1551180574 Feb 26 03:33:21 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 26 03:34:28 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 03:41:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 03:41:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 03:44:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7319d1c9-6e29-bb1b-6c4e-7953610895a2 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767c4a400, cur 1551181456 expire 1551181306 last 1551181229 Feb 26 03:44:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 03:45:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 03:45:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 03:45:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 03:45:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 03:45:32 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fd2175c4-6e22-c21f-3953-e6131baee4bf (at 10.8.18.35@o2ib6) in 217 seconds. I think it's dead, and I am evicting it. exp ffff986784b13800, cur 1551181532 expire 1551181382 last 1551181315 Feb 26 03:45:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 03:45:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client fd2175c4-6e22-c21f-3953-e6131baee4bf (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c3f400, cur 1551181542 expire 1551181392 last 1551181315 Feb 26 03:45:42 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 03:46:16 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 03:53:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0e938983-d3df-1102-1bfb-0c5d01e88c96 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d64d400, cur 1551182032 expire 1551181882 last 1551181805 Feb 26 03:54:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 03:54:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 04:03:11 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 109dabda-4797-b0a5-89ea-028612db99a3 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a99000, cur 1551182591 expire 1551182441 last 1551182364 Feb 26 04:03:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 04:04:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 26 04:04:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 04:19:49 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 04:46:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bd83aa88-0d86-0b70-e3b2-63276f1aee23 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0f800, cur 1551185186 expire 1551185036 last 1551184959 Feb 26 04:46:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 04:48:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 04:48:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 05:13:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 05:13:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 05:13:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 911bba1d-aac3-a561-aabc-a836fe660fe3 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834aeec00, cur 1551186816 expire 1551186666 last 1551186589 Feb 26 05:13:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 05:14:52 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3c8de293-496c-dae1-016a-77f5d18cf419 (at 10.8.11.9@o2ib6) in 158 seconds. I think it's dead, and I am evicting it. exp ffff984b283da000, cur 1551186892 expire 1551186742 last 1551186734 Feb 26 05:14:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 05:18:11 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 26 05:18:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:03:22 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 06:08:34 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 06:11:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 791aa87a-7b20-d4d6-3247-6ad336e4545c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867818ab800, cur 1551190272 expire 1551190122 last 1551190045 Feb 26 06:11:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:11:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 06:11:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:15:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 76c51379-dde3-4295-8312-25d0f46055a9 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780860400, cur 1551190539 expire 1551190389 last 1551190312 Feb 26 06:15:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:16:23 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 06:16:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:22:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bd2d5f7f-4b3c-0d29-45f5-a53d85119c72 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9838570c2800, cur 1551190962 expire 1551190812 last 1551190735 Feb 26 06:22:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:22:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 06:22:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:28:01 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 22a5922f-9d58-d21e-7a88-c946eba96d2d (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868d0000, cur 1551191281 expire 1551191131 last 1551191054 Feb 26 06:28:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:30:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 06:30:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:34:43 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4316e827-3e4f-df9e-9279-d416e0148338 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801a8800, cur 1551191683 expire 1551191533 last 1551191456 Feb 26 06:34:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:35:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 06:35:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:41:50 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 71bf8a1a-0452-bb7d-4be1-13ec78d48399 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986abbf65800, cur 1551192110 expire 1551191960 last 1551191883 Feb 26 06:41:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:42:11 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 71bf8a1a-0452-bb7d-4be1-13ec78d48399 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0e800, cur 1551192131 expire 1551191981 last 1551191904 Feb 26 06:42:11 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 26 06:42:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 06:42:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:46:34 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cd946d37-3a66-a68f-3462-5824ba2cb1fe (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983801b51c00, cur 1551192394 expire 1551192244 last 1551192167 Feb 26 06:46:34 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 26 06:47:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 06:47:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:51:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 21e60294-3604-74ca-986d-7feabe89217b (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852b0f49c00, cur 1551192671 expire 1551192521 last 1551192444 Feb 26 06:51:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 06:52:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 06:52:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 06:58:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d8f8750c-f7b2-c810-7c7b-ca2259742a22 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0a800, cur 1551193107 expire 1551192957 last 1551192880 Feb 26 06:58:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 07:00:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 07:00:49 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 26 07:04:36 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 550cf38d-49ff-1d72-d6e0-57f462d78193 (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98380dbfb800, cur 1551193476 expire 1551193326 last 1551193249 Feb 26 07:04:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 07:04:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 26 07:04:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 07:14:09 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a1aab7fc-9d47-9d13-6399-d761c763291c (at 10.8.18.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483b4e6400, cur 1551194049 expire 1551193899 last 1551193822 Feb 26 07:14:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 07:17:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 07:17:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 07:30:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4d299f05-b526-db6f-54d0-b5a94e0b6d6e (at 10.8.25.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848831e6800, cur 1551195000 expire 1551194850 last 1551194773 Feb 26 07:30:00 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 26 07:36:23 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 07:39:15 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 07:53:14 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 10e6bd46-05f6-9b9d-3dcb-0d8f671c6c47 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678448bc00, cur 1551196394 expire 1551196244 last 1551196167 Feb 26 07:53:14 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 26 07:54:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 07:54:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 07:58:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4d299f05-b526-db6f-54d0-b5a94e0b6d6e (at 10.8.25.33@o2ib6) Feb 26 07:58:32 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 26 08:02:59 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 081f71a8-941c-ae80-663d-2b7d00841174 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c20800, cur 1551196979 expire 1551196829 last 1551196752 Feb 26 08:02:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 08:04:04 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 08:05:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 08:05:00 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 26 08:12:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 08:12:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 08:12:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e906aeae-ab64-24f4-4215-89a5b36c51e0 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a7e400, cur 1551197555 expire 1551197405 last 1551197328 Feb 26 08:12:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 08:23:14 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7f18a6dc-08e0-1658-7449-c7cf75edac4e (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c20400, cur 1551198194 expire 1551198044 last 1551197967 Feb 26 08:23:14 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 26 08:23:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 26 08:23:47 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 26 08:26:53 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 08:29:12 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 08:37:05 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ca1af5b2-4b74-b03d-4a2b-13a823b2dc8f (at 10.8.15.10@o2ib6) Feb 26 08:37:05 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 08:47:29 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8c7c352c-0786-b576-65c8-08222987391a (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839e44c00, cur 1551199649 expire 1551199499 last 1551199422 Feb 26 08:47:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 08:47:46 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 08:50:43 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0a3359cf-f01b-f0ca-47f5-3d722a20fa29 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1554800, cur 1551199843 expire 1551199693 last 1551199616 Feb 26 08:50:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 08:51:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d2da3ac-e307-036e-f8a3-99f8b3ab4ed7 (at 10.8.15.5@o2ib6) Feb 26 08:51:09 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 26 08:59:57 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 584a550f-058e-90b1-f777-04bc395e1bdc (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f0e4c7800, cur 1551200397 expire 1551200247 last 1551200170 Feb 26 08:59:57 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 26 09:01:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 09:01:20 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 09:09:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client da75353c-11a4-d7b7-34db-4dc9a8bb77af (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480052a400, cur 1551200984 expire 1551200834 last 1551200757 Feb 26 09:09:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 09:11:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 09:11:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 09:29:21 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cd0798b5-98a5-1d8c-9fd3-a878f63429f4 (at 10.9.104.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987066f9d000, cur 1551202161 expire 1551202011 last 1551201934 Feb 26 09:29:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 09:29:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5ec5206a-1c58-eeff-df49-5fe7d326e368 (at 10.9.104.28@o2ib4) Feb 26 09:29:55 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 26 09:31:13 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 09:54:43 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 09:58:22 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4e438827-7ac6-b97d-f221-da77297fcae6 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833e70c00, cur 1551203902 expire 1551203752 last 1551203675 Feb 26 09:58:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 09:59:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8f36b2b9-a8f8-2d24-c0a3-c4a3b02a6628 (at 10.8.8.26@o2ib6) in 204 seconds. I think it's dead, and I am evicting it. exp ffff984834aec800, cur 1551203978 expire 1551203828 last 1551203774 Feb 26 09:59:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:00:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 10:00:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:08:11 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 44ed43ca-85f6-8139-47bf-2021b00e7371 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867804f2000, cur 1551204491 expire 1551204341 last 1551204264 Feb 26 10:08:11 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 26 10:09:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 10:09:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:10:20 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 10:27:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 586a6e8e-70fe-af28-3ec3-56d6983d8923 (at 10.8.9.6@o2ib6) Feb 26 10:27:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:28:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 97c8e837-ba4b-d84f-94e1-60778fc028be (at 10.8.8.37@o2ib6) Feb 26 10:28:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:30:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8f36b2b9-a8f8-2d24-c0a3-c4a3b02a6628 (at 10.8.8.26@o2ib6) Feb 26 10:30:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 10:32:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 86c6afcd-91e0-38ae-f048-b29dd98ae25f (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801a9400, cur 1551205957 expire 1551205807 last 1551205730 Feb 26 10:32:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:34:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 10:34:38 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 26 10:38:52 fir-io1-s1 kernel: Lustre: 97128:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551206321/real 1551206321] req@ffff986d94ecfb00 x1625058804893152/t0(0) o106->fir-OST0000@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551206332 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 26 10:38:52 fir-io1-s1 kernel: Lustre: 97128:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Feb 26 10:39:14 fir-io1-s1 kernel: Lustre: 96756:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551206343/real 1551206343] req@ffff986c7e71c800 x1625058804893056/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551206354 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 10:39:14 fir-io1-s1 kernel: Lustre: 96790:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551206343/real 1551206343] req@ffff98742913b300 x1625058804892976/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551206354 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 10:39:14 fir-io1-s1 kernel: Lustre: 96790:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 26 10:39:14 fir-io1-s1 kernel: Lustre: 96756:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 26 10:39:58 fir-io1-s1 kernel: Lustre: 96790:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551206387/real 1551206387] req@ffff98742913b300 x1625058804892976/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551206398 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 10:39:58 fir-io1-s1 kernel: Lustre: 96790:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Feb 26 10:41:15 fir-io1-s1 kernel: Lustre: 96790:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551206464/real 1551206464] req@ffff98742913b300 x1625058804892976/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551206475 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 10:41:15 fir-io1-s1 kernel: Lustre: 97128:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551206464/real 1551206464] req@ffff986d94ecfb00 x1625058804893152/t0(0) o106->fir-OST0000@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551206475 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 10:41:15 fir-io1-s1 kernel: Lustre: 97128:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Feb 26 10:41:15 fir-io1-s1 kernel: Lustre: 96790:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 26 10:41:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b5df26d2-8439-f08e-34c5-eb65650c2837 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a942ca000, cur 1551206508 expire 1551206358 last 1551206281 Feb 26 10:41:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:43:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 10:43:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:51:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4afa537b-8741-2d9e-bcf4-6005547fa285 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8dc00, cur 1551207093 expire 1551206943 last 1551206866 Feb 26 10:51:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 10:53:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 10:53:35 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 26 11:04:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client cbf5d06d-ba23-f67d-ac30-844ab79b193e (at 10.9.103.29@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857618e3800, cur 1551207867 expire 1551207717 last 1551207640 Feb 26 11:04:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 11:06:47 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 11:13:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c3993171-24c0-89a4-7cb1-27b4ddbf15a6 (at 10.8.6.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986838661000, cur 1551208424 expire 1551208274 last 1551208197 Feb 26 11:13:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 11:27:38 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 26 11:27:38 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Feb 26 11:27:38 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (0): c: 0, oc: 0, rc: 5 Feb 26 11:27:38 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 2 previous similar messages Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9851901dc800 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff987104e88000 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff98566091ba00 Feb 26 11:27:38 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1551209252/real 1551209258] req@ffff9852e66d2a00 x1625061613405632/t0(0) o106->fir-OST0008@10.8.6.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551209259 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff984fc329c800 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff98566091ba00 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff984fc3299a00 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff987104e88000 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff98566091ba00 Feb 26 11:27:38 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff984fc329c800 Feb 26 11:27:38 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 26 11:27:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 99c16f82-511c-281a-407d-bfbb9f83ae0f (at 10.8.0.66@o2ib6) reconnecting Feb 26 11:27:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to cf27932c-5cfb-509a-c7ce-6753e8ed5f45 (at 10.8.0.66@o2ib6) Feb 26 11:27:45 fir-io1-s1 kernel: Lustre: fir-OST0004: Client 176738e3-25c4-2c3c-9700-cb7836330618 (at 10.8.24.9@o2ib6) reconnecting Feb 26 11:28:20 fir-io1-s1 kernel: Lustre: 96944:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551209293/real 1551209293] req@ffff987320c06300 x1625061613417312/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551209300 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 11:28:20 fir-io1-s1 kernel: Lustre: 96944:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 578 previous similar messages Feb 26 11:28:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Client d9515410-0445-4a85-7660-888702fe9cec (at 10.8.13.15@o2ib6) reconnecting Feb 26 11:28:22 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 26 11:28:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 2ac12646-4ace-a274-aebb-64d4c0a4cd11 (at 10.8.16.7@o2ib6) reconnecting Feb 26 11:28:25 fir-io1-s1 kernel: LustreError: 96671:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff984cd57ad450 x1626187098349488/t0(0) o4->2ac12646-4ace-a274-aebb-64d4c0a4cd11@10.8.16.7@o2ib6:673/0 lens 488/448 e 1 to 0 dl 1551209328 ref 1 fl Interpret:/0/0 rc 0/0 Feb 26 11:28:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Bulk IO write error with 2ac12646-4ace-a274-aebb-64d4c0a4cd11 (at 10.8.16.7@o2ib6), client will retry: rc = -110 Feb 26 11:28:25 fir-io1-s1 kernel: LustreError: 96671:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message Feb 26 11:28:37 fir-io1-s1 kernel: LustreError: 96306:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ req@ffff985756f5ac50 x1626148427699120/t0(0) o3->37c5171f-2ee0-3f9b-c33d-fe1aa35295d6@10.8.9.1@o2ib6:679/0 lens 488/440 e 3 to 0 dl 1551209334 ref 1 fl Interpret:/0/0 rc 0/0 Feb 26 11:28:37 fir-io1-s1 kernel: Lustre: fir-OST0002: Bulk IO read error with 37c5171f-2ee0-3f9b-c33d-fe1aa35295d6 (at 10.8.9.1@o2ib6), client will retry: rc -110 Feb 26 11:28:37 fir-io1-s1 kernel: LustreError: 96306:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 4 previous similar messages Feb 26 11:28:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 45b8060c-e238-60ce-8b43-82cd7d128af3 (at 10.8.8.29@o2ib6) reconnecting Feb 26 11:28:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 37c5171f-2ee0-3f9b-c33d-fe1aa35295d6 (at 10.8.9.1@o2ib6) reconnecting Feb 26 11:28:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to fc60883d-f9c1-82aa-8312-f53a10d6b6ff (at 10.8.9.1@o2ib6) Feb 26 11:28:55 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 26 11:29:37 fir-io1-s1 kernel: Lustre: 96356:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551209370/real 1551209370] req@ffff984fe616cb00 x1625061613417376/t0(0) o106->fir-OST0008@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551209377 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 11:29:37 fir-io1-s1 kernel: Lustre: 96356:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1191 previous similar messages Feb 26 11:30:53 fir-io1-s1 kernel: LNet: Service thread pid 94238 was inactive for 200.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 26 11:30:53 fir-io1-s1 kernel: Pid: 94238, comm: ll_ost01_000 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 11:30:53 fir-io1-s1 kernel: Call Trace: Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 11:30:53 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 11:30:53 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551209453.94238 Feb 26 11:30:54 fir-io1-s1 kernel: LNet: Service thread pid 96251 was inactive for 201.85s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 26 11:30:54 fir-io1-s1 kernel: Pid: 96251, comm: ll_ost01_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 11:30:54 fir-io1-s1 kernel: Call Trace: Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 11:30:54 fir-io1-s1 kernel: Pid: 96524, comm: ll_ost01_062 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 11:30:54 fir-io1-s1 kernel: Call Trace: Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 11:30:54 fir-io1-s1 kernel: Pid: 96611, comm: ll_ost01_068 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 11:30:54 fir-io1-s1 kernel: Call Trace: Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 11:30:54 fir-io1-s1 kernel: Pid: 96365, comm: ll_ost01_043 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 11:30:54 fir-io1-s1 kernel: Call Trace: Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 11:30:54 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 11:30:54 fir-io1-s1 kernel: LNet: Service thread pid 94244 was inactive for 202.25s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 11:30:58 fir-io1-s1 kernel: LNet: Service thread pid 96949 was inactive for 200.29s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 11:30:58 fir-io1-s1 kernel: LNet: Skipped 30 previous similar messages Feb 26 11:30:58 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551209458.96949 Feb 26 11:30:59 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551209459.96888 Feb 26 11:30:59 fir-io1-s1 kernel: LNet: Service thread pid 96932 was inactive for 200.31s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 11:30:59 fir-io1-s1 kernel: LNet: Skipped 75 previous similar messages Feb 26 11:31:00 fir-io1-s1 kernel: LustreError: 94242:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) returned error from blocking AST (req@ffff985c25cd0300 x1625061613693344 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff984e7dd745c0/0x49e185f858fab20f lrc: 4/0,0 mode: PR/PR res: [0xc40000401:0xc8606:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xc49ee6ec8c65de35 expref: 92 pid: 96515 timeout: 1557311 lvb_type: 1 Feb 26 11:31:00 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Feb 26 11:31:00 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff984e4f903180/0x49e185f859053469 lrc: 3/0,0 mode: PR/PR res: [0xc40000401:0xc709a:0x0].0x0 rrc: 35 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xc49ee6ec8c67f9c8 expref: 93 pid: 96570 timeout: 0 lvb_type: 1 Feb 26 11:31:00 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Feb 26 11:31:00 fir-io1-s1 kernel: LNet: Service thread pid 96896 completed after 201.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 26 11:31:00 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Feb 26 11:31:00 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 26 11:32:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 11:32:14 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 26 11:33:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d9657556-3698-de72-acc2-cb9f2581779e (at 10.9.106.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf09c00, cur 1551209619 expire 1551209469 last 1551209392 Feb 26 11:33:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 11:45:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c3993171-24c0-89a4-7cb1-27b4ddbf15a6 (at 10.8.6.36@o2ib6) Feb 26 11:45:26 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 12:04:49 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d9657556-3698-de72-acc2-cb9f2581779e (at 10.9.106.5@o2ib4) Feb 26 12:04:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 12:11:43 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 12:15:31 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client dace36ca-aee2-2621-1cf1-1972e147fbdb (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868ae400, cur 1551212131 expire 1551211981 last 1551211904 Feb 26 12:15:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 12:21:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 26 12:21:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 12:26:29 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da0800, cur 1551212789 expire 1551212639 last 1551212562 Feb 26 12:26:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 12:26:29 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da4800, cur 1551212789 expire 1551212639 last 1551212562 Feb 26 12:26:29 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 26 12:26:42 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98711addfc00, cur 1551212802 expire 1551212652 last 1551212575 Feb 26 12:26:42 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 26 12:35:04 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 26 12:35:04 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 2 previous similar messages Feb 26 12:35:04 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.203@o2ib7 (0): c: 0, oc: 1, rc: 8 Feb 26 12:35:04 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 2 previous similar messages Feb 26 12:35:04 fir-io1-s1 kernel: Lustre: 2370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1551213298/real 1551213304] req@ffff986f1dab5700 x1625066301566176/t0(0) o106->fir-OST0000@10.8.2.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551213305 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Feb 26 12:35:04 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.203@o2ib7: accepting Feb 26 12:35:04 fir-io1-s1 kernel: Lustre: 2370:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2063 previous similar messages Feb 26 12:35:23 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551213316/real 1551213316] req@ffff98593cb4da00 x1625066301686896/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551213323 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 12:35:23 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551213316/real 1551213316] req@ffff98620f3b0000 x1625066301686864/t0(0) o106->fir-OST0000@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551213323 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 12:35:23 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 513 previous similar messages Feb 26 12:36:38 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551213391/real 1551213391] req@ffff984e9d278900 x1625066301705040/t0(0) o106->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551213398 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 12:36:38 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2821 previous similar messages Feb 26 12:38:18 fir-io1-s1 kernel: LNet: Service thread pid 96369 was inactive for 200.71s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 26 12:38:18 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 26 12:38:18 fir-io1-s1 kernel: Pid: 96369, comm: ll_ost01_046 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 12:38:18 fir-io1-s1 kernel: Call Trace: Feb 26 12:38:18 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 12:38:18 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 12:38:19 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213499.96369 Feb 26 12:38:19 fir-io1-s1 kernel: Pid: 97131, comm: ll_ost03_028 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 12:38:19 fir-io1-s1 kernel: Call Trace: Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 12:38:19 fir-io1-s1 kernel: Pid: 96491, comm: ll_ost01_058 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 12:38:19 fir-io1-s1 kernel: Call Trace: Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 12:38:19 fir-io1-s1 kernel: Pid: 96248, comm: ll_ost02_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 12:38:19 fir-io1-s1 kernel: Call Trace: Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 12:38:19 fir-io1-s1 kernel: LNet: Service thread pid 94242 was inactive for 201.27s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 26 12:38:19 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 26 12:38:19 fir-io1-s1 kernel: Pid: 94242, comm: ll_ost02_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 12:38:19 fir-io1-s1 kernel: Call Trace: Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 12:38:19 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 12:38:19 fir-io1-s1 kernel: LNet: Service thread pid 96915 was inactive for 201.38s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 12:38:19 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Feb 26 12:38:25 fir-io1-s1 kernel: LNet: Service thread pid 36980 was inactive for 200.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 12:38:25 fir-io1-s1 kernel: LNet: Skipped 34 previous similar messages Feb 26 12:38:25 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213505.36980 Feb 26 12:38:26 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213506.74693 Feb 26 12:38:26 fir-io1-s1 kernel: LNet: Service thread pid 49820 was inactive for 200.34s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 12:38:26 fir-io1-s1 kernel: LNet: Skipped 179 previous similar messages Feb 26 12:38:27 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213507.96255 Feb 26 12:38:28 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213508.94514 Feb 26 12:38:30 fir-io1-s1 kernel: LNet: Service thread pid 96253 was inactive for 200.42s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 12:38:30 fir-io1-s1 kernel: LNet: Skipped 15 previous similar messages Feb 26 12:38:30 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213510.96253 Feb 26 12:38:31 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213511.74749 Feb 26 12:38:32 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213512.96909 Feb 26 12:38:33 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213513.49816 Feb 26 12:38:34 fir-io1-s1 kernel: LNet: Service thread pid 96774 was inactive for 200.29s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 26 12:38:34 fir-io1-s1 kernel: LNet: Skipped 27 previous similar messages Feb 26 12:38:34 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213514.96774 Feb 26 12:38:35 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551213515.96362 Feb 26 12:38:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4bc75f04-b8d9-ad99-995d-27dacc9e399f (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a2d800, cur 1551213524 expire 1551213374 last 1551213297 Feb 26 12:38:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 26 12:38:44 fir-io1-s1 kernel: LNet: Service thread pid 96328 completed after 218.86s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 26 12:38:44 fir-io1-s1 kernel: LNet: Skipped 230 previous similar messages Feb 26 12:38:45 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4bc75f04-b8d9-ad99-995d-27dacc9e399f (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bd75c00, cur 1551213525 expire 1551213375 last 1551213298 Feb 26 12:38:45 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 26 12:38:45 fir-io1-s1 kernel: LNet: Service thread pid 36979 completed after 219.86s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 26 12:38:45 fir-io1-s1 kernel: LNet: Service thread pid 63944 completed after 214.33s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 26 12:38:45 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Feb 26 12:38:45 fir-io1-s1 kernel: LNet: Skipped 132 previous similar messages Feb 26 12:40:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 12:40:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 12:40:58 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 12:47:56 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 12:48:33 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1dd4f190-5ae4-caa5-985f-1c9f4e428645 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c3400, cur 1551214113 expire 1551213963 last 1551213886 Feb 26 12:48:33 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 26 12:50:46 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 12:51:54 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 12:58:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b9ec06be-945c-dd77-0a54-c23157571370 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a2ac00, cur 1551214738 expire 1551214588 last 1551214511 Feb 26 12:58:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 12:59:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 54802b78-b8a3-6ef2-1c4c-1f299e55d7f1 (at 10.9.103.14@o2ib4) Feb 26 12:59:57 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 26 13:03:25 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 86edcde5-a827-eaa7-4e0d-837c9d785f60 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e17400, cur 1551215005 expire 1551214855 last 1551214778 Feb 26 13:03:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 13:09:03 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 13:11:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 41de1f40-a368-cb6e-edf5-62e1463dd452 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d559800, cur 1551215473 expire 1551215323 last 1551215246 Feb 26 13:11:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 13:14:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 13:14:02 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 26 13:17:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 70a023b4-747e-4af8-d033-8c85e93a3452 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c6d400, cur 1551215846 expire 1551215696 last 1551215619 Feb 26 13:17:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 13:24:15 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 13:24:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 13:24:43 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 26 13:25:22 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7b549f94-39e7-01a6-e5cf-5436bf8dbade (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984836944800, cur 1551216322 expire 1551216172 last 1551216095 Feb 26 13:25:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 13:32:42 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 13:33:58 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d9daee9c-edb1-9d2e-619d-f21193f2215f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784bb8800, cur 1551216838 expire 1551216688 last 1551216611 Feb 26 13:33:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 13:36:54 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to (at 10.9.0.64@o2ib4) Feb 26 13:36:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 13:38:12 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a6a5e505-8da2-2047-e14c-972c90c9cbf8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a78400, cur 1551217092 expire 1551216942 last 1551216865 Feb 26 13:38:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 13:48:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 13:48:28 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 13:48:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 684ea0e7-0c08-6703-7248-c95b9a40c5aa (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825f9800, cur 1551217737 expire 1551217587 last 1551217510 Feb 26 13:48:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 13:57:43 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4b878fb5-3dc5-fb97-9a4e-5290921e42e5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825fec00, cur 1551218263 expire 1551218113 last 1551218036 Feb 26 13:57:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:05:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 14:05:57 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 14:06:13 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9fc3cd54-25e8-a4b2-3be6-326429584c71 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d200800, cur 1551218773 expire 1551218623 last 1551218546 Feb 26 14:06:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:10:03 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c666e61c-7377-927e-1e75-11a3b8689ec2 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857630fa000, cur 1551219003 expire 1551218853 last 1551218776 Feb 26 14:10:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:16:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 14:16:30 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 14:16:52 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b8941599-a431-f47d-bd8e-4cd295239b41 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f6c00, cur 1551219412 expire 1551219262 last 1551219185 Feb 26 14:16:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:19:35 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e4e1992b-645c-7fa0-95fb-f97f854d138b (at 10.8.26.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762c71400, cur 1551219575 expire 1551219425 last 1551219348 Feb 26 14:19:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:30:21 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d8aa5f17-9f20-7c2a-069a-5ce841acf2bf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983801b54000, cur 1551220221 expire 1551220071 last 1551219994 Feb 26 14:30:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:31:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 14:31:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:43:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2557db10-3079-820d-ecbc-02fa2573f957 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767577000, cur 1551221031 expire 1551220881 last 1551220804 Feb 26 14:43:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:43:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 14:43:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 14:55:23 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c294684b-f6c0-f1a6-6900-0457bc4ed9b5 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596b9800, cur 1551221723 expire 1551221573 last 1551221496 Feb 26 14:55:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 15:00:39 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 15:00:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 26 15:00:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 15:12:53 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 15:31:11 fir-io1-s1 kernel: LustreError: 96886:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff9859bc665700 x1625078992249616 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff985492947080/0x49e185fa81cefb38 lrc: 3/0,0 mode: PW/PW res: [0xc40000402:0x1d0b60:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0x12e2f7a651f8def2 expref: 7 pid: 96368 timeout: 0 lvb_type: 0 Feb 26 15:31:11 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 26 15:31:11 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 26 15:31:11 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551223871s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff983dfef03a80/0x49e185fa81cef312 lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x1d0c3b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0x12e2f7a651f8de89 expref: 8 pid: 94629 timeout: 0 lvb_type: 0 Feb 26 15:31:11 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Feb 26 15:31:11 fir-io1-s1 kernel: LustreError: 96886:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 12 previous similar messages Feb 26 15:34:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 15:34:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 15:35:05 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 15:38:50 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 15:38:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c12a1684-dc00-3104-ef93-3cf20d893979 (at 10.9.106.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904992400, cur 1551224338 expire 1551224188 last 1551224111 Feb 26 15:38:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 15:47:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b24d4d9e-3a1a-723e-14ce-1a0c1b08163a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ce8861400, cur 1551224827 expire 1551224677 last 1551224600 Feb 26 15:47:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 15:48:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 15:48:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 15:56:29 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c2806c19-f08b-a384-fd39-3948eefaeb48 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575be1e800, cur 1551225389 expire 1551225239 last 1551225162 Feb 26 15:56:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 15:57:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 15:57:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 16:06:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client d4be1d60-e9d9-7981-8e01-28dbad17c4e7 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aab1400, cur 1551225960 expire 1551225810 last 1551225733 Feb 26 16:06:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 16:07:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 16:07:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 16:10:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c12a1684-dc00-3104-ef93-3cf20d893979 (at 10.9.106.8@o2ib4) Feb 26 16:10:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 16:15:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 16:15:15 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 16:16:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d78401ad-d7f9-70f0-e382-808abf23c9bd (at 10.9.103.10@o2ib4) in 202 seconds. I think it's dead, and I am evicting it. exp ffff9877a1461000, cur 1551226606 expire 1551226456 last 1551226404 Feb 26 16:16:46 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 16:24:38 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 16:25:04 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 16:33:29 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 16:36:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 16:36:19 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 26 16:36:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 39a5491d-e10b-c19b-ffc9-b833c49037a8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e7da29c00, cur 1551227797 expire 1551227647 last 1551227570 Feb 26 16:36:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 16:41:04 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 16:50:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a37c418b-5455-262c-ca39-fe09a1d64b6a (at 10.9.106.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a1557c00, cur 1551228601 expire 1551228451 last 1551228374 Feb 26 16:50:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 16:51:14 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 16:51:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d78401ad-d7f9-70f0-e382-808abf23c9bd (at 10.9.103.10@o2ib4) Feb 26 16:51:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:00:12 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ece24e32-c30a-37fd-6034-503af11c37b1 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767c4b400, cur 1551229212 expire 1551229062 last 1551228985 Feb 26 17:00:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:10:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 17:10:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:12:31 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 17:12:31 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Feb 26 17:13:33 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 17:18:03 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 1bc1040b-c71d-b7f4-505c-0eae20c9d5b2 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c26c00, cur 1551230283 expire 1551230133 last 1551230056 Feb 26 17:18:03 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 17:21:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a37c418b-5455-262c-ca39-fe09a1d64b6a (at 10.9.106.7@o2ib4) Feb 26 17:21:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:21:22 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 17:22:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 17:22:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:31:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f05ae9bb-74a9-aaeb-6de7-2ec3e892aa1c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c0c3e8800, cur 1551231075 expire 1551230925 last 1551230848 Feb 26 17:31:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:32:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 17:32:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:41:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 17:41:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 17:46:23 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 17:50:13 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 07de05ca-2f51-d8fa-31d4-2f1d95f20c90 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da4c00, cur 1551232213 expire 1551232063 last 1551231986 Feb 26 17:50:13 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 17:52:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 17:52:15 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 17:52:16 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 26 17:57:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 488cfd93-1121-504d-019d-485c13be114d (at 10.8.14.4@o2ib6) Feb 26 17:57:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:06:05 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a51994fb-3110-bb4f-ed4e-eced3e097c15 (at 10.8.1.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984818e9f000, cur 1551233165 expire 1551233015 last 1551232938 Feb 26 18:06:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 18:10:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 18:10:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:11:07 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 18:19:37 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 18:25:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa3988d8-312e-baa0-298b-1666a8960425 (at 10.8.14.2@o2ib6) Feb 26 18:25:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:33:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e033b2fb-58ee-ad20-dbe1-c069873ac977 (at 10.9.101.47@o2ib4) Feb 26 18:33:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:34:09 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b2124be6-9114-eedd-f3c5-1909bfcb6010 (at 10.8.17.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762493000, cur 1551234849 expire 1551234699 last 1551234622 Feb 26 18:34:09 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Feb 26 18:34:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fda7a4af-47c0-0068-cddf-309c3a9c784c (at 10.9.101.13@o2ib4) Feb 26 18:34:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:34:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bf617e6a-e2c4-2972-fed8-58b2cb638da2 (at 10.9.101.25@o2ib4) Feb 26 18:34:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:34:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4c5b69ea-d1f1-0261-ea03-15f22270fb92 (at 10.9.101.2@o2ib4) Feb 26 18:34:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:37:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 054698ea-f84b-bd00-f4ed-c64e725d9902 (at 10.8.1.2@o2ib6) Feb 26 18:37:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 18:37:27 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 18:37:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bab702f8-b44a-da13-8f71-e38d2f6bf022 (at 10.8.1.13@o2ib6) Feb 26 18:37:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:41:43 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5c4d5ec9-c7a9-4090-a2e7-0e452f357bdf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575883fc00, cur 1551235303 expire 1551235153 last 1551235076 Feb 26 18:41:43 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Feb 26 18:44:19 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 18:45:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 18:45:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:46:30 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f2bd8263-263e-b013-a9c1-5f61f7b17ac2 (at 10.9.106.9@o2ib4) Feb 26 18:46:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 18:58:30 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 19:02:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b0f4a89e-7973-eb31-a1c9-fdc42b6cc4f6 (at 10.8.18.25@o2ib6) Feb 26 19:02:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 19:02:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ec7b3c21-b807-d2d6-877f-ad536dfb41e4 (at 10.8.27.26@o2ib6) Feb 26 19:02:56 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 19:03:35 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 665b3c07-f583-a6a2-a031-36b81104a696 (at 10.8.9.3@o2ib6) Feb 26 19:03:35 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 26 19:05:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 92747d3f-03b1-884a-8015-40ea9f51416f (at 10.8.2.26@o2ib6) Feb 26 19:05:47 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 19:12:52 fir-io1-s1 kernel: Lustre: 96929:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551237165/real 1551237165] req@ffff985030b3ce00 x1625095393614176/t0(0) o106->fir-OST0002@10.9.113.6@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551237172 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 26 19:12:52 fir-io1-s1 kernel: Lustre: 96929:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4722 previous similar messages Feb 26 19:13:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3932c835-e4bd-99a2-5e8c-8fdd68aa9cbf (at 10.9.106.6@o2ib4) Feb 26 19:13:03 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Feb 26 19:13:13 fir-io1-s1 kernel: Lustre: 36981:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551237186/real 1551237186] req@ffff9861fe3a4500 x1625095393614656/t0(0) o106->fir-OST0000@10.9.113.6@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551237193 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 19:13:13 fir-io1-s1 kernel: Lustre: 36981:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Feb 26 19:13:55 fir-io1-s1 kernel: Lustre: 96757:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551237228/real 1551237228] req@ffff985dbb1b2a00 x1625095393615056/t0(0) o106->fir-OST0004@10.9.113.6@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551237235 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 19:13:55 fir-io1-s1 kernel: Lustre: 96757:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 26 19:15:12 fir-io1-s1 kernel: Lustre: 96929:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551237305/real 1551237305] req@ffff985030b3ce00 x1625095393614176/t0(0) o106->fir-OST0002@10.9.113.6@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551237312 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 19:15:12 fir-io1-s1 kernel: Lustre: 96929:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Feb 26 19:16:05 fir-io1-s1 kernel: LNet: Service thread pid 96757 was inactive for 200.15s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 26 19:16:05 fir-io1-s1 kernel: Pid: 96757, comm: ll_ost02_047 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 19:16:05 fir-io1-s1 kernel: Call Trace: Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 19:16:05 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 19:16:05 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551237365.96757 Feb 26 19:16:06 fir-io1-s1 kernel: LNet: Service thread pid 36981 was inactive for 201.02s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 26 19:16:06 fir-io1-s1 kernel: Pid: 36981, comm: ll_ost02_072 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 19:16:06 fir-io1-s1 kernel: Call Trace: Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 19:16:06 fir-io1-s1 kernel: Pid: 96929, comm: ll_ost01_103 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 26 19:16:06 fir-io1-s1 kernel: Call Trace: Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 26 19:16:06 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 26 19:16:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ebee093c-8e69-4e39-2895-94a502f715cc (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b611000, cur 1551237376 expire 1551237226 last 1551237149 Feb 26 19:16:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 26 19:16:27 fir-io1-s1 kernel: LNet: Service thread pid 36981 completed after 221.84s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 26 19:16:27 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Feb 26 19:31:51 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b42f3602-820d-dd01-5406-5780a0e4a943 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318eec00, cur 1551238311 expire 1551238161 last 1551238084 Feb 26 19:31:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 19:36:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 19:36:06 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 19:39:18 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 20:37:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e3d7d750-2a26-e81f-1160-2f3ee9d7f849 (at 10.9.106.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984838609400, cur 1551242260 expire 1551242110 last 1551242033 Feb 26 20:37:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 20:38:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e78a174a-ae69-cb32-9b42-0306c7153992 (at 10.9.103.7@o2ib4) in 225 seconds. I think it's dead, and I am evicting it. exp ffff98677dc5d000, cur 1551242336 expire 1551242186 last 1551242111 Feb 26 20:38:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 20:41:43 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 20:43:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fbc086c3-52be-ac9a-e64f-08d421c53770 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576855d000, cur 1551242580 expire 1551242430 last 1551242353 Feb 26 20:43:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 20:43:06 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client fbc086c3-52be-ac9a-e64f-08d421c53770 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d28000, cur 1551242586 expire 1551242436 last 1551242359 Feb 26 20:43:06 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 20:46:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 20:46:09 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Feb 26 21:03:33 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 21:08:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e3d7d750-2a26-e81f-1160-2f3ee9d7f849 (at 10.9.106.10@o2ib4) Feb 26 21:08:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 21:11:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e78a174a-ae69-cb32-9b42-0306c7153992 (at 10.9.103.7@o2ib4) Feb 26 21:11:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 21:30:00 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 11366750-6b6e-8b38-85bb-2768074a85aa (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904996000, cur 1551245400 expire 1551245250 last 1551245173 Feb 26 21:36:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 21:36:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 21:36:45 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 52733c2f-f233-b5cc-d346-759443d05063 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd2f2000, cur 1551245805 expire 1551245655 last 1551245578 Feb 26 21:36:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 21:39:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 26 21:39:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 21:40:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ef0c3a49-59bd-2fd0-071f-d5153b593f5a (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a8a000, cur 1551246027 expire 1551245877 last 1551245800 Feb 26 21:40:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 21:41:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 26 21:41:13 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 21:57:34 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 21:58:16 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 21:58:54 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 22:08:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 22:08:00 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 26 22:08:23 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client cf9afcda-5408-c066-9de7-934feb7f8f94 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d418800, cur 1551247703 expire 1551247553 last 1551247476 Feb 26 22:08:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:11:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 22:11:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:12:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8f1df1d4-2533-a800-2b3d-cffbe6a0c6bb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83c000, cur 1551247959 expire 1551247809 last 1551247732 Feb 26 22:12:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:19:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 24778ae5-cb3e-f669-8abc-afdf158d9942 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e49400, cur 1551248364 expire 1551248214 last 1551248137 Feb 26 22:19:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:20:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 26 22:20:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:28:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 33b24dc2-6993-d831-6b12-a459c16cbaa4 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d52000, cur 1551248893 expire 1551248743 last 1551248666 Feb 26 22:28:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:28:32 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 33b24dc2-6993-d831-6b12-a459c16cbaa4 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867839e2800, cur 1551248912 expire 1551248762 last 1551248685 Feb 26 22:28:32 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 26 22:29:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 26 22:29:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:40:00 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 22:52:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Feb 26 22:52:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:52:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3664c69e-7cfa-9e0a-6163-9c0f550b7fc7 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d4c00, cur 1551250342 expire 1551250192 last 1551250115 Feb 26 22:52:22 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 26 22:52:33 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f669be61-12f7-1c40-020d-d5e177414b94 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b264000, cur 1551250353 expire 1551250203 last 1551250126 Feb 26 22:52:33 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 26 22:55:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 22:55:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 22:56:27 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a3c57bef-a739-0ea9-6582-283914517ba2 (at 10.9.103.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984803158800, cur 1551250587 expire 1551250437 last 1551250360 Feb 26 22:56:27 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 26 23:10:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1740c437-48db-49df-0c57-e20eac633aec (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762493800, cur 1551251405 expire 1551251255 last 1551251178 Feb 26 23:10:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:10:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 26 23:10:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:21:09 fir-io1-s1 kernel: Lustre: 49827:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551252058/real 1551252058] req@ffff983c14de9e00 x1625113710974320/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551252069 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 26 23:21:09 fir-io1-s1 kernel: Lustre: 49827:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 33 previous similar messages Feb 26 23:21:31 fir-io1-s1 kernel: Lustre: 96889:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551252080/real 1551252080] req@ffff98380ca0c800 x1625113710974480/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551252091 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 23:21:31 fir-io1-s1 kernel: Lustre: 96889:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 26 23:21:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a3c57bef-a739-0ea9-6582-283914517ba2 (at 10.9.103.31@o2ib4) Feb 26 23:21:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:22:15 fir-io1-s1 kernel: Lustre: 76197:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551252124/real 1551252124] req@ffff985108344e00 x1625113710973312/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551252135 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 26 23:22:15 fir-io1-s1 kernel: Lustre: 76197:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 26 23:23:21 fir-io1-s1 kernel: LustreError: 96263:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff9844f6b94200 x1625113710974592 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff984a248dbcc0/0x49e185ff10e9f928 lrc: 3/0,0 mode: PW/PW res: [0x580000401:0x87e6b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0xcd4fa679da1389ce expref: 9 pid: 96924 timeout: 0 lvb_type: 0 Feb 26 23:23:21 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 26 23:23:21 fir-io1-s1 kernel: LustreError: Skipped 6 previous similar messages Feb 26 23:23:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551252201s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff983801d24800/0x49e185ff10e9e3bf lrc: 3/0,0 mode: PW/PW res: [0x8c0000400:0x87cba:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0xcd4fa679da138926 expref: 8 pid: 96924 timeout: 0 lvb_type: 0 Feb 26 23:23:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Feb 26 23:23:21 fir-io1-s1 kernel: LustreError: 96263:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Feb 26 23:23:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 23:23:22 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 23:23:22 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 26 23:23:25 fir-io1-s1 kernel: LustreError: 96505:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff984dcc35ad00 x1625113893151216 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff984ef3e53a80/0x49e185ff10e99855 lrc: 3/0,0 mode: PW/PW res: [0x5c0000401:0x87e28:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0xcd4fa679da138870 expref: 8 pid: 96943 timeout: 0 lvb_type: 0 Feb 26 23:23:25 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 26 23:23:25 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 26 23:23:25 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551252205s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff984c69398240/0x49e185ff10e99f39 lrc: 3/0,0 mode: PW/PW res: [0x6c0000401:0x87d37:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0xcd4fa679da1388a8 expref: 9 pid: 96907 timeout: 0 lvb_type: 0 Feb 26 23:23:25 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Feb 26 23:23:25 fir-io1-s1 kernel: LustreError: 96505:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Feb 26 23:26:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 26 23:26:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:27:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4e98aca1-5176-7dc4-0f5e-99bb1a25f4d5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984804f54000, cur 1551252455 expire 1551252305 last 1551252228 Feb 26 23:27:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:33:02 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 23:43:51 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b3363dd3-6f10-4e81-f362-10865357e42b (at 10.9.106.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a161f400, cur 1551253431 expire 1551253281 last 1551253204 Feb 26 23:43:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:46:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9fb5bc8f-c015-38a2-4d97-6a68a3eb423b (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786835800, cur 1551253600 expire 1551253450 last 1551253373 Feb 26 23:46:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:47:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 26 23:47:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:52:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6d0c0e81-8f24-8d59-6a5c-99645daffcc0 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83f800, cur 1551253957 expire 1551253807 last 1551253730 Feb 26 23:52:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:53:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 26 23:53:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:53:33 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 26 23:57:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c1606760-dc37-0be3-522c-32eac969b366 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d558400, cur 1551254275 expire 1551254125 last 1551254048 Feb 26 23:57:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:58:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 26 23:58:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 26 23:59:11 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client aed98c92-7516-2fd2-ad8f-6d6af3f10bd0 (at 10.8.20.15@o2ib6) in 161 seconds. I think it's dead, and I am evicting it. exp ffff9867868fb000, cur 1551254351 expire 1551254201 last 1551254190 Feb 26 23:59:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 00:02:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 00:02:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 00:03:10 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 827df6b9-8791-e80a-263e-89c9d4786313 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d55800, cur 1551254590 expire 1551254440 last 1551254363 Feb 27 00:03:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 00:03:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 00:03:53 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 00:14:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b3363dd3-6f10-4e81-f362-10865357e42b (at 10.9.106.11@o2ib4) Feb 27 00:14:29 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 27 00:27:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 12ed72a6-9ff3-0134-3548-4cdffb71fecc (at 10.8.29.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c1a800, cur 1551256029 expire 1551255879 last 1551255802 Feb 27 00:27:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 00:33:31 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 00:38:21 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 00:39:46 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 00:41:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6792fc33-6546-9fba-3aff-deaf8715d928 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83f800, cur 1551256913 expire 1551256763 last 1551256686 Feb 27 00:41:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 00:43:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 00:43:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 00:46:14 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 00:51:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.29.2@o2ib6) Feb 27 00:51:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:01:29 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 01:02:12 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 01:21:06 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 01:24:27 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 90c91e75-46ba-4a46-2672-39282762b97c (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d7400, cur 1551259467 expire 1551259317 last 1551259240 Feb 27 01:24:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:26:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 01:26:14 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 01:27:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client afb4e635-c62a-c4c5-c50e-c4203c8fbfa8 (at 10.8.4.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986816f3d800, cur 1551259630 expire 1551259480 last 1551259403 Feb 27 01:27:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:31:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 01:31:01 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 27 01:31:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 6a1c2c98-0d47-91b2-781a-f670f1c7d341 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2f800, cur 1551259876 expire 1551259726 last 1551259649 Feb 27 01:31:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:43:06 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9cb3db7d-519e-ab2a-317b-c21da9a59f55 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985838d8cc00, cur 1551260586 expire 1551260436 last 1551260359 Feb 27 01:43:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:44:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 01:44:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:47:54 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 18219315-7b4c-908a-998b-6f526962fed4 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d7400, cur 1551260874 expire 1551260724 last 1551260647 Feb 27 01:47:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:48:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 01:48:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:52:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 299e34ad-357d-a63b-a82f-0b7a2579ea0e (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c6fc00, cur 1551261156 expire 1551261006 last 1551260929 Feb 27 01:52:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 01:53:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 01:53:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:01:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4d30b815-3d3c-7361-5607-a211379e0754 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b5d680000, cur 1551261669 expire 1551261519 last 1551261442 Feb 27 02:01:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:01:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 02:01:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:03:44 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 02:08:58 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 81f5800f-4317-2f32-0d09-cb2da5456380 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98515ab35800, cur 1551262138 expire 1551261988 last 1551261911 Feb 27 02:08:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:10:14 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ecfc1d1f-def9-0233-b190-86369353b9ea (at 10.8.20.15@o2ib6) in 179 seconds. I think it's dead, and I am evicting it. exp ffff9854318e8400, cur 1551262214 expire 1551262064 last 1551262035 Feb 27 02:10:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:10:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 02:10:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:12:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 02:12:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:13:35 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 02:16:39 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9e7293dd-ec85-5c9b-e0d5-ec0e459ab351 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784bef000, cur 1551262599 expire 1551262449 last 1551262372 Feb 27 02:16:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:17:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 02:17:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:19:37 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 02:24:10 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 02:32:18 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 02:37:39 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 02:37:39 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Feb 27 02:40:19 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 02:41:04 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3fd9c8a9-9b32-aa2a-e9eb-390c019a0b2c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985423f11000, cur 1551264064 expire 1551263914 last 1551263837 Feb 27 02:41:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 02:41:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 02:41:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 03:00:21 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 03:06:38 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 03:06:38 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Feb 27 03:11:40 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 03:12:22 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 03:17:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 03:17:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 03:17:59 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1c74969a-bec7-76ba-47aa-a9f0b48eff93 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d649000, cur 1551266279 expire 1551266129 last 1551266052 Feb 27 03:17:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 03:18:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1c74969a-bec7-76ba-47aa-a9f0b48eff93 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985836977c00, cur 1551266281 expire 1551266131 last 1551266054 Feb 27 03:18:01 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 27 03:30:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 879d94a8-a845-ea21-f6e8-a2d093701c88 (at 10.9.103.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b9e6800, cur 1551267040 expire 1551266890 last 1551266813 Feb 27 03:30:40 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 27 03:42:55 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 03:53:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c983e690-4a52-f452-dd68-e347d580a80d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767574400, cur 1551268417 expire 1551268267 last 1551268190 Feb 27 03:53:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 03:53:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 03:53:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 03:53:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c983e690-4a52-f452-dd68-e347d580a80d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986813a73c00, cur 1551268435 expire 1551268285 last 1551268208 Feb 27 03:53:55 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 27 04:02:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 879d94a8-a845-ea21-f6e8-a2d093701c88 (at 10.9.103.15@o2ib4) Feb 27 04:02:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 04:02:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 04:02:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 04:07:16 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:07:47 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:10:47 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:26:16 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:27:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7620d13c-d859-7742-88ac-47548b1f7030 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8ab000, cur 1551270464 expire 1551270314 last 1551270237 Feb 27 04:27:44 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 27 04:27:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 04:27:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 04:29:30 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:31:24 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:35:27 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551270920/real 1551270920] req@ffff98572ca04200 x1625137115227808/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551270927 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 27 04:35:27 fir-io1-s1 kernel: Lustre: 96895:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 53 previous similar messages Feb 27 04:35:35 fir-io1-s1 kernel: LustreError: 76197:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.18.31@o2ib6) returned error from glimpse AST (req@ffff9851caf0f500 x1625137115227440 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff98381d5621c0/0x49e186021b94d12e lrc: 10/0,0 mode: PW/PW res: [0xc80000401:0xd01c1:0x0].0x0 rrc: 10 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x149478fe85143d4a expref: 5 pid: 96896 timeout: 0 lvb_type: 0 Feb 27 04:35:35 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.18.31@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 27 04:35:35 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Feb 27 04:35:35 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551270934s: evicting client at 10.8.18.31@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff98526f2e60c0/0x49e186021b94f259 lrc: 10/0,0 mode: PW/PW res: [0x6c0000402:0xd01cc:0x0].0x0 rrc: 10 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x149478fe85143deb expref: 6 pid: 96909 timeout: 0 lvb_type: 0 Feb 27 04:35:35 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Feb 27 04:35:35 fir-io1-s1 kernel: LustreError: 76197:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Feb 27 04:35:38 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:35:42 fir-io1-s1 kernel: Lustre: 96265:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551270934/real 1551270934] req@ffff9841920f1e00 x1625137115230832/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551270941 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 04:35:42 fir-io1-s1 kernel: LustreError: 96404:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.18.31@o2ib6) returned error from glimpse AST (req@ffff985389af2a00 x1625137115231072 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff985379a93f00/0x49e186021b94e82d lrc: 10/0,0 mode: PW/PW res: [0x5c0000402:0xd074c:0x0].0x0 rrc: 10 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x149478fe85143db3 expref: 6 pid: 96268 timeout: 0 lvb_type: 0 Feb 27 04:35:42 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.18.31@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 27 04:35:42 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Feb 27 04:35:42 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551270941s: evicting client at 10.8.18.31@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff985379a93f00/0x49e186021b94e82d lrc: 10/0,0 mode: PW/PW res: [0x5c0000402:0xd074c:0x0].0x0 rrc: 10 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000080020000 nid: 10.8.18.31@o2ib6 remote: 0x149478fe85143db3 expref: 7 pid: 96268 timeout: 0 lvb_type: 0 Feb 27 04:35:42 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Feb 27 04:35:42 fir-io1-s1 kernel: Lustre: 96265:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Feb 27 04:35:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ad459c81-c80e-33d1-0264-0f3efe91204c (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811ff800, cur 1551270957 expire 1551270807 last 1551270730 Feb 27 04:35:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 04:35:59 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ad459c81-c80e-33d1-0264-0f3efe91204c (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762492000, cur 1551270959 expire 1551270809 last 1551270732 Feb 27 04:36:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 04:36:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 04:46:36 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 04:51:49 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:02:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 13efc053-bc5b-08f3-50f2-203848f33f8f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839350800, cur 1551272536 expire 1551272386 last 1551272309 Feb 27 05:03:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 05:03:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 05:17:16 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:17:34 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:20:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f7a0d0aa-251a-211d-7350-340df04c351e (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832bddc00, cur 1551273636 expire 1551273486 last 1551273409 Feb 27 05:20:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 05:23:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 05:23:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 05:25:46 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:34:52 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:36:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 423c99a8-9686-5e73-248a-69577cd9075f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480f61a000, cur 1551274599 expire 1551274449 last 1551274372 Feb 27 05:36:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 05:37:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 05:37:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 05:49:26 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:54:02 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:55:00 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 05:56:29 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 63d9bf75-d764-a9a0-017c-96ce55afe613 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c21000, cur 1551275789 expire 1551275639 last 1551275562 Feb 27 05:56:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 05:56:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 63d9bf75-d764-a9a0-017c-96ce55afe613 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4a400, cur 1551275799 expire 1551275649 last 1551275572 Feb 27 06:00:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 06:00:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:03:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 06:03:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:04:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1e55a8fe-1155-1620-97fd-4f31c961742b (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e14800, cur 1551276275 expire 1551276125 last 1551276048 Feb 27 06:04:35 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 06:05:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f6d62d24-c3c1-00e7-ae80-d1625c77a42d (at 10.8.20.15@o2ib6) in 175 seconds. I think it's dead, and I am evicting it. exp ffff9848801aac00, cur 1551276351 expire 1551276201 last 1551276176 Feb 27 06:05:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:08:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 07f0334b-24cb-aa64-e0c3-2e0fcb7b097e (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c27400, cur 1551276515 expire 1551276365 last 1551276288 Feb 27 06:08:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:08:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 06:08:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:09:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 06:09:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:13:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 24ed82db-57d1-547c-afe4-1faa9342eb93 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4cc00, cur 1551276816 expire 1551276666 last 1551276589 Feb 27 06:13:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:14:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 06:14:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:15:40 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 06:24:32 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 06:40:21 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 56402132-371a-7044-687a-ecd11f531e92 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0e800, cur 1551278421 expire 1551278271 last 1551278194 Feb 27 06:40:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:41:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f1a67df7-3c12-179a-1bf4-2020df1bb713 (at 10.8.20.15@o2ib6) in 195 seconds. I think it's dead, and I am evicting it. exp ffff986781f77000, cur 1551278497 expire 1551278347 last 1551278302 Feb 27 06:41:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:41:50 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 06:42:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 06:42:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:50:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 06:50:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:53:20 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 06:55:54 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client aea7f273-f3cc-2f08-48e6-ad9397da92d1 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762490000, cur 1551279354 expire 1551279204 last 1551279127 Feb 27 06:55:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 06:56:27 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 07:04:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 07:04:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:07:03 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 07:08:27 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 07:09:43 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 07:13:17 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 07:18:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 47b4e7e9-2ee3-7739-8fac-75d29b35e829 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596be000, cur 1551280690 expire 1551280540 last 1551280463 Feb 27 07:18:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:21:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 07:21:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:23:15 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fc373204-9bff-b372-8d85-a510d8cb4d1f (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f474400, cur 1551280995 expire 1551280845 last 1551280768 Feb 27 07:23:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:23:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 27 07:23:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:31:22 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 07:45:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 6f42da70-96b2-72ae-fc5d-e8c7c4539170 (at 10.8.18.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986782308c00, cur 1551282310 expire 1551282160 last 1551282083 Feb 27 07:45:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:47:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 07:47:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:52:45 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f0948fad-7908-c5b2-cb9b-405f1cbdf1c0 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596be000, cur 1551282765 expire 1551282615 last 1551282538 Feb 27 07:52:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:53:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 27 07:53:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 07:54:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 07:54:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 08:01:27 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 08:06:46 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 08:09:15 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2c2890c3-b76f-2dc9-03f1-91b8e0c7ca35 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4c400, cur 1551283755 expire 1551283605 last 1551283528 Feb 27 08:09:15 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 08:09:25 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2c2890c3-b76f-2dc9-03f1-91b8e0c7ca35 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7ef000, cur 1551283765 expire 1551283615 last 1551283538 Feb 27 08:09:25 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 27 08:09:38 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2c2890c3-b76f-2dc9-03f1-91b8e0c7ca35 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583baf2400, cur 1551283778 expire 1551283628 last 1551283551 Feb 27 08:09:45 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2c2890c3-b76f-2dc9-03f1-91b8e0c7ca35 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a2fc00, cur 1551283785 expire 1551283635 last 1551283558 Feb 27 08:12:27 fir-io1-s1 kernel: Lustre: 96241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551283940/real 1551283940] req@ffff98382a5e9b00 x1625153256225200/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551283947 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 27 08:12:27 fir-io1-s1 kernel: Lustre: 96241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Feb 27 08:12:34 fir-io1-s1 kernel: Lustre: 96280:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551283947/real 1551283947] req@ffff983ed49dd400 x1625153256225104/t0(0) o106->fir-OST000a@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551283954 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 08:12:34 fir-io1-s1 kernel: Lustre: 96280:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 27 08:12:41 fir-io1-s1 kernel: Lustre: 96765:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551283954/real 1551283954] req@ffff983866ad2700 x1625153256225312/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551283961 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 08:12:41 fir-io1-s1 kernel: Lustre: 96765:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 27 08:12:55 fir-io1-s1 kernel: Lustre: 96265:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551283968/real 1551283968] req@ffff98420e990c00 x1625153256225248/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551283975 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 08:12:55 fir-io1-s1 kernel: Lustre: 96265:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 27 08:13:16 fir-io1-s1 kernel: Lustre: 96241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551283989/real 1551283989] req@ffff98382a5e9b00 x1625153256225200/t0(0) o106->fir-OST0002@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551283996 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 08:13:16 fir-io1-s1 kernel: Lustre: 96241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Feb 27 08:13:58 fir-io1-s1 kernel: Lustre: 96265:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551284031/real 1551284031] req@ffff98420e990c00 x1625153256225248/t0(0) o106->fir-OST0000@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551284038 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 08:13:58 fir-io1-s1 kernel: Lustre: 96265:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Feb 27 08:15:15 fir-io1-s1 kernel: Lustre: 96765:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551284108/real 1551284108] req@ffff983866ad2700 x1625153256225312/t0(0) o106->fir-OST0004@10.8.18.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551284115 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 08:15:15 fir-io1-s1 kernel: Lustre: 96765:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Feb 27 08:15:39 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d1ce9bb9-28c1-286a-36e5-917eec823e48 (at 10.8.18.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f231c00, cur 1551284139 expire 1551283989 last 1551283912 Feb 27 08:19:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa4cca22-658c-61f7-260a-40c14983e220 (at 10.8.18.31@o2ib6) Feb 27 08:19:38 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 27 08:19:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.18.35@o2ib6) Feb 27 08:19:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 08:20:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c657c695-de10-8808-d9fb-1a856e787dad (at 10.8.18.34@o2ib6) Feb 27 08:20:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 08:23:09 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 08:26:17 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 65607c0b-31d0-b85c-0cd9-98dee3bda486 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ef000, cur 1551284777 expire 1551284627 last 1551284550 Feb 27 08:26:17 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 08:28:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 08:28:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 08:35:54 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 09:04:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 86ba1f57-447a-970a-e35d-f9f3e10dbc28 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b2f1400, cur 1551287088 expire 1551286938 last 1551286861 Feb 27 09:04:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:06:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 09:06:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:16:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 09:16:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:16:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bd263e8b-14cd-e651-91d0-4af7bbc2a545 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801516000, cur 1551287813 expire 1551287663 last 1551287586 Feb 27 09:16:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:22:25 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 09:25:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 09:25:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:25:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 7d683273-d398-7147-ee05-936affa82945 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483b4e6400, cur 1551288358 expire 1551288208 last 1551288131 Feb 27 09:25:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:27:39 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 09:28:23 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 09:34:51 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3cff31fc-f52f-69eb-eab0-ac3dab3d2254 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c3d400, cur 1551288891 expire 1551288741 last 1551288664 Feb 27 09:34:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:36:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 09:36:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:40:11 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 09:43:36 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 09:46:31 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 55d97700-f972-3c79-f6f3-212281f5861e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984de97c8400, cur 1551289591 expire 1551289441 last 1551289364 Feb 27 09:46:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 09:47:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 09:47:34 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 27 09:56:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 09:56:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 09:56:32 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 09:56:49 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1a7c4a85-48b4-e0ef-6a00-6789f4a82b7f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d7dc00, cur 1551290209 expire 1551290059 last 1551289982 Feb 27 09:56:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:06:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2cff157b-a3ef-8f81-5463-c2f2cc4850b1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2cc00, cur 1551290772 expire 1551290622 last 1551290545 Feb 27 10:06:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:07:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 10:07:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:09:09 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 10:15:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 10:15:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:16:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 06760105-807c-acfa-970d-fe0f142109ab (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575be1e000, cur 1551291380 expire 1551291230 last 1551291153 Feb 27 10:16:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:24:54 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a00ef1bc-8c52-d936-6bb2-fe20514b99b9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c532800, cur 1551291894 expire 1551291744 last 1551291667 Feb 27 10:24:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:24:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 10:24:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:25:05 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a00ef1bc-8c52-d936-6bb2-fe20514b99b9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985573c61c00, cur 1551291905 expire 1551291755 last 1551291678 Feb 27 10:25:05 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 27 10:34:57 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a0a153a4-af25-0e81-ae8c-7970a5f64554 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfd400, cur 1551292497 expire 1551292347 last 1551292270 Feb 27 10:34:57 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 27 10:36:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 10:36:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:41:08 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 10:44:31 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4d4c1a55-6f3c-9a92-ad05-8c201d929d06 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a04a2800, cur 1551293071 expire 1551292921 last 1551292844 Feb 27 10:44:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:45:47 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b9d5ee48-d92d-0b8f-b217-c073d8cf4946 (at 10.9.103.2@o2ib4) in 195 seconds. I think it's dead, and I am evicting it. exp ffff98480c675000, cur 1551293147 expire 1551292997 last 1551292952 Feb 27 10:45:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:45:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 10:45:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:54:21 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 10:54:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 10:54:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 10:56:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 60653fc4-7600-7253-4897-510711012c5a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ea800, cur 1551293778 expire 1551293628 last 1551293551 Feb 27 10:56:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:00:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 11:00:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:02:14 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client cbccc8dc-d0d0-7a5d-19fa-5016ce74ce38 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ea000, cur 1551294134 expire 1551293984 last 1551293907 Feb 27 11:02:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:10:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 11:10:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:11:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9c6816c3-0271-77cd-ee52-d8b7eebd9d01 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f7eec00, cur 1551294694 expire 1551294544 last 1551294467 Feb 27 11:11:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:18:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to b9d5ee48-d92d-0b8f-b217-c073d8cf4946 (at 10.9.103.2@o2ib4) Feb 27 11:18:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:19:16 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 11:25:08 fir-io1-s1 kernel: Lustre: 96913:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295501/real 1551295501] req@ffff98556d2e6600 x1625166831584784/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295508 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 27 11:25:08 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295501/real 1551295501] req@ffff98382c03a100 x1625166831584704/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295508 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 27 11:25:08 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 27 11:25:08 fir-io1-s1 kernel: Lustre: 96913:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 27 11:25:15 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295508/real 1551295508] req@ffff98382c03a100 x1625166831584704/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295515 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 11:25:15 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295508/real 1551295508] req@ffff984c1d08a100 x1625166831584848/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295515 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 11:25:15 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Feb 27 11:25:22 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295515/real 1551295515] req@ffff984c1d08a100 x1625166831584848/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295522 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 11:25:22 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Feb 27 11:25:36 fir-io1-s1 kernel: Lustre: 96354:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295529/real 1551295529] req@ffff98557f2f7500 x1625166831584912/t0(0) o106->fir-OST0002@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295536 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 11:25:36 fir-io1-s1 kernel: Lustre: 96354:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 27 11:25:57 fir-io1-s1 kernel: Lustre: 96354:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295550/real 1551295550] req@ffff98557f2f7500 x1625166831584912/t0(0) o106->fir-OST0002@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295557 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 11:25:57 fir-io1-s1 kernel: Lustre: 96354:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 27 11:26:39 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295592/real 1551295592] req@ffff984c1d08a100 x1625166831584848/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295599 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 11:26:39 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Feb 27 11:26:47 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 11:27:56 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551295669/real 1551295669] req@ffff98382c03a100 x1625166831584704/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551295676 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 11:27:56 fir-io1-s1 kernel: Lustre: 96360:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 87 previous similar messages Feb 27 11:28:07 fir-io1-s1 kernel: LustreError: 96913:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff98556d2e6600 x1625166831584784 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff985288d6bf00/0x49e18605f57ae257 lrc: 4/0,0 mode: PW/PW res: [0xc80000402:0x1d6cac:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0xf5d4ae45d8ad6393 expref: 8 pid: 96357 timeout: 0 lvb_type: 0 Feb 27 11:28:07 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 27 11:28:07 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551295687s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff984cbd598900/0x49e18605f57adf63 lrc: 4/0,0 mode: PW/PW res: [0xc40000402:0x1d6cfc:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0xf5d4ae45d8ad635b expref: 8 pid: 96928 timeout: 0 lvb_type: 0 Feb 27 11:28:07 fir-io1-s1 kernel: LustreError: 96913:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages Feb 27 11:28:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 11:28:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:28:40 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 084655ec-d855-a8eb-cec8-de4cebea897b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b743800, cur 1551295720 expire 1551295570 last 1551295493 Feb 27 11:28:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:29:00 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 11:37:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 11:37:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:45:40 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a5975f23-557d-1956-7ad0-48974842cd06 (at 10.8.9.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b5d685000, cur 1551296740 expire 1551296590 last 1551296513 Feb 27 11:45:40 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 27 11:46:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 11:46:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 11:55:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 52f3795b-786e-c941-f1fe-da136d13043c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98663589fc00, cur 1551297348 expire 1551297198 last 1551297121 Feb 27 11:55:48 fir-io1-s1 kernel: Lustre: Skipped 13 previous similar messages Feb 27 11:56:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 11:56:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 12:04:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:04:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 12:11:22 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8bff2243-5d47-aa0d-ee22-f6395580006e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581bfea800, cur 1551298282 expire 1551298132 last 1551298055 Feb 27 12:11:22 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Feb 27 12:13:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:13:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 12:22:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:22:00 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 12:22:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2aa6ece8-715d-20bc-af20-9a246b513f57 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8d800, cur 1551298962 expire 1551298812 last 1551298735 Feb 27 12:22:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 12:29:02 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 12:29:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:29:30 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 27 12:29:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4882d594-a454-41bf-1d8d-94b894128777 (at 10.9.102.7@o2ib4) Feb 27 12:29:47 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 12:36:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:36:22 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 27 12:37:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 81a79a66-9742-198c-a78e-d8b1866bc7f7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d64d400, cur 1551299824 expire 1551299674 last 1551299597 Feb 27 12:37:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 12:43:49 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:43:49 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 27 12:51:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:51:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 12:51:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 00bb6ae3-a9d5-86ec-cfe9-e83fc2e636f8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483a2e9800, cur 1551300708 expire 1551300558 last 1551300481 Feb 27 12:51:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 12:59:35 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 12:59:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 12:59:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 13:10:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 13:10:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 13:10:21 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 45f5bc21-d8f3-47fb-bbc1-d662f914bf50 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7ef000, cur 1551301821 expire 1551301671 last 1551301594 Feb 27 13:10:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 13:15:54 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 13:20:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 13:20:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 13:20:56 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6a7a623b-69df-2d9f-ef46-e123ab7ce14e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a0f400, cur 1551302456 expire 1551302306 last 1551302229 Feb 27 13:20:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 13:25:13 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 13:31:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 13:31:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 13:31:26 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 17f79f6d-3412-2976-c13a-45dd639ec025 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98546cff2800, cur 1551303086 expire 1551302936 last 1551302859 Feb 27 13:31:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 13:35:42 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 13:38:49 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 13:55:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 13:55:11 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 13:55:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a987286a-8a27-7402-0819-31f421b57640 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98546cff0400, cur 1551304551 expire 1551304401 last 1551304324 Feb 27 13:55:51 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 14:08:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 14:08:40 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 14:09:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ce9ed73a-7ee8-4c7f-fd68-05dc5ca1f1ed (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582aa2b400, cur 1551305342 expire 1551305192 last 1551305115 Feb 27 14:09:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 14:24:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7e4f58d8-6e8c-32f9-59fb-7de9af0df4b4 (at 10.9.106.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df2c00, cur 1551306267 expire 1551306117 last 1551306040 Feb 27 14:24:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 14:31:43 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 14:47:40 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 14:53:37 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 14:55:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7e4f58d8-6e8c-32f9-59fb-7de9af0df4b4 (at 10.9.106.12@o2ib4) Feb 27 14:55:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 15:30:42 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 15:37:58 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 68521de7-1794-c75f-f6e3-1d6477a534d1 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480052a800, cur 1551310678 expire 1551310528 last 1551310451 Feb 27 15:37:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 15:40:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 27 15:40:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 15:49:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3d8a67f4-2492-af32-af26-1fb7013841d0 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868d7400, cur 1551311382 expire 1551311232 last 1551311155 Feb 27 15:49:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 15:50:58 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client aa26fc7d-8d56-d044-df07-735cb4ad01f5 (at 10.8.20.15@o2ib6) in 151 seconds. I think it's dead, and I am evicting it. exp ffff986784bb8800, cur 1551311458 expire 1551311308 last 1551311307 Feb 27 15:50:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 15:51:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 15:51:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 15:55:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 27 15:55:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 16:22:59 fir-io1-s1 kernel: Lustre: 96925:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551313372/real 1551313372] req@ffff984e82e76c00 x1625184419952496/t0(0) o106->fir-OST0002@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551313379 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 27 16:22:59 fir-io1-s1 kernel: Lustre: 96925:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Feb 27 16:23:20 fir-io1-s1 kernel: Lustre: 96406:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551313393/real 1551313393] req@ffff9878056fe900 x1625184419952608/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551313400 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 16:23:20 fir-io1-s1 kernel: Lustre: 96406:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 27 16:24:02 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551313435/real 1551313435] req@ffff986a38444200 x1625184419952416/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551313442 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 27 16:24:02 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Feb 27 16:24:28 fir-io1-s1 kernel: LustreError: 96620:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff986a38444200 x1625184419952416 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff984ffb065340/0x49e18607f5015c34 lrc: 3/0,0 mode: PW/PW res: [0x580000400:0x1d8cc8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0x649cfea256792abc expref: 15 pid: 96900 timeout: 0 lvb_type: 0 Feb 27 16:24:28 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Feb 27 16:24:28 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Feb 27 16:24:28 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551313468s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff98570c0f5e80/0x49e18607f501604e lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x1d8c15:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0x649cfea256792b2c expref: 13 pid: 96897 timeout: 0 lvb_type: 0 Feb 27 16:24:28 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Feb 27 16:24:28 fir-io1-s1 kernel: LustreError: 96620:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Feb 27 16:24:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 16:24:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 16:27:25 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 92360cf1-e21f-b2b4-ad08-6e3a83ed41ed (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c3e800, cur 1551313645 expire 1551313495 last 1551313418 Feb 27 16:27:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 16:30:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Feb 27 16:30:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 17:33:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 17:33:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 17:33:57 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 75075323-e215-772e-2f5a-cb5ed5087a2d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da2c00, cur 1551317637 expire 1551317487 last 1551317410 Feb 27 17:33:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 17:58:01 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a2ccc640-0142-52e3-e3a8-9859992ac8f9 (at 10.9.102.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581bfe8c00, cur 1551319081 expire 1551318931 last 1551318854 Feb 27 17:58:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 18:02:32 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0adc7979-82ee-6805-7daa-e6743e532923 (at 10.8.6.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786921000, cur 1551319352 expire 1551319202 last 1551319125 Feb 27 18:02:32 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Feb 27 18:09:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0098b3ca-9c64-c96b-0dc4-0d4b5b3a3268 (at 10.8.4.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8abc00, cur 1551319741 expire 1551319591 last 1551319514 Feb 27 18:09:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 18:14:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 586a6e8e-70fe-af28-3ec3-56d6983d8923 (at 10.8.9.6@o2ib6) Feb 27 18:14:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 18:19:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 91f82685-8c0d-ca47-2535-2fe64ad08eab (at 10.9.107.34@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b915c00, cur 1551320352 expire 1551320202 last 1551320125 Feb 27 18:19:12 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Feb 27 18:20:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fc738f4c-fd89-da60-f357-76c857400e3c (at 10.9.115.6@o2ib4) Feb 27 18:20:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 18:26:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3c603ef7-b311-5012-38a9-d1ff9ba9b526 (at 10.9.104.13@o2ib4) Feb 27 18:26:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 18:26:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 03d523d0-d33d-b8d0-52c8-1ff235ea28e5 (at 10.9.102.16@o2ib4) Feb 27 18:26:32 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Feb 27 18:26:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 44ee34be-64b9-b529-f92c-8f8496b513f8 (at 10.9.104.16@o2ib4) Feb 27 18:26:45 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 27 18:27:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 58531951-dcfc-2dad-91c4-688aefd85811 (at 10.9.104.5@o2ib4) Feb 27 18:27:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 18:27:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2513721a-1a7a-beeb-ce96-babfef130551 (at 10.8.18.32@o2ib6) Feb 27 18:27:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 18:28:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to daa4a129-0b83-9695-ea5e-c26cf889acfd (at 10.9.104.14@o2ib4) Feb 27 18:28:01 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 27 18:32:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 400e2c70-3670-eb05-66c0-e754ea5cd280 (at 10.8.29.7@o2ib6) Feb 27 18:32:52 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 27 18:37:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72ef71e7-ecde-a7f2-d85d-452a22011f5b (at 10.9.101.14@o2ib4) Feb 27 18:37:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 19:00:12 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 02c7b380-50a1-0090-f254-abedb891ba75 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867850b7000, cur 1551322812 expire 1551322662 last 1551322585 Feb 27 19:00:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 19:00:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 27 19:00:32 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Feb 27 21:03:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client eb2d653c-5f78-e201-f039-efda162bfa3d (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d1dc40800, cur 1551330231 expire 1551330081 last 1551330004 Feb 27 21:03:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 21:04:09 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client eb2d653c-5f78-e201-f039-efda162bfa3d (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a9d000, cur 1551330249 expire 1551330099 last 1551330022 Feb 27 21:04:09 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 27 21:09:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 27 21:09:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 21:16:31 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0ee32720-6faf-1dc2-9639-7c8c0223fc6f (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f7000, cur 1551330991 expire 1551330841 last 1551330764 Feb 27 21:16:31 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 27 21:25:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 27 21:25:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 21:39:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d1755442-892e-a805-5fa2-c61746c310b0 (at 10.9.113.7@o2ib4) Feb 27 21:39:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 21:56:12 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9211fe35-0c9d-5fa3-d53b-1793ba0106aa (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780863400, cur 1551333372 expire 1551333222 last 1551333145 Feb 27 21:56:12 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 27 21:58:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 27 21:58:40 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 27 22:51:18 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0a1d480b-55e3-0f6f-5c5a-b5feda8b310f (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983851c28c00, cur 1551336678 expire 1551336528 last 1551336451 Feb 27 22:51:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 22:51:27 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0a1d480b-55e3-0f6f-5c5a-b5feda8b310f (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b264400, cur 1551336687 expire 1551336537 last 1551336460 Feb 27 22:51:36 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0a1d480b-55e3-0f6f-5c5a-b5feda8b310f (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983851c2d800, cur 1551336696 expire 1551336546 last 1551336469 Feb 27 22:56:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 27 22:56:26 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 27 23:03:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ef6e940b-980e-1ede-0e9d-fe0e36937f1c (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c676400, cur 1551337390 expire 1551337240 last 1551337163 Feb 27 23:03:10 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 27 23:04:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 27 23:04:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 23:06:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 27 23:06:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 23:07:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 789dd17b-09a7-4efa-2265-a78b0f7e8a03 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c21400, cur 1551337628 expire 1551337478 last 1551337401 Feb 27 23:07:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 23:14:42 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 27 23:47:31 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 69b34652-c357-7e9a-4564-7fad9c281069 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834454c00, cur 1551340051 expire 1551339901 last 1551339824 Feb 27 23:47:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 27 23:49:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 27 23:49:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 00:35:59 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client fd0ec97a-cf23-ffd3-7938-958f490cce96 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed03400, cur 1551342959 expire 1551342809 last 1551342732 Feb 28 00:35:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 00:38:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 28 00:38:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 00:43:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 00:43:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 00:43:20 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8aef762d-9b56-cd87-8e3a-ac3fe7c0c7e4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c20800, cur 1551343400 expire 1551343250 last 1551343173 Feb 28 00:43:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 00:46:38 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a83c90fa-9af4-f8a9-25e0-2ee34d4d33e5 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8265c00, cur 1551343598 expire 1551343448 last 1551343371 Feb 28 00:46:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 00:47:13 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Feb 28 00:49:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 28 00:49:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 01:04:26 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a415e366-7aa0-e275-36c5-cc6ca510eea6 (at 10.9.113.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985757bac400, cur 1551344666 expire 1551344516 last 1551344439 Feb 28 01:04:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 01:25:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ebee093c-8e69-4e39-2895-94a502f715cc (at 10.9.113.6@o2ib4) Feb 28 01:25:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 04:00:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to eb4e1d20-3a1f-d68d-546e-5c1cf1ecb74b (at 10.9.104.11@o2ib4) Feb 28 04:00:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 04:00:03 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8b467f67-701e-6281-4801-dec7e28f7c79 (at 10.9.104.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838a9400, cur 1551355203 expire 1551355053 last 1551354976 Feb 28 04:00:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 04:00:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8b467f67-701e-6281-4801-dec7e28f7c79 (at 10.9.104.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838a9c00, cur 1551355205 expire 1551355055 last 1551354978 Feb 28 05:30:20 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3879140b-0065-9855-1189-63f86dc8c822 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678230d400, cur 1551360620 expire 1551360470 last 1551360393 Feb 28 05:30:20 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 05:32:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 28 05:32:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 05:42:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c038235a-cde9-3cf1-6cb0-faf4262482f4 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867860bc400, cur 1551361325 expire 1551361175 last 1551361098 Feb 28 05:42:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 05:44:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 28 05:44:51 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 28 06:08:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 06:08:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 06:08:14 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1c2ad138-1eff-d341-1c94-9fdd6e306d9b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a059ec00, cur 1551362894 expire 1551362744 last 1551362667 Feb 28 06:08:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 06:15:12 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4c04773e-96c1-f080-44fd-628a8575903e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867f820b000, cur 1551363312 expire 1551363162 last 1551363085 Feb 28 06:15:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 06:16:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 06:16:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 06:23:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 06:23:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 06:23:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ac4380c7-7e83-5a11-7c23-071937dd99e4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c6e800, cur 1551363829 expire 1551363679 last 1551363602 Feb 28 06:23:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 06:48:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 06:48:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 06:48:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 22136029-f05d-65f2-d000-c90b48a2f16a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8a9400, cur 1551365320 expire 1551365170 last 1551365093 Feb 28 06:48:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:03:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 07:03:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:04:31 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7b8d0b0b-ae9b-5ddd-4c6c-79308e8c2b7a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855125db000, cur 1551366271 expire 1551366121 last 1551366044 Feb 28 07:04:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:10:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 07:10:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:10:43 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a6a8a0bd-48a0-c61a-b31f-ff9d3b3455d3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f4400, cur 1551366643 expire 1551366493 last 1551366416 Feb 28 07:10:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:16:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 07:16:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:17:27 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client bd5ffe7d-13d0-5670-bb96-f3c034eb281f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677dc5b800, cur 1551367047 expire 1551366897 last 1551366820 Feb 28 07:17:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:22:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 07:22:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:23:26 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9914b636-7eb4-b7d7-6978-3fbe68e49ac4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfc000, cur 1551367406 expire 1551367256 last 1551367179 Feb 28 07:23:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:28:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 07:28:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:29:27 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client aefa195c-5342-2e13-5254-dbf8c9b8e6bb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a9f800, cur 1551367767 expire 1551367617 last 1551367540 Feb 28 07:29:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:34:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 07:34:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 07:35:19 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b8d4d3d3-3e19-fe04-f9aa-a9e5c3d44aca (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e12800, cur 1551368119 expire 1551367969 last 1551367892 Feb 28 07:35:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 08:37:17 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 384564b1-f4b2-b616-630e-bb7cb3901fc3 (at 10.8.13.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576998a000, cur 1551371837 expire 1551371687 last 1551371610 Feb 28 08:37:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 08:52:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Feb 28 08:52:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 08:52:32 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b88b5f9c-341a-6922-1f2d-90bc070377cb (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed07800, cur 1551372752 expire 1551372602 last 1551372525 Feb 28 08:52:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 08:52:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b88b5f9c-341a-6922-1f2d-90bc070377cb (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984996393800, cur 1551372760 expire 1551372610 last 1551372533 Feb 28 09:56:46 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b72d6051-cdad-cb35-92ee-c51c9ba641cc (at 10.9.104.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858356cc000, cur 1551376606 expire 1551376456 last 1551376379 Feb 28 09:56:46 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 10:25:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 69e04fcf-27a0-cb59-92a0-ef1d06a212ef (at 10.9.104.6@o2ib4) Feb 28 10:25:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 10:25:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.104.4@o2ib4) Feb 28 10:25:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 10:25:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 54ebdcb6-100b-be98-80b4-2b807a6412a8 (at 10.9.104.10@o2ib4) Feb 28 10:25:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 10:25:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8b819495-2088-ce05-abe6-a051f7fc0b48 (at 10.9.104.7@o2ib4) Feb 28 10:25:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 10:25:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4bd8642d-f0ee-a784-85c9-bd31540eadc6 (at 10.9.104.12@o2ib4) Feb 28 10:25:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 10:25:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8d3900cc-5d6e-76ce-7239-094cf5a8f78d (at 10.9.104.8@o2ib4) Feb 28 10:25:58 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 28 10:29:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2d486816-0fee-3e02-cb25-89549e06c293 (at 10.9.104.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df0400, cur 1551378555 expire 1551378405 last 1551378328 Feb 28 10:29:15 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Feb 28 10:30:31 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8d1e8505-6f8c-2fe3-e4b1-293ce8c0c09a (at 10.9.106.61@o2ib4) in 223 seconds. I think it's dead, and I am evicting it. exp ffff9877a1464800, cur 1551378631 expire 1551378481 last 1551378408 Feb 28 10:30:31 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Feb 28 10:53:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.106.61@o2ib4) Feb 28 10:53:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 11:00:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aaf0a12e-0503-9b8d-5b98-b850f09a0ee4 (at 10.9.103.28@o2ib4) Feb 28 11:00:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 11:00:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2d486816-0fee-3e02-cb25-89549e06c293 (at 10.9.104.52@o2ib4) Feb 28 11:00:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 11:02:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7030631e-b3d2-9eed-f765-9117cb5ba8a4 (at 10.9.103.35@o2ib4) Feb 28 11:02:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 11:44:42 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3e1bfda9-cfe9-bdcb-ad30-df8e6dbb01f9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef10000, cur 1551383082 expire 1551382932 last 1551382855 Feb 28 11:44:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 11:48:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 11:48:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 11:52:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Feb 28 11:52:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 11:52:39 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3e426987-a264-d1c6-f8da-8ba52e379c08 (at 10.8.27.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c005000, cur 1551383559 expire 1551383409 last 1551383332 Feb 28 11:52:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:02:36 fir-io1-s1 kernel: Lustre: 96369:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551384149/real 1551384149] req@ffff986e7e5b2a00 x1625206463235168/t0(0) o104->fir-OST0002@10.9.108.64@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1551384156 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 28 12:02:36 fir-io1-s1 kernel: Lustre: 96369:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 28 12:02:50 fir-io1-s1 kernel: Lustre: 96369:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551384163/real 1551384163] req@ffff986e7e5b2a00 x1625206463235168/t0(0) o104->fir-OST0002@10.9.108.64@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1551384170 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 12:02:50 fir-io1-s1 kernel: Lustre: 96369:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 28 12:03:11 fir-io1-s1 kernel: Lustre: 96369:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551384184/real 1551384184] req@ffff986e7e5b2a00 x1625206463235168/t0(0) o104->fir-OST0002@10.9.108.64@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1551384191 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 12:03:11 fir-io1-s1 kernel: Lustre: 96369:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Feb 28 12:03:31 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ba234911-94f9-0f5d-7d99-92ccf7afe7e2 (at 10.9.106.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a1c00, cur 1551384211 expire 1551384061 last 1551383984 Feb 28 12:03:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:03:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ba234911-94f9-0f5d-7d99-92ccf7afe7e2 (at 10.9.106.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cdb800, cur 1551384220 expire 1551384070 last 1551383993 Feb 28 12:03:40 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 28 12:03:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ce4cb33a-923a-9cf9-f3a5-aa684c40c733 (at 10.8.29.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987446a3ac00, cur 1551384231 expire 1551384081 last 1551384004 Feb 28 12:03:51 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Feb 28 12:25:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6c3169b7-2563-1f32-7d22-6e3f2ce9c349 (at 10.9.107.66@o2ib4) Feb 28 12:25:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:26:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b3fc52f5-cc19-f1e2-5d13-43190203fae8 (at 10.9.106.22@o2ib4) Feb 28 12:26:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:26:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0343f8c1-f803-943e-238c-e83a0eb1a3ba (at 10.9.106.34@o2ib4) Feb 28 12:26:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:26:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2acb2116-5227-530a-f563-866a3449ba51 (at 10.9.106.13@o2ib4) Feb 28 12:26:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 28 12:26:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to beae83b1-f0c3-07d1-3569-d911d8777da5 (at 10.9.106.55@o2ib4) Feb 28 12:26:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:27:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 18be1149-584e-3095-4e7f-669b8a4c97d2 (at 10.8.29.6@o2ib6) Feb 28 12:27:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:28:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 73b18771-956a-ea9b-2a92-9f6c32b3eb53 (at 10.9.108.64@o2ib4) Feb 28 12:28:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 12:35:27 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551386120/real 1551386120] req@ffff986e7e5b3f00 x1625206753020080/t0(0) o104->fir-OST0002@10.9.108.68@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1551386127 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 28 12:35:27 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Feb 28 12:35:34 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551386127/real 1551386127] req@ffff986e7e5b3f00 x1625206753020080/t0(0) o104->fir-OST0002@10.9.108.68@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1551386134 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 12:35:34 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Feb 28 12:35:48 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551386141/real 1551386141] req@ffff986e7e5b3f00 x1625206753020080/t0(0) o104->fir-OST0002@10.9.108.68@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1551386148 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 12:35:48 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 28 12:36:04 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b1b05ab1-8429-7c9b-e524-6bc08a09b8fe (at 10.9.106.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481df05000, cur 1551386164 expire 1551386014 last 1551385937 Feb 28 12:36:04 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Feb 28 12:36:09 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551386162/real 1551386162] req@ffff986e7e5b3f00 x1625206753020080/t0(0) o104->fir-OST0002@10.9.108.68@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1551386169 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 12:36:09 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Feb 28 12:36:40 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client fafc7fbb-b42e-4208-974b-f946c8553275 (at 10.9.107.71@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e7da2c800, cur 1551386200 expire 1551386050 last 1551385973 Feb 28 12:36:40 fir-io1-s1 kernel: Lustre: Skipped 87 previous similar messages Feb 28 12:58:15 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 4537de6a-1feb-d906-bd43-fc3e4f1c0915 (at 10.9.108.4@o2ib4) Feb 28 12:58:15 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 4537de6a-1feb-d906-bd43-fc3e4f1c0915 (at 10.9.108.4@o2ib4) Feb 28 12:58:15 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 28 12:58:15 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 28 12:58:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f6a811d2-f77e-6c94-690d-cc60be6676e4 (at 10.9.108.7@o2ib4) Feb 28 12:58:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 13:00:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to feac05a4-716f-c34a-fd9d-1220a521af0c (at 10.9.107.69@o2ib4) Feb 28 13:00:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 13:01:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 20609818-b83c-bf65-0dd2-090d3c6e2314 (at 10.9.108.2@o2ib4) Feb 28 13:01:22 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 28 13:02:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 91f6fe25-4c34-a621-dd26-00e6ccf4cbba (at 10.9.106.32@o2ib4) Feb 28 13:02:15 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 28 13:03:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce2a5a1a-545c-760b-44a7-8c19aadb7a36 (at 10.9.107.71@o2ib4) Feb 28 13:03:21 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Feb 28 13:05:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 479d2c4f-434d-5613-7a77-8c5939e93218 (at 10.9.108.63@o2ib4) Feb 28 13:05:44 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Feb 28 13:10:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fdeab83c-a7a1-7e5b-a1b6-bc622d510300 (at 10.9.108.62@o2ib4) Feb 28 13:10:16 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Feb 28 14:25:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 14:25:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 14:47:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a73a38e8-7a0c-0288-aa98-7ef6ed07112b (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d504800, cur 1551394028 expire 1551393878 last 1551393801 Feb 28 14:47:08 fir-io1-s1 kernel: Lustre: Skipped 175 previous similar messages Feb 28 14:49:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 14:49:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 15:18:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ab6423f5-6890-6c98-d786-25703ca7a051 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762f5d000, cur 1551395937 expire 1551395787 last 1551395710 Feb 28 15:18:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 15:47:56 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 28 15:47:56 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Feb 28 15:48:47 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Feb 28 15:48:47 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Feb 28 15:48:47 fir-io1-s1 kernel: Lustre: 91454:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1551397726/real 1551397727] req@ffff985a51d63f00 x1625209701600592/t0(0) o400->fir-MDT0003-lwp-OST0002@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1551398482 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 28 15:48:47 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0002: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 28 15:48:47 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 28 15:48:47 fir-io1-s1 kernel: Lustre: 91454:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 38 previous similar messages Feb 28 15:48:50 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 3 seconds Feb 28 15:48:50 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 8 previous similar messages Feb 28 15:48:52 fir-io1-s1 kernel: Lustre: 91460:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1551397726/real 1551397732] req@ffff985a51d65100 x1625209701600448/t0(0) o400->fir-MDT0001-lwp-OST000a@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1551398482 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Feb 28 15:48:52 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0000: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Feb 28 15:48:52 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 28 15:48:52 fir-io1-s1 kernel: Lustre: 91460:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Feb 28 15:48:53 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Feb 28 15:48:53 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Feb 28 15:48:59 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds Feb 28 15:48:59 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Feb 28 15:49:02 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Feb 28 15:49:02 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 28 15:49:24 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 28 15:49:24 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (6): c: 0, oc: 0, rc: 8 Feb 28 15:49:43 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST000a: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Feb 28 15:49:43 fir-io1-s1 kernel: LustreError: Skipped 17 previous similar messages Feb 28 15:49:43 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST000a: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Feb 28 15:49:43 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Feb 28 15:49:49 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Feb 28 15:49:49 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Feb 28 15:50:05 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Feb 28 15:50:33 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST0006: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Feb 28 15:50:33 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Feb 28 15:50:33 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0002: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Feb 28 15:50:33 fir-io1-s1 kernel: Lustre: Skipped 20 previous similar messages Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:1964203 to 0x6c0000400:1964257 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:1963981 to 0xc40000402:1964129 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:1964354 to 0x580000400:1964449 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:1963989 to 0x5c0000400:1964161 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:1963678 to 0x8c0000402:1963841 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:1963963 to 0xc80000402:1964129 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1172434 to 0x0:1172513 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1172149 to 0x0:1172225 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1171945 to 0x0:1172033 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1172621 to 0x0:1172705 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1171991 to 0x0:1172033 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1171762 to 0x0:1171809 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:881071 to 0x6c0000402:881089 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:881061 to 0xc80000401:881089 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:880928 to 0xc40000401:880993 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:881179 to 0x8c0000401:881217 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:882345 to 0x580000402:882433 Feb 28 15:50:56 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:882462 to 0x5c0000402:882497 Feb 28 15:50:57 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:661404 to 0x8c0000400:661569 Feb 28 15:50:57 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:661566 to 0x6c0000401:661729 Feb 28 15:50:57 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:661530 to 0xc40000400:661889 Feb 28 15:50:57 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:661880 to 0x5c0000401:661953 Feb 28 15:50:57 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:661953 to 0x580000401:662145 Feb 28 15:50:57 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:661654 to 0xc80000400:661793 Feb 28 15:56:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ef079e70-3cb8-ddc6-a4ca-72d0e87a1b53 (at 10.8.3.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857630f9400, cur 1551398213 expire 1551398063 last 1551397986 Feb 28 15:56:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 16:39:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 16:39:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 16:40:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d63a820c-5b1c-a1eb-ab43-eec5d17a84cf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387bec00, cur 1551400816 expire 1551400666 last 1551400589 Feb 28 16:40:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 16:40:31 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d63a820c-5b1c-a1eb-ab43-eec5d17a84cf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583baf4000, cur 1551400831 expire 1551400681 last 1551400604 Feb 28 16:40:31 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 16:50:25 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 051bc37a-e3f2-75cf-ab37-f2275a37417f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986816f3b400, cur 1551401425 expire 1551401275 last 1551401198 Feb 28 16:50:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 16:50:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:01:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 17:01:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:02:17 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 17a5636d-715a-9d16-f13e-89329c5d4dd1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783fd3c00, cur 1551402137 expire 1551401987 last 1551401910 Feb 28 17:02:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:29:22 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 735f64a7-57a4-0952-f522-322b28de1841 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b2f6800, cur 1551403762 expire 1551403612 last 1551403535 Feb 28 17:29:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:30:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 17:30:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:45:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 874c608b-d57a-70f2-6a7e-5e9af693e5b6 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009ff400, cur 1551404739 expire 1551404589 last 1551404512 Feb 28 17:45:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:46:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 17:46:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:55:25 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c310426d-b177-68da-514a-b41a9b14cac1 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983801b56c00, cur 1551405325 expire 1551405175 last 1551405098 Feb 28 17:55:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:56:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 17:56:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 17:57:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 17:57:17 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 28 18:02:15 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 219eae0c-281f-10af-ca3f-1a583864f51b (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ea800, cur 1551405735 expire 1551405585 last 1551405508 Feb 28 18:02:15 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 28 18:04:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Feb 28 18:04:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 18:05:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 28 18:05:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 18:15:42 fir-io1-s1 kernel: Lustre: 96258:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406535/real 1551406535] req@ffff98380de60300 x1625212219139040/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406542 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 28 18:15:42 fir-io1-s1 kernel: Lustre: 96258:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Feb 28 18:15:43 fir-io1-s1 kernel: Lustre: 96345:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406536/real 1551406536] req@ffff9871cb37ef00 x1625212219556272/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406543 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 28 18:15:43 fir-io1-s1 kernel: Lustre: 96345:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 28 18:15:47 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406540/real 1551406540] req@ffff9850282f0000 x1625212220071280/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406547 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Feb 28 18:15:47 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Feb 28 18:15:52 fir-io1-s1 kernel: Lustre: 111263:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406544/real 1551406544] req@ffff986d52d2ef00 x1625212219733264/t0(0) o106->fir-OST0000@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406551 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 18:15:52 fir-io1-s1 kernel: Lustre: 111263:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 73 previous similar messages Feb 28 18:16:01 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406554/real 1551406554] req@ffff9856012aa700 x1625212222441744/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406561 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 18:16:01 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 149 previous similar messages Feb 28 18:16:20 fir-io1-s1 kernel: Lustre: 36980:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406573/real 1551406573] req@ffff983820229b00 x1625212224483488/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406580 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 18:16:20 fir-io1-s1 kernel: Lustre: 36980:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 396 previous similar messages Feb 28 18:16:58 fir-io1-s1 kernel: Lustre: 96927:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406610/real 1551406610] req@ffff983824ea0300 x1625212225527424/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406617 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 18:16:58 fir-io1-s1 kernel: Lustre: 96927:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 781 previous similar messages Feb 28 18:17:22 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 55d5b00f-d291-c9bf-b408-e69a398fc734 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984d22d7e000, cur 1551406642 expire 1551406492 last 1551406415 Feb 28 18:17:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 28 18:17:31 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 55d5b00f-d291-c9bf-b408-e69a398fc734 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d5b400, cur 1551406651 expire 1551406501 last 1551406424 Feb 28 18:17:31 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 28 18:18:12 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551406685/real 1551406685] req@ffff9857f123a100 x1625212221797440/t0(0) o106->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551406692 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Feb 28 18:18:12 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1627 previous similar messages Feb 28 18:18:38 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 260e6d40-cd5d-3426-6f87-ac9149a2548c (at 10.8.3.11@o2ib6) in 184 seconds. I think it's dead, and I am evicting it. exp ffff98646d55d000, cur 1551406718 expire 1551406568 last 1551406534 Feb 28 18:18:38 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 28 18:18:47 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 260e6d40-cd5d-3426-6f87-ac9149a2548c (at 10.8.3.11@o2ib6) in 193 seconds. I think it's dead, and I am evicting it. exp ffff986785da2000, cur 1551406727 expire 1551406577 last 1551406534 Feb 28 18:18:47 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 28 18:18:55 fir-io1-s1 kernel: LNet: Service thread pid 96355 was inactive for 200.18s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 28 18:18:55 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Feb 28 18:18:55 fir-io1-s1 kernel: Pid: 96355, comm: ll_ost01_033 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 28 18:18:55 fir-io1-s1 kernel: Call Trace: Feb 28 18:18:55 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 28 18:18:55 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 28 18:18:55 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 28 18:18:55 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 28 18:18:56 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 28 18:18:56 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406736.96355 Feb 28 18:18:57 fir-io1-s1 kernel: LNet: Service thread pid 96752 was inactive for 200.49s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Feb 28 18:18:57 fir-io1-s1 kernel: Pid: 96752, comm: ll_ost02_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 28 18:18:57 fir-io1-s1 kernel: Call Trace: Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 28 18:18:57 fir-io1-s1 kernel: Pid: 74743, comm: ll_ost02_078 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 28 18:18:57 fir-io1-s1 kernel: Call Trace: Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 28 18:18:57 fir-io1-s1 kernel: Pid: 96345, comm: ll_ost01_028 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 28 18:18:57 fir-io1-s1 kernel: Call Trace: Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 28 18:18:57 fir-io1-s1 kernel: Pid: 96362, comm: ll_ost01_040 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Feb 28 18:18:57 fir-io1-s1 kernel: Call Trace: Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Feb 28 18:18:57 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Feb 28 18:18:57 fir-io1-s1 kernel: LNet: Service thread pid 96258 was inactive for 202.37s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 28 18:18:57 fir-io1-s1 kernel: LNet: Skipped 7 previous similar messages Feb 28 18:18:57 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406737.49816 Feb 28 18:19:00 fir-io1-s1 kernel: LNet: Service thread pid 96892 was inactive for 200.32s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 28 18:19:00 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Feb 28 18:19:00 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406740.96892 Feb 28 18:19:02 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406742.96776 Feb 28 18:19:02 fir-io1-s1 kernel: LNet: Service thread pid 2371 was inactive for 200.45s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 28 18:19:02 fir-io1-s1 kernel: LNet: Skipped 4 previous similar messages Feb 28 18:19:03 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406743.96568 Feb 28 18:19:04 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406744.96523 Feb 28 18:19:06 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406746.96912 Feb 28 18:19:07 fir-io1-s1 kernel: LNet: Service thread pid 96564 was inactive for 200.08s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 28 18:19:07 fir-io1-s1 kernel: LNet: Skipped 10 previous similar messages Feb 28 18:19:07 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406747.96564 Feb 28 18:19:08 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406748.110000 Feb 28 18:19:09 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406749.96889 Feb 28 18:19:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 18:19:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 18:19:10 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406750.74705 Feb 28 18:19:11 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406751.96891 Feb 28 18:19:12 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406752.36980 Feb 28 18:19:14 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406754.938 Feb 28 18:19:15 fir-io1-s1 kernel: LNet: Service thread pid 96533 was inactive for 200.42s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Feb 28 18:19:15 fir-io1-s1 kernel: LNet: Skipped 9 previous similar messages Feb 28 18:19:15 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406755.96533 Feb 28 18:19:16 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406756.96886 Feb 28 18:19:19 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551406759.96940 Feb 28 18:19:21 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 260e6d40-cd5d-3426-6f87-ac9149a2548c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d558800, cur 1551406761 expire 1551406611 last 1551406534 Feb 28 18:19:21 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 28 18:19:21 fir-io1-s1 kernel: LNet: Service thread pid 49820 completed after 216.84s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Feb 28 18:19:21 fir-io1-s1 kernel: LNet: Skipped 37 previous similar messages Feb 28 18:20:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 18:20:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 18:28:39 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5903b369-0b86-edb5-e071-aa9e61da6bb0 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a30400, cur 1551407319 expire 1551407169 last 1551407092 Feb 28 18:30:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 18:30:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 18:38:14 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 52aad3c7-c216-4118-15d4-ad02110417c1 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480041d800, cur 1551407894 expire 1551407744 last 1551407667 Feb 28 18:38:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 18:39:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 18:39:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 18:52:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 18:52:28 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Feb 28 18:53:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3942d1b2-563b-e42f-e0de-7fcc8cc012d8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582aa2dc00, cur 1551408784 expire 1551408634 last 1551408557 Feb 28 18:53:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:11:45 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f678c693-9d20-da73-5e2a-d31d400bf797 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984de97ce400, cur 1551409905 expire 1551409755 last 1551409678 Feb 28 19:11:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:13:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 19:13:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:17:47 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f0ec523a-6a30-fc32-d953-11222af5eaf8 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596bd800, cur 1551410267 expire 1551410117 last 1551410040 Feb 28 19:17:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:19:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 19:19:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:26:54 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9ca32e8c-7c45-a6c0-78a4-dc8c6e2c1a51 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f007800, cur 1551410814 expire 1551410664 last 1551410587 Feb 28 19:26:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:28:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ccfcf60f-8eb0-ebb3-6a3f-f435e4ba7f7c (at 10.8.9.8@o2ib6) in 190 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c5800, cur 1551410890 expire 1551410740 last 1551410700 Feb 28 19:28:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:28:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 19:28:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:28:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Feb 28 19:28:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:29:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bb21b8eb-d3e4-bcc3-59fa-a0c40d7e5f82 (at 10.8.20.15@o2ib6) in 203 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83b800, cur 1551410966 expire 1551410816 last 1551410763 Feb 28 19:29:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:30:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 19:30:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:42:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 19:42:47 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 19:42:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a6a623b0-1dd8-f191-8778-33a1a6c6bb3c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e48800, cur 1551411778 expire 1551411628 last 1551411551 Feb 28 19:42:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:45:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 19:45:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:50:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7e076cd8-79ff-f247-8a8e-a7a1a104badb (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575e61bc00, cur 1551412200 expire 1551412050 last 1551411973 Feb 28 19:50:00 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 28 19:51:16 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 45673e93-9c5f-906c-36ac-49dbaad1815f (at 10.8.3.11@o2ib6) in 177 seconds. I think it's dead, and I am evicting it. exp ffff984d4f612400, cur 1551412276 expire 1551412126 last 1551412099 Feb 28 19:51:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:52:06 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 45673e93-9c5f-906c-36ac-49dbaad1815f (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499a65f000, cur 1551412326 expire 1551412176 last 1551412099 Feb 28 19:52:06 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 28 19:53:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 19:53:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 19:56:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3357e8e2-2ad4-c29a-a9ad-2169ee6b9f14 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786834400, cur 1551412598 expire 1551412448 last 1551412371 Feb 28 19:56:38 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Feb 28 19:57:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 19:57:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:00:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2f038ef8-a291-b7ac-b728-64a7e62c9f7e (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c04800, cur 1551412830 expire 1551412680 last 1551412603 Feb 28 20:00:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:01:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 20:01:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:09:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1617a9ce-33ff-578f-7bb2-6e61583aa4e9 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a6e400, cur 1551413391 expire 1551413241 last 1551413164 Feb 28 20:09:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:10:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 28 20:10:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:13:47 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d66add3f-81df-4844-c961-e01030dbc9a8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a7400, cur 1551413627 expire 1551413477 last 1551413400 Feb 28 20:13:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:17:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Feb 28 20:17:47 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Feb 28 20:18:30 fir-io1-s1 kernel: Lustre: fir-OST000a: Client 06a0640d-de7f-9715-947b-5ac203d15e9f (at 10.0.10.3@o2ib7) reconnecting Feb 28 20:18:30 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 28 20:32:18 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ec71568b-f268-f07f-f61e-f47c34c1468d (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801a9400, cur 1551414738 expire 1551414588 last 1551414511 Feb 28 20:32:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:34:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2d279b6b-ae49-37ac-0a12-0938de9dc4ca (at 10.8.1.29@o2ib6) Feb 28 20:34:35 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Feb 28 20:46:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 35139eaa-480f-d22c-38a2-b7c92890351e (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8265800, cur 1551415567 expire 1551415417 last 1551415340 Feb 28 20:46:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:47:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ab57e223-8202-8b5e-ed79-6ec5c4d8555c (at 10.8.3.11@o2ib6) in 156 seconds. I think it's dead, and I am evicting it. exp ffff9857573a0800, cur 1551415643 expire 1551415493 last 1551415487 Feb 28 20:47:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 20:48:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 20:48:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 20:48:26 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ab57e223-8202-8b5e-ed79-6ec5c4d8555c (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857573a3000, cur 1551415706 expire 1551415556 last 1551415479 Feb 28 20:48:26 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 20:56:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Feb 28 20:56:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:12:31 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b59c9b35-1033-a2e2-a2d0-f04ab7b94786 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811f8800, cur 1551417151 expire 1551417001 last 1551416924 Feb 28 21:14:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 21:14:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:32:09 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1fb97d42-6f2c-31eb-ac34-9428db915f5a (at 10.9.0.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4c1400, cur 1551418329 expire 1551418179 last 1551418102 Feb 28 21:32:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:34:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Feb 28 21:34:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:35:57 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 840349b7-4971-51f5-565b-da89e34a530a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b94000, cur 1551418557 expire 1551418407 last 1551418330 Feb 28 21:35:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:36:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 840349b7-4971-51f5-565b-da89e34a530a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d79c00, cur 1551418567 expire 1551418417 last 1551418340 Feb 28 21:36:07 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Feb 28 21:36:19 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 840349b7-4971-51f5-565b-da89e34a530a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480332b000, cur 1551418579 expire 1551418429 last 1551418352 Feb 28 21:37:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 21:37:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:42:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b67455a9-78cb-1575-87b7-b4a12d07f62b (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848019c7400, cur 1551418973 expire 1551418823 last 1551418746 Feb 28 21:42:53 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Feb 28 21:45:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 21:45:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:52:50 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5dc2f432-b506-d677-868d-e6c1ba995ae6 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f2800, cur 1551419570 expire 1551419420 last 1551419343 Feb 28 21:52:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 21:54:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 21:54:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 22:11:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a2169a6c-184e-8919-a60d-0846e39532ec (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986fb4e56c00, cur 1551420690 expire 1551420540 last 1551420463 Feb 28 22:11:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 22:14:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 22:14:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 22:21:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0e30fe72-2647-564a-fbe5-372359c2d20d (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984de97cc800, cur 1551421300 expire 1551421150 last 1551421073 Feb 28 22:21:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 22:23:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 22:23:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 22:41:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 464275de-acda-e759-cdd2-bcee11904c93 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983851c2d000, cur 1551422496 expire 1551422346 last 1551422269 Feb 28 22:41:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 22:42:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 22:42:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 22:58:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f1b5853c-c9c5-c984-dedf-5373387c94df (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed00400, cur 1551423531 expire 1551423381 last 1551423304 Feb 28 22:58:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 23:01:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 23:01:01 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 23:01:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 23:28:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 74a405c0-f732-0def-fd0a-e60d73fe3ac0 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d558400, cur 1551425293 expire 1551425143 last 1551425066 Feb 28 23:28:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 23:30:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 23:30:32 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Feb 28 23:51:04 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bbbfb1e6-d958-c4f7-abd8-b644e6cb44f4 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d810ff000, cur 1551426664 expire 1551426514 last 1551426437 Feb 28 23:51:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Feb 28 23:53:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Feb 28 23:53:24 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 01 00:33:13 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 01 00:44:52 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 86a5fb3b-eafc-bc55-1c22-85954c086100 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a9f800, cur 1551429892 expire 1551429742 last 1551429665 Mar 01 00:44:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 00:47:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 01 00:47:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 02:09:02 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f868b5ad-e56d-6b64-2a30-e6fa72cc85d4 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318e8000, cur 1551434942 expire 1551434792 last 1551434715 Mar 01 02:09:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 02:09:11 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f868b5ad-e56d-6b64-2a30-e6fa72cc85d4 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f4000, cur 1551434951 expire 1551434801 last 1551434724 Mar 01 02:09:11 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 01 02:09:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f868b5ad-e56d-6b64-2a30-e6fa72cc85d4 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387bc000, cur 1551434954 expire 1551434804 last 1551434727 Mar 01 02:11:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 01 02:11:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 03:45:25 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551440718/real 1551440718] req@ffff985476d03c00 x1625225879174176/t0(0) o106->fir-OST0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551440725 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 01 03:45:25 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 900 previous similar messages Mar 01 03:45:46 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551440739/real 1551440739] req@ffff986bfdb3b900 x1625225879174208/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551440746 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 01 03:45:46 fir-io1-s1 kernel: Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 01 03:46:28 fir-io1-s1 kernel: Lustre: 96887:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551440781/real 1551440781] req@ffff98384136a400 x1625225879174112/t0(0) o106->fir-OST0002@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551440788 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 01 03:46:28 fir-io1-s1 kernel: Lustre: 96887:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Mar 01 03:47:25 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7aa634bd-4c9c-76a1-5119-06c30ccebe4b (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b740000, cur 1551440845 expire 1551440695 last 1551440618 Mar 01 03:47:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 01 03:47:35 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 01 04:36:03 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 35a067da-c156-3c2f-abcf-e32726dcc88b (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed01000, cur 1551443763 expire 1551443613 last 1551443536 Mar 01 04:36:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 04:48:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 01 04:48:10 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 01 05:01:51 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 01 05:16:37 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a337eef6-0a86-8383-c324-a30dbd5ed4ff (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f54bcc000, cur 1551446197 expire 1551446047 last 1551445970 Mar 01 05:16:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 05:19:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 01 05:19:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 05:33:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bac7f25f-e00a-41b6-4b26-5d56a6fc7541 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801ab400, cur 1551447201 expire 1551447051 last 1551446974 Mar 01 05:33:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 05:36:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 01 05:36:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 07:26:30 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 01 09:14:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 665b3c07-f583-a6a2-a031-36b81104a696 (at 10.8.9.3@o2ib6) Mar 01 09:14:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 09:15:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dbe64986-3522-e2d0-d57e-b8c002fb5170 (at 10.9.106.33@o2ib4) Mar 01 09:15:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 09:15:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8d89fca0-9472-dc06-65e2-fd0a61adf564 (at 10.9.106.23@o2ib4) Mar 01 09:15:15 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 01 12:06:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 73dd7ced-107a-4de9-f63e-bb703bfbdfee (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984cd57af400, cur 1551470812 expire 1551470662 last 1551470585 Mar 01 12:06:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 12:07:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 01 12:07:21 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 01 12:07:21 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 01 13:20:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client febf13af-aaf1-e443-a6a1-ee171280402a (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d504400, cur 1551475254 expire 1551475104 last 1551475027 Mar 01 13:20:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 13:22:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 01 13:22:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 13:25:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 01 13:25:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 13:36:25 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client db64c980-0a3c-d387-945f-b2ed6fd93194 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984836940800, cur 1551476185 expire 1551476035 last 1551475958 Mar 01 13:36:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 01 13:36:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client db64c980-0a3c-d387-945f-b2ed6fd93194 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8266000, cur 1551476202 expire 1551476052 last 1551475975 Mar 01 13:36:42 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 01 13:39:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 01 13:39:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 14:02:46 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 243cb907-ec5d-451b-5ecf-830e8754cd54 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cde800, cur 1551477766 expire 1551477616 last 1551477539 Mar 01 14:02:46 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 01 14:03:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 01 14:03:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 14:10:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 01 14:10:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 14:11:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client e0d0d81e-4164-f236-7d8a-b466fb3eea50 (at 10.9.104.65@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832b4d800, cur 1551478265 expire 1551478115 last 1551478038 Mar 01 14:11:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 14:34:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7cee7bf7-9aa1-cc50-5aed-b23b669bf632 (at 10.8.15.2@o2ib6) Mar 01 14:34:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 14:37:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 1d7ed545-667f-2ef8-6bba-6c20aaec9c9f (at 10.8.14.9@o2ib6) Mar 01 14:37:32 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 01 14:42:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e0d0d81e-4164-f236-7d8a-b466fb3eea50 (at 10.9.104.65@o2ib4) Mar 01 14:42:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 14:42:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5966b900-cc54-bd33-7a41-4dba11679fd5 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762c71c00, cur 1551480162 expire 1551480012 last 1551479935 Mar 01 14:42:42 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 01 14:42:50 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5966b900-cc54-bd33-7a41-4dba11679fd5 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a059e000, cur 1551480170 expire 1551480020 last 1551479943 Mar 01 14:54:32 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7c4f8fa2-8064-a40f-7557-2bc3448a9f6c (at 10.9.105.68@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858388a5800, cur 1551480872 expire 1551480722 last 1551480645 Mar 01 14:54:32 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 01 15:27:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7c4f8fa2-8064-a40f-7557-2bc3448a9f6c (at 10.9.105.68@o2ib4) Mar 01 15:27:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:00:24 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 59838224-0b96-d82b-4d0b-c72c42caedfe (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d41c400, cur 1551484824 expire 1551484674 last 1551484597 Mar 01 16:00:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:04:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0a44ee8c-0dbc-b894-6b58-b458d94c1436 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318e9400, cur 1551485044 expire 1551484894 last 1551484817 Mar 01 16:04:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:05:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 01 16:05:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:16:31 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Mar 01 16:16:31 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (107): c: 7, oc: 0, rc: 8 Mar 01 16:16:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a9313941-962b-2ab6-84d2-e05ae27de2ef (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cda800, cur 1551485797 expire 1551485647 last 1551485570 Mar 01 16:16:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:17:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 01 16:17:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:17:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 06a0640d-de7f-9715-947b-5ac203d15e9f (at 10.0.10.3@o2ib7) in 190 seconds. I think it's dead, and I am evicting it. exp ffff9847fefec000, cur 1551485873 expire 1551485723 last 1551485683 Mar 01 16:17:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:21:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Mar 01 16:21:31 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 01 16:49:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 01 16:49:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 16:49:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9b8c1710-fb73-934b-f064-b4fe8b1f7aa1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678680c400, cur 1551487786 expire 1551487636 last 1551487559 Mar 01 16:49:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 17:00:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 01 17:00:56 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 01 17:01:42 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 7161067c-7f48-94e9-c5eb-9215957925d1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800fd9c00, cur 1551488502 expire 1551488352 last 1551488275 Mar 01 17:01:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 17:12:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 01 17:12:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 17:12:42 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 37c71cbc-1df4-726b-a1b1-aac252a53cd7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c40c00, cur 1551489162 expire 1551489012 last 1551488935 Mar 01 17:12:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 17:24:34 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 63b8301c-2489-a687-fc66-290681556ce9 (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985757ba8000, cur 1551489874 expire 1551489724 last 1551489647 Mar 01 17:24:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 01 19:52:02 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 02 03:33:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 70d85fb9-2e84-d5b2-9ee4-d1a5a5f3dc16 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a4800, cur 1551526438 expire 1551526288 last 1551526211 Mar 02 03:33:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 03:43:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 02 03:43:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 08:34:08 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 32bb871f-8b63-3302-66d5-41b8f5aa11c8 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef10000, cur 1551544448 expire 1551544298 last 1551544221 Mar 02 08:34:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 08:40:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 02 08:40:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 12:28:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2a194d99-18e6-7752-a804-1bda1a018bd1 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15cf000, cur 1551558522 expire 1551558372 last 1551558295 Mar 02 12:28:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 12:28:49 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2a194d99-18e6-7752-a804-1bda1a018bd1 (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15ca000, cur 1551558529 expire 1551558379 last 1551558302 Mar 02 12:28:49 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 02 13:48:40 fir-io1-s1 kernel: md: md10: data-check done. Mar 02 16:38:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7be6a1f6-efc0-6cc2-c1bb-485ca633e715 (at 10.9.106.41@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986c37bb5800, cur 1551573520 expire 1551573370 last 1551573293 Mar 02 17:01:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 66de0a73-cb41-c788-e30e-7505e7f80015 (at 10.9.106.41@o2ib4) Mar 02 17:01:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 18:20:17 fir-io1-s1 kernel: md: md0: data-check done. Mar 02 18:59:02 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 510f179a-ed3d-2756-0dd0-06d9bebce633 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678680dc00, cur 1551581942 expire 1551581792 last 1551581715 Mar 02 18:59:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 19:01:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 02 19:01:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 19:27:47 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9d887c4f-a9b5-f5ac-9884-16d0cb7eb9e1 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a9c384c00, cur 1551583667 expire 1551583517 last 1551583440 Mar 02 19:27:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 19:40:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 02 19:40:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 20:08:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7b1f2afe-660b-2255-e0f0-76eb03259f03 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d2cc00, cur 1551586131 expire 1551585981 last 1551585904 Mar 02 20:08:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 20:22:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 02 20:22:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 20:27:57 fir-io1-s1 kernel: md: md2: data-check done. Mar 02 22:20:26 fir-io1-s1 kernel: md: md4: data-check done. Mar 02 23:22:25 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c0857d5b-23be-e61d-3af0-f0dbe6518965 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f3064f000, cur 1551597745 expire 1551597595 last 1551597518 Mar 02 23:22:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 02 23:22:26 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c0857d5b-23be-e61d-3af0-f0dbe6518965 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f30648c00, cur 1551597746 expire 1551597596 last 1551597519 Mar 02 23:22:26 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 02 23:22:29 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c0857d5b-23be-e61d-3af0-f0dbe6518965 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762740400, cur 1551597749 expire 1551597599 last 1551597522 Mar 02 23:22:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c0857d5b-23be-e61d-3af0-f0dbe6518965 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f3064cc00, cur 1551597753 expire 1551597603 last 1551597526 Mar 02 23:22:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7751f34c-206a-edf3-4c97-12b3ac7dcb0e (at 10.8.23.14@o2ib6) Mar 02 23:22:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 01:00:01 fir-io1-s1 kernel: md: data-check of RAID array md4 Mar 03 01:00:07 fir-io1-s1 kernel: md: data-check of RAID array md2 Mar 03 01:00:13 fir-io1-s1 kernel: md: data-check of RAID array md10 Mar 03 01:00:20 fir-io1-s1 kernel: md: data-check of RAID array md0 Mar 03 03:10:22 fir-io1-s1 kernel: md: md8: data-check done. Mar 03 09:07:59 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1ca0d660-5cb5-0e63-2a12-1991c11634d4 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cb1000, cur 1551632879 expire 1551632729 last 1551632652 Mar 03 09:07:59 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 03 09:12:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:12:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:16:27 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 87ff4d9d-26d2-9741-4e0a-4525e296c8bd (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a59800, cur 1551633387 expire 1551633237 last 1551633160 Mar 03 09:16:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:17:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:17:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:21:38 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ff17c201-fc71-667b-42e9-2a32e458d304 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8265000, cur 1551633698 expire 1551633548 last 1551633471 Mar 03 09:21:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:22:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:22:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:26:35 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 691d25f9-d361-7ad5-bd91-d09318e5ee3f (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a08400, cur 1551633995 expire 1551633845 last 1551633768 Mar 03 09:26:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:28:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:28:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:31:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 86eaea9c-50b6-3df4-a813-52d0df054049 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a5ac00, cur 1551634315 expire 1551634165 last 1551634088 Mar 03 09:31:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:33:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:33:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:37:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f25507ab-4467-23f3-9cdc-d789d9dd9136 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985838d8bc00, cur 1551634629 expire 1551634479 last 1551634402 Mar 03 09:37:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:38:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:38:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:40:40 fir-io1-s1 kernel: Lustre: 110033:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551634833/real 1551634833] req@ffff98382a290f00 x1625259946732160/t0(0) o104->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1551634840 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 03 09:40:40 fir-io1-s1 kernel: Lustre: 110033:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Mar 03 09:40:54 fir-io1-s1 kernel: Lustre: 74705:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551634847/real 1551634847] req@ffff98770659f200 x1625259946736624/t0(0) o104->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1551634854 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 03 09:40:54 fir-io1-s1 kernel: Lustre: 74705:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Mar 03 09:41:15 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551634868/real 1551634868] req@ffff9876f5f74200 x1625259946731424/t0(0) o104->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1551634875 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 03 09:41:15 fir-io1-s1 kernel: Lustre: 96405:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Mar 03 09:41:57 fir-io1-s1 kernel: Lustre: 110602:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551634910/real 1551634910] req@ffff987583a9e000 x1625259946731488/t0(0) o104->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1551634917 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 03 09:41:57 fir-io1-s1 kernel: Lustre: 110602:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 103 previous similar messages Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: 94245:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff9869b792b900 x1625259946734384 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff986a28494c80/0x49e1860fecf7ce6e lrc: 4/0,0 mode: PR/PR res: [0x6c0000402:0x1252b4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xe4bc2e5a08e5d576 expref: 938 pid: 94242 timeout: 1982782 lvb_type: 1 Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff98386680dc40/0x49e1860fecf7dd4e lrc: 3/0,0 mode: PR/PR res: [0xc40000401:0x1252ca:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xe4bc2e5a08e5feb2 expref: 964 pid: 96280 timeout: 0 lvb_type: 1 Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: 96378:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff986ee5890600 x1625259946836224/t0(0) o104->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: 96378:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 9 previous similar messages Mar 03 09:42:18 fir-io1-s1 kernel: LustreError: 94245:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 8 previous similar messages Mar 03 09:42:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ae8c76ac-e130-b398-7aeb-12ff776de630 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98663589f400, cur 1551634943 expire 1551634793 last 1551634716 Mar 03 09:42:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:43:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 03 09:43:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:47:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 19dc7bab-2b71-4e0a-0013-f41774e9dd93 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a08400, cur 1551635257 expire 1551635107 last 1551635030 Mar 03 09:47:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:49:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:49:57 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 09:53:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8bcb8dc5-083b-d2a9-3ff2-ec493580687b (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfe800, cur 1551635624 expire 1551635474 last 1551635397 Mar 03 09:53:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:55:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 09:55:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 09:59:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b775c482-ef38-2316-dd46-5d28b99bed99 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98749c33c400, cur 1551635964 expire 1551635814 last 1551635737 Mar 03 09:59:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 10:01:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 10:01:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 10:05:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client cd84e569-8627-56ec-c3a2-6a94359f1896 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784488000, cur 1551636336 expire 1551636186 last 1551636109 Mar 03 10:05:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 10:07:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 10:07:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 10:16:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b6a8fde2-e4fe-0ff6-b1d6-6774b0f7bca1 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8ad800, cur 1551636964 expire 1551636814 last 1551636737 Mar 03 10:16:04 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 03 10:17:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 10:17:29 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 03 10:26:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bd93d90b-c3d1-4e8e-8fca-f94c8e58f241 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848349e5800, cur 1551637582 expire 1551637432 last 1551637355 Mar 03 10:26:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 10:27:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 10:27:49 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 03 10:36:50 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client baba7a82-45d0-7bfc-9477-255ebfe2e3cd (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a28000, cur 1551638210 expire 1551638060 last 1551637983 Mar 03 10:36:50 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 10:43:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 10:43:23 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 03 10:47:10 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c1df9137-df55-751e-2177-591b9d685916 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b763800, cur 1551638830 expire 1551638680 last 1551638603 Mar 03 10:47:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 10:53:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 10:53:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 10:57:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b1809b9d-595d-e6c1-4dad-fdffb3b194e3 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2a800, cur 1551639450 expire 1551639300 last 1551639223 Mar 03 10:57:30 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 11:11:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 11:11:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 11:15:03 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d70908cd-6ad5-9d2e-50b7-010a7c518a25 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c676400, cur 1551640503 expire 1551640353 last 1551640276 Mar 03 11:15:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 11:21:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 11:21:40 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 11:25:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 40c054d9-3fac-22fd-a8ad-8c61a4b651df (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e320c0400, cur 1551641127 expire 1551640977 last 1551640900 Mar 03 11:25:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 11:32:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 11:32:11 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 11:35:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d32cb150-5389-37f0-ffc6-659ef1d0c89f (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984803159800, cur 1551641758 expire 1551641608 last 1551641531 Mar 03 11:35:58 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 11:42:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 11:42:24 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 11:46:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f7fd6320-40f5-bfa5-53a8-cbf27646d038 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864523bd800, cur 1551642371 expire 1551642221 last 1551642144 Mar 03 11:46:11 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 11:57:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 11:57:08 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 03 12:00:55 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b4a535a3-c539-600f-fe7c-d59f43712f08 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd442400, cur 1551643255 expire 1551643105 last 1551643028 Mar 03 12:00:55 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 03 12:07:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 12:07:38 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:11:25 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c51501ee-1044-762f-8a07-4471ee136978 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8264400, cur 1551643885 expire 1551643735 last 1551643658 Mar 03 12:11:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:18:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 12:18:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:21:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 062e5baf-0a37-1c44-daac-45b5d0e3d9b2 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a9d400, cur 1551644512 expire 1551644362 last 1551644285 Mar 03 12:21:52 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:28:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 12:28:38 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:32:25 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5800e7cd-5b0a-dcdb-4060-c15f89fc5f23 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e14c00, cur 1551645145 expire 1551644995 last 1551644918 Mar 03 12:32:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:45:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 12:45:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:48:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3d4fa4eb-1b5e-6a02-2183-21249037a4c8 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4cc00, cur 1551646131 expire 1551645981 last 1551645904 Mar 03 12:48:51 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 12:56:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 12:56:23 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 13:00:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f55048a0-e6dc-b67d-b404-5d5edc64f87f (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848346a0000, cur 1551646810 expire 1551646660 last 1551646583 Mar 03 13:00:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 13:11:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 13:11:02 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 13:14:49 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4fcad4b4-e86d-cdb1-df44-678e24d9accf (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a99000, cur 1551647689 expire 1551647539 last 1551647462 Mar 03 13:14:49 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 13:25:41 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9fa7ee8d-d907-dedf-bd02-dedd8c18f1f2 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d7d400, cur 1551648341 expire 1551648191 last 1551648114 Mar 03 13:25:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 13:27:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 13:27:06 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 13:36:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2db58569-753b-5a8a-44d2-58087a2c8ede (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984836f9a800, cur 1551648960 expire 1551648810 last 1551648733 Mar 03 13:36:00 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 13:42:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 13:42:33 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 03 13:46:20 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 39d95f82-a53d-ca23-52c6-eaaa0c0ee9fd (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801bfac00, cur 1551649580 expire 1551649430 last 1551649353 Mar 03 13:46:20 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 13:53:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 13:53:05 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 03 13:56:52 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c16de1ab-bd03-21d3-5981-ad9f9b3e0933 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e8aa400, cur 1551650212 expire 1551650062 last 1551649985 Mar 03 13:56:52 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 14:03:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 14:03:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 14:07:41 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 787198e9-c687-7142-9f3b-070ba95ec682 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984830fb1000, cur 1551650861 expire 1551650711 last 1551650634 Mar 03 14:07:41 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 14:15:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 03 14:15:15 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 14:19:02 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bb4bfdd2-f5c9-da78-af4b-6886d62a5741 (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387b8800, cur 1551651542 expire 1551651392 last 1551651315 Mar 03 14:19:02 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 15:27:18 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 534999f9-98ae-b356-b86f-4286fe08e3d2 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ee000, cur 1551655638 expire 1551655488 last 1551655411 Mar 03 15:27:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 15:28:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 03 15:28:51 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 03 20:00:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 915566f6-5cf8-5402-717e-e46388b3c9a8 (at 10.8.1.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857590d0000, cur 1551672042 expire 1551671892 last 1551671815 Mar 03 20:00:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 20:01:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to be293349-45a5-91c6-c8b7-456ff508fdc0 (at 10.8.1.31@o2ib6) Mar 03 20:01:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 20:23:14 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e94a1345-0315-3234-5462-14ab93962c60 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984830fb1000, cur 1551673394 expire 1551673244 last 1551673167 Mar 03 20:23:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 20:24:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 03 20:24:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 21:23:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 16663fd5-94c4-adf1-ba40-692553dcbd4e (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f2000, cur 1551676980 expire 1551676830 last 1551676753 Mar 03 21:23:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 21:23:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 03 21:23:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 21:54:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 87c712e1-5142-11a4-c5f3-f035ebe536cf (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833da8400, cur 1551678891 expire 1551678741 last 1551678664 Mar 03 21:54:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 03 21:55:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 03 21:55:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 00:45:27 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 01:15:15 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5539da3b-51e3-2126-6ba8-0f11038263c7 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983851c2a800, cur 1551690915 expire 1551690765 last 1551690688 Mar 04 01:15:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 01:15:17 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5539da3b-51e3-2126-6ba8-0f11038263c7 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b766c00, cur 1551690917 expire 1551690767 last 1551690690 Mar 04 01:15:17 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 04 01:15:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 04 01:15:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 03:06:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1b8666e1-81bf-af07-961a-3f667c319358 (at 10.9.104.68@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987717fba000, cur 1551697612 expire 1551697462 last 1551697385 Mar 04 03:06:52 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 04 03:35:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a045669f-27d9-1372-2514-d5211db1ecd9 (at 10.9.104.68@o2ib4) Mar 04 03:35:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 04:17:44 fir-io1-s1 kernel: md: md6: data-check done. Mar 04 05:02:39 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST000a: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 04 05:02:39 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Mar 04 05:03:11 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551704584/real 1551704584] req@ffff986b61445100 x1625268083499904/t0(0) o400->fir-MDT0001-lwp-OST0008@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1551704591 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 04 05:03:11 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0006: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 04 05:03:11 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 04 05:03:11 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 52 previous similar messages Mar 04 05:03:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 04 05:03:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 04 05:03:57 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 04 05:04:07 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 04 05:04:07 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 04 05:04:26 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 04 05:04:26 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 04 05:04:26 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 04 05:04:26 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Mar 04 05:04:58 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Mar 04 05:04:58 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (51): c: 0, oc: 0, rc: 8 Mar 04 05:05:00 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 04 05:05:00 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 04 05:05:41 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0003-lwp-OST000a: This client was evicted by fir-MDT0003; in progress operations using this service will fail. Mar 04 05:05:41 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 04 05:05:41 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST000a: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 04 05:05:41 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2095907 to 0x5c0000400:2096001 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2095975 to 0x6c0000400:2096129 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2095546 to 0x8c0000402:2095617 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2095857 to 0xc40000402:2095905 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2096148 to 0x580000400:2096225 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2095863 to 0xc80000402:2095969 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1327282 to 0x0:1327361 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1327546 to 0x0:1327745 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1327994 to 0x0:1328065 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1327739 to 0x0:1327809 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1327553 to 0x0:1327585 Mar 04 05:05:52 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1328189 to 0x0:1328225 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1208765 to 0x6c0000402:1208833 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1208620 to 0xc40000401:1208673 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1210116 to 0x5c0000402:1210145 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1210147 to 0x580000402:1210241 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1208783 to 0x8c0000401:1208865 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1208738 to 0xc80000401:1208769 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:1115971 to 0x8c0000400:1116065 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:1116305 to 0x5c0000401:1116417 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:1116153 to 0x6c0000401:1116289 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:1116178 to 0xc40000400:1116545 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:1116653 to 0x580000401:1116801 Mar 04 05:05:53 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:1116174 to 0xc80000400:1116225 Mar 04 05:56:58 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 06:50:01 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 07:03:50 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 07:50:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 37c5171f-2ee0-3f9b-c33d-fe1aa35295d6 (at 10.8.9.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865175be000, cur 1551714616 expire 1551714466 last 1551714389 Mar 04 07:50:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:15:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2d34bc60-63a1-47d1-6dd1-fcc58193fe15 (at 10.9.114.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835771800, cur 1551716126 expire 1551715976 last 1551715899 Mar 04 08:15:26 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Mar 04 08:18:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72ef71e7-ecde-a7f2-d85d-452a22011f5b (at 10.9.101.14@o2ib4) Mar 04 08:18:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:19:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fc60883d-f9c1-82aa-8312-f53a10d6b6ff (at 10.8.9.1@o2ib6) Mar 04 08:19:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:21:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 664f5f86-7e37-c3dd-9009-3eec77c4bd45 (at 10.8.11.1@o2ib6) Mar 04 08:21:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:21:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e70f289f-dd27-3962-9868-2a7ca371acbb (at 10.8.10.30@o2ib6) Mar 04 08:21:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:22:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d56b93c-c51e-bd16-d798-5d8639e7069c (at 10.8.11.31@o2ib6) Mar 04 08:22:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:22:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 76c7b11b-8cf1-3938-34cb-225f4db85053 (at 10.8.12.22@o2ib6) Mar 04 08:22:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:23:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4199656a-fc55-4ace-07e9-a689e8e8d80b (at 10.8.10.7@o2ib6) Mar 04 08:23:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 08:24:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 79465c7f-aac0-bcd9-93e3-6b42fd3a6813 (at 10.8.11.7@o2ib6) Mar 04 08:24:58 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 04 08:38:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7eb786c-4da9-9431-a597-6f5f4ba4c9ed (at 10.9.114.12@o2ib4) Mar 04 08:38:47 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Mar 04 09:09:29 fir-io1-s1 kernel: Lustre: 110609:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551719362/real 1551719362] req@ffff986b962a0000 x1625273371758080/t0(0) o106->fir-OST000a@10.9.114.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551719369 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 04 09:09:29 fir-io1-s1 kernel: Lustre: 110609:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 04 09:09:36 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551719369/real 1551719369] req@ffff9852df7d1e00 x1625273371758032/t0(0) o106->fir-OST0006@10.9.114.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551719376 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 09:09:36 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 04 09:09:43 fir-io1-s1 kernel: Lustre: 2629:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551719376/real 1551719376] req@ffff987073c76f00 x1625273371758096/t0(0) o106->fir-OST0002@10.9.114.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551719383 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 09:09:43 fir-io1-s1 kernel: Lustre: 2629:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 04 09:09:50 fir-io1-s1 kernel: Lustre: 129953:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551719383/real 1551719383] req@ffff9875f14eb900 x1625273371758064/t0(0) o106->fir-OST0008@10.9.114.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551719390 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 09:10:04 fir-io1-s1 kernel: Lustre: 129953:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551719397/real 1551719397] req@ffff9875f14eb900 x1625273371758064/t0(0) o106->fir-OST0008@10.9.114.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551719404 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 09:10:04 fir-io1-s1 kernel: Lustre: 129953:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 04 09:10:25 fir-io1-s1 kernel: Lustre: 2629:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551719418/real 1551719418] req@ffff987073c76f00 x1625273371758096/t0(0) o106->fir-OST0002@10.9.114.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551719425 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 09:10:25 fir-io1-s1 kernel: Lustre: 2629:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 04 09:11:07 fir-io1-s1 kernel: Lustre: 110609:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551719460/real 1551719460] req@ffff986b962a0000 x1625273371758080/t0(0) o106->fir-OST000a@10.9.114.2@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1551719467 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 09:11:07 fir-io1-s1 kernel: Lustre: 110609:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Mar 04 09:12:01 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7e0b59f9-f094-0d9b-c246-e8f498ee0a60 (at 10.9.114.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987279af3000, cur 1551719521 expire 1551719371 last 1551719294 Mar 04 09:12:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 09:13:03 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 09:26:50 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 09:32:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 781bc9da-48cc-e7d6-4e3d-b87eade4c4d8 (at 10.9.115.5@o2ib4) Mar 04 09:32:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 09:34:47 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8f530db8-807a-fdf0-2880-6c939864abc5 (at 10.9.115.8@o2ib4) Mar 04 09:34:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 09:35:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 797f0075-ec92-4d37-f23e-cc9ca768ea89 (at 10.9.113.5@o2ib4) Mar 04 09:35:29 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 04 09:35:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1ee3037c-52dd-207d-3196-b589ce5ac006 (at 10.9.114.14@o2ib4) Mar 04 09:35:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 09:36:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f3c26b94-2261-c91e-b422-79918936510b (at 10.9.115.4@o2ib4) Mar 04 09:36:27 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 04 09:37:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fb273c41-c272-402d-98b5-3e5f91dba50e (at 10.9.114.15@o2ib4) Mar 04 09:37:02 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Mar 04 09:39:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f01f6d2d-1e5a-39ce-ec8b-7cb9b2bcde4c (at 10.8.16.2@o2ib6) Mar 04 09:39:02 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 04 09:43:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Mar 04 09:43:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 04 10:06:03 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 20e7ecad-b80c-b2bc-352b-54943f55e20a (at 10.9.114.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a83a62400, cur 1551722763 expire 1551722613 last 1551722536 Mar 04 10:06:03 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Mar 04 10:09:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4f3c448c-0dad-4290-111c-1f81c8ee3ba0 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8261800, cur 1551722948 expire 1551722798 last 1551722721 Mar 04 10:09:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 10:10:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c2297627-b160-fcef-a8e2-72f75afc06c5 (at 10.8.9.8@o2ib6) in 173 seconds. I think it's dead, and I am evicting it. exp ffff986786a5cc00, cur 1551723024 expire 1551722874 last 1551722851 Mar 04 10:10:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 10:10:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 04 10:10:34 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 04 10:11:18 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c2297627-b160-fcef-a8e2-72f75afc06c5 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835772c00, cur 1551723078 expire 1551722928 last 1551722851 Mar 04 10:11:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 04 10:11:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 04 10:11:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 10:29:22 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 10:34:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 97fc0b61-f63b-fdc3-beec-7984ab24ec08 (at 10.9.112.8@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985573c65000, cur 1551724447 expire 1551724297 last 1551724220 Mar 04 10:37:07 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 05b123c3-f140-a983-96ef-72762de6c959 (at 10.8.28.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98753eba5400, cur 1551724627 expire 1551724477 last 1551724400 Mar 04 10:37:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 10:37:31 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to e263de63-be66-7fb7-ff72-8f31aee416ec (at 10.9.114.3@o2ib4) Mar 04 10:37:31 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 04 10:56:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1b71b1a8-fdc3-550f-13e6-42b0376dd743 (at 10.9.112.8@o2ib4) Mar 04 10:56:24 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 04 11:00:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b3ef6690-dd23-2eef-0dcf-441d88950a4a (at 10.8.28.2@o2ib6) Mar 04 11:00:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:16:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e0ff77d9-5f70-2aad-3590-2374fb77dd4b (at 10.9.112.6@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e57400, cur 1551726960 expire 1551726810 last 1551726733 Mar 04 11:16:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:43:39 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0b80738d-4ce2-c944-d5ef-81b5bb637080 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481177dc00, cur 1551728619 expire 1551728469 last 1551728392 Mar 04 11:43:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:43:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0b80738d-4ce2-c944-d5ef-81b5bb637080 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834687c00, cur 1551728632 expire 1551728482 last 1551728405 Mar 04 11:43:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:46:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 142a6e04-872a-1c7a-e7fe-9168d1c1b90d (at 10.8.9.5@o2ib6) Mar 04 11:46:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:46:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aff71b74-7050-1a79-ef86-3b2a0fea26d1 (at 10.8.9.4@o2ib6) Mar 04 11:46:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:46:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 665b3c07-f583-a6a2-a031-36b81104a696 (at 10.8.9.3@o2ib6) Mar 04 11:46:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:46:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Mar 04 11:46:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:48:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a2c53951-d174-3b52-a3ee-252826248ac1 (at 10.9.112.6@o2ib4) Mar 04 11:48:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 11:48:11 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 11:54:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f45ea0af-b188-334b-c3ef-ac12644d1ea2 (at 10.8.19.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8267c00, cur 1551729263 expire 1551729113 last 1551729036 Mar 04 11:54:23 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 04 11:56:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 16726faa-6b86-b9f7-b03d-830f20c036a1 (at 10.9.113.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd264c00, cur 1551729417 expire 1551729267 last 1551729190 Mar 04 11:56:57 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 04 11:57:04 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 16726faa-6b86-b9f7-b03d-830f20c036a1 (at 10.9.113.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd264000, cur 1551729424 expire 1551729274 last 1551729197 Mar 04 11:57:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 04 12:00:47 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4fcd153b-fe70-7f2e-37a4-73933929e091 (at 10.9.113.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984835c93800, cur 1551729647 expire 1551729497 last 1551729420 Mar 04 12:00:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4fcd153b-fe70-7f2e-37a4-73933929e091 (at 10.9.113.1@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984835c91400, cur 1551729656 expire 1551729506 last 1551729429 Mar 04 12:02:03 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3965e226-6a5e-41c5-e251-993d1b14255c (at 10.8.9.5@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff9847fd2f7000, cur 1551729723 expire 1551729573 last 1551729501 Mar 04 12:02:03 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 04 12:05:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 38a104b5-26ce-5d2d-596d-9304083f888f (at 10.9.112.14@o2ib4) Mar 04 12:05:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:06:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3e279d24-e2fb-196e-d7ac-e1a73db143bd (at 10.9.112.16@o2ib4) Mar 04 12:06:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:07:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bef16258-699d-0e14-bdeb-b454fac00d89 (at 10.9.112.15@o2ib4) Mar 04 12:07:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:14:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 926fa24d-f3ab-7ad6-dbc7-f8a15bdf8c5a (at 10.8.19.8@o2ib6) Mar 04 12:14:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:15:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca78cddb-ea39-0254-4028-7b0b6c7c780d (at 10.8.19.2@o2ib6) Mar 04 12:15:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:16:56 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4deaceb0-05cb-6b79-c285-3e3009b0560f (at 10.8.28.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800e4c400, cur 1551730616 expire 1551730466 last 1551730389 Mar 04 12:17:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4deaceb0-05cb-6b79-c285-3e3009b0560f (at 10.8.28.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df7400, cur 1551730622 expire 1551730472 last 1551730395 Mar 04 12:17:02 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 04 12:17:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca78cddb-ea39-0254-4028-7b0b6c7c780d (at 10.8.19.2@o2ib6) Mar 04 12:17:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:18:12 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e3f60e59-97fd-a9e6-f7c5-6336474ffa21 (at 10.8.19.2@o2ib6) in 151 seconds. I think it's dead, and I am evicting it. exp ffff9848801ab800, cur 1551730692 expire 1551730542 last 1551730541 Mar 04 12:18:12 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 04 12:18:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 135b49e4-9bbb-43fc-66e9-1f7ec8c75a96 (at 10.9.113.3@o2ib4) Mar 04 12:18:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:19:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fe789eb3-1cd9-3594-b889-6606ba1b8e4a (at 10.9.113.2@o2ib4) Mar 04 12:19:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:24:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.113.1@o2ib4) Mar 04 12:24:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:28:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0020345d-5ca8-34a6-0ce1-c2c7b984d732 (at 10.9.115.11@o2ib4) Mar 04 12:28:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 12:39:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c243879d-6590-e58d-10d6-105c5b7b4def (at 10.8.28.1@o2ib6) Mar 04 12:39:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 04 12:42:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8e929410-082a-5385-39e8-9bbeaa9e9d26 (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865f0ecec00, cur 1551732162 expire 1551732012 last 1551731935 Mar 04 12:42:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:07:29 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f7452a41-9f0b-ebba-bd1d-d8211c44f727 (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762863c00, cur 1551733649 expire 1551733499 last 1551733422 Mar 04 13:07:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:10:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 488cfd93-1121-504d-019d-485c13be114d (at 10.8.14.4@o2ib6) Mar 04 13:10:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:10:59 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client db88ae36-758a-b4b0-de1c-20bc40573a19 (at 10.8.14.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d55e800, cur 1551733859 expire 1551733709 last 1551733632 Mar 04 13:10:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:34:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7631cb3-40a3-08be-ae74-cf548ae0665c (at 10.8.14.8@o2ib6) Mar 04 13:34:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:37:20 fir-io1-s1 kernel: Lustre: 36981:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551735433/real 1551735433] req@ffff986a7b276300 x1625283803731904/t0(0) o106->fir-OST0006@10.8.9.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551735440 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 04 13:37:20 fir-io1-s1 kernel: Lustre: 36981:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 188 previous similar messages Mar 04 13:37:31 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551735444/real 1551735444] req@ffff983803458f00 x1625283812267360/t0(0) o106->fir-OST000a@10.8.9.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551735451 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 04 13:37:31 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 04 13:37:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 86adf901-7c63-11b6-e31a-e7c278b5d0d5 (at 10.9.112.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98581ef40800, cur 1551735460 expire 1551735310 last 1551735233 Mar 04 13:37:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:37:52 fir-io1-s1 kernel: Lustre: 96354:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551735465/real 1551735465] req@ffff986c6f578600 x1625283812267504/t0(0) o106->fir-OST0004@10.8.9.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551735472 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 13:37:52 fir-io1-s1 kernel: Lustre: 96354:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Mar 04 13:38:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3dbbb061-d93c-c7e8-88c8-e262ff513397 (at 10.8.14.6@o2ib6) Mar 04 13:38:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:38:30 fir-io1-s1 kernel: Lustre: 96256:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551735503/real 1551735503] req@ffff9876c12a0f00 x1625283803731952/t0(0) o106->fir-OST0008@10.8.9.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551735510 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 13:38:30 fir-io1-s1 kernel: Lustre: 96256:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 40 previous similar messages Mar 04 13:39:47 fir-io1-s1 kernel: Lustre: 96942:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551735580/real 1551735580] req@ffff986c6f579e00 x1625283803732000/t0(0) o106->fir-OST000a@10.8.9.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551735587 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 13:39:47 fir-io1-s1 kernel: Lustre: 96942:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 83 previous similar messages Mar 04 13:40:34 fir-io1-s1 kernel: LNet: Service thread pid 96256 was inactive for 200.63s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 04 13:40:34 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 04 13:40:34 fir-io1-s1 kernel: Pid: 96256, comm: ll_ost01_015 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 04 13:40:34 fir-io1-s1 kernel: Call Trace: Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 04 13:40:34 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 04 13:40:34 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551735634.96256 Mar 04 13:40:36 fir-io1-s1 kernel: LNet: Service thread pid 75603 was inactive for 202.12s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 04 13:40:36 fir-io1-s1 kernel: Pid: 75603, comm: ll_ost02_093 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 04 13:40:36 fir-io1-s1 kernel: Call Trace: Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 04 13:40:36 fir-io1-s1 kernel: Pid: 96942, comm: ll_ost01_110 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 04 13:40:36 fir-io1-s1 kernel: Call Trace: Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 04 13:40:36 fir-io1-s1 kernel: Pid: 36981, comm: ll_ost02_072 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 04 13:40:36 fir-io1-s1 kernel: Call Trace: Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 04 13:40:36 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 04 13:40:43 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e437a2b8-91bb-d7b0-5fc6-b3c390f40701 (at 10.8.9.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984998f85000, cur 1551735643 expire 1551735493 last 1551735416 Mar 04 13:40:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 13:40:43 fir-io1-s1 kernel: LNet: Service thread pid 36981 completed after 209.17s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 04 13:40:43 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 04 13:58:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 043c3b68-4c87-dea4-1a9b-f14ca335337c (at 10.9.112.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985c3221fc00, cur 1551736696 expire 1551736546 last 1551736469 Mar 04 13:58:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:00:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6a4c2338-0ceb-a951-40ef-6ef876a157c6 (at 10.9.112.10@o2ib4) Mar 04 14:00:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:06:09 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client cd88d59b-0cf3-4fdc-1b7a-3d5a89602d5d (at 10.8.15.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984cd57ad800, cur 1551737169 expire 1551737019 last 1551736942 Mar 04 14:06:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:06:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a9bc589b-0921-525b-037b-7d74fb331c32 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98663589bc00, cur 1551737180 expire 1551737030 last 1551736953 Mar 04 14:06:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:07:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15339979-51e5-e16d-f976-ff72d24bd14f (at 10.8.9.10@o2ib6) Mar 04 14:07:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:09:04 fir-io1-s1 kernel: Lustre: 110657:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551737337/real 1551737337] req@ffff98380f97b000 x1625285059474704/t0(0) o106->fir-OST0008@10.8.15.3@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551737344 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 04 14:09:04 fir-io1-s1 kernel: Lustre: 110657:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 63 previous similar messages Mar 04 14:09:17 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 34e3da77-276c-a576-bb82-272c08d5f0d2 (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f1b2ed000, cur 1551737357 expire 1551737207 last 1551737130 Mar 04 14:09:17 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 04 14:09:25 fir-io1-s1 kernel: Lustre: 94514:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551737358/real 1551737358] req@ffff9875d3b13300 x1625285059474720/t0(0) o106->fir-OST000a@10.8.15.3@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551737365 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 14:09:25 fir-io1-s1 kernel: Lustre: 94514:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 04 14:09:28 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 34e3da77-276c-a576-bb82-272c08d5f0d2 (at 10.8.15.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811f8c00, cur 1551737368 expire 1551737218 last 1551737141 Mar 04 14:09:28 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 04 14:16:54 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a9ed82f4-26a2-3d74-3e58-90b405dfbd36 (at 10.9.112.11@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868af800, cur 1551737814 expire 1551737664 last 1551737587 Mar 04 14:16:54 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 04 14:19:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 105b4037-1bdb-9aac-e52a-dc0e974000a2 (at 10.9.112.12@o2ib4) Mar 04 14:19:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:28:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 503151d3-e911-b92b-974a-493626aee137 (at 10.8.15.8@o2ib6) Mar 04 14:28:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:30:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8c99250d-76e9-284a-5406-5556d3865e14 (at 10.8.29.5@o2ib6) Mar 04 14:30:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:32:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 04 14:32:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:33:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 519ac832-a77f-51ba-e3c7-51aa4fe15024 (at 10.8.15.3@o2ib6) Mar 04 14:33:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 14:39:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bdabca4a-b324-10cc-272f-e92e9f4e05cd (at 10.9.112.11@o2ib4) Mar 04 14:39:11 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 04 14:59:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 016e2f44-ecca-78dc-bad8-fcf67dcd3e1a (at 10.9.115.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904990800, cur 1551740382 expire 1551740232 last 1551740155 Mar 04 14:59:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 15:00:58 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b6fbec8e-13dd-42ee-7bda-89ea6fae31c3 (at 10.9.113.8@o2ib4) in 226 seconds. I think it's dead, and I am evicting it. exp ffff984b283df800, cur 1551740458 expire 1551740308 last 1551740232 Mar 04 15:00:58 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 04 15:09:09 fir-io1-s1 kernel: Lustre: 36982:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551740942/real 1551740942] req@ffff985ed9486f00 x1625287428414528/t0(0) o106->fir-OST0002@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551740949 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 04 15:09:09 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551740942/real 1551740942] req@ffff985934772100 x1625287428414336/t0(0) o106->fir-OST0006@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551740949 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 04 15:09:09 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 04 15:09:16 fir-io1-s1 kernel: Lustre: 74752:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551740949/real 1551740949] req@ffff98661e956600 x1625287428414480/t0(0) o106->fir-OST000a@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551740956 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 15:09:16 fir-io1-s1 kernel: Lustre: 74752:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 04 15:09:23 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551740956/real 1551740956] req@ffff985934772100 x1625287428414336/t0(0) o106->fir-OST0006@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551740963 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 15:09:23 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 04 15:09:30 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551740963/real 1551740963] req@ffff9861161cb300 x1625287428414432/t0(0) o106->fir-OST0008@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551740970 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 15:09:30 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 04 15:09:37 fir-io1-s1 kernel: Lustre: 74752:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551740970/real 1551740970] req@ffff98661e956600 x1625287428414480/t0(0) o106->fir-OST000a@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551740977 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 15:09:37 fir-io1-s1 kernel: Lustre: 74752:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 04 15:09:51 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551740984/real 1551740984] req@ffff985934772100 x1625287428414336/t0(0) o106->fir-OST0006@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551740991 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 15:09:51 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 04 15:10:12 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551741005/real 1551741005] req@ffff9861161cb300 x1625287428414432/t0(0) o106->fir-OST0008@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551741012 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 15:10:12 fir-io1-s1 kernel: Lustre: 94241:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 04 15:10:53 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client dcb86fb4-7db4-0ce1-485c-0eeddf39ad86 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480315fc00, cur 1551741053 expire 1551740903 last 1551740826 Mar 04 15:10:53 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 04 15:10:54 fir-io1-s1 kernel: Lustre: 36982:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551741047/real 1551741047] req@ffff985ed9486f00 x1625287428414528/t0(0) o106->fir-OST0002@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551741054 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 04 15:10:54 fir-io1-s1 kernel: Lustre: 36982:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 04 15:22:24 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 15:25:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 405a922c-acd7-226c-f53f-840190be85ca (at 10.9.115.9@o2ib4) Mar 04 15:25:08 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 04 15:31:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 10ff98e7-db66-2848-037f-7cf095a0e8cc (at 10.9.113.8@o2ib4) Mar 04 15:31:52 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 04 15:36:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b9de1fd1-ccbe-721f-e4ab-c6e06447a81c (at 10.8.15.4@o2ib6) Mar 04 15:36:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 15:51:02 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 226decb4-3af3-306b-f4f4-3292992cbdda (at 10.9.112.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b914c00, cur 1551743462 expire 1551743312 last 1551743235 Mar 04 15:51:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 15:58:16 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 76418224-15e7-b0fc-3a21-b63821db4ba2 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aabc000, cur 1551743896 expire 1551743746 last 1551743669 Mar 04 15:58:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 15:59:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 42972161-8da5-ab5e-137e-0a713bc25252 (at 10.9.115.12@o2ib4) Mar 04 15:59:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 16:10:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 42972161-8da5-ab5e-137e-0a713bc25252 (at 10.9.115.12@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872ffc11400, cur 1551744613 expire 1551744463 last 1551744386 Mar 04 16:10:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 16:17:55 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d804f75e-ba61-df60-4a59-78156f4e3979 (at 10.9.102.11@o2ib4) Mar 04 16:17:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 16:23:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.112.7@o2ib4) Mar 04 16:23:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 16:25:25 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 16:33:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ec424152-93a3-b508-3dac-8d336ae33ab0 (at 10.8.19.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834b16400, cur 1551746034 expire 1551745884 last 1551745807 Mar 04 16:33:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 16:34:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a1a32c6e-a603-c7ad-bbfa-4137583a5bae (at 10.8.17.20@o2ib6) Mar 04 16:34:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 16:43:47 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 16:55:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 890f9dc9-b9bc-0354-4c1a-b7392d8a9570 (at 10.8.19.5@o2ib6) Mar 04 16:55:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 17:01:12 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:01:20 fir-io1-s1 kernel: LNetError: 91392:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:05:23 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:09:49 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:13:23 fir-io1-s1 kernel: LNetError: 91391:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:48:15 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 59d71ff7-ade7-843e-5b69-c7624aa5f5e3 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762496c00, cur 1551750495 expire 1551750345 last 1551750268 Mar 04 17:48:15 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 04 17:54:38 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:55:22 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 269165af-a224-2f3d-aaca-da4af355fe33 (at 10.8.14.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833dac000, cur 1551750922 expire 1551750772 last 1551750695 Mar 04 17:55:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 17:55:32 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:59:27 fir-io1-s1 kernel: LNetError: 91391:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 17:59:40 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 509fbd7c-080f-2337-9b21-eae4f41debd6 (at 10.9.105.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825f8000, cur 1551751180 expire 1551751030 last 1551750953 Mar 04 17:59:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 18:02:45 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 18:16:45 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 42972161-8da5-ab5e-137e-0a713bc25252 (at 10.9.115.12@o2ib4) Mar 04 18:16:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 18:22:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa3988d8-312e-baa0-298b-1666a8960425 (at 10.8.14.2@o2ib6) Mar 04 18:22:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 18:25:08 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 18:28:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8afa4d73-b615-9d7b-2406-cc6b88c60ce6 (at 10.9.105.22@o2ib4) Mar 04 18:28:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 04 19:14:25 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 19:20:51 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 20:52:18 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 04 21:28:55 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 05 04:31:54 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 05 04:58:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 22d99b60-2a4e-2fde-f10c-47abdacf27b6 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867839e6800, cur 1551790707 expire 1551790557 last 1551790480 Mar 05 04:58:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 05:02:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 05 05:02:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 05:20:31 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client dae02b42-0317-6169-1e09-754bbef76d7e (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984818e9e400, cur 1551792031 expire 1551791881 last 1551791804 Mar 05 05:20:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 05:23:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 05 05:23:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:31:41 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ed1030ab-9163-c6e4-4799-ffcf1b8a94d2 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851091bac00, cur 1551796301 expire 1551796151 last 1551796074 Mar 05 06:31:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:33:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 05 06:33:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:36:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ccf8c675-719e-dc90-28ac-9778eed6125a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c07000, cur 1551796593 expire 1551796443 last 1551796366 Mar 05 06:36:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:37:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 05 06:37:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:46:17 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a243be12-b749-ef3f-66cf-66916b23acb6 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801a8800, cur 1551797177 expire 1551797027 last 1551796950 Mar 05 06:46:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:46:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 05 06:46:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:55:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 464e3a0e-456f-045e-6073-ff5f574d0afc (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5ac00, cur 1551797744 expire 1551797594 last 1551797517 Mar 05 06:55:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 06:58:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 05 06:58:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 07:01:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 86b7858b-210b-7c2b-a691-3063fd374c40 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0e800, cur 1551798113 expire 1551797963 last 1551797886 Mar 05 07:01:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 07:02:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 05 07:02:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:14:04 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e8acaf6a-d7fc-fda6-3737-7e52f1d4f137 (at 10.9.115.3@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499a65d800, cur 1551806044 expire 1551805894 last 1551805817 Mar 05 09:14:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:14:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aff71b74-7050-1a79-ef86-3b2a0fea26d1 (at 10.8.9.4@o2ib6) Mar 05 09:14:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:15:20 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b4992add-a87c-745a-8ea2-200cb8631aa8 (at 10.9.113.11@o2ib4) in 180 seconds. I think it's dead, and I am evicting it. exp ffff9872e8fc3000, cur 1551806120 expire 1551805970 last 1551805940 Mar 05 09:15:20 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Mar 05 09:36:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d9d3ac4e-0fb3-be83-7c67-dfe4c97facfb (at 10.9.114.9@o2ib4) Mar 05 09:36:10 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 05 09:37:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9d90a6a6-e463-02e9-3fef-fe0fa60e4307 (at 10.9.114.13@o2ib4) Mar 05 09:37:07 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 05 09:37:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 78f38211-beb8-aca3-b985-9281f7d5f62c (at 10.8.29.8@o2ib6) Mar 05 09:37:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:37:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fbe187bd-3c7e-1c1e-2397-90b673b213a7 (at 10.9.115.3@o2ib4) Mar 05 09:37:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:38:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 18759a8b-1d8a-beb8-df84-89689c8aa9e2 (at 10.9.113.11@o2ib4) Mar 05 09:38:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:39:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e265f84a-d19d-6fce-343c-d86c6eba2d5b (at 10.8.29.3@o2ib6) Mar 05 09:39:12 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 05 09:39:17 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d9061385-b84d-90d5-bed4-40468f5bd328 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780d58800, cur 1551807557 expire 1551807407 last 1551807330 Mar 05 09:39:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:40:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 05 09:40:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 09:41:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 05 09:41:14 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 05 09:43:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ff20c7b7-4d81-8856-8ae3-3529cc25f5bc (at 10.9.114.6@o2ib4) Mar 05 09:43:42 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 05 09:45:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25940368-3cb8-9782-644f-f19d28d165f3 (at 10.8.28.11@o2ib6) Mar 05 09:45:59 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 05 09:54:13 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 05 11:47:34 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 05 11:47:34 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.203@o2ib7 (0): c: 0, oc: 0, rc: 8 Mar 05 11:47:34 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.203@o2ib7: accepting Mar 05 11:47:34 fir-io1-s1 kernel: Lustre: 110618:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551815247/real 1551815247] req@ffff98381ee73900 x1625322916599680/t0(0) o106->fir-OST0008@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551815254 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 05 11:47:34 fir-io1-s1 kernel: Lustre: 110618:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Mar 05 11:47:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Client 4c980a2e-dcb3-1f84-3d84-3d72a4ae40d6 (at 10.8.10.22@o2ib6) reconnecting Mar 05 11:47:37 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 14c211dc-def9-30b3-8703-6d8fa95aeff3 (at 10.8.10.22@o2ib6) Mar 05 11:47:37 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 05 11:47:39 fir-io1-s1 kernel: Lustre: fir-OST000a: Client ac6b7868-9631-d06b-4e97-5105f55c80aa (at 10.8.21.20@o2ib6) reconnecting Mar 05 11:47:45 fir-io1-s1 kernel: Lustre: 110615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551815258/real 1551815258] req@ffff98380db98c00 x1625322917847056/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551815265 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 05 11:47:45 fir-io1-s1 kernel: Lustre: 110615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 100 previous similar messages Mar 05 11:47:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 9832d5da-7e28-bd8c-fe17-34d245fb168a (at 10.8.3.13@o2ib6) reconnecting Mar 05 11:47:57 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 05 11:48:05 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551815277/real 1551815277] req@ffff986edcdcc200 x1625322916792848/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551815284 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 05 11:48:05 fir-io1-s1 kernel: Lustre: 97131:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 316 previous similar messages Mar 05 11:48:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 6c4d496b-8615-5f70-05e8-3d4c968a99d2 (at 10.8.4.3@o2ib6) reconnecting Mar 05 11:48:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 56748acc-0114-c8d8-7a9c-c6362a34cc0f (at 10.8.4.3@o2ib6) Mar 05 11:48:16 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 05 11:48:42 fir-io1-s1 kernel: Lustre: 96277:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551815315/real 1551815315] req@ffff9864eb505700 x1625322922325216/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551815322 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 05 11:48:42 fir-io1-s1 kernel: Lustre: 96277:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 838 previous similar messages Mar 05 11:49:36 fir-io1-s1 kernel: LustreError: 96332:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff9838416c3f00 x1625322920491872 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff9857319bcc80/0x49e1861740ec5e68 lrc: 4/0,0 mode: PR/PR res: [0x6c0000402:0x1286a3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x3eaa6240a2422b79 expref: 401 pid: 96345 timeout: 2163220 lvb_type: 1 Mar 05 11:49:36 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 05 11:49:36 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 05 11:49:36 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff9857319bcc80/0x49e1861740ec5e68 lrc: 3/0,0 mode: PR/PR res: [0x6c0000402:0x1286a3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x3eaa6240a2422b79 expref: 402 pid: 96345 timeout: 0 lvb_type: 1 Mar 05 11:49:37 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Mar 05 11:49:57 fir-io1-s1 kernel: Lustre: 74817:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551815390/real 1551815390] req@ffff9866e1651b00 x1625322918678464/t0(0) o106->fir-OST0008@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551815397 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 05 11:49:57 fir-io1-s1 kernel: Lustre: 74817:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1954 previous similar messages Mar 05 11:50:48 fir-io1-s1 kernel: LNet: Service thread pid 96913 was inactive for 200.36s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 05 11:50:48 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 05 11:50:48 fir-io1-s1 kernel: Pid: 96913, comm: ll_ost01_093 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 05 11:50:48 fir-io1-s1 kernel: Call Trace: Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 05 11:50:48 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 05 11:50:48 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815448.96913 Mar 05 11:50:49 fir-io1-s1 kernel: LNet: Service thread pid 96491 was inactive for 200.80s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 05 11:50:49 fir-io1-s1 kernel: Pid: 96491, comm: ll_ost01_058 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 05 11:50:49 fir-io1-s1 kernel: Call Trace: Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 05 11:50:49 fir-io1-s1 kernel: Pid: 74693, comm: ll_ost03_044 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 05 11:50:49 fir-io1-s1 kernel: Call Trace: Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 05 11:50:49 fir-io1-s1 kernel: Pid: 96894, comm: ll_ost01_083 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 05 11:50:49 fir-io1-s1 kernel: Call Trace: Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 05 11:50:49 fir-io1-s1 kernel: Pid: 110655, comm: ll_ost03_097 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 05 11:50:49 fir-io1-s1 kernel: Call Trace: Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 05 11:50:49 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 05 11:50:49 fir-io1-s1 kernel: LNet: Service thread pid 96941 was inactive for 201.22s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 05 11:50:49 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 05 11:50:50 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815450.97131 Mar 05 11:50:52 fir-io1-s1 kernel: LNet: Service thread pid 96281 was inactive for 200.20s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 05 11:50:52 fir-io1-s1 kernel: LNet: Skipped 26 previous similar messages Mar 05 11:50:52 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815452.96281 Mar 05 11:50:53 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815453.96272 Mar 05 11:50:54 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815454.96284 Mar 05 11:50:55 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815455.75602 Mar 05 11:50:56 fir-io1-s1 kernel: LNet: Service thread pid 110657 was inactive for 200.28s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 05 11:50:56 fir-io1-s1 kernel: LNet: Skipped 39 previous similar messages Mar 05 11:50:56 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815456.110657 Mar 05 11:50:58 fir-io1-s1 kernel: LustreError: 110034:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff98380d6c9500 x1625322924493184 status 0 rc -110), evict it ns: filter-fir-OST000a_UUID lock: ffff9843affd21c0/0x49e1861740ebb02c lrc: 4/0,0 mode: PR/PR res: [0x580000402:0x128cf5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x3eaa6240a2414d16 expref: 403 pid: 96763 timeout: 2163301 lvb_type: 1 Mar 05 11:50:58 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 05 11:50:58 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 05 11:50:58 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff9857d37d3180/0x49e1861740ebfbc7 lrc: 3/0,0 mode: PR/PR res: [0xc80000401:0x12891c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x3eaa6240a241b464 expref: 397 pid: 96366 timeout: 0 lvb_type: 1 Mar 05 11:50:58 fir-io1-s1 kernel: LNet: Service thread pid 110618 completed after 210.27s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 05 11:50:58 fir-io1-s1 kernel: LustreError: 110034:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Mar 05 11:50:59 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815459.110659 Mar 05 11:51:00 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815460.96262 Mar 05 11:51:02 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815462.110035 Mar 05 11:51:04 fir-io1-s1 kernel: LNet: Service thread pid 96374 was inactive for 200.03s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 05 11:51:04 fir-io1-s1 kernel: LNet: Skipped 17 previous similar messages Mar 05 11:51:04 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815464.96374 Mar 05 11:51:06 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815466.96355 Mar 05 11:51:07 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815467.96753 Mar 05 11:51:09 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815469.96897 Mar 05 11:51:10 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551815470.96893 Mar 05 11:51:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a45cc417-6e9d-7879-c92e-7d387da5101f (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ecc00, cur 1551815472 expire 1551815322 last 1551815245 Mar 05 11:51:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 11:51:12 fir-io1-s1 kernel: LNet: Service thread pid 110574 completed after 216.57s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 05 11:51:12 fir-io1-s1 kernel: LNet: Skipped 96 previous similar messages Mar 05 11:54:58 fir-io1-s1 kernel: Lustre: 94315:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551815687/real 1551815687] req@ffff985efd243900 x1625323021340512/t0(0) o106->fir-OST0008@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551815698 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 05 11:54:58 fir-io1-s1 kernel: Lustre: 94315:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1870 previous similar messages Mar 05 11:57:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 05 11:57:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 11:57:29 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 42fb20f9-8193-98e6-771a-3cdc42eb63ab (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833dac000, cur 1551815849 expire 1551815699 last 1551815622 Mar 05 11:57:29 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 05 12:04:18 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:04:18 fir-io1-s1 kernel: Lustre: Skipped 19 previous similar messages Mar 05 12:04:52 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 27271a2a-2e8d-26b8-c55a-a4a039c2287f (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853ec086c00, cur 1551816292 expire 1551816142 last 1551816065 Mar 05 12:04:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 12:05:09 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 6 seconds Mar 05 12:05:09 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 1 previous similar message Mar 05 12:05:09 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1551816308/real 1551816309] req@ffff98381ccc1b00 x1625323160862896/t0(0) o400->fir-MDT0003-lwp-OST0002@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1551817064 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Mar 05 12:05:09 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0002: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:05:09 fir-io1-s1 kernel: Lustre: 91457:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 56 previous similar messages Mar 05 12:05:12 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 9 seconds Mar 05 12:05:12 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 8 previous similar messages Mar 05 12:05:14 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:05:14 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 05 12:05:14 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 05 12:05:14 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 05 12:05:15 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 05 12:05:15 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 9 previous similar messages Mar 05 12:06:05 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST0000: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Mar 05 12:06:05 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 05 12:09:07 fir-io1-s1 kernel: Lustre: DEBUG MARKER: Tue Mar 5 12:09:07 2019 Mar 05 12:09:51 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 05 12:09:51 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Mar 05 12:09:51 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST000a: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:09:51 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 05 12:09:56 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:09:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 12:09:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 05 12:09:56 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 05 12:10:21 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST000a: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:10:21 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:10:21 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST0000: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Mar 05 12:10:21 fir-io1-s1 kernel: LustreError: Skipped 23 previous similar messages Mar 05 12:10:21 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 05 12:10:46 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST0006: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 05 12:10:46 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 05 12:12:26 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0000: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:12:26 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 05 12:12:26 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Mar 05 12:12:52 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 20 seconds Mar 05 12:12:52 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 22 previous similar messages Mar 05 12:13:16 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST000a: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Mar 05 12:13:16 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1339822 to 0x0:1339841 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1340050 to 0x0:1340065 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1339759 to 0x0:1339777 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1339369 to 0x0:1339393 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1340219 to 0x0:1340257 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1339578 to 0x0:1339617 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2127559 to 0x5c0000400:2127681 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2127695 to 0x6c0000400:2127809 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2127158 to 0x8c0000402:2127201 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2127573 to 0xc80000402:2127649 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2127728 to 0x580000400:2127841 Mar 05 12:27:34 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2127488 to 0xc40000402:2127649 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:1567998 to 0x6c0000401:1568161 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:1568456 to 0x580000401:1568609 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:1567877 to 0x8c0000400:1568257 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:1568285 to 0x5c0000401:1568577 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:1567843 to 0xc80000400:1568097 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:1568248 to 0xc40000400:1568449 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1218950 to 0x5c0000402:1219009 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1217610 to 0x6c0000402:1217633 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1217557 to 0xc80000401:1217601 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1217478 to 0xc40000401:1217505 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1219035 to 0x580000402:1219073 Mar 05 12:27:35 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1217656 to 0x8c0000401:1217697 Mar 05 12:27:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 89311c25-5811-466c-95ed-d7a183bd4753 (at 10.9.113.15@o2ib4) Mar 05 12:27:50 fir-io1-s1 kernel: Lustre: Skipped 97 previous similar messages Mar 05 13:10:48 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 45f975f5-f789-a873-c015-ae15bc675a94 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848831e5400, cur 1551820248 expire 1551820098 last 1551820021 Mar 05 13:10:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 05 13:31:02 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e32f72f4-c5c5-f90b-cd2e-68cad1fab269 (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756581c00, cur 1551821462 expire 1551821312 last 1551821235 Mar 05 13:31:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 14:05:27 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client efe9b694-4084-eb0c-3f2e-2e0f9e94b243 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98663589f000, cur 1551823527 expire 1551823377 last 1551823300 Mar 05 14:05:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 14:05:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 05 14:05:53 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Mar 05 14:30:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dafe1c8-3c93-e104-2f19-ffcaca2d90cd (at 10.8.16.5@o2ib6) Mar 05 14:30:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 14:55:04 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6e62f6e7-5644-5697-4ad0-73cc186d6667 (at 10.8.19.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852e4704c00, cur 1551826504 expire 1551826354 last 1551826277 Mar 05 14:55:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 05 15:15:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6f23ad32-0dd1-26f7-1bbe-7fefdeb50a2a (at 10.8.19.1@o2ib6) Mar 05 15:15:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 15:15:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a4f2c4c1-03c5-819a-6852-1875c7d76a33 (at 10.8.19.7@o2ib6) Mar 05 15:15:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 15:15:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7d64396e-7a86-2e01-38b5-8f4fd2cfeb04 (at 10.8.19.4@o2ib6) Mar 05 15:15:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 05 15:16:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7b159774-9739-84d0-f0ac-fcc62a72d585 (at 10.8.19.6@o2ib6) Mar 05 15:16:09 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 05 15:51:29 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2ac12646-4ace-a274-aebb-64d4c0a4cd11 (at 10.8.16.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499a658800, cur 1551829889 expire 1551829739 last 1551829662 Mar 05 15:51:29 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 05 16:12:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 24d9d171-71c4-5c61-0044-18351be88bb7 (at 10.9.114.7@o2ib4) Mar 05 16:12:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:14:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2e8b1a97-514f-63aa-1bc8-051eadecacf0 (at 10.9.112.9@o2ib4) Mar 05 16:14:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:15:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.16.7@o2ib6) Mar 05 16:15:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:16:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.112.17@o2ib4) Mar 05 16:16:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:16:11 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to cfaf9282-7f7c-20ef-2a1a-f242e378dd7c (at 10.8.16.8@o2ib6) Mar 05 16:16:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:17:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0cda1ca9-b849-4b59-7ce7-48abe2de3c2e (at 10.8.15.6@o2ib6) Mar 05 16:17:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:18:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8feac3a9-5d0e-b456-91aa-b72196a5e39e (at 10.8.29.4@o2ib6) Mar 05 16:18:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:20:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e18eed75-ce52-cc42-be69-772ded053e90 (at 10.8.13.23@o2ib6) Mar 05 16:20:14 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 05 16:22:59 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 194ae90b-dda5-aeec-e623-aa1c27f6c383 (at 10.8.17.21@o2ib6) Mar 05 16:22:59 fir-io1-s1 kernel: Lustre: Skipped 32 previous similar messages Mar 05 16:35:45 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1cbf77f1-25f5-e39f-f802-ab9685fcc565 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983bb7f1b800, cur 1551832545 expire 1551832395 last 1551832318 Mar 05 16:35:45 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 05 16:36:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 05 16:36:57 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Mar 05 16:45:22 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 81f119af-ef4c-a5ba-2165-3ffecd9adf1d (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768928c00, cur 1551833122 expire 1551832972 last 1551832895 Mar 05 16:45:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 16:45:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 81f119af-ef4c-a5ba-2165-3ffecd9adf1d (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576892c000, cur 1551833127 expire 1551832977 last 1551832900 Mar 05 16:45:27 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 05 16:47:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 05 16:47:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 17:09:09 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client eaadbbe1-299f-9091-7703-87d51ddf2f5e (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd261800, cur 1551834549 expire 1551834399 last 1551834322 Mar 05 17:10:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 05 17:10:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 17:17:46 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5842c050-f05d-e125-ef5c-cbec4dcaeec1 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678680f800, cur 1551835066 expire 1551834916 last 1551834839 Mar 05 17:17:46 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 05 17:17:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5842c050-f05d-e125-ef5c-cbec4dcaeec1 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683528c000, cur 1551835075 expire 1551834925 last 1551834848 Mar 05 17:17:55 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 05 17:18:09 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 05 17:18:09 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 05 17:18:30 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 05 17:19:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 1ddc34e0-8d72-2819-3f0e-cf5cc38a3f01 (at 10.8.3.11@o2ib6) in 223 seconds. I think it's dead, and I am evicting it. exp ffff986786a89400, cur 1551835142 expire 1551834992 last 1551834919 Mar 05 17:19:02 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 05 17:21:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 05 17:21:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 05 17:43:16 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 356145f6-88c7-4312-bf14-a1c8e6241d11 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c7000, cur 1551836596 expire 1551836446 last 1551836369 Mar 05 17:43:16 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 05 17:44:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 05 17:44:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 20:28:26 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 454fbbf9-12da-520a-173d-fd5848df4987 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986abbf65c00, cur 1551846506 expire 1551846356 last 1551846279 Mar 05 20:28:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 20:28:30 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 454fbbf9-12da-520a-173d-fd5848df4987 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986abbf63800, cur 1551846510 expire 1551846360 last 1551846283 Mar 05 20:28:30 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 05 20:29:42 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9bf83cf6-e3e2-0c7d-81d3-d11c1f5d0772 (at 10.8.26.33@o2ib6) in 213 seconds. I think it's dead, and I am evicting it. exp ffff98622f65f800, cur 1551846582 expire 1551846432 last 1551846369 Mar 05 20:29:42 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 05 20:29:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9bf83cf6-e3e2-0c7d-81d3-d11c1f5d0772 (at 10.8.26.33@o2ib6) in 217 seconds. I think it's dead, and I am evicting it. exp ffff98622f65f400, cur 1551846586 expire 1551846436 last 1551846369 Mar 05 20:29:46 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 05 20:30:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 05 20:30:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 05 21:36:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 56b68211-62eb-4598-300c-1cfaa46678f5 (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b1dbc00, cur 1551850595 expire 1551850445 last 1551850368 Mar 05 21:36:35 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 05 23:01:47 fir-io1-s1 kernel: Lustre: fir-OST000a: Client 36e39237-5b34-0fcd-d3aa-6021119926c9 (at 10.8.2.3@o2ib6) reconnecting Mar 05 23:01:47 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to (at 10.8.2.3@o2ib6) Mar 05 23:02:28 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.0.10.52@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 05 23:02:28 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 05 23:47:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2551084f-85c2-eb45-3339-ecc679dbd245 (at 10.9.112.16@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c7c00, cur 1551858469 expire 1551858319 last 1551858242 Mar 05 23:47:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 00:28:20 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client aa69e66b-7b29-b583-b8a4-1b71ae4db96b (at 10.8.29.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b09c00, cur 1551860900 expire 1551860750 last 1551860673 Mar 06 00:28:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 00:51:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4f6b42bd-8393-db19-0238-71ebc8ff53fb (at 10.8.29.1@o2ib6) Mar 06 00:51:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 02:32:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:32:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:32:47 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:32:47 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:32:47 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:32:49 fir-io1-s1 kernel: Lustre: fir-OST0004: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:32:49 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:32:49 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:32:49 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 02:32:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:32:57 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:33:11 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:33:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:33:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:33:33 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:33:42 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000b_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:33:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:33:53 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:33:53 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:33:53 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:34:18 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:34:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:34:18 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:34:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:34:18 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 02:34:59 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:34:59 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:34:59 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:34:59 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 02:36:06 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000b_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:36:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:36:30 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:36:30 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:36:30 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 02:39:55 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0005_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:40:14 fir-io1-s1 kernel: LustreError: 137-5: fir-OST000b_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:40:17 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df5800, cur 1551868817 expire 1551868667 last 1551868590 Mar 06 02:40:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 02:40:40 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986781f70800, cur 1551868840 expire 1551868690 last 1551868613 Mar 06 02:40:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984836f9dc00, cur 1551868857 expire 1551868707 last 1551868630 Mar 06 02:40:57 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:41:20 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:41:20 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 06 02:41:20 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:41:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 02:41:45 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9838570c5800, cur 1551868905 expire 1551868755 last 1551868678 Mar 06 02:41:45 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:45:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a04a5c00, cur 1551869107 expire 1551868957 last 1551868880 Mar 06 02:45:33 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0003_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:45:47 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:45:47 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 02:47:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855e70d8400, cur 1551869243 expire 1551869093 last 1551869016 Mar 06 02:48:14 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785785000, cur 1551869294 expire 1551869144 last 1551869067 Mar 06 02:49:34 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d05c00, cur 1551869374 expire 1551869224 last 1551869147 Mar 06 02:51:33 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0005_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:51:33 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 06 02:54:33 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0003_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 02:54:33 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 06 02:54:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 06 02:54:33 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 06 02:55:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98729e1cdc00, cur 1551869752 expire 1551869602 last 1551869525 Mar 06 02:55:52 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 02:56:33 fir-io1-s1 kernel: Lustre: fir-OST0004: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:56:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 02:57:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 06 02:57:03 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 02:59:16 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 06 03:01:18 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 06 04:16:19 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 06 08:03:19 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 955a12ad-d47f-2e77-4d0f-4e609867b4e4 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872e2c8b000, cur 1551888199 expire 1551888049 last 1551887972 Mar 06 08:05:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 06 08:05:26 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Mar 06 08:51:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) Mar 06 08:51:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 08:52:34 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 03274ab3-51e1-91f4-7aab-02414a98d375 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c6dc000, cur 1551891154 expire 1551891004 last 1551890927 Mar 06 08:52:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 08:57:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 665b3c07-f583-a6a2-a031-36b81104a696 (at 10.8.9.3@o2ib6) Mar 06 08:57:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 08:58:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3e279d24-e2fb-196e-d7ac-e1a73db143bd (at 10.9.112.16@o2ib4) Mar 06 08:58:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 09:54:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2f54289c-64ae-044a-7714-538c5857dcb6 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833e2ec00, cur 1551894877 expire 1551894727 last 1551894650 Mar 06 09:54:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 09:55:05 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2f54289c-64ae-044a-7714-538c5857dcb6 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851091be000, cur 1551894905 expire 1551894755 last 1551894678 Mar 06 09:55:05 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 09:57:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 06 09:57:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 09:59:35 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7cfee449-cbe1-eeb7-e2d4-9eae3b1ec7fa (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da5c00, cur 1551895175 expire 1551895025 last 1551894948 Mar 06 09:59:35 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 06 10:02:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 06 10:02:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 10:07:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 06 10:07:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 10:07:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3779e905-f536-2bd5-31bb-d82fcaa2a331 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984830fb1000, cur 1551895676 expire 1551895526 last 1551895449 Mar 06 10:07:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 10:10:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 06 10:10:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 10:21:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f86aeb09-ae8e-4ad0-4fed-e311276ed408 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986abbf65400, cur 1551896516 expire 1551896366 last 1551896289 Mar 06 10:21:56 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 06 10:26:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e2c3a1c5-1166-f4ba-7363-9f11ce6ad590 (at 10.8.1.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983bb7f1f400, cur 1551896802 expire 1551896652 last 1551896575 Mar 06 10:26:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 11:41:21 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 06 11:41:21 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (0): c: 0, oc: 1, rc: 8 Mar 06 11:41:21 fir-io1-s1 kernel: Lustre: 96758:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551901274/real 1551901274] req@ffff986cc7eada00 x1625358987312752/t0(0) o104->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1551901281 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 06 11:41:21 fir-io1-s1 kernel: Lustre: 96758:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 26 previous similar messages Mar 06 11:41:23 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 788839fa-568f-a17f-42e0-7b342b16adad (at 10.8.30.36@o2ib6) reconnecting Mar 06 11:41:23 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 06 11:41:23 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 71464f83-f435-3a33-e9d6-ef54166e95b7 (at 10.8.30.36@o2ib6) Mar 06 11:41:28 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 7e37805b-730d-90a2-5253-267b5633a1b9 (at 10.8.26.24@o2ib6) reconnecting Mar 06 11:41:28 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 06 11:42:00 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 0c8ae5c6-8ce2-6ec4-7c6f-f7830a242d1d (at 10.8.25.20@o2ib6) reconnecting Mar 06 11:42:00 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 11:42:00 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to d7132a19-51b8-e098-d0d8-a2755039375a (at 10.8.25.20@o2ib6) Mar 06 11:42:00 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 06 11:42:21 fir-io1-s1 kernel: LustreError: 96502:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ truncated bulk READ 0(131072) req@ffff98677f471850 x1626126789625008/t0(0) o3->922abd5f-de8e-e1b5-0240-342fbac1018b@10.8.7.15@o2ib6:359/0 lens 488/440 e 1 to 0 dl 1551901349 ref 1 fl Interpret:/0/0 rc 0/0 Mar 06 11:42:21 fir-io1-s1 kernel: Lustre: fir-OST0002: Bulk IO read error with 922abd5f-de8e-e1b5-0240-342fbac1018b (at 10.8.7.15@o2ib6), client will retry: rc -110 Mar 06 11:42:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 06 11:42:25 fir-io1-s1 kernel: Lustre: fir-OST000a: Client 21625bfb-5fdd-2e81-aee5-6854266bb8c6 (at 10.8.1.4@o2ib6) reconnecting Mar 06 11:42:25 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 028d6433-9e7d-1b84-c8b7-1bb2a8570ec4 (at 10.8.1.4@o2ib6) Mar 06 11:42:36 fir-io1-s1 kernel: Lustre: 96892:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551901349/real 1551901349] req@ffff9869d308ce00 x1625358987880720/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551901356 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 06 11:42:36 fir-io1-s1 kernel: Lustre: 96892:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1402 previous similar messages Mar 06 11:42:59 fir-io1-s1 kernel: LustreError: 96758:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff986cc7eada00 x1625358987312752 status 0 rc -110), evict it ns: filter-fir-OST0002_UUID lock: ffff9867270cb180/0x49e1861b7fba936d lrc: 4/0,0 mode: PR/PR res: [0x5c0000402:0x128d72:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xbb83aa3b740921b0 expref: 475 pid: 77317 timeout: 2249223 lvb_type: 1 Mar 06 11:42:59 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 06 11:42:59 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 06 11:42:59 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff9867270cb180/0x49e1861b7fba936d lrc: 3/0,0 mode: PR/PR res: [0x5c0000402:0x128d72:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xbb83aa3b740921b0 expref: 476 pid: 77317 timeout: 0 lvb_type: 1 Mar 06 11:42:59 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Mar 06 11:43:42 fir-io1-s1 kernel: LustreError: 96406:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff98652b93e600 x1625358987642000 status 0 rc -110), evict it ns: filter-fir-OST000a_UUID lock: ffff984c38795e80/0x49e1861b809cc8d7 lrc: 4/0,0 mode: PR/PR res: [0x580000402:0x128e46:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xbb83aa3b742175d4 expref: 482 pid: 96352 timeout: 2249266 lvb_type: 1 Mar 06 11:43:42 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 06 11:43:42 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff984c38795e80/0x49e1861b809cc8d7 lrc: 3/0,0 mode: PR/PR res: [0x580000402:0x128e46:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xbb83aa3b742175d4 expref: 483 pid: 96352 timeout: 0 lvb_type: 1 Mar 06 11:44:34 fir-io1-s1 kernel: LNet: Service thread pid 96378 was inactive for 200.04s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 06 11:44:34 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 06 11:44:34 fir-io1-s1 kernel: Pid: 96378, comm: ll_ost01_051 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 11:44:34 fir-io1-s1 kernel: Call Trace: Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 11:44:34 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 11:44:34 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551901474.96378 Mar 06 11:44:35 fir-io1-s1 kernel: LNet: Service thread pid 96355 was inactive for 200.47s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 06 11:44:35 fir-io1-s1 kernel: Pid: 96355, comm: ll_ost01_033 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 11:44:35 fir-io1-s1 kernel: Call Trace: Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 11:44:35 fir-io1-s1 kernel: Pid: 96251, comm: ll_ost01_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 11:44:35 fir-io1-s1 kernel: Call Trace: Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 11:44:35 fir-io1-s1 kernel: Pid: 96895, comm: ll_ost01_084 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 11:44:35 fir-io1-s1 kernel: Call Trace: Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 11:44:35 fir-io1-s1 kernel: Pid: 110611, comm: ll_ost03_096 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 11:44:35 fir-io1-s1 kernel: Call Trace: Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 11:44:35 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 11:44:35 fir-io1-s1 kernel: LNet: Service thread pid 96370 was inactive for 200.42s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 06 11:44:35 fir-io1-s1 kernel: LNet: Skipped 9 previous similar messages Mar 06 11:44:36 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551901476.94242 Mar 06 11:44:41 fir-io1-s1 kernel: LNet: Service thread pid 49830 was inactive for 200.70s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 06 11:44:41 fir-io1-s1 kernel: LNet: Skipped 14 previous similar messages Mar 06 11:44:41 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551901481.49830 Mar 06 11:44:42 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551901482.96375 Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: 76197:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff986d43930300 x1625358988114320 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff98573c42bcc0/0x49e1861b80704881 lrc: 4/0,0 mode: PR/PR res: [0xc80000401:0x15d86e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xbb83aa3b74148fef expref: 468 pid: 96274 timeout: 2249328 lvb_type: 1 Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff985d99f23600/0x49e1861b80705420 lrc: 3/0,0 mode: PR/PR res: [0xc40000401:0x15d7b5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0xbb83aa3b741496f6 expref: 470 pid: 96275 timeout: 0 lvb_type: 1 Mar 06 11:44:44 fir-io1-s1 kernel: LNet: Service thread pid 96779 completed after 202.46s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: 96768:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff98382a6a2100 x1625358988360992/t0(0) o104->fir-OST0006@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: 96768:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 15 previous similar messages Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551901484.96277 Mar 06 11:44:44 fir-io1-s1 kernel: LustreError: 76197:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 7 previous similar messages Mar 06 11:45:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 06 11:45:56 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 06 11:48:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 174708c4-2035-5f47-9345-1e2b543b6ecd (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868053e8000, cur 1551901738 expire 1551901588 last 1551901511 Mar 06 11:48:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 12:08:33 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ea1c6a9b-5f49-871d-cbdb-4024a9d55acd (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499a65e400, cur 1551902913 expire 1551902763 last 1551902686 Mar 06 12:08:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 12:10:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 06 12:10:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 12:12:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4e6ca6b6-e639-4e3e-8f61-daee777c74af (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009f9400, cur 1551903121 expire 1551902971 last 1551902894 Mar 06 12:12:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 12:12:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 519ac832-a77f-51ba-e3c7-51aa4fe15024 (at 10.8.15.3@o2ib6) Mar 06 12:12:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 06 12:12:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 06 12:12:30 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 06 12:16:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0cda1ca9-b849-4b59-7ce7-48abe2de3c2e (at 10.8.15.6@o2ib6) Mar 06 12:16:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 12:17:27 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c43e1e7e-8fd7-4682-8a75-e3b63df6bc56 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785cdb800, cur 1551903447 expire 1551903297 last 1551903220 Mar 06 12:17:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 06 13:46:24 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 39a58eae-eb29-e7fd-c4d5-e4f8c77d3ee1 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387bd800, cur 1551908784 expire 1551908634 last 1551908557 Mar 06 13:46:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 13:46:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 39a58eae-eb29-e7fd-c4d5-e4f8c77d3ee1 (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043fd000, cur 1551908790 expire 1551908640 last 1551908563 Mar 06 13:46:30 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 14:10:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b2490c9c-99c6-33de-b033-7716166c911d (at 10.8.17.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848349e2c00, cur 1551910257 expire 1551910107 last 1551910030 Mar 06 14:10:57 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 06 15:00:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7b7e02d1-56ce-1646-fdf2-fbf074562774 (at 10.8.17.29@o2ib6) Mar 06 15:00:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 15:59:37 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 97d16432-8aea-593a-5164-e9768bd80f35 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4ed800, cur 1551916777 expire 1551916627 last 1551916550 Mar 06 15:59:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 15:59:52 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 97d16432-8aea-593a-5164-e9768bd80f35 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864683df400, cur 1551916792 expire 1551916642 last 1551916565 Mar 06 16:00:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 06 16:00:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 16:11:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 629af712-405c-38fc-fd16-12b82a1542fa (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986bc31cfc00, cur 1551917483 expire 1551917333 last 1551917256 Mar 06 16:11:23 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 06 16:44:16 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9feff43d-52bc-edd9-6b96-08d840556472 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df1000, cur 1551919456 expire 1551919306 last 1551919229 Mar 06 16:44:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 16:44:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 06 16:44:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 17:19:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6cbee5ee-2697-c3ae-18b4-1709bf98ed42 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009fcc00, cur 1551921564 expire 1551921414 last 1551921337 Mar 06 17:19:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 17:19:43 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 6cbee5ee-2697-c3ae-18b4-1709bf98ed42 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853ec083800, cur 1551921583 expire 1551921433 last 1551921356 Mar 06 17:19:43 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 17:19:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6cbee5ee-2697-c3ae-18b4-1709bf98ed42 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851091bd400, cur 1551921584 expire 1551921434 last 1551921357 Mar 06 17:19:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 17:19:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 06 17:19:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 17:34:19 fir-io1-s1 kernel: Lustre: 96920:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551922452/real 1551922452] req@ffff986765393c00 x1625368887847952/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551922459 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 06 17:34:19 fir-io1-s1 kernel: Lustre: 96920:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3199 previous similar messages Mar 06 17:34:38 fir-io1-s1 kernel: Lustre: 96897:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551922471/real 1551922471] req@ffff9862d7974500 x1625368887861024/t0(0) o106->fir-OST000a@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551922478 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 06 17:34:38 fir-io1-s1 kernel: Lustre: 96897:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 202 previous similar messages Mar 06 17:35:17 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551922510/real 1551922510] req@ffff985b58b74e00 x1625368887857296/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551922517 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 06 17:35:17 fir-io1-s1 kernel: Lustre: 96524:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 488 previous similar messages Mar 06 17:36:32 fir-io1-s1 kernel: Lustre: 110035:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551922585/real 1551922585] req@ffff98383b409b00 x1625368887864720/t0(0) o106->fir-OST0004@10.8.3.11@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551922592 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 06 17:36:32 fir-io1-s1 kernel: Lustre: 110035:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1420 previous similar messages Mar 06 17:37:12 fir-io1-s1 kernel: LustreError: 96277:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff985b58b77b00 x1625368887962448 status 0 rc -110), evict it ns: filter-fir-OST000a_UUID lock: ffff984e2b147bc0/0x49e1861c9c5b0292 lrc: 4/0,0 mode: PR/PR res: [0x580000402:0x15de43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x8659065f1f9e8091 expref: 685 pid: 96615 timeout: 2270476 lvb_type: 1 Mar 06 17:37:12 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 06 17:37:12 fir-io1-s1 kernel: LustreError: Skipped 6 previous similar messages Mar 06 17:37:12 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff984e2b147bc0/0x49e1861c9c5b0292 lrc: 3/0,0 mode: PR/PR res: [0x580000402:0x15de43:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x8659065f1f9e8091 expref: 686 pid: 96615 timeout: 0 lvb_type: 1 Mar 06 17:37:12 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 7 previous similar messages Mar 06 17:37:32 fir-io1-s1 kernel: LNet: Service thread pid 96409 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 06 17:37:32 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 06 17:37:32 fir-io1-s1 kernel: Pid: 96409, comm: ll_ost02_029 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 17:37:32 fir-io1-s1 kernel: Call Trace: Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 17:37:32 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 17:37:32 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551922652.96409 Mar 06 17:37:33 fir-io1-s1 kernel: LNet: Service thread pid 96750 was inactive for 200.77s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 06 17:37:33 fir-io1-s1 kernel: Pid: 96750, comm: ll_ost03_013 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 17:37:33 fir-io1-s1 kernel: Call Trace: Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 17:37:33 fir-io1-s1 kernel: Pid: 109956, comm: ll_ost03_062 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 17:37:33 fir-io1-s1 kernel: Call Trace: Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 17:37:33 fir-io1-s1 kernel: Pid: 110637, comm: ll_ost02_103 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 17:37:33 fir-io1-s1 kernel: Call Trace: Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 17:37:33 fir-io1-s1 kernel: Pid: 96766, comm: ll_ost02_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 17:37:33 fir-io1-s1 kernel: Call Trace: Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 17:37:33 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 17:37:34 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 17:37:34 fir-io1-s1 kernel: LNet: Service thread pid 110635 was inactive for 201.04s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 06 17:37:34 fir-io1-s1 kernel: LNet: Skipped 49 previous similar messages Mar 06 17:37:34 fir-io1-s1 kernel: LNet: Service thread pid 96353 was inactive for 200.32s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 06 17:37:34 fir-io1-s1 kernel: LNet: Skipped 16 previous similar messages Mar 06 17:37:34 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551922654.96353 Mar 06 17:37:38 fir-io1-s1 kernel: LNet: Service thread pid 94931 was inactive for 200.30s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 06 17:37:38 fir-io1-s1 kernel: LNet: Skipped 5 previous similar messages Mar 06 17:37:38 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551922658.94931 Mar 06 17:37:39 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551922659.97126 Mar 06 17:37:40 fir-io1-s1 kernel: LNet: Service thread pid 96264 was inactive for 200.56s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 06 17:37:40 fir-io1-s1 kernel: LNet: Skipped 26 previous similar messages Mar 06 17:37:40 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551922660.96264 Mar 06 17:37:41 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551922661.74749 Mar 06 17:37:42 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551922662.96912 Mar 06 17:37:42 fir-io1-s1 kernel: LustreError: 96260:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.11@o2ib6) failed to reply to blocking AST (req@ffff9838163cb000 x1625368887975568 status 0 rc -110), evict it ns: filter-fir-OST0002_UUID lock: ffff983d8d964c80/0x49e1861c9c5bba2f lrc: 4/0,0 mode: PR/PR res: [0x5c0000402:0x128cfc:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x8659065f1f9f3f97 expref: 670 pid: 49822 timeout: 2270506 lvb_type: 1 Mar 06 17:37:42 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.3.11@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 06 17:37:42 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.11@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff983850009200/0x49e1861c9c5b0906 lrc: 3/0,0 mode: PR/PR res: [0x5c0000402:0x15de32:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.11@o2ib6 remote: 0x8659065f1f9e86c6 expref: 671 pid: 49822 timeout: 0 lvb_type: 1 Mar 06 17:37:42 fir-io1-s1 kernel: LNet: Service thread pid 49824 completed after 210.18s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 06 17:37:42 fir-io1-s1 kernel: LNet: Skipped 64 previous similar messages Mar 06 17:37:42 fir-io1-s1 kernel: LustreError: 111317:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff983838348000 x1625368888131120/t0(0) o104->fir-OST0002@10.8.3.11@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 06 17:37:42 fir-io1-s1 kernel: LustreError: 111317:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 53 previous similar messages Mar 06 17:37:42 fir-io1-s1 kernel: LustreError: 96260:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 9 previous similar messages Mar 06 17:44:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 06 17:44:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 17:51:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e2a8fad8-2ee0-ac58-a48b-f3ad3293e7f7 (at 10.8.3.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986671180400, cur 1551923504 expire 1551923354 last 1551923277 Mar 06 17:53:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ba4f6ab-da5b-8d5b-7e10-cb73b415cd02 (at 10.8.3.11@o2ib6) Mar 06 17:53:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 18:19:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 15347712-fa4a-1343-e88d-528cd4677216 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f232800, cur 1551925149 expire 1551924999 last 1551924922 Mar 06 18:19:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 18:21:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 06 18:21:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 18:27:25 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5a13b3c1-e031-151c-91bd-b11b654f56ed (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dffe400, cur 1551925645 expire 1551925495 last 1551925418 Mar 06 18:27:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 18:30:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 06 18:30:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 18:53:41 fir-io1-s1 kernel: LustreError: 96366:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from blocking AST (req@ffff984e0a6e3f00 x1625370734641792 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff98514e9d5580/0x49e1861ccd572593 lrc: 4/0,0 mode: PR/PR res: [0x5c0000400:0x20af7b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.27.23@o2ib6 remote: 0x83b5b92692b0441 expref: 470 pid: 96939 timeout: 2275122 lvb_type: 1 Mar 06 18:53:41 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 06 18:53:41 fir-io1-s1 kernel: LustreError: Skipped 9 previous similar messages Mar 06 18:53:41 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff98514e9d5580/0x49e1861ccd572593 lrc: 3/0,0 mode: PR/PR res: [0x5c0000400:0x20af7b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.27.23@o2ib6 remote: 0x83b5b92692b0441 expref: 471 pid: 96939 timeout: 0 lvb_type: 1 Mar 06 18:53:41 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 8 previous similar messages Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: 113340:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9870b617c500 x1625370734815824/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: 113340:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 90 previous similar messages Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: 75602:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from glimpse AST (req@ffff9866f2f93600 x1625370734921984 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff985fd02aa880/0x49e1861ccd69e0a8 lrc: 3/0,0 mode: PW/PW res: [0x580000400:0x20b1f5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x40000000000000 nid: 10.8.27.23@o2ib6 remote: 0x83b5b92692ce320 expref: 495 pid: 94316 timeout: 0 lvb_type: 0 Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: 75602:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 5 previous similar messages Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1551927222s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff985fd02aa880/0x49e1861ccd69e0a8 lrc: 3/0,0 mode: PW/PW res: [0x580000400:0x20b1f5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x40000000000000 nid: 10.8.27.23@o2ib6 remote: 0x83b5b92692ce320 expref: 496 pid: 94316 timeout: 0 lvb_type: 0 Mar 06 18:53:42 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages Mar 06 18:54:19 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 20bfa6d3-7d6c-dd24-caf5-5767ec86945c (at 10.8.11.9@o2ib6) in 200 seconds. I think it's dead, and I am evicting it. exp ffff985767577800, cur 1551927259 expire 1551927109 last 1551927059 Mar 06 18:54:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 18:54:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 06 18:54:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 18:59:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 06 18:59:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 19:02:43 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.8.17@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 06 19:02:43 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 06 19:03:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Client d8f0824e-3e17-1861-9567-873d13c6a482 (at 10.8.8.17@o2ib6) reconnecting Mar 06 19:03:24 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to dd0510e7-9ffb-771c-249b-7b72018d8d01 (at 10.8.8.17@o2ib6) Mar 06 19:03:24 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to dd0510e7-9ffb-771c-249b-7b72018d8d01 (at 10.8.8.17@o2ib6) Mar 06 19:03:24 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 19:03:24 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 19:03:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 19:19:39 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ab27b6dd-4be0-e6cc-cbea-663cfc7aa2a4 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801ab400, cur 1551928779 expire 1551928629 last 1551928552 Mar 06 19:19:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 19:23:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 06 19:23:18 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Mar 06 19:31:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 882ea567-b387-7339-4a34-4e3ebe57f8da (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867844b0400, cur 1551929500 expire 1551929350 last 1551929273 Mar 06 19:31:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 19:34:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 06 19:34:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 19:45:28 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0872fb2c-3070-5151-1654-62990b8a4800 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984de97cb000, cur 1551930328 expire 1551930178 last 1551930101 Mar 06 19:45:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 19:46:05 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 06 19:46:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 19:46:15 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 06 20:04:26 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 06 20:24:31 fir-io1-s1 kernel: Lustre: 96934:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551932664/real 1551932664] req@ffff986e557cfb00 x1625372696541904/t0(0) o106->fir-OST0002@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551932671 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 06 20:24:31 fir-io1-s1 kernel: Lustre: 96934:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1770 previous similar messages Mar 06 20:24:51 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551932683/real 1551932683] req@ffff98380a3d1800 x1625372697996304/t0(0) o106->fir-OST000a@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551932690 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 06 20:24:51 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 06 20:25:33 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551932726/real 1551932726] req@ffff98380a3d1800 x1625372697996304/t0(0) o106->fir-OST000a@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551932733 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 06 20:25:33 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Mar 06 20:26:50 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551932803/real 1551932803] req@ffff98380a3d1800 x1625372697996304/t0(0) o106->fir-OST000a@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551932810 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 06 20:26:50 fir-io1-s1 kernel: Lustre: 49820:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 54 previous similar messages Mar 06 20:27:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 07f8dac1-fada-74a2-c38a-46ab693624c2 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800e4ec00, cur 1551932857 expire 1551932707 last 1551932630 Mar 06 20:27:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 20:27:45 fir-io1-s1 kernel: LNet: Service thread pid 96515 was inactive for 200.25s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 06 20:27:45 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 06 20:27:45 fir-io1-s1 kernel: Pid: 96515, comm: ll_ost01_059 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 20:27:45 fir-io1-s1 kernel: Call Trace: Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 20:27:45 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1551932865.96515 Mar 06 20:27:45 fir-io1-s1 kernel: Pid: 49824, comm: ll_ost00_074 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 06 20:27:45 fir-io1-s1 kernel: Call Trace: Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 06 20:27:45 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 06 20:27:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 07f8dac1-fada-74a2-c38a-46ab693624c2 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c25c00, cur 1551932868 expire 1551932718 last 1551932641 Mar 06 20:27:48 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 20:27:49 fir-io1-s1 kernel: LNet: Service thread pid 96515 completed after 204.16s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 06 20:27:49 fir-io1-s1 kernel: LNet: Skipped 69 previous similar messages Mar 06 20:27:50 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 07f8dac1-fada-74a2-c38a-46ab693624c2 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800e4b000, cur 1551932870 expire 1551932720 last 1551932643 Mar 06 20:27:50 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 20:27:50 fir-io1-s1 kernel: LNet: Service thread pid 49824 completed after 205.19s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 06 20:30:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 06 20:30:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 20:45:02 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9ff1e81a-4451-e3af-dac0-7b467b953562 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986025047800, cur 1551933902 expire 1551933752 last 1551933675 Mar 06 20:45:02 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 06 21:40:22 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 270badd4-2c5d-850f-f81e-6e2f88765c8a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c674400, cur 1551937222 expire 1551937072 last 1551936995 Mar 06 21:40:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 21:40:32 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 270badd4-2c5d-850f-f81e-6e2f88765c8a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318e9800, cur 1551937232 expire 1551937082 last 1551937005 Mar 06 21:40:32 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 21:40:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 06 21:40:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 21:51:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Mar 06 21:51:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 21:51:32 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1baeec15-ec06-5bfc-7415-4ec528dfc90c (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785784400, cur 1551937892 expire 1551937742 last 1551937665 Mar 06 21:51:32 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 22:02:59 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5bd8f075-dc7c-2f9a-f3b8-1d76b21b787d (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9838570c1400, cur 1551938579 expire 1551938429 last 1551938352 Mar 06 22:02:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 06 22:03:11 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5bd8f075-dc7c-2f9a-f3b8-1d76b21b787d (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985764d64800, cur 1551938591 expire 1551938441 last 1551938364 Mar 06 22:03:11 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 06 22:03:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 06 22:03:25 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 06 23:34:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 06 23:34:41 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 06 23:35:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 85f8d21d-7213-dd4e-cc12-7bbf08b14c89 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b9c800, cur 1551944113 expire 1551943963 last 1551943886 Mar 06 23:35:13 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 07 01:07:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 63a7d96b-fa0d-42ed-39ab-3ad67ce49214 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4eb000, cur 1551949625 expire 1551949475 last 1551949398 Mar 07 01:07:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 01:07:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 07 01:07:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 01:59:47 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.9.9@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 07 01:59:47 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 07 02:00:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 07 02:00:12 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 07 02:00:12 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 07 02:00:57 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767c4a000, cur 1551952857 expire 1551952707 last 1551952630 Mar 07 02:00:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 02:01:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) reconnecting Mar 07 02:01:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 07 02:01:02 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 07 02:01:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 289e9466-48b8-1720-3fb2-fad6998826d4 (at 10.8.9.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bd0a000, cur 1551952865 expire 1551952715 last 1551952638 Mar 07 02:02:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 07 02:02:35 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to a650fb27-cfe4-742f-7959-475c51cb8a54 (at 10.8.9.9@o2ib6) Mar 07 02:21:34 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 04:07:48 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 59c9e16b-29d3-2697-b81f-b4d01e8c5927 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e910000, cur 1551960468 expire 1551960318 last 1551960241 Mar 07 04:10:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 07 04:10:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 04:17:09 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client cd87368f-9291-fe59-bc18-8bb024f45e39 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987686cdac00, cur 1551961029 expire 1551960879 last 1551960802 Mar 07 04:17:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 04:19:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 07 04:19:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 08:15:01 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 7b7e02d1-56ce-1646-fdf2-fbf074562774 (at 10.8.17.29@o2ib6) Mar 07 08:15:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 08:25:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Mar 07 08:25:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 08:55:41 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 11:30:03 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 11:43:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c22e763b-2712-8624-a4bb-1c3145d32fd9 (at 10.8.1.14@o2ib6) Mar 07 11:43:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 11:49:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0da28f8e-2048-4a68-a8f0-a79e798fb7bf (at 10.9.113.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986025044c00, cur 1551988158 expire 1551988008 last 1551987931 Mar 07 11:49:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 11:55:25 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0347d150-6ffe-e388-b010-ef256fe6fea0 (at 10.8.22.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985dff65f800, cur 1551988525 expire 1551988375 last 1551988298 Mar 07 11:55:25 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 07 11:55:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b15137cf-311a-3865-c282-8f1cad0a5e07 (at 10.8.30.14@o2ib6) Mar 07 11:55:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 11:55:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ab1f55c6-25d8-b2cf-818d-4cc69ca36dd0 (at 10.8.22.28@o2ib6) Mar 07 11:55:52 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 07 11:55:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 71eebf52-b0fc-514f-7aaf-aca66e4f2af1 (at 10.8.27.27@o2ib6) Mar 07 11:55:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 11:56:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to eca85409-9804-5945-41db-99b8b236d7bc (at 10.8.7.30@o2ib6) Mar 07 11:56:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 11:56:41 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d9d268bc-88cb-c9fc-e4bf-68c5c0db87a8 (at 10.8.18.18@o2ib6) in 203 seconds. I think it's dead, and I am evicting it. exp ffff9857590d5000, cur 1551988601 expire 1551988451 last 1551988398 Mar 07 11:56:41 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 07 11:57:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 531debd8-6dde-8101-c0c5-b86120a894b1 (at 10.8.18.20@o2ib6) Mar 07 11:57:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 07 11:58:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 880716fb-6a2b-3d87-95fb-03534cabe92d (at 10.8.8.28@o2ib6) Mar 07 11:58:25 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 07 11:59:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6bfa800a-c0c5-71b7-464d-e3efc0c0229b (at 10.8.8.11@o2ib6) Mar 07 11:59:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 12:04:01 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f2e69785-833e-4c3c-f013-d1ee8a07730d (at 10.8.27.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b917000, cur 1551989041 expire 1551988891 last 1551988814 Mar 07 12:04:01 fir-io1-s1 kernel: Lustre: Skipped 101 previous similar messages Mar 07 12:07:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c017fa08-6b03-f233-f83d-69ef225b6bd8 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd443400, cur 1551989255 expire 1551989105 last 1551989028 Mar 07 12:07:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 12:08:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 07 12:08:01 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 07 12:13:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6179b8f9-2b15-63f8-671c-eb8d9ecd9187 (at 10.8.16.3@o2ib6) Mar 07 12:13:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 12:19:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 70fba524-1486-a60e-6bc9-6cdcc41e09a1 (at 10.9.113.13@o2ib4) Mar 07 12:19:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 12:20:42 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a0e541fb-4a1c-dd39-1cf5-d380241f0613 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576998cc00, cur 1551990042 expire 1551989892 last 1551989815 Mar 07 12:20:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 12:23:14 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 12:34:46 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Mar 07 12:34:46 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Mar 07 12:37:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c6e5f4cd-a836-ec33-a945-cc77fbc895d4 (at 10.9.114.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499a65e000, cur 1551991050 expire 1551990900 last 1551990823 Mar 07 12:37:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 12:39:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3702fcd2-63ea-b9a2-4b87-e2016e25ec8d (at 10.9.114.4@o2ib4) Mar 07 12:39:18 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 07 13:18:32 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 13:38:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 142a6e04-872a-1c7a-e7fe-9168d1c1b90d (at 10.8.9.5@o2ib6) Mar 07 13:38:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 13:46:26 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 14:06:43 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 77c42f3f-9f3b-7360-ccfc-1c8ee877d09f (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864683da800, cur 1551996403 expire 1551996253 last 1551996176 Mar 07 14:06:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 14:07:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 07 14:07:18 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 07 14:16:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 71979329-1d78-fe3f-70ff-8f19fdbea09c (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98567fe63000, cur 1551997016 expire 1551996866 last 1551996789 Mar 07 14:16:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 14:17:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 07 14:17:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 14:32:54 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 091b1288-4aa8-27d5-148d-05b47d0021aa (at 10.9.103.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800b0b400, cur 1551997974 expire 1551997824 last 1551997747 Mar 07 14:32:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 14:34:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 935e7eb3-1ff6-7dda-ab9a-d14a4b5f1855 (at 10.9.103.32@o2ib4) Mar 07 14:34:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 14:45:32 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551998725/real 1551998725] req@ffff985b3fc79e00 x1625402159197920/t0(0) o106->fir-OST0004@10.8.3.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551998732 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 07 14:45:32 fir-io1-s1 kernel: Lustre: 96778:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Mar 07 14:45:52 fir-io1-s1 kernel: Lustre: 96920:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551998745/real 1551998745] req@ffff985a5e3c3900 x1625402160879744/t0(0) o106->fir-OST0004@10.8.3.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551998752 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 07 14:45:52 fir-io1-s1 kernel: Lustre: 96920:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 244 previous similar messages Mar 07 14:46:30 fir-io1-s1 kernel: Lustre: 129902:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551998783/real 1551998783] req@ffff986e44634800 x1625402161024848/t0(0) o106->fir-OST0006@10.8.3.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551998790 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 07 14:46:30 fir-io1-s1 kernel: Lustre: 129902:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 903 previous similar messages Mar 07 14:46:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0d03aecb-1fd1-5c04-b891-e5290855a03a (at 10.8.2.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848001fd800, cur 1551998800 expire 1551998650 last 1551998573 Mar 07 14:46:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 14:47:16 fir-io1-s1 kernel: LustreError: 94316:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.3@o2ib6) failed to reply to blocking AST (req@ffff985be46c6000 x1625402159402416 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff9840b359da00/0x49e1862023c2b302 lrc: 4/0,0 mode: PR/PR res: [0xc80000401:0x16dc58:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.3@o2ib6 remote: 0x753cfe35077104f6 expref: 705 pid: 96899 timeout: 2346680 lvb_type: 1 Mar 07 14:47:16 fir-io1-s1 kernel: LustreError: 94316:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 07 14:47:16 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.3.3@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 07 14:47:16 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 07 14:47:16 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.3@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff9840b359da00/0x49e1862023c2b302 lrc: 3/0,0 mode: PR/PR res: [0xc80000401:0x16dc58:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.3@o2ib6 remote: 0x753cfe35077104f6 expref: 706 pid: 96899 timeout: 0 lvb_type: 1 Mar 07 14:47:16 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Mar 07 14:47:25 fir-io1-s1 kernel: LustreError: 49825:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.3@o2ib6) failed to reply to blocking AST (req@ffff98381ae6d700 x1625402160551552 status 0 rc -110), evict it ns: filter-fir-OST000a_UUID lock: ffff98382c8b69c0/0x49e1862023ad2add lrc: 4/0,0 mode: PR/PR res: [0x580000402:0x15de69:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.3@o2ib6 remote: 0x753cfe3507686d28 expref: 702 pid: 96899 timeout: 2346688 lvb_type: 1 Mar 07 14:47:25 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.3.3@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 07 14:47:25 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.3@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff98382c8b69c0/0x49e1862023ad2add lrc: 3/0,0 mode: PR/PR res: [0x580000402:0x15de69:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.3@o2ib6 remote: 0x753cfe3507686d28 expref: 703 pid: 96899 timeout: 0 lvb_type: 1 Mar 07 14:47:45 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551998858/real 1551998858] req@ffff985e1b1f2a00 x1625402160457328/t0(0) o106->fir-OST0006@10.8.3.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551998865 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 07 14:47:45 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1551998858/real 1551998858] req@ffff985b3fc78f00 x1625402160457296/t0(0) o106->fir-OST0002@10.8.3.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1551998865 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 07 14:47:45 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2224 previous similar messages Mar 07 14:47:56 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4b284abe-1cd8-3760-00fb-5ecc097613ba (at 10.8.3.18@o2ib6) in 155 seconds. I think it's dead, and I am evicting it. exp ffff9867811fd400, cur 1551998876 expire 1551998726 last 1551998721 Mar 07 14:47:56 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 07 14:49:00 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4b284abe-1cd8-3760-00fb-5ecc097613ba (at 10.8.3.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858356c9c00, cur 1551998940 expire 1551998790 last 1551998713 Mar 07 14:49:00 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Mar 07 14:58:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 44a2c2a6-781f-61cb-fd7a-5b68752a7816 (at 10.8.6.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986cabdc0000, cur 1551999531 expire 1551999381 last 1551999304 Mar 07 14:59:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 07 14:59:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:19:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 23d2bcaf-f181-5f6e-6636-b07b46e525e0 (at 10.8.3.3@o2ib6) Mar 07 15:19:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:20:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a253e634-6f73-cada-6654-0ab67e1de2bb (at 10.8.3.5@o2ib6) Mar 07 15:20:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:20:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f3bddf34-379e-90f9-b8a3-37dd2323157d (at 10.8.3.18@o2ib6) Mar 07 15:20:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:21:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4128d1d0-036f-499f-293b-8ff4e9479ccf (at 10.8.2.1@o2ib6) Mar 07 15:21:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 07 15:21:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9ca88d6a-bbf2-01c1-11ca-c3f6715dc691 (at 10.8.2.12@o2ib6) Mar 07 15:21:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:23:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e88f100f-7377-9adc-e510-b37b360d1f8e (at 10.8.2.2@o2ib6) Mar 07 15:23:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:24:13 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 18383fc4-5c87-c9a7-2b73-d92b724d96b5 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fb83d000, cur 1552001053 expire 1552000903 last 1552000826 Mar 07 15:24:13 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Mar 07 15:24:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 07 15:24:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:25:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2140fd4e-c658-422a-8c2d-f85ad5c9184c (at 10.8.2.9@o2ib6) Mar 07 15:25:09 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 07 15:33:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7ecda3d5-6e86-7907-ec95-3ef4a7a0262e (at 10.8.6.8@o2ib6) Mar 07 15:33:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:36:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c7d587fd-2878-3a53-bb0b-89a81458bb83 (at 10.8.6.5@o2ib6) Mar 07 15:36:19 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 07 15:37:43 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 15:53:23 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client cc52b071-de23-2dbd-843a-1b9788ff3cfd (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b16800, cur 1552002803 expire 1552002653 last 1552002576 Mar 07 15:53:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 15:55:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 07 15:55:46 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Mar 07 15:58:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client eaeb1c8f-0f27-d224-4778-e2a4ed683084 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868147b1c00, cur 1552003129 expire 1552002979 last 1552002902 Mar 07 15:58:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 16:01:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 07 16:01:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 16:04:36 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 6962a116-6600-ff87-b2bc-57a3843495f1 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986671181000, cur 1552003476 expire 1552003326 last 1552003249 Mar 07 16:04:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 16:08:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 07 16:08:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 17:43:40 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 60bcced1-b8a2-4637-4a76-d20d32ccf3af (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678685f400, cur 1552009420 expire 1552009270 last 1552009193 Mar 07 17:43:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 17:44:56 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8d8c791e-6323-b4c1-3a54-54df121248a0 (at 10.8.3.4@o2ib6) in 160 seconds. I think it's dead, and I am evicting it. exp ffff986c35909000, cur 1552009496 expire 1552009346 last 1552009336 Mar 07 17:44:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 17:46:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 07 17:46:01 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 07 18:18:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ed13e4be-ee2a-09d6-f2c5-8f181f229aea (at 10.8.3.33@o2ib6) Mar 07 18:18:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 18:19:09 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 881bf6fc-5822-2b3d-8911-0d976664a01f (at 10.8.3.24@o2ib6) Mar 07 18:19:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 18:19:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6a2da33a-cfce-6abc-c437-26a227c18c4c (at 10.8.3.4@o2ib6) Mar 07 18:19:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 18:22:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6716b36f-bd58-8af6-cb0b-59c9d183b99d (at 10.8.3.15@o2ib6) Mar 07 18:22:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 18:43:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 30c5eca5-6b78-4ced-60dc-6a5ed6c80363 (at 10.8.6.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987259f5f000, cur 1552013020 expire 1552012870 last 1552012793 Mar 07 18:43:40 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 07 18:45:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f7d79af9-3424-b0d8-6dc6-f23e0df4e16a (at 10.8.6.32@o2ib6) Mar 07 18:45:28 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 07 19:59:15 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 93d3fb60-1179-8522-bef9-642d72744215 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800fd8400, cur 1552017555 expire 1552017405 last 1552017328 Mar 07 19:59:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 20:00:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 07 20:00:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 21:09:52 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 7dadf289-5c47-d17d-d159-563dfce707ed (at 10.8.17.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480f619800, cur 1552021792 expire 1552021642 last 1552021565 Mar 07 21:09:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 21:17:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3a402832-0036-d3a1-814f-0c25fe87ef98 (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98763803c400, cur 1552022230 expire 1552022080 last 1552022003 Mar 07 21:17:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 21:17:19 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3a402832-0036-d3a1-814f-0c25fe87ef98 (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576dc9f400, cur 1552022239 expire 1552022089 last 1552022012 Mar 07 21:17:19 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 07 21:42:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 722fc89e-4fb6-0519-04d0-92f8091a9aa0 (at 10.8.14.5@o2ib6) Mar 07 21:42:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 22:41:23 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 07 23:03:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0cbaddac-7047-2ee7-d70e-1c50532fb4db (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98707f699c00, cur 1552028581 expire 1552028431 last 1552028354 Mar 07 23:03:01 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 07 23:03:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 07 23:03:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 07 23:17:11 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 08 00:38:39 fir-io1-s1 kernel: LNetError: 91392:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 08 00:42:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2da8582c-ee0b-2f7e-e63e-1cda25e5d379 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864d267fc00, cur 1552034520 expire 1552034370 last 1552034293 Mar 08 00:42:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 00:44:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 08 00:44:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 01:42:02 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 08 01:45:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 01:45:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 01:46:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client cb512052-ae1f-fa74-79c4-a40fae633001 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984de97cec00, cur 1552038370 expire 1552038220 last 1552038143 Mar 08 01:46:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 01:48:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e47e6a73-971a-c567-07f7-ec5b5b6c20bb (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848346a7c00, cur 1552038531 expire 1552038381 last 1552038304 Mar 08 01:48:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 01:51:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 01:51:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 01:52:45 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c372b906-6a05-8f19-f7f1-eec327dec93c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762742000, cur 1552038765 expire 1552038615 last 1552038538 Mar 08 01:52:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 01:53:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 08 01:53:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 02:19:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 72f12478-df10-de89-fbe7-dc1e969f4731 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98567fe62400, cur 1552040397 expire 1552040247 last 1552040170 Mar 08 02:19:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 02:22:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 08 02:22:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:00:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:00:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:01:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 51f30e2d-7ae7-e044-77d3-8a1e8104b27d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b908800, cur 1552046507 expire 1552046357 last 1552046280 Mar 08 04:01:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:08:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:08:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:09:23 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 54c4f616-5986-263f-3ea5-956bedc3145e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801513000, cur 1552046963 expire 1552046813 last 1552046736 Mar 08 04:09:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:17:11 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2decda46-4c66-d6b5-aacd-45d466694cb9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4eb000, cur 1552047431 expire 1552047281 last 1552047204 Mar 08 04:17:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:18:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:18:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:23:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:23:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:24:24 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 74423170-edf7-ed47-efb5-a06b23fcfc16 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857590d7c00, cur 1552047864 expire 1552047714 last 1552047637 Mar 08 04:24:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:31:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:31:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:32:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 306fe30f-d210-5954-c988-3146116f9573 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677dc5e000, cur 1552048336 expire 1552048186 last 1552048109 Mar 08 04:32:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:38:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:38:48 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 08 04:39:29 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b2dd119e-7c05-5c62-08fd-15fb62e4c7a9 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848346a4800, cur 1552048769 expire 1552048619 last 1552048542 Mar 08 04:39:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:46:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:46:08 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 08 04:46:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5d3db54d-5a5c-e8fc-b6ac-f7a42609b4f3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868053ea000, cur 1552049208 expire 1552049058 last 1552048981 Mar 08 04:46:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:50:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 34c048ca-f7f5-abef-2125-67a4e23c4ce9 (at 10.8.19.3@o2ib6) Mar 08 04:50:17 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 08 04:50:59 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 30bf6b89-0e78-426d-551f-adfc38aada87 (at 10.8.19.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858356cb400, cur 1552049459 expire 1552049309 last 1552049232 Mar 08 04:50:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 04:53:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 04:53:41 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 08 04:54:33 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 25adcd41-8512-c175-d1e1-6b1d2b1cd24a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677c23f800, cur 1552049673 expire 1552049523 last 1552049446 Mar 08 04:54:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 05:01:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c8c946ed-0648-8f90-8489-24c849b19fea (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987686cddc00, cur 1552050100 expire 1552049950 last 1552049873 Mar 08 05:01:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 05:04:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 05:04:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 05:12:36 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0726594c-853f-71b0-2963-e57a283e30db (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00bf000, cur 1552050756 expire 1552050606 last 1552050529 Mar 08 05:12:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 05:12:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 05:12:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 05:22:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 05:22:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 05:22:54 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c24b7264-4e5b-e8e2-139e-028a96ad2fde (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848387bd800, cur 1552051374 expire 1552051224 last 1552051147 Mar 08 05:22:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 05:36:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 05:36:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 05:36:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client eb5ee943-8747-74a8-e85e-e10fcea14e09 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481a6c2000, cur 1552052179 expire 1552052029 last 1552051952 Mar 08 05:36:19 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 05:57:05 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8dc20368-5420-80a7-65da-50514468552b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b14400, cur 1552053425 expire 1552053275 last 1552053198 Mar 08 05:57:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 05:57:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 05:57:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 06:04:09 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7021eb0f-8d3d-c1c1-ee5e-2d56bc94db5a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283dec00, cur 1552053849 expire 1552053699 last 1552053622 Mar 08 06:04:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:05:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 06:05:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:10:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 06:10:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:11:00 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client dbb6c760-9b6b-276e-ff46-04e77fdee44a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d78400, cur 1552054260 expire 1552054110 last 1552054033 Mar 08 06:11:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:17:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 06:17:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:17:43 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d1d6454a-66a5-d945-aeca-4b0e2b567acb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811f9000, cur 1552054663 expire 1552054513 last 1552054436 Mar 08 06:17:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:32:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 06:32:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:32:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5a3a3f90-e4eb-aa88-45be-f50285e8c43e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864683db400, cur 1552055564 expire 1552055414 last 1552055337 Mar 08 06:32:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 06:50:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 06:50:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 06:51:08 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4a1021e2-e2df-a09e-b75d-d1e3fb23a66b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b615000, cur 1552056668 expire 1552056518 last 1552056441 Mar 08 06:51:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 07:00:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 07:00:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 07:11:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 07:11:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 07:11:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f4ca1eae-b75f-7b4e-4ff3-e87f5ae9e6c2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758dbec00, cur 1552057901 expire 1552057751 last 1552057674 Mar 08 07:11:41 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 07:18:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client da99cdd0-ba72-9852-eb42-0c444201a132 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985c3d661c00, cur 1552058322 expire 1552058172 last 1552058095 Mar 08 07:18:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 07:24:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 07:24:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 07:25:16 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0958a15e-b5b4-6240-842a-cab69c76ead8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984de97cb000, cur 1552058716 expire 1552058566 last 1552058489 Mar 08 07:25:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 07:31:58 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9c032755-1a3f-19bf-e5f6-8c5986acdf87 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987637921000, cur 1552059118 expire 1552058968 last 1552058891 Mar 08 07:31:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 07:44:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 07:44:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 07:45:40 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 90777fc1-f9d6-0238-bcb5-17c4d0e3b68f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d76400, cur 1552059940 expire 1552059790 last 1552059713 Mar 08 07:45:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 07:48:02 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552060075/real 1552060075] req@ffff986a7a9cf800 x1625427997157056/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552060082 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 08 07:48:02 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 395 previous similar messages Mar 08 07:48:23 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552060096/real 1552060096] req@ffff986a7a9cf800 x1625427997157056/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552060103 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 08 07:48:23 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 08 07:49:05 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552060138/real 1552060138] req@ffff986a7a9cf800 x1625427997157056/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552060145 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 08 07:49:05 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 08 07:50:22 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552060215/real 1552060215] req@ffff986a7a9cf800 x1625427997157056/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552060222 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 08 07:50:22 fir-io1-s1 kernel: Lustre: 77323:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 08 07:50:31 fir-io1-s1 kernel: LustreError: 77323:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff986a7a9cf800 x1625427997157056 status -107 rc -107), evict it ns: filter-fir-OST0004_UUID lock: ffff98675392ee40/0x49e18622dcf84428 lrc: 3/0,0 mode: PW/PW res: [0x8c0000401:0x192b7c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0x1fab2d5e29308d16 expref: 6 pid: 74799 timeout: 0 lvb_type: 0 Mar 08 07:50:32 fir-io1-s1 kernel: LustreError: 77323:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Mar 08 07:50:32 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 08 07:50:32 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 08 07:50:32 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552060232s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff98675392ee40/0x49e18622dcf84428 lrc: 3/0,0 mode: PW/PW res: [0x8c0000401:0x192b7c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0x1fab2d5e29308d16 expref: 7 pid: 74799 timeout: 0 lvb_type: 0 Mar 08 07:50:32 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Mar 08 07:58:57 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d50f39f4-7ac5-0662-181e-3f5159301b62 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986112875c00, cur 1552060737 expire 1552060587 last 1552060510 Mar 08 07:58:57 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 08 07:59:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 07:59:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 08:19:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0cbd69f8-7db9-4e74-f386-0be09d9f13df (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed05400, cur 1552061962 expire 1552061812 last 1552061735 Mar 08 08:19:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 08:20:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 08:20:00 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 08:25:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ffc33f5a-4083-9888-23b8-cdb08c8008e6 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c348d8400, cur 1552062304 expire 1552062154 last 1552062077 Mar 08 08:25:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 08:25:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 08:25:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 08:32:03 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1cf22e07-9c32-adf6-a3d7-7fc6d6995041 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c26400, cur 1552062723 expire 1552062573 last 1552062496 Mar 08 08:32:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 08:33:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 08:33:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 08:40:34 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 514e1ad2-830d-b992-d22a-6ef3b4584ecb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043fac00, cur 1552063234 expire 1552063084 last 1552063007 Mar 08 08:40:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 08:41:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 08:41:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 08:51:59 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b50c9ccd-239a-945b-c0b0-1950904e1e43 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009fbc00, cur 1552063919 expire 1552063769 last 1552063692 Mar 08 08:51:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 08:53:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 08:53:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 09:12:06 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2095c080-1330-6c68-15cd-a283a1e4475a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786972800, cur 1552065126 expire 1552064976 last 1552064899 Mar 08 09:12:06 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 09:13:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 09:13:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 09:25:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 09:25:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 09:32:46 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 84237bde-e2c5-4c7b-183a-5b8dd6409a09 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c26400, cur 1552066366 expire 1552066216 last 1552066139 Mar 08 09:32:46 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 10:16:25 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 73f5ff4c-a60a-da9d-d561-392011648e82 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b90bc00, cur 1552068985 expire 1552068835 last 1552068758 Mar 08 10:16:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 10:18:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 10:18:14 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 08 10:22:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) Mar 08 10:22:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:23:10 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 29c98107-7ff9-7c72-992d-aadaebcf1e74 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885f800, cur 1552069390 expire 1552069240 last 1552069163 Mar 08 10:23:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:24:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 05b5c064-83e9-ff93-834e-85f45f17fbc2 (at 10.8.20.15@o2ib6) in 220 seconds. I think it's dead, and I am evicting it. exp ffff9867868fe000, cur 1552069466 expire 1552069316 last 1552069246 Mar 08 10:24:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:25:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 10:25:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:37:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 10:37:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:37:59 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 65f22875-06db-bf8b-8df4-646bb79e1608 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758dbec00, cur 1552070279 expire 1552070129 last 1552070052 Mar 08 10:37:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:45:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 10201175-75f7-c520-6e68-b0f406ab519f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ea47400, cur 1552070720 expire 1552070570 last 1552070493 Mar 08 10:45:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:58:38 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client af11da21-69e0-3df1-e886-bb4753ef7ebe (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858356c9000, cur 1552071518 expire 1552071368 last 1552071291 Mar 08 10:58:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 10:59:54 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0773a2df-8ddc-60dc-b1be-285a06036993 (at 10.8.21.21@o2ib6) in 183 seconds. I think it's dead, and I am evicting it. exp ffff985762a5c000, cur 1552071594 expire 1552071444 last 1552071411 Mar 08 10:59:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:00:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 08 11:00:59 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 11:11:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 11:11:44 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 11:12:02 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client cf63db3d-df6d-0b01-a19e-6c42ec7a3cde (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d71000, cur 1552072322 expire 1552072172 last 1552072095 Mar 08 11:12:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:30:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9138a1d2-4693-575e-f914-a45c81e70f27 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985765730400, cur 1552073458 expire 1552073308 last 1552073231 Mar 08 11:30:58 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 11:31:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 08 11:31:30 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 11:34:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 29a66ca3-a611-f9d1-074d-4f159ef2007b (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f54bcd000, cur 1552073644 expire 1552073494 last 1552073417 Mar 08 11:34:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:34:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 08 11:34:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:40:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8d5d6766-8cdf-2906-1530-54584a9c8235 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985761f47000, cur 1552074012 expire 1552073862 last 1552073785 Mar 08 11:40:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:40:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 11:40:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:45:31 fir-io1-s1 kernel: LustreError: 96921:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff98383837cb00 x1625431370309152 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff985c9a4033c0/0x49e1862335c8a9f5 lrc: 3/0,0 mode: PW/PW res: [0x6c0000402:0x19688e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000080020000 nid: 10.8.20.15@o2ib6 remote: 0x1e7349c90fde8e0e expref: 6 pid: 94241 timeout: 0 lvb_type: 0 Mar 08 11:45:31 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 08 11:45:31 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 08 11:45:31 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552074331s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff985c9a400fc0/0x49e1862335c8a9cb lrc: 3/0,0 mode: PW/PW res: [0x5c0000402:0x196da1:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000080020000 nid: 10.8.20.15@o2ib6 remote: 0x1e7349c90fde8dd6 expref: 8 pid: 94241 timeout: 0 lvb_type: 0 Mar 08 11:45:31 fir-io1-s1 kernel: LustreError: 96921:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages Mar 08 11:45:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 11:45:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:45:46 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1eb15444-a3ac-6a7d-1a0d-eb88358fc955 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811fec00, cur 1552074346 expire 1552074196 last 1552074119 Mar 08 11:45:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 11:52:17 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 4bacdc82-be13-20e0-ec12-202e1016a961 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833f7e000, cur 1552074737 expire 1552074587 last 1552074510 Mar 08 11:52:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 11:52:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:00:28 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6577b7b3-184b-6b60-301a-9252fe99eed0 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a04a6000, cur 1552075228 expire 1552075078 last 1552075001 Mar 08 12:00:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:01:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 12:01:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:19:31 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 956bbe85-13fe-5f3a-789d-985a056ddc21 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985836971400, cur 1552076371 expire 1552076221 last 1552076144 Mar 08 12:19:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:20:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 12:20:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:33:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 12:33:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:35:06 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5ff344b6-ae8d-9e87-ba37-8ecb2d2c32fe (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd443400, cur 1552077306 expire 1552077156 last 1552077079 Mar 08 12:35:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:37:49 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d23552c3-c23d-f86d-ae18-a23f82dd6387 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857590d0800, cur 1552077469 expire 1552077319 last 1552077242 Mar 08 12:37:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:47:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 12:47:33 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 08 12:48:06 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7011d526-5784-7b3e-00e5-8a04606692b0 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c3000, cur 1552078086 expire 1552077936 last 1552077859 Mar 08 12:48:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:55:57 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1c01938d-9f3a-948b-ee02-a94cba4cd272 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985836972400, cur 1552078557 expire 1552078407 last 1552078330 Mar 08 12:55:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 12:56:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1c01938d-9f3a-948b-ee02-a94cba4cd272 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762a58400, cur 1552078560 expire 1552078410 last 1552078333 Mar 08 12:56:00 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 08 13:02:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 423f42d6-7acb-835e-7eaa-aa6b6f1b3566 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864683db400, cur 1552078955 expire 1552078805 last 1552078728 Mar 08 13:02:35 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 08 13:03:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 08 13:03:53 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Mar 08 13:10:14 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d841120d-5e92-6c00-1562-a9d909111766 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d72000, cur 1552079414 expire 1552079264 last 1552079187 Mar 08 13:10:14 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 13:16:38 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 58c079e5-63fb-0d27-5e46-13db4d475f06 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d77400, cur 1552079798 expire 1552079648 last 1552079571 Mar 08 13:16:38 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 13:17:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 13:17:47 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 08 13:29:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 91ca7d78-e45a-aee3-a1a8-5b93e19151ef (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98760f60f400, cur 1552080597 expire 1552080447 last 1552080370 Mar 08 13:29:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 13:31:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f4711ff6-c650-0dc0-da7f-c0dfd15d97eb (at 10.8.21.21@o2ib6) in 205 seconds. I think it's dead, and I am evicting it. exp ffff985838d8b400, cur 1552080673 expire 1552080523 last 1552080468 Mar 08 13:31:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 13:31:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 13:31:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 13:38:49 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9572b543-7f22-2164-c5e9-5eb03b29e9e6 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847ae51f400, cur 1552081129 expire 1552080979 last 1552080902 Mar 08 13:38:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 13:49:10 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cfc18b91-9bde-9b25-9434-24ba94adfe48 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855e70dcc00, cur 1552081750 expire 1552081600 last 1552081523 Mar 08 13:49:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 13:51:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 13:51:15 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Mar 08 13:58:54 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552082327/real 1552082327] req@ffff98754b7d1e00 x1625432732591968/t0(0) o106->fir-OST0002@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552082334 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 08 13:58:54 fir-io1-s1 kernel: Lustre: 96904:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 08 13:59:13 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 81b15ba9-6eba-8e33-5365-d01fe5b11e7e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15f2000, cur 1552082353 expire 1552082203 last 1552082126 Mar 08 13:59:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 13:59:15 fir-io1-s1 kernel: Lustre: 96326:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552082348/real 1552082348] req@ffff9848938bce00 x1625432732591936/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552082355 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 08 13:59:15 fir-io1-s1 kernel: Lustre: 96326:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 08 13:59:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 81b15ba9-6eba-8e33-5365-d01fe5b11e7e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5a400, cur 1552082356 expire 1552082206 last 1552082129 Mar 08 13:59:16 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 08 14:06:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f22edfed-d08d-6746-7c54-8386863e7015 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8265400, cur 1552082813 expire 1552082663 last 1552082586 Mar 08 14:06:53 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 08 14:11:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 08 14:11:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 14:13:59 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 424c4ae7-32b4-ea10-6174-c34db4f9a453 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f7ef800, cur 1552083239 expire 1552083089 last 1552083012 Mar 08 14:13:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 14:16:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 08 14:16:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 14:18:32 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7c250a90-23cc-7e09-6f6b-abe8baca9042 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832bdd000, cur 1552083512 expire 1552083362 last 1552083285 Mar 08 14:18:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 14:22:59 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 53517eb5-2786-0012-6e5a-e2c9ebf3ee90 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98567fe64800, cur 1552083779 expire 1552083629 last 1552083552 Mar 08 14:22:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 14:23:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 08 14:23:48 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 14:34:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 14:34:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 14:34:38 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cd0460e6-3caa-bea2-0dd6-2751a6991196 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043fe400, cur 1552084478 expire 1552084328 last 1552084251 Mar 08 14:34:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 14:41:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d7451531-ed40-8279-f867-2066ab7262a1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8267c00, cur 1552084903 expire 1552084753 last 1552084676 Mar 08 14:41:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 14:57:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 14:57:28 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 08 14:57:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2c89558c-a007-ee48-1ca3-adde1563fd14 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986cce295400, cur 1552085878 expire 1552085728 last 1552085651 Mar 08 14:57:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 15:04:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 15:04:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 15:05:03 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 07371e8b-609b-348d-3453-e7d819abab5d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872ac81cc00, cur 1552086303 expire 1552086153 last 1552086076 Mar 08 15:05:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 15:20:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 454fd1f5-90f9-dc6e-2b34-df3abb039311 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a00bf400, cur 1552087244 expire 1552087094 last 1552087017 Mar 08 15:20:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 15:21:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 15:21:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 15:35:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3c88106a-1e8e-886d-0959-c1292e65a172 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d75800, cur 1552088109 expire 1552087959 last 1552087882 Mar 08 15:35:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 15:35:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 15:35:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 16:34:45 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST000a: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 08 16:34:45 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 08 16:35:12 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 21 seconds Mar 08 16:35:12 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages Mar 08 16:35:16 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 08 16:35:16 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Mar 08 16:35:37 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 4 seconds Mar 08 16:35:37 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Mar 08 16:35:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 08 16:35:56 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 08 16:35:56 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 08 16:36:26 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 08 16:36:26 fir-io1-s1 kernel: LustreError: Skipped 23 previous similar messages Mar 08 16:36:26 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 08 16:36:26 fir-io1-s1 kernel: Lustre: Skipped 34 previous similar messages Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2198927 to 0x8c0000402:2199009 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2199527 to 0x6c0000400:2199713 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2199412 to 0x5c0000400:2199489 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2199573 to 0x580000400:2199713 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2199373 to 0xc40000402:2199457 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2199403 to 0xc80000402:2199553 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1477992 to 0x0:1478177 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1478010 to 0x0:1478177 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1478491 to 0x0:1478561 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1478243 to 0x0:1478433 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1477599 to 0x0:1477953 Mar 08 16:36:47 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1477880 to 0x0:1478049 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1734730 to 0x6c0000402:1735105 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1734560 to 0xc80000401:1734913 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1734497 to 0xc40000401:1735169 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1735952 to 0x580000402:1736481 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1734560 to 0x8c0000401:1734977 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1736081 to 0x5c0000402:1736609 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:1974846 to 0xc40000400:1974945 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:1975065 to 0x5c0000401:1975137 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:1974590 to 0x6c0000401:1974657 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:1974656 to 0x8c0000400:1974721 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:1974600 to 0xc80000400:1974689 Mar 08 16:36:48 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:1975065 to 0x580000401:1975169 Mar 08 16:47:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 16:47:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 16:49:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ef15fc40-06ad-4d81-8849-08792aa7b11d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a7e915c00, cur 1552092595 expire 1552092445 last 1552092368 Mar 08 16:49:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 16:51:11 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8d0e87a1-8476-96de-9fc9-8732dda3bd3b (at 10.8.21.21@o2ib6) in 178 seconds. I think it's dead, and I am evicting it. exp ffff98582bba6c00, cur 1552092671 expire 1552092521 last 1552092493 Mar 08 16:51:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 16:52:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 08 16:52:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 16:57:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1cf51eae-9595-bf46-2b77-937c9cca4d38 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4f000, cur 1552093076 expire 1552092926 last 1552092849 Mar 08 16:57:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 17:03:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 08 17:03:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 17:32:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9dfa6a95-6868-29a0-3538-576ab8510edb (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984838f86c00, cur 1552095121 expire 1552094971 last 1552094894 Mar 08 17:32:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 17:32:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 08 17:32:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 17:38:19 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client fdbf9ec0-7c1e-a6ec-6370-40b8fbe80566 (at 10.8.15.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904997400, cur 1552095499 expire 1552095349 last 1552095272 Mar 08 17:38:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 17:58:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6) Mar 08 17:58:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 19:12:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c9bd13fe-0c4c-6d29-4d76-419583bbbc0f (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986112870400, cur 1552101141 expire 1552100991 last 1552100914 Mar 08 19:12:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 19:12:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 08 19:12:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 19:33:04 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a8a00305-46cd-f718-c752-a3ee83114eb0 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983f32adfc00, cur 1552102384 expire 1552102234 last 1552102157 Mar 08 19:33:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 19:33:43 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 08 19:33:43 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 08 19:43:22 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1d014e40-f906-ff28-4143-a4a4f53e0556 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575de9b400, cur 1552103002 expire 1552102852 last 1552102775 Mar 08 19:43:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 19:43:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 08 19:43:41 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 08 19:48:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) Mar 08 19:48:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 19:48:22 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d13fb412-792d-7d07-0514-c72219005886 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868f8000, cur 1552103302 expire 1552103152 last 1552103075 Mar 08 19:48:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 20:36:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 56ca1f9f-e666-c26c-fcc9-59b490d74659 (at 10.8.29.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985c3d662800, cur 1552106215 expire 1552106065 last 1552105988 Mar 08 20:36:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 08 20:37:13 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 56ca1f9f-e666-c26c-fcc9-59b490d74659 (at 10.8.29.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983851c2a400, cur 1552106233 expire 1552106083 last 1552106006 Mar 08 20:37:13 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 08 20:38:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 400e2c70-3670-eb05-66c0-e754ea5cd280 (at 10.8.29.7@o2ib6) Mar 08 20:38:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 02:26:03 fir-io1-s1 kernel: AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0003 address=0xfffffffdf8210000 flags=0x0008] Mar 09 10:11:23 fir-io1-s1 kernel: mlx5_0:dump_cqe:286:(pid 91383): dump error cqe Mar 09 10:11:23 fir-io1-s1 kernel: 00000000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Mar 09 10:11:23 fir-io1-s1 kernel: 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Mar 09 10:11:23 fir-io1-s1 kernel: 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Mar 09 10:11:23 fir-io1-s1 kernel: 00000030: 00 00 00 00 00 00 89 14 0a 00 01 cd 2e f4 ef d2 Mar 09 10:11:23 fir-io1-s1 kernel: Lustre: 91463:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1552155083/real 1552155083] req@ffff987584ea8f00 x1625434459303904/t0(0) o400->fir-MDT0001-lwp-OST0006@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1552155090 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Mar 09 10:11:23 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0006: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 09 10:11:23 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 09 10:11:23 fir-io1-s1 kernel: Lustre: 91463:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 09 10:11:30 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Mar 09 10:11:30 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.51@o2ib7 (6): c: 0, oc: 0, rc: 8 Mar 09 10:11:30 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds Mar 09 10:11:30 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 11 previous similar messages Mar 09 10:11:30 fir-io1-s1 kernel: Lustre: 91453:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1552155083/real 1552155090] req@ffff987584ead400 x1625434459303936/t0(0) o400->fir-MDT0002-lwp-OST0002@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1552155090 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Mar 09 10:11:30 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0004: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 09 10:11:30 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 09 10:11:30 fir-io1-s1 kernel: Lustre: 91453:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 09 10:11:30 fir-io1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail Mar 09 10:11:32 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 3 seconds Mar 09 10:11:32 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 13 previous similar messages Mar 09 10:11:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds Mar 09 10:11:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Mar 09 10:11:56 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 15 seconds Mar 09 10:11:56 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 5 previous similar messages Mar 09 10:12:21 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 14 seconds Mar 09 10:12:21 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 24 previous similar messages Mar 09 10:12:47 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 16 seconds Mar 09 10:12:47 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 24 previous similar messages Mar 09 10:13:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 4 seconds Mar 09 10:13:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 49 previous similar messages Mar 09 10:14:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986cce290000, cur 1552155292 expire 1552155142 last 1552155065 Mar 09 10:14:52 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 3 seconds Mar 09 10:14:52 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 74 previous similar messages Mar 09 10:14:52 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 09 10:14:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fir-MDT0001-mdtlov_UUID (at 10.0.10.52@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986cce295800, cur 1552155295 expire 1552155145 last 1552155068 Mar 09 10:14:55 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 09 10:17:45 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 09 10:17:49 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 09 10:17:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 10:18:36 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST0000: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Mar 09 10:18:36 fir-io1-s1 kernel: LustreError: Skipped 23 previous similar messages Mar 09 10:18:36 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 09 10:18:36 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Mar 09 10:18:41 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 09 10:18:41 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 09 10:19:27 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 09 10:19:27 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 09 10:19:27 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 09 10:19:27 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Mar 09 10:19:48 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1741891 to 0x5c0000402:1741921 Mar 09 10:19:48 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1740262 to 0x8c0000401:1740289 Mar 09 10:19:48 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1740390 to 0x6c0000402:1740417 Mar 09 10:19:48 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1740448 to 0xc40000401:1740481 Mar 09 10:19:48 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1740189 to 0xc80000401:1740225 Mar 09 10:19:48 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1741777 to 0x580000402:1741793 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:1988609 to 0x6c0000401:1988641 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:1988642 to 0xc80000400:1988673 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:1988656 to 0x8c0000400:1988673 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:1988893 to 0xc40000400:1988929 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:1989081 to 0x5c0000401:1989121 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:1989116 to 0x580000401:1989153 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1478858 to 0x0:1478881 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1479114 to 0x0:1479137 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1479246 to 0x0:1479265 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1478632 to 0x0:1478657 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1478859 to 0x0:1478881 Mar 09 10:19:49 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1478732 to 0x0:1478753 Mar 09 10:19:50 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2224324 to 0x6c0000400:2224513 Mar 09 10:19:50 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2223598 to 0x8c0000402:2223681 Mar 09 10:19:50 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2224099 to 0x5c0000400:2224161 Mar 09 10:19:50 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2224157 to 0xc80000402:2224225 Mar 09 10:19:50 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2224276 to 0x580000400:2224321 Mar 09 10:19:50 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2224053 to 0xc40000402:2224129 Mar 09 10:25:39 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7176c6f9-3353-7b24-7e87-90f8349019b0 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b658000, cur 1552155939 expire 1552155789 last 1552155712 Mar 09 10:25:39 fir-io1-s1 kernel: Lustre: Skipped 19 previous similar messages Mar 09 14:32:00 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 09 14:32:00 fir-io1-s1 kernel: Lustre: Skipped 36 previous similar messages Mar 09 14:32:26 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 3 seconds Mar 09 14:32:26 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 74 previous similar messages Mar 09 14:33:48 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 09 14:34:30 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 09 14:34:30 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 09 14:34:30 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 09 14:34:30 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 09 14:34:40 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 09 14:35:20 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST000a: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Mar 09 14:35:20 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 09 14:35:20 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST000a: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 09 14:35:20 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1742616 to 0x5c0000402:1742657 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1741112 to 0x6c0000402:1741153 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1741180 to 0xc40000401:1741217 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1740981 to 0x8c0000401:1741025 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1742491 to 0x580000402:1742529 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1740921 to 0xc80000401:1740961 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2227203 to 0x6c0000400:2227233 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2226822 to 0x5c0000400:2226913 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2226368 to 0x8c0000402:2226401 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2226770 to 0xc40000402:2226849 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2227005 to 0x580000400:2227041 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2226903 to 0xc80000402:2226945 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:1994216 to 0x580000401:1994241 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:1993996 to 0xc40000400:1994049 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:1994183 to 0x5c0000401:1994209 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:1993740 to 0x8c0000400:1993825 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:1993735 to 0xc80000400:1993761 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:1993699 to 0x6c0000401:1993729 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1479092 to 0x0:1479137 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1479096 to 0x0:1479137 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1478871 to 0x0:1478913 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1479350 to 0x0:1479393 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1479479 to 0x0:1479521 Mar 09 14:35:49 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1478967 to 0x0:1479009 Mar 09 15:21:46 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7981ec06-5d17-6c17-6f40-ca14f6184db2 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f2c00, cur 1552173706 expire 1552173556 last 1552173479 Mar 09 15:21:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 15:29:57 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 262c736e-815a-6500-9b27-6059240a12b1 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a79400, cur 1552174197 expire 1552174047 last 1552173970 Mar 09 15:29:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 15:38:03 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Mar 09 15:38:03 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (107): c: 8, oc: 0, rc: 8 Mar 09 16:11:46 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 09 17:02:50 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f193cc04-eb04-2533-4647-386f011e30ba (at 10.8.18.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9863d7956800, cur 1552179770 expire 1552179620 last 1552179543 Mar 09 17:02:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 17:04:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f4b4a4f5-e37f-502c-6e38-2cf5d9bf12ad (at 10.8.18.13@o2ib6) in 191 seconds. I think it's dead, and I am evicting it. exp ffff98677e79b400, cur 1552179846 expire 1552179696 last 1552179655 Mar 09 17:04:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 18:29:23 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 10.8.17.12@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff984ef2848fc0/0x49e18621ce6c01b3 lrc: 3/0,0 mode: PW/PW res: [0xc80000402:0x1bbddd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000480030020 nid: 10.8.17.12@o2ib6 remote: 0xa7a0bb5fda12aa58 expref: 78 pid: 96925 timeout: 2532714 lvb_type: 0 Mar 09 18:29:23 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages Mar 09 18:29:48 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to ea7e114e-3d27-0438-c912-7927f0cdf6fc (at 10.8.17.12@o2ib6) Mar 09 18:50:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d5162907-0710-f955-5785-49f22e8d992d (at 10.8.3.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864ac6eb000, cur 1552186251 expire 1552186101 last 1552186024 Mar 09 18:50:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 19:15:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 05eee491-224c-56d9-e69a-55da391ad687 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d7dc00, cur 1552187758 expire 1552187608 last 1552187531 Mar 09 19:15:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 19:17:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 63a3d0a9-3d76-72da-1801-efa0f02606ff (at 10.8.9.3@o2ib6) in 177 seconds. I think it's dead, and I am evicting it. exp ffff9854e2653c00, cur 1552187834 expire 1552187684 last 1552187657 Mar 09 19:17:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 09 19:18:04 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 63a3d0a9-3d76-72da-1801-efa0f02606ff (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985815684800, cur 1552187884 expire 1552187734 last 1552187657 Mar 09 19:18:04 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 10 01:00:01 fir-io1-s1 kernel: md: data-check of RAID array md6 Mar 10 03:22:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 23e9b720-ea05-13e5-fc9c-a5a8c0687a8a (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857596bcc00, cur 1552213375 expire 1552213225 last 1552213148 Mar 10 03:22:55 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 10 04:19:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client e1850785-18c4-9efd-e338-2f845529c6b1 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a0ae800, cur 1552216791 expire 1552216641 last 1552216564 Mar 10 04:19:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 12:17:20 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552245433/real 1552245433] req@ffff984cb6c0a700 x1625436483934512/t0(0) o104->fir-OST0008@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1552245440 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 12:17:20 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Mar 10 12:17:27 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552245440/real 1552245440] req@ffff984cb6c0a700 x1625436483934512/t0(0) o104->fir-OST0008@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1552245447 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 12:17:34 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552245447/real 1552245447] req@ffff984cb6c0a700 x1625436483934512/t0(0) o104->fir-OST0008@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1552245454 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 12:17:41 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552245454/real 1552245454] req@ffff984cb6c0a700 x1625436483934512/t0(0) o104->fir-OST0008@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1552245461 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 12:17:55 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552245468/real 1552245468] req@ffff984cb6c0a700 x1625436483934512/t0(0) o104->fir-OST0008@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1552245475 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 12:17:55 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 10 12:18:16 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552245489/real 1552245489] req@ffff984cb6c0a700 x1625436483934512/t0(0) o104->fir-OST0008@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1552245496 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 12:18:16 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 10 12:18:58 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552245531/real 1552245531] req@ffff984cb6c0a700 x1625436483934512/t0(0) o104->fir-OST0008@10.9.0.62@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1552245538 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 12:18:58 fir-io1-s1 kernel: Lustre: 96620:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 10 12:18:58 fir-io1-s1 kernel: LustreError: 96620:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.62@o2ib4) failed to reply to blocking AST (req@ffff984cb6c0a700 x1625436483934512 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff983856ada1c0/0x49e18623c34f1ae7 lrc: 4/0,0 mode: PR/PR res: [0xc80000402:0x2237cf:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd31e7c2f expref: 744 pid: 110571 timeout: 2593382 lvb_type: 1 Mar 10 12:18:58 fir-io1-s1 kernel: LustreError: 96620:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 10 12:18:58 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.9.0.62@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Mar 10 12:18:58 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Mar 10 12:18:58 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.9.0.62@o2ib4 ns: filter-fir-OST0008_UUID lock: ffff983856ada1c0/0x49e18623c34f1ae7 lrc: 3/0,0 mode: PR/PR res: [0xc80000402:0x2237cf:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd31e7c2f expref: 745 pid: 110571 timeout: 0 lvb_type: 1 Mar 10 12:19:48 fir-io1-s1 kernel: LustreError: 96367:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.62@o2ib4) returned error from blocking AST (req@ffff985d75c71b00 x1625436491268480 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff985f80ae5a00/0x49e18623bf27960f lrc: 4/0,0 mode: PR/PR res: [0xc40000402:0x222cbd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd30ed3a5 expref: 752 pid: 96242 timeout: 2593438 lvb_type: 1 Mar 10 12:19:48 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.9.0.62@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Mar 10 12:19:48 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 21s: evicting client at 10.9.0.62@o2ib4 ns: filter-fir-OST0006_UUID lock: ffff985f80ae5a00/0x49e18623bf27960f lrc: 3/0,0 mode: PR/PR res: [0xc40000402:0x222cbd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd30ed3a5 expref: 753 pid: 96242 timeout: 0 lvb_type: 1 Mar 10 12:19:52 fir-io1-s1 kernel: LustreError: 96242:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.62@o2ib4) returned error from blocking AST (req@ffff9872b9c96000 x1625436492503136 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff985ff1fcde80/0x49e18623c3a7151a lrc: 4/0,0 mode: PR/PR res: [0x6c0000400:0x223314:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd31ec507 expref: 767 pid: 96755 timeout: 2593443 lvb_type: 1 Mar 10 12:19:52 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.9.0.62@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Mar 10 12:19:52 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.9.0.62@o2ib4 ns: filter-fir-OST0000_UUID lock: ffff9875b5129f80/0x49e18623c3a7201f lrc: 3/0,0 mode: PR/PR res: [0x6c0000400:0x223999:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd31ecea0 expref: 768 pid: 96749 timeout: 0 lvb_type: 1 Mar 10 12:19:52 fir-io1-s1 kernel: LustreError: 96242:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 10 12:19:57 fir-io1-s1 kernel: LustreError: 96514:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.62@o2ib4) returned error from blocking AST (req@ffff984f542ad100 x1625436492590032 status -107 rc -107), evict it ns: filter-fir-OST0004_UUID lock: ffff985aab5b3a80/0x49e18623bf27a471 lrc: 4/0,0 mode: PR/PR res: [0x8c0000402:0x222840:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd30f4dc3 expref: 747 pid: 96405 timeout: 2593447 lvb_type: 1 Mar 10 12:19:57 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.9.0.62@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Mar 10 12:19:57 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 10 12:19:57 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.9.0.62@o2ib4 ns: filter-fir-OST0004_UUID lock: ffff985aab5b3a80/0x49e18623bf27a471 lrc: 3/0,0 mode: PR/PR res: [0x8c0000402:0x222840:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd30f4dc3 expref: 748 pid: 96405 timeout: 0 lvb_type: 1 Mar 10 12:20:27 fir-io1-s1 kernel: LustreError: 94242:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.62@o2ib4) returned error from blocking AST (req@ffff985b64800900 x1625436493964672 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff9838025a5a00/0x49e18623bf279dbe lrc: 4/0,0 mode: PR/PR res: [0x5c0000400:0x222e6c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd30f14b2 expref: 751 pid: 96505 timeout: 2593477 lvb_type: 1 Mar 10 12:20:27 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.9.0.62@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Mar 10 12:20:27 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.9.0.62@o2ib4 ns: filter-fir-OST0002_UUID lock: ffff9838025a5a00/0x49e18623bf279dbe lrc: 3/0,0 mode: PR/PR res: [0x5c0000400:0x222e6c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.9.0.62@o2ib4 remote: 0x6737cffdd30f14b2 expref: 752 pid: 96505 timeout: 0 lvb_type: 1 Mar 10 12:20:34 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0f82de9c-d37a-1eea-ad4a-b15f87c0fd06 (at 10.9.0.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c85dc00, cur 1552245634 expire 1552245484 last 1552245407 Mar 10 12:20:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 14:05:33 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 14:19:20 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 14:36:46 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 14:59:13 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 416aee25-fddc-b3b7-596a-ffb1b377c458 (at 10.8.15.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b91800, cur 1552255153 expire 1552255003 last 1552254926 Mar 10 16:53:10 fir-io1-s1 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0xb7044c4b134c3dec to 0x253aba2c0652a450 Mar 10 16:53:10 fir-io1-s1 kernel: Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 10 16:53:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 10 16:53:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 16:53:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 519ac832-a77f-51ba-e3c7-51aa4fe15024 (at 10.8.15.3@o2ib6) Mar 10 16:53:50 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 10 16:54:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca1af5b2-4b74-b03d-4a2b-13a823b2dc8f (at 10.8.15.10@o2ib6) Mar 10 16:54:02 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 10 16:54:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7a28562b-3b5d-09cc-c2f6-568559bac302 (at 10.8.18.15@o2ib6) Mar 10 16:54:08 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 10 16:54:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d2da3ac-e307-036e-f8a3-99f8b3ab4ed7 (at 10.8.15.5@o2ib6) Mar 10 16:54:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 16:56:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bef16258-699d-0e14-bdeb-b454fac00d89 (at 10.9.112.15@o2ib4) Mar 10 16:56:00 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 10 16:58:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa7f73a8-b3bd-8e08-6f60-12dfe67e15ff (at 10.8.4.15@o2ib6) Mar 10 16:58:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 17:00:19 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 10 17:00:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 17:04:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b9de1fd1-ccbe-721f-e4ab-c6e06447a81c (at 10.8.15.4@o2ib6) Mar 10 17:04:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 17:05:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client fa7f73a8-b3bd-8e08-6f60-12dfe67e15ff (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e02b000, cur 1552262713 expire 1552262563 last 1552262486 Mar 10 17:05:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 17:12:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6) Mar 10 17:12:28 fir-io1-s1 kernel: Lustre: Skipped 27 previous similar messages Mar 10 17:13:17 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b39088c6-a82c-77ef-84c9-6f95445049ff (at 10.8.15.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a1c00, cur 1552263197 expire 1552263047 last 1552262970 Mar 10 17:13:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 17:19:30 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 10 17:19:30 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 10 17:20:22 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 4 seconds Mar 10 17:20:22 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 11 previous similar messages Mar 10 17:20:27 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 10 17:20:27 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages Mar 10 17:20:52 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 10 17:20:52 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (6): c: 0, oc: 0, rc: 8 Mar 10 17:21:11 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST000a: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 10 17:21:11 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 10 17:21:11 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST000a: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 10 17:21:11 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 10 17:21:17 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 10 17:21:17 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 10 17:22:01 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 10 17:22:01 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1486267 to 0x0:1486305 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1486267 to 0x0:1486305 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1486131 to 0x0:1486177 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1486533 to 0x0:1486561 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1486047 to 0x0:1486113 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1486641 to 0x0:1486657 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2254915 to 0x5c0000400:2255009 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2254401 to 0x8c0000402:2254433 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2255032 to 0x580000400:2255265 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2255244 to 0x6c0000400:2255329 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2254944 to 0xc80000402:2254977 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2254857 to 0xc40000402:2254881 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2030620 to 0xc40000400:2030657 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2030342 to 0x6c0000401:2030433 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2030792 to 0x5c0000401:2030817 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2030835 to 0x580000401:2031009 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2030420 to 0x8c0000400:2030465 Mar 10 17:22:29 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2030373 to 0xc80000400:2030401 Mar 10 17:22:30 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1754961 to 0x8c0000401:1754977 Mar 10 17:22:30 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1756583 to 0x5c0000402:1756673 Mar 10 17:22:30 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1755078 to 0x6c0000402:1755169 Mar 10 17:22:30 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1755140 to 0xc40000401:1755169 Mar 10 17:22:30 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1754898 to 0xc80000401:1754913 Mar 10 17:22:30 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1756455 to 0x580000402:1756545 Mar 10 17:32:32 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 076c4071-7bcb-b7de-12f9-282e573303a9 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043f8400, cur 1552264352 expire 1552264202 last 1552264125 Mar 10 17:32:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 17:33:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa7f73a8-b3bd-8e08-6f60-12dfe67e15ff (at 10.8.4.15@o2ib6) Mar 10 17:33:02 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 10 17:36:03 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ec0cbea4-ecef-04d8-8a88-e7790cb02488 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786886c00, cur 1552264563 expire 1552264413 last 1552264336 Mar 10 17:36:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 17:46:41 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 10 17:46:41 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 10 17:46:57 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 10 17:46:57 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 10 17:47:06 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 10 17:47:06 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 10 17:47:06 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 10 17:47:06 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Mar 10 17:47:37 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 10 17:47:37 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 10 17:48:21 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0003-lwp-OST0000: This client was evicted by fir-MDT0003; in progress operations using this service will fail. Mar 10 17:48:21 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1486241 to 0x0:1486273 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1486431 to 0x0:1486465 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1486432 to 0x0:1486465 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1486785 to 0x0:1486817 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1486302 to 0x0:1486337 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1486688 to 0x0:1486721 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2256670 to 0x6c0000400:2256737 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2256364 to 0x5c0000400:2256385 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2255800 to 0x8c0000402:2255841 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2256227 to 0xc40000402:2256321 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2256611 to 0x580000400:2256641 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2256330 to 0xc80000402:2256417 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1756705 to 0x5c0000402:1756737 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1755203 to 0x6c0000402:1755233 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1755011 to 0x8c0000401:1755041 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1756578 to 0x580000402:1756609 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1754947 to 0xc80000401:1754977 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1755203 to 0xc40000401:1755233 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2030832 to 0x6c0000401:2030849 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2030863 to 0x8c0000400:2030881 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2031406 to 0x580000401:2031425 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2030797 to 0xc80000400:2030817 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2031214 to 0x5c0000401:2031233 Mar 10 17:48:51 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2031057 to 0xc40000400:2031073 Mar 10 18:00:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 10 18:00:31 fir-io1-s1 kernel: Lustre: Skipped 51 previous similar messages Mar 10 18:01:06 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 425098e4-e563-0f40-279f-3ea08948d444 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985763311c00, cur 1552266066 expire 1552265916 last 1552265839 Mar 10 18:01:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 18:05:48 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 10 18:05:48 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.201@o2ib7 (0): c: 0, oc: 2, rc: 8 Mar 10 18:05:49 fir-io1-s1 kernel: Lustre: 96774:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552266342/real 1552266342] req@ffff98382a386600 x1625438295775872/t0(0) o106->fir-OST0004@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552266349 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 18:05:49 fir-io1-s1 kernel: Lustre: 96774:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Mar 10 18:05:59 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552266352/real 1552266352] req@ffff984f542afb00 x1625438295918720/t0(0) o106->fir-OST0006@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552266359 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 18:05:59 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552266352/real 1552266352] req@ffff98728765aa00 x1625438295918688/t0(0) o106->fir-OST0000@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552266359 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 18:05:59 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 51 previous similar messages Mar 10 18:06:28 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 088be137-c6a7-2b43-2183-5ec9349d8421 (at 10.8.8.33@o2ib6) reconnecting Mar 10 18:06:28 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 10 18:06:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a792c2c8-fe7f-0c11-b333-ebd11e33240c (at 10.8.27.26@o2ib6) reconnecting Mar 10 18:06:32 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 10 18:06:34 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 4c5155a8-c01c-cc55-c4b5-c794d0043bbe (at 10.8.6.22@o2ib6) reconnecting Mar 10 18:06:34 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 10 18:06:36 fir-io1-s1 kernel: Lustre: 96913:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552266389/real 1552266389] req@ffff9854c9a5b300 x1625438298925680/t0(0) o106->fir-OST0002@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552266396 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 18:06:36 fir-io1-s1 kernel: Lustre: 96913:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 380 previous similar messages Mar 10 18:07:35 fir-io1-s1 kernel: Lustre: fir-OST000a: Client a5e7688e-10db-f09e-c5a3-4fd49cb8bff8 (at 10.8.7.18@o2ib6) reconnecting Mar 10 18:07:52 fir-io1-s1 kernel: Lustre: 96259:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552266465/real 1552266465] req@ffff986f915f0c00 x1625438299570000/t0(0) o106->fir-OST0000@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552266472 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 18:07:52 fir-io1-s1 kernel: Lustre: 96259:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1128 previous similar messages Mar 10 18:07:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Client b5bdacfb-4b25-b3c2-5f11-89c3781f97a5 (at 10.8.7.20@o2ib6) reconnecting Mar 10 18:07:55 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 10 18:09:02 fir-io1-s1 kernel: LNet: Service thread pid 96905 was inactive for 200.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 10 18:09:02 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Mar 10 18:09:02 fir-io1-s1 kernel: Pid: 96905, comm: ll_ost00_058 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:09:02 fir-io1-s1 kernel: Call Trace: Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:09:02 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:09:02 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266542.96905 Mar 10 18:09:04 fir-io1-s1 kernel: LNet: Service thread pid 96927 was inactive for 200.92s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 10 18:09:04 fir-io1-s1 kernel: Pid: 96927, comm: ll_ost01_101 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:09:04 fir-io1-s1 kernel: Call Trace: Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:09:04 fir-io1-s1 kernel: Pid: 96245, comm: ll_ost00_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:09:04 fir-io1-s1 kernel: Call Trace: Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:09:04 fir-io1-s1 kernel: Pid: 96783, comm: ll_ost01_076 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:09:04 fir-io1-s1 kernel: Call Trace: Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:09:04 fir-io1-s1 kernel: Pid: 96374, comm: ll_ost02_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:09:04 fir-io1-s1 kernel: Call Trace: Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:09:04 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:09:04 fir-io1-s1 kernel: LNet: Service thread pid 96562 was inactive for 201.33s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 18:09:04 fir-io1-s1 kernel: LNet: Skipped 14 previous similar messages Mar 10 18:09:05 fir-io1-s1 kernel: LNet: Service thread pid 96360 was inactive for 200.50s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 18:09:05 fir-io1-s1 kernel: LNet: Skipped 18 previous similar messages Mar 10 18:09:05 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266545.96360 Mar 10 18:09:09 fir-io1-s1 kernel: LNet: Service thread pid 96249 was inactive for 200.73s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 18:09:09 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 10 18:09:09 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266549.96249 Mar 10 18:09:10 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266550.96764 Mar 10 18:09:15 fir-io1-s1 kernel: LNet: Service thread pid 96357 was inactive for 200.30s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 18:09:15 fir-io1-s1 kernel: LNet: Skipped 7 previous similar messages Mar 10 18:09:15 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266555.96357 Mar 10 18:09:18 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266558.96378 Mar 10 18:09:19 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266559.49830 Mar 10 18:09:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e196843a-03ac-4b1a-eba2-eb6f1ff2b1cb (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786886000, cur 1552266561 expire 1552266411 last 1552266334 Mar 10 18:09:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 18:09:21 fir-io1-s1 kernel: LNet: Service thread pid 96368 completed after 201.89s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 10 18:09:21 fir-io1-s1 kernel: LNet: Skipped 22 previous similar messages Mar 10 18:09:22 fir-io1-s1 kernel: LNet: Service thread pid 96375 was inactive for 200.38s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 18:09:22 fir-io1-s1 kernel: LNet: Skipped 11 previous similar messages Mar 10 18:09:22 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266562.96375 Mar 10 18:09:24 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266564.110657 Mar 10 18:09:25 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552266565.110618 Mar 10 18:09:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e196843a-03ac-4b1a-eba2-eb6f1ff2b1cb (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed07800, cur 1552266566 expire 1552266416 last 1552266339 Mar 10 18:09:26 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 10 18:09:26 fir-io1-s1 kernel: LNet: Service thread pid 96355 completed after 216.93s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 10 18:09:26 fir-io1-s1 kernel: LNet: Skipped 28 previous similar messages Mar 10 18:09:32 fir-io1-s1 kernel: Lustre: fir-OST000a: Client e18fad02-59fc-284f-8b23-341e1d56114f (at 10.8.7.7@o2ib6) reconnecting Mar 10 18:11:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 70328a4f-1771-1c4a-41a9-4ca1153337e8 (at 10.8.7.12@o2ib6) reconnecting Mar 10 18:11:57 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 09e96f09-88f3-17cc-ba26-88dee3b61d1c (at 10.8.7.12@o2ib6) Mar 10 18:11:57 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Mar 10 18:11:57 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 10 18:13:28 fir-io1-s1 kernel: LustreError: 96927:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff984cbce1c800 x1625438349907904 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff984b87cd9200/0x49e186240897858f lrc: 4/0,0 mode: PR/PR res: [0xc40000402:0x2253f5:0x0].0x0 rrc: 20 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f3020 expref: 11 pid: 96247 timeout: 2614708 lvb_type: 1 Mar 10 18:13:28 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 18:13:28 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff984b87cd9200/0x49e186240897858f lrc: 3/0,0 mode: PR/PR res: [0xc40000402:0x2253f5:0x0].0x0 rrc: 19 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f3020 expref: 12 pid: 96247 timeout: 0 lvb_type: 1 Mar 10 18:13:29 fir-io1-s1 kernel: LustreError: 96516:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff985a1e616300 x1625438350346224 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff98384caad100/0x49e18624089785a4 lrc: 4/0,0 mode: PR/PR res: [0x580000400:0x2254be:0x0].0x0 rrc: 22 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f3097 expref: 11 pid: 96499 timeout: 2614709 lvb_type: 1 Mar 10 18:13:29 fir-io1-s1 kernel: LustreError: 96516:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 10 18:13:29 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 18:13:29 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 10 18:13:29 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff98384caad100/0x49e18624089785a4 lrc: 3/0,0 mode: PR/PR res: [0x580000400:0x2254be:0x0].0x0 rrc: 19 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f3097 expref: 12 pid: 96499 timeout: 0 lvb_type: 1 Mar 10 18:13:29 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Mar 10 18:13:34 fir-io1-s1 kernel: LustreError: 96268:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff98754e3f7500 x1625438350673904 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff9848e72ce300/0x49e18624089781bb lrc: 4/0,0 mode: PR/PR res: [0x6c0000400:0x2252e8:0x0].0x0 rrc: 23 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f1959 expref: 12 pid: 96247 timeout: 2614714 lvb_type: 1 Mar 10 18:13:34 fir-io1-s1 kernel: LustreError: 96268:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 10 18:13:34 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 18:13:34 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 10 18:13:34 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff9848e72ce300/0x49e18624089781bb lrc: 3/0,0 mode: PR/PR res: [0x6c0000400:0x2252e8:0x0].0x0 rrc: 21 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f1959 expref: 13 pid: 96247 timeout: 0 lvb_type: 1 Mar 10 18:13:34 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Mar 10 18:13:40 fir-io1-s1 kernel: LustreError: 75602:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff985b8080b900 x1625438351050608 status -107 rc -107), evict it ns: filter-fir-OST0004_UUID lock: ffff983848b54140/0x49e1862408978183 lrc: 4/0,0 mode: PR/PR res: [0x8c0000402:0x224ce9:0x0].0x0 rrc: 9 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f17fb expref: 11 pid: 49827 timeout: 2614721 lvb_type: 1 Mar 10 18:13:40 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 18:13:40 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff983848b54140/0x49e1862408978183 lrc: 3/0,0 mode: PR/PR res: [0x8c0000402:0x224ce9:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x347d8268b09f17fb expref: 12 pid: 49827 timeout: 0 lvb_type: 1 Mar 10 18:15:40 fir-io1-s1 kernel: Lustre: fir-OST0008: Client e18fad02-59fc-284f-8b23-341e1d56114f (at 10.8.7.7@o2ib6) reconnecting Mar 10 18:15:40 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 10 18:28:21 fir-io1-s1 kernel: LustreError: 96254:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff986eec63f200 x1625438413518016 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff9857205421c0/0x49e186240dfa3325 lrc: 4/0,0 mode: PR/PR res: [0x6c0000400:0x225040:0x0].0x0 rrc: 36 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x819f7ef7b8e3d2d3 expref: 20 pid: 96361 timeout: 2615602 lvb_type: 1 Mar 10 18:28:21 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 18:28:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff983851c50000/0x49e186240dfa30b6 lrc: 3/0,0 mode: PR/PR res: [0x580000400:0x224f71:0x0].0x0 rrc: 30 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x819f7ef7b8e3c176 expref: 18 pid: 110658 timeout: 0 lvb_type: 1 Mar 10 18:28:21 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 10 18:29:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 10 18:29:04 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Mar 10 18:30:41 fir-io1-s1 kernel: Lustre: 96246:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552267834/real 1552267834] req@ffff9863e373b300 x1625438423885296/t0(0) o106->fir-OST0008@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552267841 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 18:30:41 fir-io1-s1 kernel: Lustre: 96246:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1497 previous similar messages Mar 10 18:31:00 fir-io1-s1 kernel: Lustre: 96523:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552267853/real 1552267853] req@ffff987704139500 x1625438424874352/t0(0) o106->fir-OST0006@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552267860 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 18:31:00 fir-io1-s1 kernel: Lustre: 96523:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 33 previous similar messages Mar 10 18:31:40 fir-io1-s1 kernel: Lustre: 94539:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552267893/real 1552267893] req@ffff984968b8f200 x1625438424180368/t0(0) o106->fir-OST0002@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552267900 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 18:31:40 fir-io1-s1 kernel: Lustre: 94539:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 111 previous similar messages Mar 10 18:32:57 fir-io1-s1 kernel: Lustre: 96572:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552267970/real 1552267970] req@ffff98697e658c00 x1625438424180336/t0(0) o106->fir-OST0008@10.8.21.21@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552267977 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 18:32:57 fir-io1-s1 kernel: Lustre: 96572:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 208 previous similar messages Mar 10 18:33:55 fir-io1-s1 kernel: LNet: Service thread pid 96250 was inactive for 200.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 10 18:33:55 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 10 18:33:55 fir-io1-s1 kernel: Pid: 96250, comm: ll_ost02_011 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:33:55 fir-io1-s1 kernel: Call Trace: Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:33:55 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552268035.96250 Mar 10 18:33:55 fir-io1-s1 kernel: Pid: 96244, comm: ll_ost01_010 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:33:55 fir-io1-s1 kernel: Call Trace: Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:33:55 fir-io1-s1 kernel: Pid: 96246, comm: ll_ost02_009 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:33:55 fir-io1-s1 kernel: Call Trace: Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:33:55 fir-io1-s1 kernel: Pid: 96286, comm: ll_ost01_024 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:33:55 fir-io1-s1 kernel: Call Trace: Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:33:55 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:33:56 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d0e2cded-9f93-c6ac-9c3d-82cf975adfa3 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dffa800, cur 1552268036 expire 1552267886 last 1552267809 Mar 10 18:33:56 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 10 18:33:57 fir-io1-s1 kernel: LNet: Service thread pid 96934 was inactive for 200.14s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 10 18:33:57 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 10 18:33:57 fir-io1-s1 kernel: Pid: 96934, comm: ll_ost01_106 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 18:33:57 fir-io1-s1 kernel: Call Trace: Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 18:33:57 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 18:33:57 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552268037.96934 Mar 10 18:33:57 fir-io1-s1 kernel: LNet: Service thread pid 96904 was inactive for 200.27s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 18:33:57 fir-io1-s1 kernel: LNet: Skipped 5 previous similar messages Mar 10 18:34:00 fir-io1-s1 kernel: LNet: Service thread pid 96929 was inactive for 200.10s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 18:34:00 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 10 18:34:00 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552268040.96929 Mar 10 18:34:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client d0e2cded-9f93-c6ac-9c3d-82cf975adfa3 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575f4dac00, cur 1552268046 expire 1552267896 last 1552267819 Mar 10 18:34:06 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 10 18:34:06 fir-io1-s1 kernel: LNet: Service thread pid 96904 completed after 208.47s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 10 18:34:06 fir-io1-s1 kernel: LNet: Skipped 11 previous similar messages Mar 10 18:40:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 10 18:40:51 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 10 18:41:14 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 728c5b5e-4140-bb39-4fc0-b0b354cf14ec (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c6000, cur 1552268474 expire 1552268324 last 1552268247 Mar 10 18:41:14 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 10 18:47:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d38bb286-09b2-3020-955d-c86af4ee6479 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985769988c00, cur 1552268830 expire 1552268680 last 1552268603 Mar 10 18:47:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 18:48:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b5993e34-577c-d793-0498-d6696e1908f3 (at 10.8.9.4@o2ib6) in 151 seconds. I think it's dead, and I am evicting it. exp ffff986785d2d800, cur 1552268906 expire 1552268756 last 1552268755 Mar 10 18:48:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 18:49:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b5993e34-577c-d793-0498-d6696e1908f3 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868d0000, cur 1552268982 expire 1552268832 last 1552268755 Mar 10 18:49:42 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 10 18:50:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 142a6e04-872a-1c7a-e7fe-9168d1c1b90d (at 10.8.9.5@o2ib6) Mar 10 18:50:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 10 18:50:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ba02a2b3-7ab8-8981-35d1-b48ec04f7bfa (at 10.8.9.5@o2ib6) in 188 seconds. I think it's dead, and I am evicting it. exp ffff984ebed02c00, cur 1552269058 expire 1552268908 last 1552268870 Mar 10 18:51:33 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ba02a2b3-7ab8-8981-35d1-b48ec04f7bfa (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed01800, cur 1552269093 expire 1552268943 last 1552268866 Mar 10 18:51:33 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 10 18:53:38 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 19:00:43 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8b27fd9e-5d2b-40d4-c0d0-6c2eea919349 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98546cff2800, cur 1552269643 expire 1552269493 last 1552269416 Mar 10 19:00:43 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 10 19:02:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 10 19:02:43 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 10 19:06:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3af317f1-c1a6-1d9b-8def-be1d70807f6f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785e43c00, cur 1552269987 expire 1552269837 last 1552269760 Mar 10 19:06:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 10 19:14:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aff71b74-7050-1a79-ef86-3b2a0fea26d1 (at 10.8.9.4@o2ib6) Mar 10 19:14:25 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 10 19:21:14 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552270867/real 1552270867] req@ffff986ff815c200 x1625438725848992/t0(0) o104->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552270874 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 19:21:14 fir-io1-s1 kernel: Lustre: 96365:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 198 previous similar messages Mar 10 19:21:35 fir-io1-s1 kernel: Lustre: 96917:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552270888/real 1552270888] req@ffff986bd4484500 x1625438725849312/t0(0) o104->fir-OST0004@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552270895 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 19:21:35 fir-io1-s1 kernel: Lustre: 96917:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 10 19:22:03 fir-io1-s1 kernel: LustreError: 96357:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from blocking AST (req@ffff9852a3253f00 x1625438725852576 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff984f8361e9c0/0x49e186241a383306 lrc: 4/0,0 mode: PR/PR res: [0x6c0000400:0x227e0a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.27.23@o2ib6 remote: 0x591309be52572c2d expref: 16160 pid: 96516 timeout: 2618824 lvb_type: 1 Mar 10 19:22:03 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 19:22:03 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 10 19:22:03 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 56s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff984f4a91d340/0x49e186241a301d91 lrc: 3/0,0 mode: PR/PR res: [0x6c0000400:0x2280e4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.27.23@o2ib6 remote: 0x591309be525697ac expref: 16161 pid: 96894 timeout: 0 lvb_type: 1 Mar 10 19:22:03 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Mar 10 19:22:03 fir-io1-s1 kernel: LustreError: 96357:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 8 previous similar messages Mar 10 19:22:06 fir-io1-s1 kernel: LustreError: 96781:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from blocking AST (req@ffff987429702700 x1625438728948416 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff984add6ec5c0/0x49e186241a890cd2 lrc: 4/0,0 mode: PR/PR res: [0xc80000402:0x2280d1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.27.23@o2ib6 remote: 0x591309be525aef1d expref: 16139 pid: 96404 timeout: 2618777 lvb_type: 1 Mar 10 19:22:06 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 19:22:06 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 10 19:22:06 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 14s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff984add6ec5c0/0x49e186241a890cd2 lrc: 3/0,0 mode: PR/PR res: [0xc80000402:0x2280d1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.27.23@o2ib6 remote: 0x591309be525aef1d expref: 16140 pid: 96404 timeout: 0 lvb_type: 1 Mar 10 19:22:06 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Mar 10 19:22:31 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ee253e03-4ea9-d4e3-1517-81d16e1c4acb (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984996393400, cur 1552270951 expire 1552270801 last 1552270724 Mar 10 19:22:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 19:22:31 fir-io1-s1 kernel: LustreError: 110660:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff98386a16f500 x1625438732206240/t0(0) o104->fir-OST0002@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 10 19:22:31 fir-io1-s1 kernel: LustreError: 110660:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 3 previous similar messages Mar 10 19:23:47 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a49a82a8-fadf-d582-34b4-f94d3725c566 (at 10.8.9.4@o2ib6) in 184 seconds. I think it's dead, and I am evicting it. exp ffff9847fae7b400, cur 1552271027 expire 1552270877 last 1552270843 Mar 10 19:23:47 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Mar 10 19:33:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 23c1de71-d2eb-cd1d-0c15-edefa3d3741a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767577800, cur 1552271589 expire 1552271439 last 1552271362 Mar 10 19:33:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 19:33:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 10 19:33:20 fir-io1-s1 kernel: Lustre: Skipped 16 previous similar messages Mar 10 19:41:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 221c3640-48a6-c137-26b9-1cdf99db72fc (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8c400, cur 1552272109 expire 1552271959 last 1552271882 Mar 10 19:41:49 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 10 19:54:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 10 19:54:26 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Mar 10 19:55:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1b9dbe1e-f272-1dbd-1cab-6cd6dc06b22f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762496400, cur 1552272905 expire 1552272755 last 1552272678 Mar 10 19:55:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 19:59:36 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552273169/real 1552273169] req@ffff9874fd2c8300 x1625438981045248/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552273176 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 19:59:36 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 10 19:59:43 fir-io1-s1 kernel: Lustre: 96781:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552273176/real 1552273176] req@ffff9872c428f800 x1625438981045232/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552273183 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 19:59:43 fir-io1-s1 kernel: Lustre: 96781:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 10 19:59:57 fir-io1-s1 kernel: Lustre: 96913:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552273190/real 1552273190] req@ffff986c783e8600 x1625438981045216/t0(0) o106->fir-OST0002@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552273197 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 19:59:57 fir-io1-s1 kernel: Lustre: 96913:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 13 previous similar messages Mar 10 20:00:18 fir-io1-s1 kernel: Lustre: 96909:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552273211/real 1552273211] req@ffff9856a86e1200 x1625438981045264/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552273218 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 20:00:18 fir-io1-s1 kernel: Lustre: 96909:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Mar 10 20:01:00 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552273253/real 1552273253] req@ffff9874fd2c8300 x1625438981045248/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552273260 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 10 20:01:00 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 91 previous similar messages Mar 10 20:01:57 fir-io1-s1 kernel: LustreError: 96365:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff986c783e9b00 x1625438981925808 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff984ec8ecc800/0x49e18624315f8ec7 lrc: 6/0,0 mode: PW/PW res: [0xc40000402:0x228a1d:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0x3f41a4d78143105c expref: 26 pid: 96358 timeout: 0 lvb_type: 0 Mar 10 20:01:57 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 10 20:01:57 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552273317s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff984ec8ecc800/0x49e18624315f8ec7 lrc: 6/0,0 mode: PW/PW res: [0xc40000402:0x228a1d:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0x3f41a4d78143105c expref: 27 pid: 96358 timeout: 0 lvb_type: 0 Mar 10 20:01:58 fir-io1-s1 kernel: LustreError: 96573:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff9872c482f200 x1625438981191648 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff98382b46b600/0x49e18624315f8d23 lrc: 6/0,0 mode: PW/PW res: [0x6c0000400:0x228bd9:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0x3f41a4d781430fe5 expref: 26 pid: 96941 timeout: 0 lvb_type: 0 Mar 10 20:01:58 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 10 20:01:58 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552273318s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff9854e4147740/0x49e18624315f8e0a lrc: 6/0,0 mode: PW/PW res: [0x8c0000402:0x228852:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0x3f41a4d781431024 expref: 26 pid: 96365 timeout: 0 lvb_type: 0 Mar 10 20:01:58 fir-io1-s1 kernel: LustreError: 96573:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Mar 10 20:02:08 fir-io1-s1 kernel: LustreError: 96359:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff98748be98000 x1625438997971120 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff98602106f980/0x49e186242f442b2a lrc: 4/0,0 mode: PR/PR res: [0xc80000402:0x226ea1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x3f41a4d7813e776e expref: 23 pid: 77317 timeout: 2621228 lvb_type: 1 Mar 10 20:02:08 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 20:02:08 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 10 20:02:08 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff98602106f980/0x49e186242f442b2a lrc: 3/0,0 mode: PR/PR res: [0xc80000402:0x226ea1:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x3f41a4d7813e776e expref: 24 pid: 77317 timeout: 0 lvb_type: 1 Mar 10 20:02:08 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Mar 10 20:03:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 10 20:03:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 20:08:15 fir-io1-s1 kernel: Lustre: 110617:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552273651/real 1552273651] req@ffff9838405b6c00 x1625439025806352/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552273695 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 20:08:15 fir-io1-s1 kernel: Lustre: 49818:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552273651/real 1552273651] req@ffff98380c02da00 x1625439025806368/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552273695 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 20:08:15 fir-io1-s1 kernel: Lustre: 49818:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 140 previous similar messages Mar 10 20:10:02 fir-io1-s1 kernel: LustreError: 96911:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff985e6c518600 x1625439043869872 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff984b775cf080/0x49e18624360351d1 lrc: 7/0,0 mode: PW/PW res: [0xc40000402:0x228c2c:0x0].0x0 rrc: 7 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0xe52d021006eaadd expref: 29 pid: 96362 timeout: 0 lvb_type: 0 Mar 10 20:10:02 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 10 20:10:02 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552273802s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff984dcdced100/0x49e1862436035424 lrc: 7/0,0 mode: PW/PW res: [0x580000400:0x228d7f:0x0].0x0 rrc: 7 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.20.15@o2ib6 remote: 0xe52d021006eab62 expref: 28 pid: 96268 timeout: 0 lvb_type: 0 Mar 10 20:10:02 fir-io1-s1 kernel: LustreError: 96911:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Mar 10 20:10:32 fir-io1-s1 kernel: LustreError: 74745:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff987071c8ce00 x1625439048595456 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff984a5b3c1f80/0x49e186243484f15a lrc: 4/0,0 mode: PR/PR res: [0x6c0000400:0x225c89:0x0].0x0 rrc: 19 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0xe52d021006b5d6c expref: 28 pid: 96619 timeout: 2621733 lvb_type: 1 Mar 10 20:10:32 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 10 20:10:32 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 10 20:10:32 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff984a5b3c1f80/0x49e186243484f15a lrc: 3/0,0 mode: PR/PR res: [0x6c0000400:0x225c89:0x0].0x0 rrc: 16 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0xe52d021006b5d6c expref: 29 pid: 96619 timeout: 0 lvb_type: 1 Mar 10 20:10:32 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Mar 10 20:11:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 10 20:11:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 20:16:31 fir-io1-s1 kernel: Lustre: 49816:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552274100/real 1552274100] req@ffff98383e6f6600 x1625439074242032/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552274191 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 10 20:16:31 fir-io1-s1 kernel: Lustre: 49816:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Mar 10 20:18:22 fir-io1-s1 kernel: LNet: Service thread pid 96278 was inactive for 200.37s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 10 20:18:22 fir-io1-s1 kernel: Pid: 96278, comm: ll_ost02_018 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 20:18:22 fir-io1-s1 kernel: Call Trace: Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 20:18:22 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 20:18:22 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552274302.96278 Mar 10 20:18:23 fir-io1-s1 kernel: LNet: Service thread pid 110639 was inactive for 200.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 10 20:18:23 fir-io1-s1 kernel: Pid: 110639, comm: ll_ost02_105 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 20:18:23 fir-io1-s1 kernel: Call Trace: Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 20:18:23 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552274303.110639 Mar 10 20:18:23 fir-io1-s1 kernel: Pid: 74818, comm: ll_ost02_091 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 20:18:23 fir-io1-s1 kernel: Call Trace: Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 20:18:23 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 20:18:26 fir-io1-s1 kernel: LNet: Service thread pid 110634 was inactive for 200.30s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 10 20:18:26 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Mar 10 20:18:26 fir-io1-s1 kernel: Pid: 110634, comm: ll_ost02_100 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 20:18:26 fir-io1-s1 kernel: Call Trace: Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 20:18:26 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552274306.110634 Mar 10 20:18:26 fir-io1-s1 kernel: Pid: 96328, comm: ll_ost02_022 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 10 20:18:26 fir-io1-s1 kernel: Call Trace: Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 10 20:18:26 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 10 20:18:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c72587fc-bd72-7167-3cc4-1b600a7aa10a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838ae800, cur 1552274317 expire 1552274167 last 1552274090 Mar 10 20:18:37 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Mar 10 20:18:40 fir-io1-s1 kernel: LNet: Service thread pid 82278 was inactive for 200.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 10 20:18:40 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 10 20:18:40 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552274320.82278 Mar 10 20:18:42 fir-io1-s1 kernel: LNet: Service thread pid 96328 completed after 215.67s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 10 20:18:42 fir-io1-s1 kernel: LustreError: 96405:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9874fd2cda00 x1625439104048208/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 10 20:18:42 fir-io1-s1 kernel: LNet: Skipped 5 previous similar messages Mar 10 20:19:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 10 20:19:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 20:51:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7854289e-9030-bf9d-83e0-3d0fcec4b62a (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855e70de800, cur 1552276294 expire 1552276144 last 1552276067 Mar 10 20:51:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 20:52:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 10 20:52:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 20:53:29 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 20:53:47 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 20:54:16 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 21:16:22 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 21:31:07 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 21:43:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) Mar 10 21:43:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 21:43:45 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3cdd48f3-f959-c076-7072-825a05e7a49e (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4c000, cur 1552279425 expire 1552279275 last 1552279198 Mar 10 21:43:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 22:14:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f9b5af76-8553-6040-bf62-cde94bae7628 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c05c00, cur 1552281241 expire 1552281091 last 1552281014 Mar 10 22:14:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 22:14:02 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f9b5af76-8553-6040-bf62-cde94bae7628 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c3400, cur 1552281242 expire 1552281092 last 1552281015 Mar 10 22:15:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 10 22:15:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 10 22:31:51 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 22:33:26 fir-io1-s1 kernel: perf: interrupt took too long (5006 > 4898), lowering kernel.perf_event_max_sample_rate to 39000 Mar 10 22:33:29 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client da019f79-0339-447f-231c-d8a46cb6eb8f (at 10.9.112.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868fc400, cur 1552282409 expire 1552282259 last 1552282182 Mar 10 22:33:29 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 10 23:08:02 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:21:17 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:30:57 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:31:06 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:34:48 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:38:58 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:41:42 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:47:06 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:50:11 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:50:11 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 10 23:51:20 fir-io1-s1 kernel: LNetError: 91390:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 10 23:51:47 fir-io1-s1 kernel: perf: interrupt took too long (6302 > 6257), lowering kernel.perf_event_max_sample_rate to 31000 Mar 10 23:56:22 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 00:06:49 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 00:06:49 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 11 00:15:36 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 00:22:35 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bf0a34a5-7ae5-75fe-763e-8cdfef028857 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800fdac00, cur 1552288955 expire 1552288805 last 1552288728 Mar 11 00:22:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 00:26:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 00:26:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 00:30:13 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 00:30:13 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Mar 11 00:33:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 00:33:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 00:34:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0df6b618-aad4-4397-4ae6-47689b286004 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5cc00, cur 1552289647 expire 1552289497 last 1552289420 Mar 11 00:34:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 00:40:28 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 00:40:28 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 00:40:44 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e919de91-30be-62bb-3014-b5d13a83c52a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852b0f4f000, cur 1552290044 expire 1552289894 last 1552289817 Mar 11 00:40:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 00:44:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 00:44:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 00:53:08 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 00:53:08 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 00:55:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 00:55:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 00:55:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 289aa212-1a7c-0329-c21f-0cfe03abfd13 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867851e7800, cur 1552290942 expire 1552290792 last 1552290715 Mar 11 00:55:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:04:47 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 01:04:47 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 11 01:17:11 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 88510f76-5dd8-4af2-83c5-7804df13facd (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801a8c00, cur 1552292231 expire 1552292081 last 1552292004 Mar 11 01:17:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:17:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 01:17:28 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 01:18:38 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 01:18:38 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages Mar 11 01:23:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 01:23:26 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 11 01:24:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9a9f37ec-0a37-6549-b499-9c5318554b99 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864523bcc00, cur 1552292653 expire 1552292503 last 1552292426 Mar 11 01:24:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:29:09 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 01:29:09 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 8 previous similar messages Mar 11 01:32:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 01:32:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:33:06 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b1e06b14-a4c0-18e7-fd4d-de291fc2386d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f0e4c1400, cur 1552293186 expire 1552293036 last 1552292959 Mar 11 01:33:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:40:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 01:40:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:40:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e5e5a418-769e-93e0-fe13-b9726ebeefc3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c21000, cur 1552293652 expire 1552293502 last 1552293425 Mar 11 01:40:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:45:56 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 01:45:56 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages Mar 11 01:48:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 01:48:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:49:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9751d7f1-b362-46a3-6215-fe5fdb3d6122 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868fe000, cur 1552294158 expire 1552294008 last 1552293931 Mar 11 01:49:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:56:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 01:56:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:56:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0afa35ce-533f-27ca-2871-ef020c8db52b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985767574400, cur 1552294603 expire 1552294453 last 1552294376 Mar 11 01:56:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 01:57:25 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 01:57:25 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 02:08:02 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 02:08:02 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 02:14:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 30f72fd2-ef2b-e585-376d-6b1b25c9cb5d (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a92d99000, cur 1552295684 expire 1552295534 last 1552295457 Mar 11 02:14:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 02:15:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 11 02:15:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 02:25:43 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 02:25:43 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages Mar 11 02:39:27 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 02:39:27 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Mar 11 02:57:54 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 02:57:54 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages Mar 11 03:07:58 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 03:07:58 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages Mar 11 03:20:17 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 03:20:17 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 11 03:21:06 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0c95cb5e-6c67-1f77-03e3-912d0053a0a3 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c40c00, cur 1552299666 expire 1552299516 last 1552299439 Mar 11 03:21:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 03:21:21 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0c95cb5e-6c67-1f77-03e3-912d0053a0a3 (at 10.8.4.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864ac6eb000, cur 1552299681 expire 1552299531 last 1552299454 Mar 11 03:30:20 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 03:30:20 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages Mar 11 03:42:11 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 03:42:11 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages Mar 11 03:52:24 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 03:52:24 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 11 04:07:41 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 04:07:41 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 6 previous similar messages Mar 11 04:12:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 79273579-8e8e-5b74-3679-6d4cd1e0f345 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4ed800, cur 1552302760 expire 1552302610 last 1552302533 Mar 11 04:12:40 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 04:12:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 46ffc5c0-5221-592c-b4b8-0937c3c0dccb (at 10.8.14.7@o2ib6) Mar 11 04:12:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 04:18:06 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 04:18:06 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 6 previous similar messages Mar 11 04:28:17 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 04:28:17 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Mar 11 04:39:50 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 04:39:50 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Mar 11 04:50:36 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 04:50:36 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 05:01:05 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 05:01:05 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 6 previous similar messages Mar 11 05:14:55 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 05:14:55 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Mar 11 05:22:56 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 37d33bdf-13f2-c3a1-ad5d-e5e9c8e622a6 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480cf28c00, cur 1552306976 expire 1552306826 last 1552306749 Mar 11 05:22:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 05:27:02 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 11 05:27:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 11 05:27:02 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 05:27:36 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 05:27:36 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 6 previous similar messages Mar 11 05:38:54 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 05:38:54 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages Mar 11 05:53:16 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 05:53:16 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Mar 11 05:55:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Client c68a6880-ce72-d94b-8cb3-ca3e5702fcfe (at 10.8.8.35@o2ib6) reconnecting Mar 11 05:55:12 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 103f9da3-989e-ad73-cfdb-75395d4c9148 (at 10.8.8.35@o2ib6) Mar 11 05:55:12 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 11 05:55:12 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 11 06:04:55 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 06:04:55 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 06:15:12 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 06:15:12 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 5 previous similar messages Mar 11 06:25:44 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 06:25:44 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 11 06:38:06 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 06:38:06 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 06:43:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0b424515-9200-0eb3-96f8-e9c1832e5048 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f1b2e9c00, cur 1552311789 expire 1552311639 last 1552311562 Mar 11 06:43:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 06:43:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0b424515-9200-0eb3-96f8-e9c1832e5048 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d76800, cur 1552311790 expire 1552311640 last 1552311563 Mar 11 06:43:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0b424515-9200-0eb3-96f8-e9c1832e5048 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f1b2ed800, cur 1552311793 expire 1552311643 last 1552311566 Mar 11 06:43:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 11 06:43:44 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Mar 11 06:52:43 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 06:52:43 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Mar 11 07:04:49 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 07:04:49 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 4 previous similar messages Mar 11 07:17:18 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 07:27:53 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 07:27:53 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 11 07:33:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 07:33:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 07:34:15 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a3bc538e-71c1-61c3-5c21-04e877c87a7c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b88dc00, cur 1552314855 expire 1552314705 last 1552314628 Mar 11 07:34:15 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 11 07:38:00 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 07:38:00 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages Mar 11 07:39:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 07:39:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 07:40:05 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e33b8c61-06c6-66e8-f105-bbbabb140d6d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677bc9e000, cur 1552315205 expire 1552315055 last 1552314978 Mar 11 07:40:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 07:47:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 07:47:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 07:49:07 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d908b009-05ff-6a5c-267b-148d9e5592cf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd443000, cur 1552315747 expire 1552315597 last 1552315520 Mar 11 07:49:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 07:49:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 11 07:49:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 07:50:32 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 07:50:32 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 7 previous similar messages Mar 11 07:54:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 07:54:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 07:55:01 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 59211fa6-068f-1f6c-1ab8-6de7f6552da2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8262c00, cur 1552316101 expire 1552315951 last 1552315874 Mar 11 07:55:01 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 08:01:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 08:01:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 08:01:41 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c4e9f7eb-3009-7aca-76ae-0bc614803c9f (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987825608800, cur 1552316501 expire 1552316351 last 1552316274 Mar 11 08:01:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 08:01:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 11 08:01:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 08:06:43 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 11 08:06:43 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 11 08:07:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 08:07:29 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 08:07:30 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f7f788c2-4239-444e-0cfd-e1691cd437b4 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848801abc00, cur 1552316850 expire 1552316700 last 1552316623 Mar 11 08:07:30 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 08:51:18 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3d7ad993-1b65-b302-675b-3bdf200b6842 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b65a800, cur 1552319478 expire 1552319328 last 1552319251 Mar 11 08:51:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 08:52:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 11 08:52:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 08:59:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 08:59:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 09:00:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3831e1f5-2649-7fac-3a50-815441cfe5c1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c3000, cur 1552320013 expire 1552319863 last 1552319786 Mar 11 09:00:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:09:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 10:09:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:10:30 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8e82a987-de42-189c-34d1-546971ad129d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fae7b400, cur 1552324230 expire 1552324080 last 1552324003 Mar 11 10:10:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:11:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b50ec62b-95f8-2400-b29b-66ee84451d22 (at 10.8.21.21@o2ib6) in 193 seconds. I think it's dead, and I am evicting it. exp ffff986838664000, cur 1552324306 expire 1552324156 last 1552324113 Mar 11 10:11:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:12:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 11 10:12:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:18:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 10:18:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:18:53 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3ab8f3bc-654a-e713-35ef-b4807c836b21 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985837fe4800, cur 1552324733 expire 1552324583 last 1552324506 Mar 11 10:18:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:25:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 10:25:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:26:25 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2d3a66e2-2b9a-91b8-ff3f-6022acb3cc4c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c3c00, cur 1552325185 expire 1552325035 last 1552324958 Mar 11 10:26:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 10:32:16 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Mar 11 10:32:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Mar 11 10:32:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 12:39:21 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5fb1ed66-d0df-0136-c46d-455449705a53 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987734fbb800, cur 1552333161 expire 1552333011 last 1552332934 Mar 11 12:39:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 12:42:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 11 12:42:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 12:44:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 12:44:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 12:45:24 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 901414bf-a0d2-3130-bf9c-ac498dd1f89c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758dbf400, cur 1552333524 expire 1552333374 last 1552333297 Mar 11 12:45:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 12:52:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 12:52:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 12:53:21 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c402e588-2569-0ebd-b5a7-f689e536967d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7ef000, cur 1552334001 expire 1552333851 last 1552333774 Mar 11 12:53:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 12:59:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 12:59:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 13:01:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client cd728322-8b15-6516-fd62-8c9abf9cb500 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683bb59400, cur 1552334518 expire 1552334368 last 1552334291 Mar 11 13:01:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 13:10:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 13:10:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 13:10:37 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d08eb4c1-07f6-b429-322f-87b184800d43 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984998f84400, cur 1552335037 expire 1552334887 last 1552334810 Mar 11 13:10:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 13:56:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7d3e76e9-6580-1b72-971c-a650fe1dd3bc (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f54bc8800, cur 1552337814 expire 1552337664 last 1552337587 Mar 11 13:56:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 13:57:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 11 13:57:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:05:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4cf71e0b-905a-f752-f037-085e129bdb64 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283df800, cur 1552338336 expire 1552338186 last 1552338109 Mar 11 14:05:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:06:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 63979246-0505-d581-264f-4be47a160c12 (at 10.8.27.23@o2ib6) in 175 seconds. I think it's dead, and I am evicting it. exp ffff98483bd71000, cur 1552338412 expire 1552338262 last 1552338237 Mar 11 14:06:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:08:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 14:08:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:08:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 14:08:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:17:00 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4dd22d6e-9614-73ee-b08b-bb6300d9268a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a36800, cur 1552339020 expire 1552338870 last 1552338793 Mar 11 14:17:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:17:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 14:17:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:20:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a8fb0e14-3938-2e07-839d-43d64c70f798 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c7400, cur 1552339257 expire 1552339107 last 1552339030 Mar 11 14:20:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:23:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 14:23:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:23:36 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 29240b12-2ece-93aa-1784-2e12cb3c0ae9 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d58800, cur 1552339416 expire 1552339266 last 1552339189 Mar 11 14:23:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:24:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 11 14:24:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:24:52 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4a3ee296-b478-53ff-ad0a-c96571faf04a (at 10.8.9.8@o2ib6) in 211 seconds. I think it's dead, and I am evicting it. exp ffff985f9da99400, cur 1552339492 expire 1552339342 last 1552339281 Mar 11 14:24:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:25:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 11 14:25:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:29:28 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e1651c4f-a3af-cc3b-827f-05eb7f299765 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c27c22400, cur 1552339768 expire 1552339618 last 1552339541 Mar 11 14:29:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:29:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 14:29:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:36:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 14:36:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:37:07 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b0950990-da1d-4ec5-aba8-681332f3926c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a15f0800, cur 1552340227 expire 1552340077 last 1552340000 Mar 11 14:37:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:44:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 14:44:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:44:59 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 97a5253a-319c-81e1-7f2d-1964a2665053 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986abbf67000, cur 1552340699 expire 1552340549 last 1552340472 Mar 11 14:44:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 14:53:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ab70ee0e-6c8c-3a09-f8ec-374f5f2e5036 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d56000, cur 1552341236 expire 1552341086 last 1552341009 Mar 11 14:53:56 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 14:55:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 14:55:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 15:05:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 15:05:55 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 11 15:09:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f41f1d9f-5012-95db-1071-9313eccb7447 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d1dc44800, cur 1552342156 expire 1552342006 last 1552341929 Mar 11 15:09:16 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 11 15:18:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 15:18:25 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 11 15:20:33 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f452693b-a841-79c0-9b04-7000caa9b47e (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e7d078000, cur 1552342833 expire 1552342683 last 1552342606 Mar 11 15:20:33 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 11 15:32:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 1f153d74-46d8-784e-d6c4-b1e50ea3209d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a8cc00, cur 1552343533 expire 1552343383 last 1552343306 Mar 11 15:32:13 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 11 15:33:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 15:33:36 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 11 15:42:51 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f6aefdaa-5b81-69a9-d530-d9631f1e1c6b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864683dc800, cur 1552344171 expire 1552344021 last 1552343944 Mar 11 15:42:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:00:55 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bbb69f4e-b2db-d926-6e7a-e4c2e64d848e (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855e70d9c00, cur 1552345255 expire 1552345105 last 1552345028 Mar 11 16:00:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:01:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 16:01:18 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 11 16:13:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a4a20c66-e936-b11a-5d9c-e26bed2c5125 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986ed5c4e400, cur 1552345984 expire 1552345834 last 1552345757 Mar 11 16:13:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:13:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 16:13:39 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 11 16:27:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 16:27:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 16:27:30 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b44c850a-e6f8-beb8-d2a7-c9fe53741873 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ebed00c00, cur 1552346850 expire 1552346700 last 1552346623 Mar 11 16:27:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:39:07 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 4d24aa60-38e5-801b-9189-738674f01934 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784412000, cur 1552347547 expire 1552347397 last 1552347320 Mar 11 16:39:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:39:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 11 16:39:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:40:23 fir-io1-s1 kernel: LustreError: 74752:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff987625b83600 x1625443404600960/t0(0) o104->fir-OST0002@10.8.6.33@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 11 16:40:23 fir-io1-s1 kernel: LustreError: 74752:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 48 previous similar messages Mar 11 16:40:29 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552347622/real 1552347622] req@ffff986dd54d5d00 x1625443404599808/t0(0) o104->fir-OST0008@10.8.6.33@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552347629 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 11 16:40:29 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 44 previous similar messages Mar 11 16:40:39 fir-io1-s1 kernel: LustreError: 96560:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9868932e5d00 x1625443404612080/t0(0) o104->fir-OST0006@10.8.6.33@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 11 16:40:39 fir-io1-s1 kernel: LustreError: 96560:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 1 previous similar message Mar 11 16:40:41 fir-io1-s1 kernel: LustreError: 110702:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff986f94c7cb00 x1625443404617840/t0(0) o104->fir-OST0006@10.8.6.33@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 11 16:42:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ae00caa-cbf0-7459-d290-f56a87e71bb5 (at 10.8.6.33@o2ib6) Mar 11 16:42:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:53:12 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 11 16:53:12 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 11 16:53:52 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 11 16:53:53 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 11 16:53:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 16:53:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 11 16:53:54 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 16:54:02 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST0000: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Mar 11 16:54:02 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 11 16:54:02 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 11 16:54:02 fir-io1-s1 kernel: Lustre: Skipped 34 previous similar messages Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1522189 to 0x0:1522337 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1521955 to 0x0:1522049 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1522458 to 0x0:1522593 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1522545 to 0x0:1522689 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1522178 to 0x0:1522241 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1522072 to 0x0:1522209 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2307894 to 0x6c0000400:2308033 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2307342 to 0x5c0000400:2307841 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2306949 to 0x8c0000402:2307073 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2307419 to 0xc40000402:2307489 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2307725 to 0x580000400:2307905 Mar 11 16:54:45 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2307549 to 0xc80000402:2307681 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2120669 to 0x8c0000400:2120705 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2120692 to 0x6c0000401:2120737 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2121076 to 0x5c0000401:2121153 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2120673 to 0xc80000400:2120705 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2120873 to 0xc40000400:2120897 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2121279 to 0x580000401:2121313 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1765898 to 0x5c0000402:1766177 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1764411 to 0x6c0000402:1764705 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1764196 to 0x8c0000401:1764481 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1764141 to 0xc80000401:1764481 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1764407 to 0xc40000401:1764705 Mar 11 16:54:46 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1765774 to 0x580000402:1766049 Mar 11 17:49:41 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 5 seconds Mar 11 17:49:41 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Mar 11 17:49:41 fir-io1-s1 kernel: Lustre: 91458:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1552351779/real 1552351781] req@ffff986179383000 x1625443453360928/t0(0) o400->fir-MDT0003-lwp-OST0002@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1552351786 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Mar 11 17:49:41 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0002: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 11 17:49:41 fir-io1-s1 kernel: Lustre: 91458:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 11 17:49:42 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 22 seconds Mar 11 17:49:42 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 4 previous similar messages Mar 11 17:49:42 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0004: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 11 17:49:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 17:49:45 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 11 17:49:45 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages Mar 11 17:49:45 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 11 17:49:45 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 11 17:49:48 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 11 17:49:48 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Mar 11 17:50:38 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 42 seconds Mar 11 17:50:38 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 3 previous similar messages Mar 11 17:50:40 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 11 17:50:41 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 11 17:50:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 17:51:01 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 11 17:51:01 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0006: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 11 17:51:01 fir-io1-s1 kernel: LustreError: Skipped 23 previous similar messages Mar 11 17:51:01 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0006: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 11 17:51:01 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 17:51:01 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 17:51:32 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 11 17:51:32 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 11 17:51:43 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 11 17:51:43 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 17:52:16 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 11 17:52:16 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 11 17:52:16 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST000a: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 11 17:52:16 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 11 17:52:34 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1764742 to 0x6c0000402:1764769 Mar 11 17:52:34 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1764518 to 0xc80000401:1764545 Mar 11 17:52:34 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1766086 to 0x580000402:1766113 Mar 11 17:52:34 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1764519 to 0x8c0000401:1764545 Mar 11 17:52:34 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1766213 to 0x5c0000402:1766241 Mar 11 17:52:34 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1764741 to 0xc40000401:1764769 Mar 11 17:52:35 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2121283 to 0x5c0000401:2121313 Mar 11 17:52:35 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2120836 to 0x8c0000400:2120865 Mar 11 17:52:35 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2120865 to 0x6c0000401:2120897 Mar 11 17:52:35 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2121028 to 0xc40000400:2121057 Mar 11 17:52:35 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2121445 to 0x580000401:2121473 Mar 11 17:52:35 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2120835 to 0xc80000400:2120865 Mar 11 17:55:34 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client dd4676e2-1049-5102-d72d-7ea95888008f (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480073d400, cur 1552352134 expire 1552351984 last 1552351907 Mar 11 17:55:34 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 17:58:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e8d2fb7b-0f2d-5cac-c206-394ecb7a140a (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855125dd800, cur 1552352313 expire 1552352163 last 1552352086 Mar 11 17:58:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 18:05:18 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1524524 to 0x0:1524609 Mar 11 18:05:18 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1524444 to 0x0:1524481 Mar 11 18:05:18 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1524793 to 0x0:1524833 Mar 11 18:05:18 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1524256 to 0x0:1524289 Mar 11 18:05:18 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1524879 to 0x0:1524897 Mar 11 18:05:18 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1524411 to 0x0:1524449 Mar 11 18:05:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 18:05:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 18:05:43 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2308453 to 0x6c0000400:2308481 Mar 11 18:05:43 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2307489 to 0x8c0000402:2307521 Mar 11 18:05:43 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2308261 to 0x5c0000400:2308289 Mar 11 18:05:43 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2307905 to 0xc40000402:2307937 Mar 11 18:05:43 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2308322 to 0x580000400:2308353 Mar 11 18:05:43 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2308100 to 0xc80000402:2308129 Mar 11 18:09:08 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cafc6174-bf79-7815-2601-960b22eb58c0 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a47fbe400, cur 1552352948 expire 1552352798 last 1552352721 Mar 11 18:09:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 18:11:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 11 18:11:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 19:51:12 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client bbb95397-e102-1a5d-948e-96a002c7e7b7 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98782e6d3800, cur 1552359072 expire 1552358922 last 1552358845 Mar 11 19:51:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 19:53:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 19:53:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:04:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client bf5c17b9-ed8b-d871-84e5-963a91b1e83e (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98707f69b800, cur 1552359840 expire 1552359690 last 1552359613 Mar 11 20:04:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:08:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 20:08:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:17:19 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 56018bcd-3f65-35d1-3cb8-9d32ec341895 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867860b9000, cur 1552360639 expire 1552360489 last 1552360412 Mar 11 20:17:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:17:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 20:17:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:24:54 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 012dfd41-d0ea-9eeb-2aff-b692188b5e28 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481df07400, cur 1552361094 expire 1552360944 last 1552360867 Mar 11 20:24:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:26:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 81f235e0-f3ab-37af-5570-4de4e905b5ae (at 10.8.10.29@o2ib6) in 195 seconds. I think it's dead, and I am evicting it. exp ffff986671180400, cur 1552361170 expire 1552361020 last 1552360975 Mar 11 20:26:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:26:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 81f235e0-f3ab-37af-5570-4de4e905b5ae (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867850b7000, cur 1552361202 expire 1552361052 last 1552360975 Mar 11 20:26:42 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 11 20:27:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 11 20:27:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:28:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 20:28:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:33:27 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552361596/real 1552361596] req@ffff9849aebe8000 x1625444013845920/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552361607 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 11 20:33:27 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 11 20:33:38 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552361607/real 1552361607] req@ffff9849aebe8000 x1625444013845920/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552361618 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 20:33:49 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552361618/real 1552361618] req@ffff9849aebe8000 x1625444013845920/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552361629 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 20:34:00 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552361629/real 1552361629] req@ffff9849aebe8000 x1625444013845920/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552361640 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 20:34:03 fir-io1-s1 kernel: LustreError: 96683:0:(ldlm_lib.c:3273:target_bulk_io()) @@@ truncated bulk READ 0(131072) req@ffff9848831e7450 x1627759050495296/t0(0) o3->15dccff9-46b6-c7dc-e3a4-f085a24f0eb3@10.8.27.23@o2ib6:124/0 lens 488/440 e 3 to 0 dl 1552361664 ref 1 fl Interpret:/0/0 rc 0/0 Mar 11 20:34:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Bulk IO read error with 15dccff9-46b6-c7dc-e3a4-f085a24f0eb3 (at 10.8.27.23@o2ib6), client will retry: rc -110 Mar 11 20:34:11 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552361640/real 1552361640] req@ffff9849aebe8000 x1625444013845920/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552361651 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 20:34:33 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552361662/real 1552361662] req@ffff9849aebe8000 x1625444013845920/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552361673 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 20:34:33 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 11 20:35:17 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552361706/real 1552361706] req@ffff9849aebe8000 x1625444013845920/t0(0) o106->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552361717 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 20:35:17 fir-io1-s1 kernel: Lustre: 96274:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 11 20:35:50 fir-io1-s1 kernel: LustreError: 96274:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from glimpse AST (req@ffff9849aebe8000 x1625444013845920 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff98386bd4ee40/0x49e1862975da73b7 lrc: 3/0,0 mode: PW/PW res: [0x6c0000401:0x208a24:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x40000000000000 nid: 10.8.27.23@o2ib6 remote: 0xc55ba83cbac5eea7 expref: 8 pid: 110617 timeout: 0 lvb_type: 0 Mar 11 20:35:50 fir-io1-s1 kernel: LustreError: 96274:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Mar 11 20:35:50 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 11 20:35:50 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 11 20:35:50 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552361750s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff98386bd4ee40/0x49e1862975da73b7 lrc: 3/0,0 mode: PW/PW res: [0x6c0000401:0x208a24:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x40000000000000 nid: 10.8.27.23@o2ib6 remote: 0xc55ba83cbac5eea7 expref: 9 pid: 110617 timeout: 0 lvb_type: 0 Mar 11 20:35:50 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Mar 11 20:36:37 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 15dccff9-46b6-c7dc-e3a4-f085a24f0eb3 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868ff400, cur 1552361797 expire 1552361647 last 1552361570 Mar 11 20:36:37 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 11 20:36:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 15dccff9-46b6-c7dc-e3a4-f085a24f0eb3 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868ff800, cur 1552361809 expire 1552361659 last 1552361582 Mar 11 20:36:49 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 11 20:37:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 20:37:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:37:53 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 045fdfb5-03eb-62f2-da49-431dbfaa4608 (at 10.8.10.29@o2ib6) in 210 seconds. I think it's dead, and I am evicting it. exp ffff98480073bc00, cur 1552361873 expire 1552361723 last 1552361663 Mar 11 20:40:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 20:40:11 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 20:56:33 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cba662c5-504c-bad0-9317-3d60047dd718 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984904991c00, cur 1552362993 expire 1552362843 last 1552362766 Mar 11 20:56:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 20:59:59 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fa97181c-c3fd-320d-d911-2f4ba6803604 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0df0000, cur 1552363199 expire 1552363049 last 1552362972 Mar 11 20:59:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:01:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 21:01:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:04:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 11 21:04:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:10:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 69f75046-043d-f0ff-551a-5ad862bbeb54 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a285c00, cur 1552363837 expire 1552363687 last 1552363610 Mar 11 21:10:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:15:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 21:15:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:17:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 992ffb60-3268-a53f-b30a-69dc7103f8b0 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867844b6400, cur 1552364257 expire 1552364107 last 1552364030 Mar 11 21:17:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:18:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 21:18:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:25:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client dcf25227-12f1-7060-dca8-3e037a4a1e20 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b909400, cur 1552364713 expire 1552364563 last 1552364486 Mar 11 21:25:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:25:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 21:25:52 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 11 21:33:26 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 24b86b39-dd15-ef54-8593-b237ad9a6e8c (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481a6c7800, cur 1552365206 expire 1552365056 last 1552364979 Mar 11 21:33:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:33:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 11 21:33:41 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 11 21:38:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 21:38:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 21:45:14 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4f8f37dc-a95d-1f61-7379-5cde58195b87 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f67c000, cur 1552365914 expire 1552365764 last 1552365687 Mar 11 21:45:14 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 11 21:45:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 21:45:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 22:01:03 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c4a4efab-6dfa-0d33-fad6-8921f13b1a12 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a7d000, cur 1552366863 expire 1552366713 last 1552366636 Mar 11 22:01:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 22:03:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 22:03:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 22:21:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e6b7344d-4fc8-5a41-1314-028eebe1009a (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857f2766800, cur 1552368066 expire 1552367916 last 1552367839 Mar 11 22:21:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 22:26:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 22:26:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 22:35:19 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 12e60230-4daf-1eb8-ea5d-b8b6e76f1626 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d28c00, cur 1552368919 expire 1552368769 last 1552368692 Mar 11 22:35:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 22:38:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 22:38:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 23:04:51 fir-io1-s1 kernel: Lustre: 49829:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552370684/real 1552370684] req@ffff984386cb3f00 x1625444750464912/t0(0) o106->fir-OST0008@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552370691 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 11 23:04:51 fir-io1-s1 kernel: Lustre: 49833:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552370684/real 1552370684] req@ffff983d53d10900 x1625444750464864/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552370691 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 11 23:04:51 fir-io1-s1 kernel: Lustre: 49833:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 11 23:05:05 fir-io1-s1 kernel: Lustre: 49818:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552370698/real 1552370698] req@ffff983fbeeeec00 x1625444750464928/t0(0) o106->fir-OST000a@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552370705 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 23:05:05 fir-io1-s1 kernel: Lustre: 96914:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552370698/real 1552370698] req@ffff9840c5ee8000 x1625444750464896/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552370705 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 23:05:05 fir-io1-s1 kernel: Lustre: 96914:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 11 23:05:05 fir-io1-s1 kernel: Lustre: 49818:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 11 23:05:26 fir-io1-s1 kernel: Lustre: 49833:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552370719/real 1552370719] req@ffff983d53d10900 x1625444750464864/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552370726 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 23:05:26 fir-io1-s1 kernel: Lustre: 49833:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 11 23:06:08 fir-io1-s1 kernel: Lustre: 96914:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552370761/real 1552370761] req@ffff9840c5ee8000 x1625444750464896/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552370768 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 23:06:08 fir-io1-s1 kernel: Lustre: 96914:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Mar 11 23:07:25 fir-io1-s1 kernel: Lustre: 96914:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552370838/real 1552370838] req@ffff9840c5ee8000 x1625444750464896/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552370845 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 11 23:07:25 fir-io1-s1 kernel: Lustre: 96914:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 44 previous similar messages Mar 11 23:07:45 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b0196eba-5be0-be72-aa34-563ee175617d (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858aac90c00, cur 1552370865 expire 1552370715 last 1552370638 Mar 11 23:07:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 23:07:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b0196eba-5be0-be72-aa34-563ee175617d (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d73800, cur 1552370868 expire 1552370718 last 1552370641 Mar 11 23:07:48 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 11 23:08:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 11 23:08:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 11 23:20:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 46ff3438-10ee-6520-2cba-3ead3a06c7b2 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a6b800, cur 1552371619 expire 1552371469 last 1552371392 Mar 11 23:20:19 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 11 23:22:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 11 23:22:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 00:13:53 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 00:30:30 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 00:39:41 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 00:46:22 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 00:56:30 fir-io1-s1 kernel: LNetError: 91391:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 01:00:55 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 01:13:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 01:13:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 01:15:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 16268e6b-ebaa-77e7-1e68-4faa2d988082 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848341fe800, cur 1552378510 expire 1552378360 last 1552378283 Mar 12 01:15:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 01:23:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 01:23:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 01:23:59 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5507b13b-ce30-7c94-f7e7-87196317b96a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98497c8c0000, cur 1552379039 expire 1552378889 last 1552378812 Mar 12 01:23:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 01:36:16 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 01:38:49 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 01:47:29 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 02:03:27 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 02:12:40 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 02:23:09 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 02:26:21 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ad66a615-6365-752b-e49f-ea0aaaac0ff2 (at 10.8.14.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c85bc00, cur 1552382781 expire 1552382631 last 1552382554 Mar 12 02:26:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 02:32:27 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 02:32:42 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 60f0bbfc-53b4-88d1-4419-840662d6eeef (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98765c5d6000, cur 1552383162 expire 1552383012 last 1552382935 Mar 12 02:32:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 02:32:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 60f0bbfc-53b4-88d1-4419-840662d6eeef (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983bb7f1e400, cur 1552383178 expire 1552383028 last 1552382951 Mar 12 02:32:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 02:33:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 12 02:33:11 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 02:39:30 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 61b168d8-d9d3-b4ef-2b7a-2d5460492d35 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575eaa6c00, cur 1552383570 expire 1552383420 last 1552383343 Mar 12 02:39:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 12 02:39:37 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 12 02:39:47 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 02:45:06 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4b0b62e0-bc54-8928-1803-006c24f7d2c5 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e12400, cur 1552383906 expire 1552383756 last 1552383679 Mar 12 02:45:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 02:45:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 12 02:45:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 03:01:23 fir-io1-s1 kernel: LNetError: 91393:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 03:05:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Mar 12 03:05:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 03:06:28 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 648d0e11-2a53-216e-6d6d-244480c5b301 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984831a9f800, cur 1552385188 expire 1552385038 last 1552384961 Mar 12 03:06:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 03:09:49 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 03:12:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f3a357e0-f66c-d7f2-ee4c-00445e5c0c19 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f54bcfc00, cur 1552385558 expire 1552385408 last 1552385331 Mar 12 03:12:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 03:12:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f3a357e0-f66c-d7f2-ee4c-00445e5c0c19 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e7d07cc00, cur 1552385564 expire 1552385414 last 1552385337 Mar 12 03:12:44 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 03:26:52 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 03:35:49 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 03:45:23 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 03:52:35 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 03:53:21 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 03:56:19 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:07:10 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:08:56 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:10:15 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:13:36 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:14:00 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:16:04 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:19:48 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:22:36 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:22:36 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Mar 12 04:27:17 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 04:27:17 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 12 04:38:32 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 05:03:27 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 05:03:27 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 12 05:10:40 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 05:16:48 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 05:31:27 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 05:31:27 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 12 05:45:05 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 05:45:05 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1 previous similar message Mar 12 05:50:12 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 7a486713-dfdf-5c48-4264-3ceccf540748 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984996397c00, cur 1552395012 expire 1552394862 last 1552394785 Mar 12 05:51:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 05:51:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 05:55:18 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 05:55:18 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages Mar 12 06:18:01 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 06:18:01 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 2 previous similar messages Mar 12 06:21:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a580d0b8-d5ef-80ac-c3d3-bd1fa1a2138b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c004800, cur 1552396897 expire 1552396747 last 1552396670 Mar 12 06:21:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 06:24:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 06:24:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 06:29:06 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 06:45:01 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 07:19:14 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 642d1b42-396c-3252-a42a-4638aa80e945 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98654faafc00, cur 1552400354 expire 1552400204 last 1552400127 Mar 12 07:19:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 07:22:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 07:22:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 07:40:15 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 07:47:50 fir-io1-s1 kernel: LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 07:50:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3f1b49d5-f8a1-109f-8839-2462863b601d (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762602c00, cur 1552402226 expire 1552402076 last 1552401999 Mar 12 07:50:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 07:53:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 07:53:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 07:54:33 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 421d0c5e-a17c-91b7-c1b5-b7f75f589190 (at 10.8.12.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985acab68000, cur 1552402473 expire 1552402323 last 1552402246 Mar 12 07:54:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 07:57:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 80a4d388-dfdd-5a6f-35e8-374e461c44ba (at 10.8.12.35@o2ib6) Mar 12 07:57:25 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 07:57:43 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 08:03:16 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 08:36:16 fir-io1-s1 kernel: LNetError: 91387:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 08:38:54 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 08:46:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 46c0728c-191a-67e5-c3e0-bcb47862d3f2 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d70000, cur 1552405594 expire 1552405444 last 1552405367 Mar 12 08:46:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 08:48:54 fir-io1-s1 kernel: LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 08:49:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 08:49:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 08:55:51 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 08:55:51 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 08:55:58 fir-io1-s1 kernel: Lustre: 91463:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552406151/real 1552406151] req@ffff98666ff26c00 x1625467294545856/t0(0) o400->fir-MDT0000-lwp-OST0002@10.0.10.51@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1552406158 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 12 08:55:58 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0008: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 08:55:58 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0008: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 08:55:58 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 12 08:55:58 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 12 08:55:58 fir-io1-s1 kernel: Lustre: 91463:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Mar 12 08:56:24 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 20 seconds Mar 12 08:56:24 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Mar 12 08:56:26 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 41 seconds Mar 12 08:56:26 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 10 previous similar messages Mar 12 08:56:29 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 0 seconds Mar 12 08:56:29 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages Mar 12 08:56:29 fir-io1-s1 kernel: Lustre: 91462:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1552406183/real 1552406189] req@ffff9860f358bc00 x1625467294546528/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1552406190 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Mar 12 08:56:29 fir-io1-s1 kernel: Lustre: 91462:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 12 08:56:29 fir-io1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail Mar 12 08:56:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 1 seconds Mar 12 08:56:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 6 previous similar messages Mar 12 08:56:55 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 11 seconds Mar 12 08:57:19 fir-io1-s1 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x253aba2c0652a450 to 0x4cd4038d7c2c19a0 Mar 12 08:57:19 fir-io1-s1 kernel: Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 12 08:57:25 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 12 08:57:25 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (6): c: 0, oc: 0, rc: 8 Mar 12 08:57:50 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 12 08:57:50 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 12 08:58:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 12 08:58:09 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 12 08:58:09 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 12 08:58:09 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 12 08:58:09 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Mar 12 08:58:34 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0003-lwp-OST0000: This client was evicted by fir-MDT0003; in progress operations using this service will fail. Mar 12 08:58:34 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 12 08:58:34 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0000: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 12 08:58:34 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1541594 to 0x0:1541633 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1541453 to 0x0:1541633 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1541593 to 0x0:1541633 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1542059 to 0x0:1542209 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1541740 to 0x0:1541793 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1541971 to 0x0:1542145 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2322516 to 0x6c0000400:2322593 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2322311 to 0x5c0000400:2322337 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2322162 to 0xc80000402:2322177 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2321505 to 0x8c0000402:2321537 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2322385 to 0x580000400:2322401 Mar 12 08:58:58 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2321957 to 0xc40000402:2321985 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1816034 to 0x6c0000402:1816353 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1817557 to 0x5c0000402:1817729 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1817415 to 0x580000402:1817537 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1815823 to 0x8c0000401:1816033 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1816041 to 0xc40000401:1816193 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1815799 to 0xc80000401:1815969 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2178169 to 0x6c0000401:2178209 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2178352 to 0xc40000400:2178369 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2178131 to 0x8c0000400:2178177 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2178561 to 0x5c0000401:2178593 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2178146 to 0xc80000400:2178177 Mar 12 08:58:59 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2178729 to 0x580000401:2178753 Mar 12 09:05:54 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:07:34 fir-io1-s1 kernel: LNetError: 91388:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:16:46 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:23:21 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:25:10 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:26:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 46b4e840-29f4-9bc6-9140-4fedbe95f338 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd0f3800, cur 1552407999 expire 1552407849 last 1552407772 Mar 12 09:26:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 09:26:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 12 09:26:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 09:30:19 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:44:24 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:47:44 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:48:22 fir-io1-s1 kernel: LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 09:52:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 8f4d1584-ccf7-99f3-3ded-a6a55bf3e318 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867869e5000, cur 1552409532 expire 1552409382 last 1552409305 Mar 12 09:52:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 09:54:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 09:54:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 10:03:44 fir-io1-s1 kernel: LNetError: 91384:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 10:11:51 fir-io1-s1 kernel: LNetError: 91386:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 10:16:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client dcf3b1de-8bc6-8ef2-b656-8d02255ebbd6 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd440800, cur 1552411001 expire 1552410851 last 1552410774 Mar 12 10:16:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 10:19:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 10:19:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 10:32:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client dd89ba42-e479-8760-ca23-3ee3f55e2533 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762e4c800, cur 1552411956 expire 1552411806 last 1552411729 Mar 12 10:32:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 10:34:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 10:34:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 10:40:13 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a2728456-b445-3f9a-0337-d45dd1a9d361 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857590d2000, cur 1552412413 expire 1552412263 last 1552412186 Mar 12 10:40:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 10:44:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 10:44:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 10:48:58 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 10:59:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9d3d77ed-d29e-9f13-e89a-345a7254f2bf (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857598aac00, cur 1552413574 expire 1552413424 last 1552413347 Mar 12 10:59:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:00:17 fir-io1-s1 kernel: LNetError: 91385:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 11:02:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 11:02:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:11:28 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f4c946f3-e24f-e714-9a44-9fd3d5ada9e9 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832b4d800, cur 1552414288 expire 1552414138 last 1552414061 Mar 12 11:11:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:14:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 11:14:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:26:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6e92841e-b558-6b98-94bb-73bba9fa7daf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481a6c7000, cur 1552415200 expire 1552415050 last 1552414973 Mar 12 11:26:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:27:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 11:27:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:28:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.10.29@o2ib6) Mar 12 11:28:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:30:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Mar 12 11:30:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:35:44 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 11:36:49 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 78ea01e9-52f1-d082-da29-1b41f2e65ed8 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857618e2800, cur 1552415809 expire 1552415659 last 1552415582 Mar 12 11:36:49 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 11:36:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 11:36:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:45:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 11:45:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:46:33 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 70c0da46-e4a6-d49a-b6d4-a60e1a8f75fb (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a36800, cur 1552416393 expire 1552416243 last 1552416166 Mar 12 11:46:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:51:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client beaaa05c-05cd-8445-43af-0c799ce2709c (at 10.8.8.37@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762747c00, cur 1552416690 expire 1552416540 last 1552416463 Mar 12 11:51:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:52:46 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 83296373-083a-aa5f-e28d-5f36172537d8 (at 10.8.20.15@o2ib6) in 185 seconds. I think it's dead, and I am evicting it. exp ffff98582bba6000, cur 1552416766 expire 1552416616 last 1552416581 Mar 12 11:52:46 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Mar 12 11:55:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 11:55:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:57:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2aa90518-227d-4557-9238-cd8cd884ba59 (at 10.8.26.19@o2ib6) Mar 12 11:57:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 11:58:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce2a5a1a-545c-760b-44a7-8c19aadb7a36 (at 10.9.107.71@o2ib4) Mar 12 11:58:15 fir-io1-s1 kernel: Lustre: Skipped 58 previous similar messages Mar 12 11:59:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4c4f9b6e-ffa8-1fee-3df9-3f645a83c731 (at 10.8.24.6@o2ib6) Mar 12 11:59:36 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Mar 12 12:01:38 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 72cd44c0-4110-2ad2-9bb8-649a4532b77f (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985769986800, cur 1552417298 expire 1552417148 last 1552417071 Mar 12 12:01:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 12:02:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f7d79af9-3424-b0d8-6dc6-f23e0df4e16a (at 10.8.6.32@o2ib6) Mar 12 12:02:45 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Mar 12 12:08:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 94e44c44-6d41-294c-102c-43da8bc36188 (at 10.8.10.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4c4800, cur 1552417731 expire 1552417581 last 1552417504 Mar 12 12:08:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 12:31:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a8839ca2-c0ce-9339-9ffb-7fca81f9a92a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678680a800, cur 1552419061 expire 1552418911 last 1552418834 Mar 12 12:31:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 12:31:54 fir-io1-s1 kernel: LNetError: 91381:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 12:33:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 12:33:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 77323:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.61@o2ib4) returned error from blocking AST (req@ffff986200aa2d00 x1625475636682848 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff9865f36321c0/0x49e1862c6fc285ae lrc: 4/0,0 mode: PR/PR res: [0xc40000402:0x22972a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400000020 nid: 10.9.0.61@o2ib4 remote: 0x2d8d53d7181c4bdb expref: 53 pid: 96248 timeout: 2767288 lvb_type: 1 Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 77323:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.9.0.61@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1s: evicting client at 10.9.0.61@o2ib4 ns: filter-fir-OST0006_UUID lock: ffff9865f36321c0/0x49e1862c6fc285ae lrc: 3/0,0 mode: PR/PR res: [0xc40000402:0x22972a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400000020 nid: 10.9.0.61@o2ib4 remote: 0x2d8d53d7181c4bdb expref: 54 pid: 96248 timeout: 0 lvb_type: 1 Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 96268:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.0.61@o2ib4) returned error from blocking AST (req@ffff984969f93c00 x1625475637112048 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff98694d6bd580/0x49e1862c6fc285e6 lrc: 4/0,0 mode: PR/PR res: [0xc80000402:0x2297ac:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x60000400000020 nid: 10.9.0.61@o2ib4 remote: 0x2d8d53d7181c4c67 expref: 54 pid: 97132 timeout: 2767289 lvb_type: 1 Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.9.0.61@o2ib4 was evicted due to a lock blocking callback time out: rc -107 Mar 12 12:37:18 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.9.0.61@o2ib4 ns: filter-fir-OST0008_UUID lock: ffff98694d6bd580/0x49e1862c6fc285e6 lrc: 3/0,0 mode: PR/PR res: [0xc80000402:0x2297ac:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x60000400000020 nid: 10.9.0.61@o2ib4 remote: 0x2d8d53d7181c4c67 expref: 55 pid: 97132 timeout: 0 lvb_type: 1 Mar 12 12:37:22 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552419435/real 1552419435] req@ffff984e68009200 x1625475635379728/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552419442 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 12:37:29 fir-io1-s1 kernel: Lustre: 96896:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552419442/real 1552419442] req@ffff9847015f1500 x1625475639256528/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552419449 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 12:37:29 fir-io1-s1 kernel: Lustre: 96896:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 12 12:37:31 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 7d62cb0a-533f-0457-60c7-febdd8c21b70 (at 10.8.0.68@o2ib6) in 208 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a7e000, cur 1552419451 expire 1552419301 last 1552419243 Mar 12 12:37:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 12:37:39 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552419452/real 1552419452] req@ffff985219e87b00 x1625475641137216/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552419459 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 12:37:39 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Mar 12 12:37:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Mar 12 12:37:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 12:37:57 fir-io1-s1 kernel: Lustre: 96897:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552419470/real 1552419470] req@ffff985567290600 x1625475649029376/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552419477 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 12:37:57 fir-io1-s1 kernel: Lustre: 96897:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 60 previous similar messages Mar 12 12:38:02 fir-io1-s1 kernel: LNetError: 91380:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 12:38:35 fir-io1-s1 kernel: Lustre: 96566:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552419508/real 1552419508] req@ffff98609b3eb900 x1625475641340320/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552419515 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 12:38:35 fir-io1-s1 kernel: Lustre: 96566:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 119 previous similar messages Mar 12 12:39:50 fir-io1-s1 kernel: Lustre: 110633:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552419583/real 1552419583] req@ffff985b08a03f00 x1625475695846640/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552419590 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 12:39:50 fir-io1-s1 kernel: Lustre: 110633:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 464 previous similar messages Mar 12 12:40:23 fir-io1-s1 kernel: LustreError: 94237:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff984170d63c00 x1625475698431632 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff985344616540/0x49e1862c9a484309 lrc: 56/0,0 mode: PW/PW res: [0x1789bf:0x0:0x0].0x0 rrc: 56 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0x282f29ed39ed2ae9 expref: 5 pid: 96493 timeout: 0 lvb_type: 0 Mar 12 12:40:23 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 12 12:40:23 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 12 12:40:23 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552419623s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff985344616540/0x49e1862c9a484309 lrc: 56/0,0 mode: PW/PW res: [0x1789bf:0x0:0x0].0x0 rrc: 57 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x40000000000000 nid: 10.8.20.15@o2ib6 remote: 0x282f29ed39ed2ae9 expref: 6 pid: 96493 timeout: 0 lvb_type: 0 Mar 12 12:40:23 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 2 previous similar messages Mar 12 12:40:23 fir-io1-s1 kernel: LustreError: 94237:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Mar 12 12:40:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9e9c0882-3f15-dd96-7d4f-45564e48c021 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7ee800, cur 1552419639 expire 1552419489 last 1552419412 Mar 12 12:40:39 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 12 12:41:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 12:41:37 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 13:03:13 fir-io1-s1 kernel: LNetError: 91383:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 13:15:13 fir-io1-s1 kernel: LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 5) Mar 12 13:39:18 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client aba0eef5-a1cf-34df-15d9-59fa4ab202a4 (at 10.9.0.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9862a0328000, cur 1552423158 expire 1552423008 last 1552422931 Mar 12 13:39:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 13:51:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fc1bc7ce-5d0d-3dd1-97c3-5b7a2adca326 (at 10.9.0.2@o2ib4) Mar 12 13:51:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 14:13:01 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 2 seconds Mar 12 14:13:01 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 12 previous similar messages Mar 12 14:13:01 fir-io1-s1 kernel: Lustre: 91458:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1552425180/real 1552425181] req@ffff985dc6cce600 x1625479013655088/t0(0) o400->fir-MDT0001-lwp-OST0002@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1552425188 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Mar 12 14:13:01 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0006: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 14:13:01 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 12 14:13:01 fir-io1-s1 kernel: Lustre: 91458:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 245 previous similar messages Mar 12 14:13:02 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0008: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 14:13:02 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0004: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 14:13:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 14:13:04 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 12 14:13:04 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Mar 12 14:13:04 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 14:13:07 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0004: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 14:13:07 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0008: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 12 14:13:07 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 12 14:13:07 fir-io1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail Mar 12 14:13:07 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 12 14:13:09 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 103 seconds Mar 12 14:13:09 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 19 previous similar messages Mar 12 14:13:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 12 14:13:58 fir-io1-s1 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x4cd4038d7c2c19a0 to 0x974d7e52601ddf Mar 12 14:13:58 fir-io1-s1 kernel: Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 12 14:13:58 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 12 14:13:58 fir-io1-s1 kernel: LustreError: Skipped 17 previous similar messages Mar 12 14:13:58 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1838185 to 0x5c0000402:1838305 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1836765 to 0x6c0000402:1837121 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1837995 to 0x580000402:1838369 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1836676 to 0xc40000401:1836801 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1836474 to 0x8c0000401:1836545 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1836433 to 0xc80000401:1836545 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2194400 to 0x6c0000401:2194433 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2194339 to 0x8c0000400:2194369 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2194356 to 0xc80000400:2194401 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2194562 to 0xc40000400:2194593 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2194749 to 0x5c0000401:2194785 Mar 12 14:14:16 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2194911 to 0x580000401:2194945 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1542848 to 0x0:1542881 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1542850 to 0x0:1542881 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1543426 to 0x0:1543457 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1543359 to 0x0:1543393 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1543010 to 0x0:1543041 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1542840 to 0x0:1542881 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2330287 to 0x5c0000400:2330433 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2329497 to 0x8c0000402:2329537 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2330542 to 0x6c0000400:2330625 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2330130 to 0xc80000402:2330177 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2330371 to 0x580000400:2330401 Mar 12 14:14:19 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2329950 to 0xc40000402:2330017 Mar 12 14:18:10 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 6f635f45-ae2a-0aad-843c-d6486afb74d2 (at 10.8.10.33@o2ib6) Mar 12 14:18:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 14:33:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Mar 12 14:33:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 14:35:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.0.64@o2ib4) Mar 12 14:35:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 14:58:08 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 78e05fd4-5b0f-34da-43a8-f275cce0e43c (at 10.8.20.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865f0ece400, cur 1552427888 expire 1552427738 last 1552427661 Mar 12 14:58:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 15:02:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dcc55e80-12ea-0589-b9b3-fd1f0daa7f90 (at 10.9.108.65@o2ib4) Mar 12 15:02:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 15:03:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Mar 12 15:03:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 15:04:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c97311b2-9622-d925-9ad2-505310c59617 (at 10.9.108.43@o2ib4) Mar 12 15:04:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 15:04:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 479d2c4f-434d-5613-7a77-8c5939e93218 (at 10.9.108.63@o2ib4) Mar 12 15:04:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 15:04:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ed540f5a-5df6-5998-8f5c-40181564f690 (at 10.9.106.50@o2ib4) Mar 12 15:04:12 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 12 15:04:19 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9d363e4e-932d-cf45-d2fc-abcbef2a550d (at 10.9.106.50@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d7de51000, cur 1552428259 expire 1552428109 last 1552428032 Mar 12 15:04:19 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 15:04:20 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 166c4791-1a84-8206-fe53-c61fa08583c2 (at 10.9.103.19@o2ib4) Mar 12 15:04:20 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 12 15:04:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 31f9f1e5-0053-c9bc-655f-d68cfd64847e (at 10.8.21.32@o2ib6) Mar 12 15:04:38 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 12 15:06:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 69023f0b-28b9-1d08-eb1b-f3a097e42672 (at 10.8.3.14@o2ib6) Mar 12 15:06:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 15:26:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4abc7b2d-5f6a-da5f-d550-baa3bf9eb296 (at 10.8.20.8@o2ib6) Mar 12 15:26:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 15:26:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7a5d42a3-072b-ada7-8bd9-6b223c35b055 (at 10.8.20.9@o2ib6) Mar 12 15:26:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 16:16:46 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 975b15b6-cdfe-af36-a121-a3b805d090c5 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683b88d800, cur 1552432606 expire 1552432456 last 1552432379 Mar 12 16:16:46 fir-io1-s1 kernel: Lustre: Skipped 131 previous similar messages Mar 12 16:29:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5e4b60f-fe33-b991-7d48-5b8db7e07ab0 (at 10.8.0.67@o2ib6) Mar 12 16:29:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 16:29:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1f874194-eb5d-819e-2a70-00a794c8bff7 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984998f86800, cur 1552433391 expire 1552433241 last 1552433164 Mar 12 16:29:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 16:29:52 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 1f874194-eb5d-819e-2a70-00a794c8bff7 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f356400, cur 1552433392 expire 1552433242 last 1552433165 Mar 12 16:29:52 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 12 16:29:53 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 1f874194-eb5d-819e-2a70-00a794c8bff7 (at 10.8.0.67@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d1800, cur 1552433393 expire 1552433243 last 1552433166 Mar 12 16:29:53 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 12 16:31:07 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 17a41fbf-e40b-a6a4-6b12-32d00492793d (at 10.9.0.61@o2ib4) in 213 seconds. I think it's dead, and I am evicting it. exp ffff985756f5bc00, cur 1552433467 expire 1552433317 last 1552433254 Mar 12 16:31:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 17a41fbf-e40b-a6a4-6b12-32d00492793d (at 10.9.0.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9876b0151000, cur 1552433481 expire 1552433331 last 1552433254 Mar 12 16:31:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 16:31:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Mar 12 16:31:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 16:45:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 97bc8e0c-1614-4de0-a593-98b585b7fd0b (at 10.9.103.30@o2ib4) Mar 12 16:45:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 16:45:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 302c6952-5b6c-1588-cbd4-2c54f063f559 (at 10.9.103.37@o2ib4) Mar 12 16:45:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 16:45:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 40113a8d-c41f-508e-9772-7563fba01286 (at 10.9.105.5@o2ib4) Mar 12 16:45:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 16:45:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 01d5ada9-338b-fc1a-1541-7fe86bd87ddc (at 10.9.105.18@o2ib4) Mar 12 16:45:46 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 12 16:46:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1e03043d-571f-f709-9076-72fb7d056ac3 (at 10.9.103.43@o2ib4) Mar 12 16:46:34 fir-io1-s1 kernel: Lustre: Skipped 24 previous similar messages Mar 12 16:46:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 10ff98e7-db66-2848-037f-7cf095a0e8cc (at 10.9.113.8@o2ib4) Mar 12 16:46:42 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Mar 12 16:46:53 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 55cc7ce4-dbe0-16d5-81f3-b58ec238df6d (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851091ba800, cur 1552434413 expire 1552434263 last 1552434186 Mar 12 16:46:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3730432f-bcc1-09aa-920b-fdcba043b544 (at 10.9.105.25@o2ib4) Mar 12 16:46:59 fir-io1-s1 kernel: Lustre: Skipped 230 previous similar messages Mar 12 16:47:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8a305ec8-58cd-38d7-7085-a98f5d22aa5b (at 10.9.107.47@o2ib4) Mar 12 16:47:32 fir-io1-s1 kernel: Lustre: Skipped 239 previous similar messages Mar 12 16:49:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7b3b6f2b-764c-4642-1783-263f87e59249 (at 10.8.1.10@o2ib6) Mar 12 16:49:03 fir-io1-s1 kernel: Lustre: Skipped 160 previous similar messages Mar 12 16:54:52 fir-io1-s1 kernel: Lustre: 96946:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434885/real 1552434885] req@ffff985b2cd34b00 x1625479416707856/t0(0) o106->fir-OST0008@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434892 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 16:54:52 fir-io1-s1 kernel: Lustre: 77316:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434885/real 1552434885] req@ffff986266a85100 x1625479416707872/t0(0) o106->fir-OST000a@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434892 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 16:54:52 fir-io1-s1 kernel: Lustre: 77316:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Mar 12 16:54:55 fir-io1-s1 kernel: Lustre: 110034:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434888/real 1552434888] req@ffff9845c8300300 x1625479416772192/t0(0) o106->fir-OST0008@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434895 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 16:54:55 fir-io1-s1 kernel: Lustre: 110034:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 12 16:55:02 fir-io1-s1 kernel: Lustre: 96271:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434895/real 1552434895] req@ffff98408f3ce600 x1625479416772240/t0(0) o106->fir-OST0002@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434902 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 16:55:02 fir-io1-s1 kernel: Lustre: 96271:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 12 16:55:14 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434906/real 1552434906] req@ffff98590e498f00 x1625479416707904/t0(0) o106->fir-OST0002@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434913 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 16:55:14 fir-io1-s1 kernel: Lustre: 96514:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434906/real 1552434906] req@ffff985cba3a5700 x1625479416707824/t0(0) o106->fir-OST0006@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434913 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 16:55:14 fir-io1-s1 kernel: Lustre: 77316:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434906/real 1552434906] req@ffff986266a85100 x1625479416707872/t0(0) o106->fir-OST000a@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434913 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 16:55:14 fir-io1-s1 kernel: Lustre: 96514:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Mar 12 16:55:14 fir-io1-s1 kernel: Lustre: 77316:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Mar 12 16:55:14 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 12 16:55:51 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552434944/real 1552434944] req@ffff9855112f6000 x1625479417845088/t0(0) o106->fir-OST0002@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552434951 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 16:55:51 fir-io1-s1 kernel: Lustre: 96368:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 89 previous similar messages Mar 12 16:57:05 fir-io1-s1 kernel: LustreError: 96927:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.3.25@o2ib6) failed to reply to blocking AST (req@ffff984974107200 x1625479417451616 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff98384bb069c0/0x49e1862cf6ade97a lrc: 4/0,0 mode: PR/PR res: [0x6c0000402:0x1b6930:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.25@o2ib6 remote: 0x6ab6839011c2588e expref: 592 pid: 49822 timeout: 2782869 lvb_type: 1 Mar 12 16:57:05 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.3.25@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 12 16:57:05 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 12 16:57:05 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.3.25@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff98384bb069c0/0x49e1862cf6ade97a lrc: 3/0,0 mode: PR/PR res: [0x6c0000402:0x1b6930:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.3.25@o2ib6 remote: 0x6ab6839011c2588e expref: 593 pid: 49822 timeout: 0 lvb_type: 1 Mar 12 16:57:06 fir-io1-s1 kernel: Lustre: 96890:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552435019/real 1552435019] req@ffff9855112f2d00 x1625479417957104/t0(0) o106->fir-OST0002@10.8.3.25@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552435026 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 16:57:06 fir-io1-s1 kernel: Lustre: 96890:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 744 previous similar messages Mar 12 16:57:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 609a75ae-bcd8-62a8-78c3-eceb01368f0a (at 10.8.22.12@o2ib6) Mar 12 16:57:12 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Mar 12 16:58:06 fir-io1-s1 kernel: LNet: Service thread pid 96374 was inactive for 200.49s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 12 16:58:06 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Mar 12 16:58:06 fir-io1-s1 kernel: Pid: 96374, comm: ll_ost02_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 16:58:06 fir-io1-s1 kernel: Call Trace: Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 16:58:06 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 16:58:06 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552435086.96374 Mar 12 16:58:07 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f1b5bd10-52bd-7857-9b03-4cd0a53ba2a5 (at 10.8.22.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d200000, cur 1552435087 expire 1552434937 last 1552434860 Mar 12 16:58:07 fir-io1-s1 kernel: Lustre: Skipped 1571 previous similar messages Mar 12 16:58:07 fir-io1-s1 kernel: LNet: Service thread pid 96946 completed after 201.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 12 16:58:07 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 12 16:58:07 fir-io1-s1 kernel: LustreError: 94512:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff984965534b00 x1625479421227264/t0(0) o104->fir-OST000a@10.8.3.25@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 12 16:58:07 fir-io1-s1 kernel: LustreError: 94512:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 5 previous similar messages Mar 12 17:01:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8fb7db49-dbd0-262c-c529-3bbd4a2a2cec (at 10.8.22.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f9da9b400, cur 1552435309 expire 1552435159 last 1552435082 Mar 12 17:01:49 fir-io1-s1 kernel: Lustre: Skipped 352 previous similar messages Mar 12 17:10:08 fir-io1-s1 kernel: Lustre: 110571:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552435801/real 1552435801] req@ffff9844f1f55700 x1625479446990640/t0(0) o106->fir-OST0000@10.9.103.41@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552435808 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 17:10:08 fir-io1-s1 kernel: Lustre: 110571:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 712 previous similar messages Mar 12 17:10:29 fir-io1-s1 kernel: Lustre: 110620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552435822/real 1552435822] req@ffff983c22784800 x1625479446990672/t0(0) o106->fir-OST0006@10.9.103.41@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552435829 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 17:10:29 fir-io1-s1 kernel: Lustre: 110620:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Mar 12 17:11:11 fir-io1-s1 kernel: Lustre: 96784:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552435864/real 1552435864] req@ffff98383d50bf00 x1625479446990656/t0(0) o106->fir-OST0004@10.9.103.41@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552435871 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 17:11:11 fir-io1-s1 kernel: Lustre: 96784:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Mar 12 17:12:28 fir-io1-s1 kernel: Lustre: 96905:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552435941/real 1552435941] req@ffff98408f3cb300 x1625479446990608/t0(0) o106->fir-OST0002@10.9.103.41@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552435948 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 17:12:28 fir-io1-s1 kernel: Lustre: 96905:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 79 previous similar messages Mar 12 17:13:21 fir-io1-s1 kernel: LNet: Service thread pid 110620 was inactive for 200.11s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 12 17:13:21 fir-io1-s1 kernel: Pid: 110620, comm: ll_ost00_101 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 17:13:21 fir-io1-s1 kernel: Call Trace: Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 17:13:21 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552436001.110620 Mar 12 17:13:21 fir-io1-s1 kernel: Pid: 110571, comm: ll_ost00_092 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 17:13:21 fir-io1-s1 kernel: Call Trace: Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 17:13:21 fir-io1-s1 kernel: Pid: 96905, comm: ll_ost00_058 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 17:13:21 fir-io1-s1 kernel: Call Trace: Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 17:13:21 fir-io1-s1 kernel: Pid: 96784, comm: ll_ost00_050 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 17:13:21 fir-io1-s1 kernel: Call Trace: Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 17:13:21 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 17:13:23 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c4cd6672-8515-f7be-5507-e8598509eecf (at 10.9.103.41@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762b64400, cur 1552436003 expire 1552435853 last 1552435776 Mar 12 17:13:23 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 12 17:13:25 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c4cd6672-8515-f7be-5507-e8598509eecf (at 10.9.103.41@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785df5000, cur 1552436005 expire 1552435855 last 1552435778 Mar 12 17:13:25 fir-io1-s1 kernel: Lustre: Skipped 31 previous similar messages Mar 12 17:13:25 fir-io1-s1 kernel: LNet: Service thread pid 110571 completed after 203.81s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 12 17:13:25 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 12 17:20:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ad5813bc-0e1b-f4a4-6b7d-92ba9e63b92f (at 10.9.107.68@o2ib4) Mar 12 17:20:10 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 12 17:20:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f39d45bf-d6dd-b23d-2197-e7717f302794 (at 10.9.108.36@o2ib4) Mar 12 17:20:47 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 17:22:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 40010d6e-d29b-c686-3f5e-dc2316139f55 (at 10.9.107.27@o2ib4) Mar 12 17:22:03 fir-io1-s1 kernel: Lustre: Skipped 142 previous similar messages Mar 12 17:26:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c8bbd681-6d42-3196-83fc-d269a276781d (at 10.9.105.55@o2ib4) Mar 12 17:26:35 fir-io1-s1 kernel: Lustre: Skipped 77 previous similar messages Mar 12 17:31:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ef406a33-dbdc-c381-15c5-4fb662abecc1 (at 10.8.27.10@o2ib6) Mar 12 17:31:09 fir-io1-s1 kernel: Lustre: Skipped 293 previous similar messages Mar 12 17:36:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 713abe37-dbc6-d614-017d-c665f17ff374 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784489800, cur 1552437411 expire 1552437261 last 1552437184 Mar 12 17:36:51 fir-io1-s1 kernel: Lustre: Skipped 63 previous similar messages Mar 12 17:40:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ac6b7868-9631-d06b-4e97-5105f55c80aa (at 10.8.21.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984fcccfe800, cur 1552437607 expire 1552437457 last 1552437380 Mar 12 17:40:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 17:40:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ac6b7868-9631-d06b-4e97-5105f55c80aa (at 10.8.21.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f236800, cur 1552437609 expire 1552437459 last 1552437382 Mar 12 17:40:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ac6b7868-9631-d06b-4e97-5105f55c80aa (at 10.8.21.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784488400, cur 1552437613 expire 1552437463 last 1552437386 Mar 12 17:40:13 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 12 17:40:18 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ac6b7868-9631-d06b-4e97-5105f55c80aa (at 10.8.21.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985573c61800, cur 1552437618 expire 1552437468 last 1552437391 Mar 12 17:40:18 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 12 17:42:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 826cadf5-386e-17e1-e989-7d6c22fc9192 (at 10.9.105.4@o2ib4) Mar 12 17:42:07 fir-io1-s1 kernel: Lustre: Skipped 370 previous similar messages Mar 12 17:43:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d2227cbe-3f54-2a4b-22a8-b0c8ac46a853 (at 10.8.11.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868053efc00, cur 1552437831 expire 1552437681 last 1552437604 Mar 12 17:45:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4c971ad5-f601-ef4d-1fe3-69c32fde1ff6 (at 10.8.15.1@o2ib6) in 177 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a79400, cur 1552437907 expire 1552437757 last 1552437730 Mar 12 17:45:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 17:45:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4c971ad5-f601-ef4d-1fe3-69c32fde1ff6 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9855125d8000, cur 1552437957 expire 1552437807 last 1552437730 Mar 12 17:47:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 662f7e26-873b-2cc5-a201-233bb3eeb208 (at 10.9.104.31@o2ib4) in 197 seconds. I think it's dead, and I am evicting it. exp ffff986b6d280c00, cur 1552438033 expire 1552437883 last 1552437836 Mar 12 17:47:13 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 18:08:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e43da944-d239-923f-8f68-10646264727b (at 10.8.21.20@o2ib6) Mar 12 18:08:06 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Mar 12 18:12:33 fir-io1-s1 kernel: Lustre: 96784:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552439546/real 1552439546] req@ffff983c95c22100 x1625479529498400/t0(0) o106->fir-OST0006@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552439553 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 18:12:33 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552439546/real 1552439546] req@ffff98420df09800 x1625479529498352/t0(0) o106->fir-OST0004@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552439553 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 18:12:33 fir-io1-s1 kernel: Lustre: 96332:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 60 previous similar messages Mar 12 18:12:33 fir-io1-s1 kernel: Lustre: 96784:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 12 18:12:54 fir-io1-s1 kernel: Lustre: 110034:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552439567/real 1552439567] req@ffff983a34ff8c00 x1625479529498320/t0(0) o106->fir-OST0000@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552439574 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 18:12:54 fir-io1-s1 kernel: Lustre: 111318:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552439567/real 1552439567] req@ffff983e87670c00 x1625479529498416/t0(0) o106->fir-OST0008@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552439574 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 18:12:54 fir-io1-s1 kernel: Lustre: 111318:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Mar 12 18:14:11 fir-io1-s1 kernel: Lustre: 111318:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552439644/real 1552439644] req@ffff983e87670c00 x1625479529498416/t0(0) o106->fir-OST0008@10.9.113.3@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552439651 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 18:14:11 fir-io1-s1 kernel: Lustre: 111318:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Mar 12 18:15:11 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5373418f-c199-48a4-ed10-fe0e00ae8bd8 (at 10.8.2.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867804f6000, cur 1552439711 expire 1552439561 last 1552439484 Mar 12 18:15:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 18:15:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 79465c7f-aac0-bcd9-93e3-6b42fd3a6813 (at 10.8.11.7@o2ib6) Mar 12 18:15:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 18:27:26 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552440439/real 1552440439] req@ffff984965533000 x1625479540227552/t0(0) o106->fir-OST0004@10.8.11.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552440446 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 18:27:26 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 37 previous similar messages Mar 12 18:27:46 fir-io1-s1 kernel: Lustre: 96569:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552440459/real 1552440459] req@ffff98612975b600 x1625479540305296/t0(0) o106->fir-OST000a@10.8.11.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552440466 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 18:27:46 fir-io1-s1 kernel: Lustre: 49823:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552440459/real 1552440459] req@ffff98612975d100 x1625479540305280/t0(0) o106->fir-OST0008@10.8.11.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552440466 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 18:27:46 fir-io1-s1 kernel: Lustre: 96784:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552440459/real 1552440459] req@ffff983c22786300 x1625479540305264/t0(0) o106->fir-OST0006@10.8.11.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552440466 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 18:27:46 fir-io1-s1 kernel: Lustre: 49823:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Mar 12 18:27:46 fir-io1-s1 kernel: Lustre: 96784:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Mar 12 18:29:03 fir-io1-s1 kernel: Lustre: 96894:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552440536/real 1552440536] req@ffff984b8704f800 x1625479540305248/t0(0) o106->fir-OST0004@10.8.11.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552440543 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 18:29:03 fir-io1-s1 kernel: Lustre: 49823:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552440536/real 1552440536] req@ffff98612975d100 x1625479540305280/t0(0) o106->fir-OST0008@10.8.11.12@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552440543 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 18:29:03 fir-io1-s1 kernel: Lustre: 49823:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 118 previous similar messages Mar 12 18:29:44 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 04238d0b-b633-4623-59fd-2443a4678dba (at 10.8.8.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9874ab1fa400, cur 1552440584 expire 1552440434 last 1552440357 Mar 12 18:29:44 fir-io1-s1 kernel: Lustre: Skipped 665 previous similar messages Mar 12 18:31:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9a689d17-d4b2-4205-f263-4b7a7e0d799c (at 10.9.108.35@o2ib4) in 209 seconds. I think it's dead, and I am evicting it. exp ffff984830f6b400, cur 1552440660 expire 1552440510 last 1552440451 Mar 12 18:31:00 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Mar 12 18:31:18 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9a689d17-d4b2-4205-f263-4b7a7e0d799c (at 10.9.108.35@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984830f6c000, cur 1552440678 expire 1552440528 last 1552440451 Mar 12 18:31:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 18:32:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 801be733-785b-a675-6116-74f5d07a121a (at 10.9.0.61@o2ib4) Mar 12 18:32:13 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 12 18:34:19 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8af16c22-f4d7-c31e-bbcd-6c0c27655fb0 (at 10.9.108.53@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d8fecd000, cur 1552440859 expire 1552440709 last 1552440632 Mar 12 18:34:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 18:36:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6c3169b7-2563-1f32-7d22-6e3f2ce9c349 (at 10.9.107.66@o2ib4) Mar 12 18:36:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 18:37:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dbe64986-3522-e2d0-d57e-b8c002fb5170 (at 10.9.106.33@o2ib4) Mar 12 18:37:27 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 18:38:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ea8bf8d4-a8e9-b686-d235-744a0baed4dd (at 10.9.108.22@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e799400, cur 1552441082 expire 1552440932 last 1552440855 Mar 12 18:38:02 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 12 18:38:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 135b49e4-9bbb-43fc-66e9-1f7ec8c75a96 (at 10.9.113.3@o2ib4) Mar 12 18:38:52 fir-io1-s1 kernel: Lustre: Skipped 70 previous similar messages Mar 12 18:39:18 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3b18b3ba-a169-785d-8fe7-7826fb5724a8 (at 10.8.22.18@o2ib6) in 157 seconds. I think it's dead, and I am evicting it. exp ffff98693edfc400, cur 1552441158 expire 1552441008 last 1552441001 Mar 12 18:39:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 18:42:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fd4739f5-61ac-b1c8-a4a0-9ad8b819daba (at 10.9.101.38@o2ib4) Mar 12 18:42:03 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Mar 12 18:47:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5a2d9e29-056b-95ad-717a-5193768c9b7c (at 10.8.23.29@o2ib6) Mar 12 18:47:08 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Mar 12 18:48:26 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0e0f15e5-5ab9-e4cd-5d82-bfd76a4b3f39 (at 10.9.102.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f06878400, cur 1552441706 expire 1552441556 last 1552441479 Mar 12 18:48:26 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 12 18:52:57 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f337aa5c-3989-ad65-70f1-6422db5ea508 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c19400, cur 1552441977 expire 1552441827 last 1552441750 Mar 12 18:52:57 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 12 18:57:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0127092e-ab70-ddef-6a66-286028d84f5d (at 10.9.102.43@o2ib4) Mar 12 18:57:17 fir-io1-s1 kernel: Lustre: Skipped 429 previous similar messages Mar 12 18:59:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c5c845ee-f5dc-4b01-7931-dfbd71c3c151 (at 10.8.11.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a5d800, cur 1552442395 expire 1552442245 last 1552442168 Mar 12 18:59:55 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 12 19:08:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 323d9e9b-f4db-fb48-a1bd-689a69067782 (at 10.8.25.2@o2ib6) Mar 12 19:08:33 fir-io1-s1 kernel: Lustre: Skipped 179 previous similar messages Mar 12 19:09:02 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e91dc4d5-453a-dd86-b0a3-608a191ab3d6 (at 10.8.23.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985763317000, cur 1552442942 expire 1552442792 last 1552442715 Mar 12 19:09:02 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 12 19:19:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 19:19:34 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Mar 12 19:27:35 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 614c7208-a46c-f363-3d6a-61d327796bc4 (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dffe000, cur 1552444055 expire 1552443905 last 1552443828 Mar 12 19:27:35 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 12 19:30:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c4f458ee-079e-1f6b-715d-4cc60d32c4b8 (at 10.8.11.4@o2ib6) Mar 12 19:30:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 19:40:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 3fcbba1a-8a26-3581-0ebf-ed57fe529918 (at 10.8.7.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0df3000, cur 1552444837 expire 1552444687 last 1552444610 Mar 12 19:40:37 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 12 19:46:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 75c103c5-8c22-70ed-cfb0-bd07e014990e (at 10.8.11.36@o2ib6) Mar 12 19:46:11 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 12 19:51:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client bd03bd68-7ea1-a33b-6409-106d6fa60b7c (at 10.8.10.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfa800, cur 1552445480 expire 1552445330 last 1552445253 Mar 12 19:51:20 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 19:53:47 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552445620/real 1552445620] req@ffff9860c355ec00 x1625479749568160/t0(0) o104->fir-OST000a@10.8.17.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552445627 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 19:53:47 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 57 previous similar messages Mar 12 19:54:29 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552445662/real 1552445662] req@ffff9860c355ec00 x1625479749568160/t0(0) o104->fir-OST000a@10.8.17.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552445669 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 19:54:29 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 12 19:55:25 fir-io1-s1 kernel: LustreError: 96253:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.17.3@o2ib6) failed to reply to blocking AST (req@ffff9860c355ec00 x1625479749568160 status 0 rc -110), evict it ns: filter-fir-OST000a_UUID lock: ffff984dcfd418c0/0x49e1862d00068b18 lrc: 4/0,0 mode: PW/PW res: [0x17880c:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400030020 nid: 10.8.17.3@o2ib6 remote: 0x8b5719db647d85cb expref: 5 pid: 96378 timeout: 2793568 lvb_type: 0 Mar 12 19:55:25 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.17.3@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 12 19:55:25 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.17.3@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff984dcfd418c0/0x49e1862d00068b18 lrc: 3/0,0 mode: PW/PW res: [0x17880c:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x60000400030020 nid: 10.8.17.3@o2ib6 remote: 0x8b5719db647d85cb expref: 6 pid: 96378 timeout: 0 lvb_type: 0 Mar 12 20:01:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fb0e351b-5ab8-b43d-813c-60db20cd78c1 (at 10.9.101.45@o2ib4) Mar 12 20:01:05 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 12 20:12:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 40a4c4f2-a940-462d-8df0-96fdeeb554f6 (at 10.9.101.33@o2ib4) Mar 12 20:12:20 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 12 20:22:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 86d44073-2a86-f18b-4a0f-e98051cdbb2e (at 10.9.105.51@o2ib4) Mar 12 20:22:37 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 20:38:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c46b7f3d-ec55-7fd7-e207-e9c6f5525b60 (at 10.8.0.68@o2ib6) Mar 12 20:38:27 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 12 20:50:02 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client cdea270d-db54-d524-f577-70542cdd6aac (at 10.9.106.32@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868aa000, cur 1552449002 expire 1552448852 last 1552448775 Mar 12 20:50:02 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Mar 12 20:59:10 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 44ddaef8-497b-24c9-aa94-19385d125d57 (at 10.8.13.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985c3221ec00, cur 1552449550 expire 1552449400 last 1552449323 Mar 12 20:59:10 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 12 20:59:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7cee7bf7-9aa1-cc50-5aed-b23b669bf632 (at 10.8.15.2@o2ib6) Mar 12 20:59:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 21:10:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 91f6fe25-4c34-a621-dd26-00e6ccf4cbba (at 10.9.106.32@o2ib4) Mar 12 21:10:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 21:12:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7f14aec-d465-a06a-f9b6-045d2e3bc764 (at 10.9.107.39@o2ib4) Mar 12 21:12:33 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 12 21:18:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 142a6e04-872a-1c7a-e7fe-9168d1c1b90d (at 10.8.9.5@o2ib6) Mar 12 21:18:09 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Mar 12 21:20:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 28f8d35e-1ad3-e136-5aaa-6c04ba23b3c5 (at 10.9.0.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864523bd800, cur 1552450856 expire 1552450706 last 1552450629 Mar 12 21:20:56 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 12 21:22:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e34b7e2a-029e-8f4d-15d7-359ba4013616 (at 10.8.20.15@o2ib6) in 153 seconds. I think it's dead, and I am evicting it. exp ffff9848947a5000, cur 1552450932 expire 1552450782 last 1552450779 Mar 12 21:22:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 21:30:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 98b7035a-35b8-cd21-9b10-3e7f2a49b7a7 (at 10.8.13.9@o2ib6) Mar 12 21:30:00 fir-io1-s1 kernel: Lustre: Skipped 132 previous similar messages Mar 12 21:34:30 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client e29706e4-56c0-a8b9-1ecb-e0b39a3d233b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef14c00, cur 1552451670 expire 1552451520 last 1552451443 Mar 12 21:34:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 21:39:58 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 041f82e7-36fd-3d04-fea2-1ca597a55bf1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a942c9000, cur 1552451998 expire 1552451848 last 1552451771 Mar 12 21:39:58 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 12 21:47:08 fir-io1-s1 kernel: LustreError: 96783:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from blocking AST (req@ffff9856daf80c00 x1625480308612480 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff98494ab11b00/0x49e1862d0f74baaf lrc: 4/0,0 mode: PR/PR res: [0x580000400:0x239c16:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x6b9830d1fad10ab1 expref: 8 pid: 96916 timeout: 2800329 lvb_type: 1 Mar 12 21:47:08 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 12 21:47:08 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff98494ab16c00/0x49e1862d0f74ba7e lrc: 3/0,0 mode: PR/PR res: [0x5c0000400:0x23a54b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400000020 nid: 10.8.20.15@o2ib6 remote: 0x6b9830d1fad1098b expref: 10 pid: 96916 timeout: 0 lvb_type: 1 Mar 12 21:47:08 fir-io1-s1 kernel: LustreError: 96783:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Mar 12 21:47:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 21:47:56 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 12 22:01:37 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 40b14a78-bba0-845b-c191-b42810beca2a (at 10.9.0.64@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9877a14f8c00, cur 1552453297 expire 1552453147 last 1552453070 Mar 12 22:01:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 22:01:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.0.64@o2ib4) Mar 12 22:01:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 22:07:02 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552453615/real 1552453615] req@ffff98575bafcb00 x1625480397771840/t0(0) o106->fir-OST0004@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552453622 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 12 22:07:02 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Mar 12 22:07:16 fir-io1-s1 kernel: Lustre: 77316:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552453629/real 1552453629] req@ffff985b991c8600 x1625480397771728/t0(0) o106->fir-OST0004@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552453636 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 22:07:16 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552453629/real 1552453629] req@ffff985f5d003f00 x1625480397771872/t0(0) o106->fir-OST0008@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552453636 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 22:07:16 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 12 22:07:16 fir-io1-s1 kernel: Lustre: 77316:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 12 22:07:37 fir-io1-s1 kernel: Lustre: 110633:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552453650/real 1552453650] req@ffff98612975c800 x1625480397771888/t0(0) o106->fir-OST000a@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552453657 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 22:07:37 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552453650/real 1552453650] req@ffff985f5d003f00 x1625480397771872/t0(0) o106->fir-OST0008@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552453657 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 22:07:37 fir-io1-s1 kernel: Lustre: 96374:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Mar 12 22:07:37 fir-io1-s1 kernel: Lustre: 110633:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 12 22:08:54 fir-io1-s1 kernel: Lustre: 94236:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552453727/real 1552453727] req@ffff98381460cb00 x1625480397771808/t0(0) o106->fir-OST0008@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552453734 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 22:08:54 fir-io1-s1 kernel: Lustre: 49822:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552453727/real 1552453727] req@ffff98390bb81b00 x1625480397771824/t0(0) o106->fir-OST000a@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552453734 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 12 22:08:54 fir-io1-s1 kernel: Lustre: 49822:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 74 previous similar messages Mar 12 22:08:54 fir-io1-s1 kernel: Lustre: 94236:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 12 22:10:15 fir-io1-s1 kernel: LNet: Service thread pid 110633 was inactive for 200.32s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 12 22:10:15 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 12 22:10:15 fir-io1-s1 kernel: Pid: 110633, comm: ll_ost02_099 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 22:10:15 fir-io1-s1 kernel: Call Trace: Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 22:10:15 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 22:10:15 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552453815.110633 Mar 12 22:10:16 fir-io1-s1 kernel: LNet: Service thread pid 96374 was inactive for 201.21s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 12 22:10:16 fir-io1-s1 kernel: Pid: 96374, comm: ll_ost02_027 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 22:10:16 fir-io1-s1 kernel: Call Trace: Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 22:10:16 fir-io1-s1 kernel: Pid: 96329, comm: ll_ost00_022 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 22:10:16 fir-io1-s1 kernel: Call Trace: Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 22:10:16 fir-io1-s1 kernel: Pid: 49822, comm: ll_ost00_072 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 22:10:16 fir-io1-s1 kernel: Call Trace: Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 22:10:16 fir-io1-s1 kernel: Pid: 94236, comm: ll_ost00_001 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 12 22:10:16 fir-io1-s1 kernel: Call Trace: Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 12 22:10:16 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 12 22:10:16 fir-io1-s1 kernel: LNet: Service thread pid 96515 was inactive for 201.63s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 12 22:10:24 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ab9b8dac-8eac-2446-5c44-81adc082eda6 (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98622f65f000, cur 1552453824 expire 1552453674 last 1552453597 Mar 12 22:10:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 22:10:24 fir-io1-s1 kernel: LNet: Service thread pid 96374 completed after 208.99s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 12 22:10:24 fir-io1-s1 kernel: LNet: Skipped 7 previous similar messages Mar 12 22:18:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 22:18:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 22:19:17 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client de58bc14-bd13-7244-e9b0-e49418653258 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575de9f800, cur 1552454357 expire 1552454207 last 1552454130 Mar 12 22:19:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 22:30:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 12 22:30:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 22:31:39 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f322dd8a-870f-988a-9a09-9eddb45f324d (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756f5e800, cur 1552455099 expire 1552454949 last 1552454872 Mar 12 22:31:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 22:41:57 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 7bf142b2-6af5-c26c-9508-0f42a126314d (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984804dbd800, cur 1552455717 expire 1552455567 last 1552455490 Mar 12 22:41:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 12 23:12:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ece3ba5f-d1f7-e120-60ce-a1205e7d3f4d (at 10.9.101.57@o2ib4) Mar 12 23:12:39 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 13 00:32:53 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552462366/real 1552462366] req@ffff9865e9a00000 x1625480890064048/t0(0) o104->fir-OST0008@10.8.2.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552462373 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 00:32:53 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 96 previous similar messages Mar 13 00:33:35 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552462408/real 1552462408] req@ffff9865e9a00000 x1625480890064048/t0(0) o104->fir-OST0008@10.8.2.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552462415 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 00:33:35 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 13 00:34:31 fir-io1-s1 kernel: LustreError: 110701:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.2.4@o2ib6) failed to reply to blocking AST (req@ffff9865e9a00000 x1625480890064048 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff983826ea0240/0x49e1862d216f8e54 lrc: 4/0,0 mode: PR/PR res: [0xc80000401:0x1bb5b2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.2.4@o2ib6 remote: 0xc4007edde3302850 expref: 349 pid: 96567 timeout: 2810315 lvb_type: 1 Mar 13 00:34:31 fir-io1-s1 kernel: LustreError: 110701:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Mar 13 00:34:31 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.2.4@o2ib6 was evicted due to a lock blocking callback time out: rc -110 Mar 13 00:34:31 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 13 00:34:31 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.2.4@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff983826ea0240/0x49e1862d216f8e54 lrc: 3/0,0 mode: PR/PR res: [0xc80000401:0x1bb5b2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.2.4@o2ib6 remote: 0xc4007edde3302850 expref: 350 pid: 96567 timeout: 0 lvb_type: 1 Mar 13 00:34:31 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages Mar 13 00:34:52 fir-io1-s1 kernel: Lustre: 96366:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552462485/real 1552462485] req@ffff984fea649500 x1625480893596880/t0(0) o104->fir-OST0006@10.8.2.4@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552462492 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 00:34:52 fir-io1-s1 kernel: Lustre: 96366:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Mar 13 00:35:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c682a673-9d42-ab04-4f07-1f00ebc11325 (at 10.8.2.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de7400, cur 1552462556 expire 1552462406 last 1552462329 Mar 13 00:35:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 00:35:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c682a673-9d42-ab04-4f07-1f00ebc11325 (at 10.8.2.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984838082000, cur 1552462557 expire 1552462407 last 1552462330 Mar 13 00:35:57 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 13 00:38:36 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a09f3227-937d-ce2a-ad7f-844813ad7354 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985e74163400, cur 1552462716 expire 1552462566 last 1552462489 Mar 13 00:38:36 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 13 00:39:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 00:39:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 00:39:52 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 548acead-cb88-47af-a034-82c7075f143a (at 10.9.101.65@o2ib4) in 163 seconds. I think it's dead, and I am evicting it. exp ffff98677e02d000, cur 1552462792 expire 1552462642 last 1552462629 Mar 13 00:39:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 00:41:08 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 602baf1a-0fbb-48de-6fe5-fde70e9e74c7 (at 10.9.101.66@o2ib4) in 166 seconds. I think it's dead, and I am evicting it. exp ffff9871058d3c00, cur 1552462868 expire 1552462718 last 1552462702 Mar 13 00:41:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 00:41:22 fir-io1-s1 kernel: Lustre: 96765:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552462875/real 1552462875] req@ffff983841c7d400 x1625480910138896/t0(0) o106->fir-OST0008@10.9.101.66@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552462882 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 00:41:22 fir-io1-s1 kernel: Lustre: 96765:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Mar 13 00:42:09 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 602baf1a-0fbb-48de-6fe5-fde70e9e74c7 (at 10.9.101.66@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dff8400, cur 1552462929 expire 1552462779 last 1552462702 Mar 13 00:42:09 fir-io1-s1 kernel: Lustre: Skipped 9 previous similar messages Mar 13 00:50:05 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 229ea3b4-75f9-b9c6-7e06-59180c69799b (at 10.8.14.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c64c400, cur 1552463405 expire 1552463255 last 1552463178 Mar 13 00:50:05 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 13 00:54:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 08bba89a-0d39-8322-0e1e-c061bb0ee70a (at 10.8.2.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d55c000, cur 1552463656 expire 1552463506 last 1552463429 Mar 13 00:54:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:05:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f3023e70-967c-cdca-d170-324afefea199 (at 10.9.106.49@o2ib4) Mar 13 01:05:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:05:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5aa81f57-e870-e6bb-de05-2c8d24b54371 (at 10.8.2.4@o2ib6) Mar 13 01:05:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:08:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 91c3019e-2880-7f92-0b14-6eb0f2cbe1dd (at 10.9.101.65@o2ib4) Mar 13 01:08:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:10:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 37aa7d65-aee3-56c5-fbe1-78e40cec7bbc (at 10.9.101.66@o2ib4) Mar 13 01:10:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:19:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 51250c5a-971c-dc1f-b9af-0d839de6bbe8 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576e917c00, cur 1552465166 expire 1552465016 last 1552464939 Mar 13 01:19:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:19:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 51250c5a-971c-dc1f-b9af-0d839de6bbe8 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d5e000, cur 1552465178 expire 1552465028 last 1552464951 Mar 13 01:19:38 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 13 01:25:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b5ad8834-3caa-3b3a-aeae-c877aabb1ef0 (at 10.8.2.6@o2ib6) Mar 13 01:25:03 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 13 01:27:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 01:27:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:27:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client efe95f45-7c8c-6e6c-b499-8b863f1919a2 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983851c28c00, cur 1552465674 expire 1552465524 last 1552465447 Mar 13 01:27:54 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 13 01:33:25 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 883db5e3-06ed-7b40-60d9-2bc762b7d342 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a0afc00, cur 1552466005 expire 1552465855 last 1552465778 Mar 13 01:33:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:34:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 01:34:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:42:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 01:42:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:42:15 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bb818948-3683-0ce9-2f51-8c221361d7a5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c9069b400, cur 1552466535 expire 1552466385 last 1552466308 Mar 13 01:42:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:51:07 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6b46c3b4-60b0-7871-5ad6-3c634761feaf (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c85c400, cur 1552467067 expire 1552466917 last 1552466840 Mar 13 01:51:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 01:58:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 01:58:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 02:32:44 fir-io1-s1 kernel: Lustre: 110661:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552469557/real 1552469557] req@ffff98380f1fc500 x1625481178871792/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552469564 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 02:32:44 fir-io1-s1 kernel: Lustre: 110034:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552469557/real 1552469557] req@ffff9838451b8600 x1625481178871776/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552469564 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 02:32:44 fir-io1-s1 kernel: Lustre: 110034:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Mar 13 02:32:51 fir-io1-s1 kernel: Lustre: 49817:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552469564/real 1552469564] req@ffff9838474c3900 x1625481178871824/t0(0) o106->fir-OST000a@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552469571 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 02:32:51 fir-io1-s1 kernel: Lustre: 110034:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552469564/real 1552469564] req@ffff9838451b8600 x1625481178871776/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552469571 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 02:32:51 fir-io1-s1 kernel: Lustre: 110034:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 13 02:32:51 fir-io1-s1 kernel: Lustre: 49817:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 13 02:33:12 fir-io1-s1 kernel: Lustre: 96779:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552469585/real 1552469585] req@ffff983a6a296000 x1625481178871808/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552469592 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 02:33:12 fir-io1-s1 kernel: Lustre: 96779:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Mar 13 02:33:50 fir-io1-s1 kernel: Lustre: 96567:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552469623/real 1552469623] req@ffff98380f1fe000 x1625481179005040/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552469630 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 02:33:50 fir-io1-s1 kernel: Lustre: 96567:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Mar 13 02:35:07 fir-io1-s1 kernel: Lustre: 96261:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552469700/real 1552469700] req@ffff984e1929bf00 x1625481179005008/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552469707 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 02:35:07 fir-io1-s1 kernel: Lustre: 96261:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 101 previous similar messages Mar 13 02:35:57 fir-io1-s1 kernel: LNet: Service thread pid 96779 was inactive for 200.06s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 13 02:35:57 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 13 02:35:57 fir-io1-s1 kernel: Pid: 96779, comm: ll_ost00_049 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 13 02:35:57 fir-io1-s1 kernel: Call Trace: Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 13 02:35:57 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 13 02:35:57 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552469757.96779 Mar 13 02:35:58 fir-io1-s1 kernel: LNet: Service thread pid 110034 was inactive for 200.81s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 13 02:35:58 fir-io1-s1 kernel: Pid: 110034, comm: ll_ost00_087 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 13 02:35:58 fir-io1-s1 kernel: Call Trace: Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 13 02:35:58 fir-io1-s1 kernel: Pid: 110661, comm: ll_ost00_108 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 13 02:35:58 fir-io1-s1 kernel: Call Trace: Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 13 02:35:58 fir-io1-s1 kernel: Pid: 49817, comm: ll_ost00_067 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 13 02:35:58 fir-io1-s1 kernel: Call Trace: Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 13 02:35:58 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 13 02:36:01 fir-io1-s1 kernel: LNet: Service thread pid 96567 was inactive for 200.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 13 02:36:01 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 13 02:36:01 fir-io1-s1 kernel: Pid: 96567, comm: ll_ost00_031 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 13 02:36:01 fir-io1-s1 kernel: Call Trace: Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 13 02:36:01 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 13 02:36:01 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552469761.96567 Mar 13 02:36:01 fir-io1-s1 kernel: LNet: Service thread pid 96769 was inactive for 200.86s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 13 02:36:01 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 13 02:36:07 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9ab206be-5713-3e50-183f-d0ddb3f486a3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b3d000, cur 1552469767 expire 1552469617 last 1552469540 Mar 13 02:36:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 02:36:18 fir-io1-s1 kernel: LNet: Service thread pid 110034 completed after 220.55s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 13 02:36:18 fir-io1-s1 kernel: LNet: Skipped 5 previous similar messages Mar 13 02:36:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 02:36:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 02:37:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ca227231-27cd-48a6-c9a9-559ba898fce6 (at 10.9.112.16@o2ib4) in 209 seconds. I think it's dead, and I am evicting it. exp ffff98699ec88400, cur 1552469843 expire 1552469693 last 1552469634 Mar 13 02:37:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 02:49:08 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3d37f31d-6d03-38bf-8e04-628bd053ff83 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984fcccfb400, cur 1552470548 expire 1552470398 last 1552470321 Mar 13 02:49:08 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 13 02:49:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 02:49:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:02:39 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f033d518-0edd-d2e8-9cc4-cc4267a07bf3 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854e2657000, cur 1552471359 expire 1552471209 last 1552471132 Mar 13 03:02:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:03:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 03:03:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:06:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f71b46a9-a3f7-5ee6-507a-4f7f60af7482 (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e17000, cur 1552471606 expire 1552471456 last 1552471379 Mar 13 03:06:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:16:33 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 767a14b7-94b4-fe7b-746a-9499b123149b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984832b4a800, cur 1552472193 expire 1552472043 last 1552471966 Mar 13 03:16:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:16:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 03:16:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:23:24 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 00e2ff2b-73fc-7514-fe6f-cae9256f79fc (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785da0800, cur 1552472604 expire 1552472454 last 1552472377 Mar 13 03:23:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:25:11 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 03:25:11 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 13 03:31:31 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b3651020-45ee-7d51-469a-1cc69eebe110 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986ee5fbb000, cur 1552473091 expire 1552472941 last 1552472864 Mar 13 03:31:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:31:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 03:31:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:36:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 03:36:46 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 13 03:37:34 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 89266edb-e95f-7a1c-9e90-34d6218b81e1 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885dc00, cur 1552473454 expire 1552473304 last 1552473227 Mar 13 03:37:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:42:15 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6dfe4444-f935-0281-1c43-e715a9b7d432 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575de9f800, cur 1552473735 expire 1552473585 last 1552473508 Mar 13 03:42:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 03:47:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 03:47:56 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 13 04:11:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b3e1f6f7-a6fd-b37f-50ea-857a7a905b89 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983851c2f800, cur 1552475482 expire 1552475332 last 1552475255 Mar 13 04:11:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:11:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 04:11:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:17:24 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client bc30242d-2c90-1517-4c4a-a4e1412e45ee (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857f2765400, cur 1552475844 expire 1552475694 last 1552475617 Mar 13 04:17:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:17:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 04:17:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:24:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 04:24:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:24:24 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a8c6f73a-36a9-9557-53c7-512eb751dcbc (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d51400, cur 1552476264 expire 1552476114 last 1552476037 Mar 13 04:24:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:32:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e0075083-9864-b743-1d39-1a996e71793c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575cd50c00, cur 1552476746 expire 1552476596 last 1552476519 Mar 13 04:32:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:34:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 04:34:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:41:41 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2256f493-4f8b-d77f-c9b3-c855dc480e81 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c673000, cur 1552477301 expire 1552477151 last 1552477074 Mar 13 04:41:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:42:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 04:42:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:47:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 04:47:59 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 13 04:48:13 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 07c611f0-71f7-ce13-3066-bb55b339718c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582bba5800, cur 1552477693 expire 1552477543 last 1552477466 Mar 13 04:48:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 04:53:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 13 04:53:03 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 13 04:53:28 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 310e503d-0973-1719-ca53-fb76a4485d78 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854e2656800, cur 1552478008 expire 1552477858 last 1552477781 Mar 13 04:53:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 06:49:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d2da3ac-e307-036e-f8a3-99f8b3ab4ed7 (at 10.8.15.5@o2ib6) Mar 13 06:49:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 06:50:06 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client bce67404-12df-7d79-8fed-10d426bd5f19 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987823704800, cur 1552485006 expire 1552484856 last 1552484779 Mar 13 06:50:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 06:53:31 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b4948f92-eca6-7eac-474b-7b73ab3033aa (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d71c00, cur 1552485211 expire 1552485061 last 1552484984 Mar 13 06:53:31 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 13 09:36:00 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ff66990f-e07b-21f2-703b-8e9acd1f5030 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785dce800, cur 1552494960 expire 1552494810 last 1552494733 Mar 13 09:36:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 09:50:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 6da36f58-0431-e291-21aa-cb3e24483fd5 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683ea43800, cur 1552495851 expire 1552495701 last 1552495624 Mar 13 09:50:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 09:51:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 6da36f58-0431-e291-21aa-cb3e24483fd5 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b85d400, cur 1552495861 expire 1552495711 last 1552495634 Mar 13 09:51:01 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 13 09:51:11 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6da36f58-0431-e291-21aa-cb3e24483fd5 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784bb9800, cur 1552495871 expire 1552495721 last 1552495644 Mar 13 09:51:11 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 13 10:09:56 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 79513297-eca2-eaee-d6e9-bc0f9baa1c1c (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678445fc00, cur 1552496996 expire 1552496846 last 1552496769 Mar 13 10:16:43 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 59616314-0a36-7bc8-0c60-5e3b82ba7d95 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ad81acc00, cur 1552497403 expire 1552497253 last 1552497176 Mar 13 10:16:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 10:16:46 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 59616314-0a36-7bc8-0c60-5e3b82ba7d95 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987787176400, cur 1552497406 expire 1552497256 last 1552497179 Mar 13 10:17:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 13 10:17:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 10:23:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 13 10:23:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 10:45:35 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552499128/real 1552499128] req@ffff98695b28a100 x1625481809603104/t0(0) o104->fir-OST0008@10.8.10.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552499135 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 10:45:35 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 271 previous similar messages Mar 13 10:45:56 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552499149/real 1552499149] req@ffff98695b28a100 x1625481809603104/t0(0) o104->fir-OST0008@10.8.10.9@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552499156 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 10:45:56 fir-io1-s1 kernel: Lustre: 96926:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Mar 13 10:46:16 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f105b6fe-6822-03a8-0409-29be12e05d98 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985445e17c00, cur 1552499176 expire 1552499026 last 1552498949 Mar 13 10:46:16 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 13 10:46:19 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f105b6fe-6822-03a8-0409-29be12e05d98 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480f61b800, cur 1552499179 expire 1552499029 last 1552498952 Mar 13 10:46:21 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client f105b6fe-6822-03a8-0409-29be12e05d98 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857f2765800, cur 1552499181 expire 1552499031 last 1552498954 Mar 13 10:48:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 13 10:48:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 10:54:30 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 33f5ecfe-14ad-df52-bced-4a0d454d3f75 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de7000, cur 1552499670 expire 1552499520 last 1552499443 Mar 13 10:54:30 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 13 10:54:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 13 10:54:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 11:03:59 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a6aad304-4130-d16e-cec6-a4b564c6338e (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985e74162400, cur 1552500239 expire 1552500089 last 1552500012 Mar 13 11:03:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 11:06:02 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 13 11:06:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 11:41:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 11:41:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 11:42:28 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e94837d6-b1ec-f909-8ba8-2fd218c7293a (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786976800, cur 1552502548 expire 1552502398 last 1552502321 Mar 13 11:42:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 11:45:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 13 11:45:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 11:46:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 13 11:46:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 12:13:32 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client cfc69763-917e-d825-40ca-deca596998aa (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984bb4fe6400, cur 1552504412 expire 1552504262 last 1552504185 Mar 13 12:13:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 12:17:01 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 9ca05c8d-6694-9127-3fba-69ffc89899b7 (at 10.8.4.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987660f96800, cur 1552504621 expire 1552504471 last 1552504394 Mar 13 12:17:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 12:18:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 13 12:18:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 12:47:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b29100f0-7f9b-f4f6-2c85-7505f2641dbf (at 10.8.6.22@o2ib6) Mar 13 12:47:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 12:48:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6d0ee7e8-906e-1c92-e48e-46358f4f2bf8 (at 10.8.4.1@o2ib6) Mar 13 12:48:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 12:48:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd16151d-af0e-00df-69f0-bc73398a9c87 (at 10.8.4.5@o2ib6) Mar 13 12:48:36 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 13 12:54:57 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client cceed633-680e-9af2-fdd6-a2d5e989bd5a (at 10.8.4.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811fd1000, cur 1552506897 expire 1552506747 last 1552506670 Mar 13 12:54:57 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 13 12:59:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 76adebb8-0f8e-2758-6912-726467c6e858 (at 10.8.26.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b615c00, cur 1552507198 expire 1552507048 last 1552506971 Mar 13 12:59:58 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 13 13:03:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5dce2671-4a1a-0111-86ae-091a2a13785c (at 10.8.26.33@o2ib6) Mar 13 13:03:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 13:25:51 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 414dcc40-1a1f-dafe-b9b7-84383e8013e5 (at 10.8.4.29@o2ib6) Mar 13 13:25:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 13:26:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5874979c-2d42-d4b8-b0f7-48cd970c494d (at 10.8.4.28@o2ib6) Mar 13 13:26:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 14:16:07 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3fd73720-b448-c5ce-6538-f225679b76ff (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a0ad800, cur 1552511767 expire 1552511617 last 1552511540 Mar 13 14:16:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 14:16:25 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3fd73720-b448-c5ce-6538-f225679b76ff (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985ef8cf9800, cur 1552511785 expire 1552511635 last 1552511558 Mar 13 14:16:25 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 13 14:22:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 13 14:22:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 14:33:59 fir-io1-s1 kernel: Lustre: 96927:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552512832/real 1552512832] req@ffff9865559cfb00 x1625481972558496/t0(0) o104->fir-OST0006@10.8.0.68@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552512839 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 14:33:59 fir-io1-s1 kernel: Lustre: 96927:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 13 14:34:05 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552512838/real 1552512838] req@ffff986b9f191500 x1625481972635552/t0(0) o104->fir-OST0008@10.8.0.68@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552512845 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 14:34:17 fir-io1-s1 kernel: Lustre: 96908:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552512850/real 1552512850] req@ffff98380541f200 x1625481972800144/t0(0) o104->fir-OST000a@10.8.0.68@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552512857 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 14:34:17 fir-io1-s1 kernel: Lustre: 96908:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 13 14:34:38 fir-io1-s1 kernel: Lustre: 96908:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552512871/real 1552512871] req@ffff98380541f200 x1625481972800144/t0(0) o104->fir-OST000a@10.8.0.68@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552512878 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 14:34:38 fir-io1-s1 kernel: Lustre: 96908:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 13 14:35:16 fir-io1-s1 kernel: Lustre: 96927:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552512909/real 1552512909] req@ffff9865559cfb00 x1625481972558496/t0(0) o104->fir-OST0006@10.8.0.68@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552512916 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 14:35:16 fir-io1-s1 kernel: Lustre: 96927:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 33 previous similar messages Mar 13 14:35:54 fir-io1-s1 kernel: LustreError: 74799:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.0.68@o2ib6) returned error from blocking AST (req@ffff9864fb82da00 x1625481973355280 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff984986800b40/0x49e1862d42826675 lrc: 4/0,0 mode: PR/PR res: [0xc80000400:0x22874b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.0.68@o2ib6 remote: 0xfe7c9402d60e3a66 expref: 1040 pid: 96897 timeout: 2860855 lvb_type: 1 Mar 13 14:35:54 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.0.68@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 13 14:35:54 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 63s: evicting client at 10.8.0.68@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff984986800b40/0x49e1862d42826675 lrc: 3/0,0 mode: PR/PR res: [0xc80000400:0x22874b:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.0.68@o2ib6 remote: 0xfe7c9402d60e3a66 expref: 1041 pid: 96897 timeout: 0 lvb_type: 1 Mar 13 14:35:54 fir-io1-s1 kernel: LustreError: 96250:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff986126e7d700 x1625481974004304/t0(0) o104->fir-OST000a@10.8.0.68@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 Mar 13 14:35:54 fir-io1-s1 kernel: LustreError: 96250:0:(client.c:1175:ptlrpc_import_delay_req()) Skipped 13 previous similar messages Mar 13 14:37:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c46b7f3d-ec55-7fd7-e207-e9c6f5525b60 (at 10.8.0.68@o2ib6) Mar 13 14:37:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 14:51:01 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c11420a4-ed64-fc04-ee55-8267450f8ddb (at 10.8.0.68@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d7a800, cur 1552513861 expire 1552513711 last 1552513634 Mar 13 14:51:01 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 13 14:51:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c46b7f3d-ec55-7fd7-e207-e9c6f5525b60 (at 10.8.0.68@o2ib6) Mar 13 14:51:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 14:55:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 055bf6e6-6e4e-ec05-1339-93ce725baab4 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678445a800, cur 1552514158 expire 1552514008 last 1552513931 Mar 13 14:55:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 14:57:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 14:57:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 15:40:40 fir-io1-s1 kernel: Lustre: 77317:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552516833/real 1552516833] req@ffff9863c1a55700 x1625482017392112/t0(0) o106->fir-OST0004@10.8.9.3@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552516840 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 15:40:40 fir-io1-s1 kernel: Lustre: 77317:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 47 previous similar messages Mar 13 15:40:54 fir-io1-s1 kernel: Lustre: 1015:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552516847/real 1552516847] req@ffff986e39417200 x1625482017392144/t0(0) o106->fir-OST0008@10.8.9.3@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552516854 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 15:40:54 fir-io1-s1 kernel: Lustre: 1015:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Mar 13 15:41:15 fir-io1-s1 kernel: Lustre: 109960:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552516868/real 1552516868] req@ffff9849a5e20f00 x1625482017392128/t0(0) o106->fir-OST0006@10.8.9.3@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552516875 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 15:41:15 fir-io1-s1 kernel: Lustre: 109960:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 13 15:41:57 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552516909/real 1552516909] req@ffff984e1f213600 x1625482017798384/t0(0) o106->fir-OST0008@10.8.9.3@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552516916 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 15:41:57 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 24 previous similar messages Mar 13 15:43:14 fir-io1-s1 kernel: Lustre: 96897:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552516986/real 1552516986] req@ffff9849e1b80f00 x1625482017798368/t0(0) o106->fir-OST0006@10.8.9.3@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552516993 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 15:43:14 fir-io1-s1 kernel: Lustre: 96897:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 81 previous similar messages Mar 13 15:43:53 fir-io1-s1 kernel: LNet: Service thread pid 109960 was inactive for 200.53s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 13 15:43:53 fir-io1-s1 kernel: Pid: 109960, comm: ll_ost03_066 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 13 15:43:53 fir-io1-s1 kernel: Call Trace: Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 13 15:43:53 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 13 15:43:53 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552517033.109960 Mar 13 15:43:54 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3120dd7d-cbc9-4142-25f0-5a27cf4b37aa (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009fc800, cur 1552517034 expire 1552516884 last 1552516807 Mar 13 15:43:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 15:43:54 fir-io1-s1 kernel: LNet: Service thread pid 109960 completed after 201.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 13 15:43:54 fir-io1-s1 kernel: LNet: Skipped 4 previous similar messages Mar 13 15:47:58 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 635508b5-34fc-6e88-49e3-e482b5de4546 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ad81a8c00, cur 1552517278 expire 1552517128 last 1552517051 Mar 13 15:47:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 15:48:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 15:48:19 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 13 16:02:35 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a9042613-824c-7ff5-4680-9479fc855731 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838a9000, cur 1552518155 expire 1552518005 last 1552517928 Mar 13 16:02:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:03:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 16:03:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:03:51 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0cd44f52-db95-791b-2f1b-5f016078ded7 (at 10.8.15.6@o2ib6) in 160 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a7cc00, cur 1552518231 expire 1552518081 last 1552518071 Mar 13 16:03:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:11:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8c7862be-eedf-a9bb-3f95-114b32e73ddc (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984fcccfa800, cur 1552518702 expire 1552518552 last 1552518475 Mar 13 16:11:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:14:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 13 16:14:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:20:52 fir-io1-s1 kernel: mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) Mar 13 16:35:25 fir-io1-s1 kernel: Lustre: 74750:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552520118/real 1552520118] req@ffff985fa0806c00 x1625482047340384/t0(0) o106->fir-OST0004@10.8.9.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552520125 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 16:35:25 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552520118/real 1552520118] req@ffff9863c1a55100 x1625482047340400/t0(0) o106->fir-OST0006@10.8.9.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552520125 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 13 16:35:25 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 64 previous similar messages Mar 13 16:35:46 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552520139/real 1552520139] req@ffff9863c1a55100 x1625482047340400/t0(0) o106->fir-OST0006@10.8.9.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552520146 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 16:35:46 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 13 16:36:28 fir-io1-s1 kernel: Lustre: 96259:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552520181/real 1552520181] req@ffff98381ef53600 x1625482047340416/t0(0) o106->fir-OST0008@10.8.9.5@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552520188 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 13 16:36:28 fir-io1-s1 kernel: Lustre: 96259:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 13 16:36:56 fir-io1-s1 kernel: LustreError: 49822:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.5@o2ib6) returned error from glimpse AST (req@ffff983810507800 x1625482047340432 status -107 rc -107), evict it ns: filter-fir-OST000a_UUID lock: ffff98383b48dc40/0x49e1862d450776f7 lrc: 4/0,0 mode: PW/PW res: [0x17ad34:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.5@o2ib6 remote: 0x121fde5f71a3508b expref: 11 pid: 96891 timeout: 0 lvb_type: 0 Mar 13 16:36:56 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.8.9.5@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 13 16:36:56 fir-io1-s1 kernel: LustreError: Skipped 6 previous similar messages Mar 13 16:36:56 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552520216s: evicting client at 10.8.9.5@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff98382b3fdc40/0x49e1862d450776db lrc: 4/0,0 mode: PW/PW res: [0x17af40:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000020000 nid: 10.8.9.5@o2ib6 remote: 0x121fde5f71a3501b expref: 11 pid: 96891 timeout: 0 lvb_type: 0 Mar 13 16:36:56 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 6 previous similar messages Mar 13 16:36:56 fir-io1-s1 kernel: LustreError: 49822:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 7 previous similar messages Mar 13 16:36:57 fir-io1-s1 kernel: LustreError: 96914:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.5@o2ib6) returned error from glimpse AST (req@ffff98381edfa100 x1625482048203456 status -107 rc -107), evict it ns: filter-fir-OST0004_UUID lock: ffff984a7e98de80/0x49e1862d450776aa lrc: 4/0,0 mode: PW/PW res: [0x17ad3f:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000020000 nid: 10.8.9.5@o2ib6 remote: 0x121fde5f71a34fe3 expref: 11 pid: 96361 timeout: 0 lvb_type: 0 Mar 13 16:36:57 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.9.5@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 13 16:36:57 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 13 16:36:57 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552520217s: evicting client at 10.8.9.5@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff984a7e98de80/0x49e1862d450776aa lrc: 4/0,0 mode: PW/PW res: [0x17ad3f:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000020000 nid: 10.8.9.5@o2ib6 remote: 0x121fde5f71a34fe3 expref: 12 pid: 96361 timeout: 0 lvb_type: 0 Mar 13 16:36:57 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Mar 13 16:37:03 fir-io1-s1 kernel: LustreError: 96915:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.5@o2ib6) returned error from glimpse AST (req@ffff984e1f211800 x1625482048203504 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff984b28950d80/0x49e1862d450776f0 lrc: 4/0,0 mode: PW/PW res: [0x17af6f:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.5@o2ib6 remote: 0x121fde5f71a35053 expref: 10 pid: 96891 timeout: 0 lvb_type: 0 Mar 13 16:37:03 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.9.5@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 13 16:37:03 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552520223s: evicting client at 10.8.9.5@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff984b28950d80/0x49e1862d450776f0 lrc: 4/0,0 mode: PW/PW res: [0x17af6f:0x0:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.5@o2ib6 remote: 0x121fde5f71a35053 expref: 11 pid: 96891 timeout: 0 lvb_type: 0 Mar 13 16:37:03 fir-io1-s1 kernel: LustreError: 96915:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 13 16:38:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 142a6e04-872a-1c7a-e7fe-9168d1c1b90d (at 10.8.9.5@o2ib6) Mar 13 16:38:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:38:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7519e33d-a7c1-9fe6-4000-28e230b9b28c (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849be958800, cur 1552520337 expire 1552520187 last 1552520110 Mar 13 16:38:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:46:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a719eae9-d67d-2753-a4c4-82063a8e9002 (at 10.8.9.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984839a66800, cur 1552520795 expire 1552520645 last 1552520568 Mar 13 16:46:35 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 13 16:50:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ceaf4d08-2b11-802e-a726-633beb62e830 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835773400, cur 1552521044 expire 1552520894 last 1552520817 Mar 13 16:50:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:50:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 16:50:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:57:17 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e49c0f4e-0249-809b-fca8-798228273bd9 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483462a400, cur 1552521437 expire 1552521287 last 1552521210 Mar 13 16:57:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 16:57:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 16:57:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 17:41:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 13 17:41:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 17:42:02 fir-io1-s1 kernel: LustreError: 74750:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from blocking AST (req@ffff9863165ec500 x1625482080643008 status -107 rc -107), evict it ns: filter-fir-OST0008_UUID lock: ffff98382c63a640/0x49e1862d458c20ea lrc: 4/0,0 mode: PR/PR res: [0x17aefd:0x0:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0xfe7f8ee7d2c5f9cd expref: 52 pid: 96499 timeout: 2872022 lvb_type: 1 Mar 13 17:42:02 fir-io1-s1 kernel: LustreError: 138-a: fir-OST000a: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 13 17:42:02 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 13 17:42:02 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST000a_UUID lock: ffff984aa4f99b00/0x49e1862d458c20e3 lrc: 3/0,0 mode: PR/PR res: [0x17acb5:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0xfe7f8ee7d2c5f9b1 expref: 47 pid: 96405 timeout: 0 lvb_type: 1 Mar 13 17:42:02 fir-io1-s1 kernel: LustreError: 74750:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 1 previous similar message Mar 13 17:42:22 fir-io1-s1 kernel: LustreError: 96328:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from blocking AST (req@ffff986126e7b900 x1625482080898240 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff98381ac03600/0x49e1862d458b0f49 lrc: 4/0,0 mode: PR/PR res: [0x17ad76:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0xfe7f8ee7d2c5d2a5 expref: 52 pid: 96499 timeout: 2872043 lvb_type: 1 Mar 13 17:42:22 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0002: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -107 Mar 13 17:42:22 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 13 17:42:22 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff98381ac03600/0x49e1862d458b0f49 lrc: 3/0,0 mode: PR/PR res: [0x17ad76:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0xfe7f8ee7d2c5d2a5 expref: 53 pid: 96499 timeout: 0 lvb_type: 1 Mar 13 17:42:22 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message Mar 13 17:42:26 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2c37257b-9a2d-892e-d329-758e95cdbc8b (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480f619800, cur 1552524146 expire 1552523996 last 1552523919 Mar 13 17:42:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 13 17:42:30 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2c37257b-9a2d-892e-d329-758e95cdbc8b (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b13400, cur 1552524150 expire 1552524000 last 1552523923 Mar 13 17:42:54 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2c37257b-9a2d-892e-d329-758e95cdbc8b (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480eeb6800, cur 1552524174 expire 1552524024 last 1552523947 Mar 13 18:02:30 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 4d3a72b8-bdf3-2b35-da0b-326f2115ca40 (at 10.8.26.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985573c62800, cur 1552525350 expire 1552525200 last 1552525123 Mar 13 18:30:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 336a74b6-8c8d-f751-b9b8-b5703f982a84 (at 10.8.26.14@o2ib6) Mar 13 18:30:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 00:31:57 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552548710/real 1552548710] req@ffff985fa0800600 x1625482364704000/t0(0) o104->fir-OST0000@10.8.0.82@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1552548717 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 14 00:31:57 fir-io1-s1 kernel: Lustre: 96568:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 24 previous similar messages Mar 14 00:32:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 401b5a6f-4f28-4de3-924c-a052b917cb46 (at 10.8.0.82@o2ib6) reconnecting Mar 14 00:32:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 45f81bad-ba79-a5b8-97f4-c718cac35552 (at 10.8.0.82@o2ib6) Mar 14 02:20:38 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2369c486-d7d3-0fee-03c0-881024607903 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c348dfc00, cur 1552555238 expire 1552555088 last 1552555011 Mar 14 02:20:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 02:21:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 14 02:21:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 03:05:10 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 123994e8-9f70-6fbd-0ebf-ebfcc77039b7 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d53800, cur 1552557910 expire 1552557760 last 1552557683 Mar 14 03:05:10 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 03:07:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 14 03:07:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 03:35:57 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a814e10e-72a9-f905-9672-6a0c6b815aa3 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c64e400, cur 1552559757 expire 1552559607 last 1552559530 Mar 14 03:35:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 03:36:33 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 14 03:36:33 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 14 03:42:53 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client fa00476e-a2c5-2d95-613d-036a32933cd2 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868043fdc00, cur 1552560173 expire 1552560023 last 1552559946 Mar 14 03:42:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 03:43:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 14 03:43:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 03:52:59 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c099485c-fe17-56e6-f12b-afd5fe495a9d (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984ad81ad400, cur 1552560779 expire 1552560629 last 1552560552 Mar 14 03:52:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 03:53:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 14 03:53:24 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 14 04:02:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ab2a7189-387d-c630-0d6d-0a22313efb0a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483ed4b800, cur 1552561333 expire 1552561183 last 1552561106 Mar 14 04:02:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 04:02:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 14 04:02:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 04:25:21 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 47a93a5d-683e-1091-fb5f-c82254dcfacd (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576885e000, cur 1552562721 expire 1552562571 last 1552562494 Mar 14 04:25:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 04:25:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 14 04:25:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 07:58:10 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 14 07:58:10 fir-io1-s1 kernel: Lustre: Skipped 12 previous similar messages Mar 14 07:58:35 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST0000: Connection to fir-MDT0001 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 14 07:58:35 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 14 07:59:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 14 07:59:06 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 14 07:59:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 07:59:31 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 14 07:59:31 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 14 07:59:50 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 14 07:59:50 fir-io1-s1 kernel: LustreError: Skipped 23 previous similar messages Mar 14 07:59:50 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 14 07:59:50 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Mar 14 08:00:09 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 14 08:00:15 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 14 08:00:15 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 14 08:00:15 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 14 08:00:15 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 14 08:00:59 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1573967 to 0x0:1574049 Mar 14 08:00:59 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1574121 to 0x0:1574177 Mar 14 08:00:59 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1574492 to 0x0:1574561 Mar 14 08:00:59 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1573970 to 0x0:1574017 Mar 14 08:00:59 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1574502 to 0x0:1574593 Mar 14 08:00:59 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1573986 to 0x0:1574017 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2413891 to 0x6c0000400:2414145 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2413703 to 0x5c0000400:2413857 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2413700 to 0x580000400:2413985 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2412669 to 0x8c0000402:2412833 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2413268 to 0xc40000402:2413473 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2413487 to 0xc80000402:2413633 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1855000 to 0x6c0000402:1855105 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1856201 to 0x5c0000402:1856289 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1854444 to 0x8c0000401:1854561 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1854667 to 0xc40000401:1854977 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1854446 to 0xc80000401:1854529 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1856242 to 0x580000402:1856321 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2573393 to 0x6c0000401:2573729 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2573514 to 0x5c0000401:2573665 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2573132 to 0x8c0000400:2573409 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2573802 to 0x580000401:2574145 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2573299 to 0xc80000400:2573633 Mar 14 08:01:00 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2573381 to 0xc40000400:2573473 Mar 14 08:36:17 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d330338f-15d0-4bfd-6ec6-2cc124128de5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800f56c00, cur 1552577777 expire 1552577627 last 1552577550 Mar 14 08:36:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 08:36:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d330338f-15d0-4bfd-6ec6-2cc124128de5 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800f50c00, cur 1552577780 expire 1552577630 last 1552577553 Mar 14 08:36:20 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 14 08:36:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 14 08:36:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 08:39:54 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552577987/real 1552577987] req@ffff98575baf8f00 x1625482532788288/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552577994 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 14 08:39:54 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Mar 14 08:40:01 fir-io1-s1 kernel: Lustre: 96890:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552577994/real 1552577994] req@ffff986b9f191b00 x1625482532788256/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578001 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 08:40:01 fir-io1-s1 kernel: Lustre: 96890:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 14 08:40:08 fir-io1-s1 kernel: Lustre: 110035:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552578001/real 1552578001] req@ffff984067fd0300 x1625482532788272/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578008 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 08:40:08 fir-io1-s1 kernel: Lustre: 110035:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 14 08:40:15 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552578008/real 1552578008] req@ffff98747bf84800 x1625482532788240/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578015 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 08:40:15 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Mar 14 08:40:29 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552578022/real 1552578022] req@ffff98747bf84800 x1625482532788240/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578029 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 08:40:29 fir-io1-s1 kernel: Lustre: 96515:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Mar 14 08:40:50 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552578043/real 1552578043] req@ffff98575baf8f00 x1625482532788288/t0(0) o106->fir-OST0008@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578050 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 08:40:50 fir-io1-s1 kernel: Lustre: 96247:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 14 08:41:31 fir-io1-s1 kernel: Lustre: 96288:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552578084/real 1552578084] req@ffff98747bf80600 x1625482534107472/t0(0) o106->fir-OST0000@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578091 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 14 08:41:31 fir-io1-s1 kernel: Lustre: 96288:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Mar 14 08:42:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c91cee48-9db1-95b3-e271-1c2be8873d3a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858aac90800, cur 1552578168 expire 1552578018 last 1552577941 Mar 14 08:42:48 fir-io1-s1 kernel: Lustre: 110615:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552578161/real 1552578161] req@ffff9840b04d0c00 x1625482534107504/t0(0) o106->fir-OST0006@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578168 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 08:42:48 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552578161/real 1552578161] req@ffff985f38d94b00 x1625482534107488/t0(0) o106->fir-OST0004@10.8.20.15@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552578168 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 08:42:48 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 201 previous similar messages Mar 14 08:42:48 fir-io1-s1 kernel: Lustre: 110615:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 14 08:42:50 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c91cee48-9db1-95b3-e271-1c2be8873d3a (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480eeb7c00, cur 1552578170 expire 1552578020 last 1552577943 Mar 14 08:42:50 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 14 08:43:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 14 08:43:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 08:49:10 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ba45a24e-e4ce-6956-e9e5-62fd59f7c8db (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864523be800, cur 1552578550 expire 1552578400 last 1552578323 Mar 14 08:49:10 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 14 08:49:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 14 08:49:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 08:49:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ba45a24e-e4ce-6956-e9e5-62fd59f7c8db (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864523bb400, cur 1552578559 expire 1552578409 last 1552578332 Mar 14 08:49:19 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 14 08:54:41 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 48fae193-b84c-3e22-86e9-7e21e52d070b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fc51c000, cur 1552578881 expire 1552578731 last 1552578654 Mar 14 09:00:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 14 09:00:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 09:04:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 14 09:04:28 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 14 09:04:35 fir-io1-s1 kernel: LustreError: 96928:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.20.15@o2ib6) returned error from glimpse AST (req@ffff985913750f00 x1625482550185328 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff986faa822400/0x49e1862d564e7f73 lrc: 3/0,0 mode: PW/PW res: [0x6c0000402:0x1a946e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000080020000 nid: 10.8.20.15@o2ib6 remote: 0x4a1c1ec6186d2d61 expref: 5 pid: 63944 timeout: 0 lvb_type: 0 Mar 14 09:04:35 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.20.15@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 14 09:04:35 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552579475s: evicting client at 10.8.20.15@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff986faa8221c0/0x49e1862d564e7f7a lrc: 3/0,0 mode: PW/PW res: [0x8c0000401:0x1a93f3:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000080020000 nid: 10.8.20.15@o2ib6 remote: 0x4a1c1ec6186d2d99 expref: 7 pid: 63944 timeout: 0 lvb_type: 0 Mar 14 09:04:35 fir-io1-s1 kernel: LustreError: 96928:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages Mar 14 09:05:49 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 48f07312-8826-775f-35d0-fc2124c4d94b (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d5c000, cur 1552579549 expire 1552579399 last 1552579322 Mar 14 09:05:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 09:10:44 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 394871c4-7231-a5d6-54b0-401cac6ae938 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d58000, cur 1552579844 expire 1552579694 last 1552579617 Mar 14 09:10:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 14 09:13:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 14 09:13:21 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 14 11:20:20 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ba74176b-5489-97ce-f535-a27ed1c25931 (at 10.0.10.3@o2ib7) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a1bcf7000, cur 1552587620 expire 1552587470 last 1552587393 Mar 14 11:20:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 11:23:22 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Mar 14 11:23:22 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (104): c: 8, oc: 0, rc: 8 Mar 14 12:27:52 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0006: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 14 12:27:52 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Mar 14 12:28:46 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 14 12:28:47 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 14 12:28:47 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 14 12:28:53 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 14 12:28:53 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (6): c: 0, oc: 0, rc: 8 Mar 14 12:29:13 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 14 12:29:13 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 14 12:29:32 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0000-lwp-OST0000: This client was evicted by fir-MDT0000; in progress operations using this service will fail. Mar 14 12:29:32 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 14 12:29:32 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST000a: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 14 12:29:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 14 12:29:38 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 14 12:29:38 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 14 12:29:49 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 14 12:30:22 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST000a: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 14 12:30:22 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 14 12:30:22 fir-io1-s1 kernel: Lustre: fir-MDT0001-lwp-OST000a: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 14 12:30:22 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1593320 to 0x0:1593473 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1593149 to 0x0:1593249 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1593168 to 0x0:1593249 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1593137 to 0x0:1593313 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1593702 to 0x0:1593889 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1593670 to 0x0:1593857 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2426256 to 0x6c0000400:2426273 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2425759 to 0xc80000402:2425825 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2425598 to 0xc40000402:2425633 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2426098 to 0x580000400:2426177 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2425968 to 0x5c0000400:2425985 Mar 14 12:30:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2424951 to 0x8c0000402:2424993 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2669127 to 0x580000401:2669377 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2668514 to 0x8c0000400:2668865 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2668810 to 0x6c0000401:2669057 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2668714 to 0x5c0000401:2669537 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2668658 to 0xc80000400:2668865 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2668552 to 0xc40000400:2668897 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:1859115 to 0x5c0000402:1859137 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:1857389 to 0x8c0000401:1857409 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:1857919 to 0x6c0000402:1857953 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:1857807 to 0xc40000401:1857825 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:1857352 to 0xc80000401:1857377 Mar 14 12:34:00 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:1859148 to 0x580000402:1859169 Mar 14 12:34:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1ff06b81-caa6-4355-af83-ae1341529a49 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c9069a800, cur 1552592044 expire 1552591894 last 1552591817 Mar 14 12:34:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 12:34:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 14 12:34:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 13:43:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to a9d1952c-c35e-2d17-0ac7-abfee221f073 (at 10.8.9.2@o2ib6) Mar 14 13:43:03 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 14 13:43:38 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client bb583c77-029e-a4a6-7f72-7f3ee0c0e319 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fbda2c00, cur 1552596218 expire 1552596068 last 1552595991 Mar 14 13:43:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 14:00:14 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c59faddd-f6ea-f344-32a7-bb7f25e4a3f1 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983812d76800, cur 1552597214 expire 1552597064 last 1552596987 Mar 14 14:00:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 14:00:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 14 14:00:39 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 14 14:32:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0ccdc4e2-9749-c9a5-afb4-85874ce74d6c (at 10.0.10.3@o2ib7) Mar 14 14:32:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 15:10:55 fir-io1-s1 kernel: mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) Mar 14 15:11:52 fir-io1-s1 kernel: mpt3sas_cm0: log_info(0x31200205): originator(PL), code(0x20), sub_code(0x0205) Mar 14 18:43:20 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 0515aad5-e596-a7ef-fd9c-556b4ec3aa38 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b616000, cur 1552614200 expire 1552614050 last 1552613973 Mar 14 18:43:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 18:43:25 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0515aad5-e596-a7ef-fd9c-556b4ec3aa38 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a78000, cur 1552614205 expire 1552614055 last 1552613978 Mar 14 18:43:27 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0515aad5-e596-a7ef-fd9c-556b4ec3aa38 (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c85a000, cur 1552614207 expire 1552614057 last 1552613980 Mar 14 18:43:27 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 14 18:59:25 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8fb32098-b090-7502-ca18-8fe8b2db5bc8 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847fd264400, cur 1552615165 expire 1552615015 last 1552614938 Mar 14 19:28:29 fir-io1-s1 kernel: md: md2: data-check done. Mar 14 19:35:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9cc869db-86b5-83f4-3d1d-91c30da3920c (at 10.8.14.1@o2ib6) Mar 14 19:35:46 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 14 21:42:24 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 063a8ac1-845f-60fa-9e6a-5de7ca7baa70 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9849c8262000, cur 1552624944 expire 1552624794 last 1552624717 Mar 14 21:42:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 21:43:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 14 21:43:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 21:48:52 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 10d0f34c-e9fb-f48c-2275-5605494f6413 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d28c00, cur 1552625332 expire 1552625182 last 1552625105 Mar 14 21:48:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 21:55:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 14 21:55:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 22:04:35 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 626cade1-d0d1-f574-3ff3-a6fc597070a5 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b616800, cur 1552626275 expire 1552626125 last 1552626048 Mar 14 22:04:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 22:19:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 14 22:19:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 22:34:45 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552628078/real 1552628078] req@ffff9859ba66d400 x1625483292512960/t0(0) o106->fir-OST0008@10.9.113.7@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552628085 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 14 22:34:45 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 14 22:35:27 fir-io1-s1 kernel: Lustre: 109998:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552628120/real 1552628120] req@ffff986e59006c00 x1625483292512976/t0(0) o106->fir-OST000a@10.9.113.7@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552628127 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 22:35:27 fir-io1-s1 kernel: Lustre: 109998:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 14 22:36:44 fir-io1-s1 kernel: Lustre: 96514:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552628197/real 1552628197] req@ffff985fe3c39200 x1625483292512992/t0(0) o106->fir-OST0002@10.9.113.7@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552628204 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 22:36:44 fir-io1-s1 kernel: Lustre: 74743:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552628197/real 1552628197] req@ffff9876a78b9b00 x1625483292513008/t0(0) o106->fir-OST0000@10.9.113.7@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552628204 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 14 22:36:44 fir-io1-s1 kernel: Lustre: 74743:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 36 previous similar messages Mar 14 22:36:44 fir-io1-s1 kernel: Lustre: 96514:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 14 22:37:58 fir-io1-s1 kernel: LNet: Service thread pid 110701 was inactive for 200.35s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 14 22:37:58 fir-io1-s1 kernel: Pid: 110701, comm: ll_ost02_110 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 14 22:37:58 fir-io1-s1 kernel: Call Trace: Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 14 22:37:58 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 14 22:37:58 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552628278.110701 Mar 14 22:38:00 fir-io1-s1 kernel: LNet: Service thread pid 96514 was inactive for 201.69s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 14 22:38:00 fir-io1-s1 kernel: Pid: 96514, comm: ll_ost02_032 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 14 22:38:00 fir-io1-s1 kernel: Call Trace: Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 14 22:38:00 fir-io1-s1 kernel: Pid: 74743, comm: ll_ost02_078 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 14 22:38:00 fir-io1-s1 kernel: Call Trace: Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 14 22:38:00 fir-io1-s1 kernel: Pid: 109998, comm: ll_ost03_078 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 14 22:38:00 fir-io1-s1 kernel: Call Trace: Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 14 22:38:00 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 14 22:38:03 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a076b4ca-8622-3fae-15b0-24498eea221e (at 10.9.113.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984fcccfc800, cur 1552628283 expire 1552628133 last 1552628056 Mar 14 22:38:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 22:38:03 fir-io1-s1 kernel: LNet: Service thread pid 109998 completed after 204.53s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 14 22:38:04 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 65c113f8-0dec-cad2-88c8-0f33d3b2f0bf (at 10.9.108.41@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a896e4800, cur 1552628284 expire 1552628134 last 1552628057 Mar 14 22:38:04 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 14 22:38:04 fir-io1-s1 kernel: LNet: Service thread pid 96514 completed after 205.53s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 14 22:38:04 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 14 22:51:04 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 04beb61e-c757-d4c1-db6e-79ef6a5f0316 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483ed4d800, cur 1552629064 expire 1552628914 last 1552628837 Mar 14 22:51:04 fir-io1-s1 kernel: Lustre: Skipped 14 previous similar messages Mar 14 22:59:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d1755442-892e-a805-5fa2-c61746c310b0 (at 10.9.113.7@o2ib4) Mar 14 22:59:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 22:59:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a4e427fa-d968-911a-150f-37f69bc4903c (at 10.9.106.3@o2ib4) Mar 14 22:59:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 23:02:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 65c113f8-0dec-cad2-88c8-0f33d3b2f0bf (at 10.9.108.41@o2ib4) Mar 14 23:02:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 23:05:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 14 23:05:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 23:06:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 616573c5-7a05-f902-eaf8-6320cb0c16b6 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986781f72400, cur 1552630013 expire 1552629863 last 1552629786 Mar 14 23:06:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 14 23:16:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 14 23:16:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 03:49:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8ba1b18f-72db-d059-bab7-d3e649c57271 (at 10.9.108.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857590d4800, cur 1552646947 expire 1552646797 last 1552646720 Mar 15 03:49:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 04:11:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8ba1b18f-72db-d059-bab7-d3e649c57271 (at 10.9.108.55@o2ib4) Mar 15 04:11:22 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 15 06:44:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9841fa39-d025-10ee-2419-bf9ab776057c (at 10.8.15.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984998f85800, cur 1552657475 expire 1552657325 last 1552657248 Mar 15 06:44:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 08:10:51 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 54cab5eb-169f-56b2-e738-967aa49b2920 (at 10.8.7.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575871d000, cur 1552662651 expire 1552662501 last 1552662424 Mar 15 08:10:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 08:12:07 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c1106011-f63d-329c-7050-c55778229aed (at 10.8.3.2@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff98499a65b000, cur 1552662727 expire 1552662577 last 1552662501 Mar 15 08:12:07 fir-io1-s1 kernel: Lustre: Skipped 106 previous similar messages Mar 15 08:25:35 fir-io1-s1 kernel: md: md10: data-check done. Mar 15 08:32:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0020345d-5ca8-34a6-0ce1-c2c7b984d732 (at 10.9.115.11@o2ib4) Mar 15 08:32:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 08:32:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a4f2c4c1-03c5-819a-6852-1875c7d76a33 (at 10.8.19.7@o2ib6) Mar 15 08:32:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 08:33:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 725d787f-ad1b-18f8-1091-f9fac1b511d9 (at 10.9.106.62@o2ib4) Mar 15 08:33:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 08:34:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 50f28a0e-eb03-3ed4-df6b-96db06d3f42b (at 10.9.107.34@o2ib4) Mar 15 08:34:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 08:37:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 37cc079a-c37c-cf86-ab23-66b4c360eeae (at 10.8.22.4@o2ib6) Mar 15 08:37:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 08:38:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 85cfcf77-29ac-d755-b385-af543ebdafc6 (at 10.9.101.31@o2ib4) Mar 15 08:38:36 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 15 08:39:06 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to de25656c-fa14-1cbc-c67d-a4c1e6d26c7a (at 10.8.7.34@o2ib6) Mar 15 08:39:06 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 15 08:39:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 22505c78-b9c2-e28a-88c4-7dadc4be41e9 (at 10.9.101.28@o2ib4) Mar 15 08:39:57 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 15 08:41:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 493ebf88-512d-5d74-9d04-1d0b668cf490 (at 10.8.11.26@o2ib6) Mar 15 08:41:28 fir-io1-s1 kernel: Lustre: Skipped 65 previous similar messages Mar 15 09:24:43 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bd4c9cbc-2845-7b73-e6bd-dce18f860234 (at 10.8.25.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a83a66800, cur 1552667083 expire 1552666933 last 1552666856 Mar 15 09:24:43 fir-io1-s1 kernel: Lustre: Skipped 36 previous similar messages Mar 15 09:25:59 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 67d9b08d-f7d8-4fab-de12-c69c561046c1 (at 10.8.25.27@o2ib6) in 212 seconds. I think it's dead, and I am evicting it. exp ffff984804f54c00, cur 1552667159 expire 1552667009 last 1552666947 Mar 15 09:25:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 09:38:38 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8a95c791-5f46-532f-0660-5d36a65ca51d (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f5800, cur 1552667918 expire 1552667768 last 1552667691 Mar 15 09:38:38 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 15 09:38:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) Mar 15 09:38:58 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 15 09:52:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a10633db-3389-9c5a-c26d-af02296e5868 (at 10.8.25.16@o2ib6) Mar 15 09:52:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 09:54:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 79496ef6-ef75-cc5f-9bd3-cf20df3c8e1d (at 10.8.25.27@o2ib6) Mar 15 09:54:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 09:55:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 71464f83-f435-3a33-e9d6-ef54166e95b7 (at 10.8.30.36@o2ib6) Mar 15 09:55:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 09:57:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 696ec4e4-4f79-f247-b0f6-742daa25e948 (at 10.8.11.8@o2ib6) Mar 15 09:57:41 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 15 11:46:02 fir-io1-s1 kernel: Lustre: 74776:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552675555/real 1552675555] req@ffff985d739c4200 x1625483952981392/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552675562 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 15 11:46:02 fir-io1-s1 kernel: Lustre: 74776:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 70 previous similar messages Mar 15 11:46:44 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552675597/real 1552675597] req@ffff984bbdf55100 x1625483952981424/t0(0) o106->fir-OST0008@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552675604 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 15 11:46:44 fir-io1-s1 kernel: Lustre: 96352:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 15 11:48:01 fir-io1-s1 kernel: Lustre: 110574:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552675674/real 1552675674] req@ffff98382c51ad00 x1625483952981376/t0(0) o106->fir-OST0000@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552675681 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 15 11:48:01 fir-io1-s1 kernel: Lustre: 110574:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Mar 15 11:48:08 fir-io1-s1 kernel: LustreError: 74763:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.9.8@o2ib6) returned error from glimpse AST (req@ffff985dd2607800 x1625483952981408 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff985a41988900/0x49e1862d85a789b3 lrc: 3/0,0 mode: PW/PW res: [0x146473:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0xe4d75032635dd2f4 expref: 5 pid: 96895 timeout: 0 lvb_type: 0 Mar 15 11:48:08 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.8.9.8@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 15 11:48:08 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Mar 15 11:48:08 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552675688s: evicting client at 10.8.9.8@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff9852514d8d80/0x49e1862d85a789ba lrc: 3/0,0 mode: PW/PW res: [0x146515:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.9.8@o2ib6 remote: 0xe4d75032635dd32c expref: 7 pid: 96895 timeout: 0 lvb_type: 0 Mar 15 11:48:08 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Mar 15 11:48:08 fir-io1-s1 kernel: LustreError: 74763:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Mar 15 11:48:57 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e8896e03-9fb5-6968-3f3d-9beb40e4091c (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801545000, cur 1552675737 expire 1552675587 last 1552675510 Mar 15 11:48:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 11:49:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8521bd31-1a2d-c432-8053-5d8be1fc3492 (at 10.8.9.8@o2ib6) Mar 15 11:49:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 15 12:05:23 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4c047bfb-3018-be08-f511-bc023c7cf53d (at 10.8.3.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575d204c00, cur 1552676723 expire 1552676573 last 1552676496 Mar 15 12:05:23 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 15 12:37:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bf62c7df-fe55-c26d-63f8-adbf89ed0ecb (at 10.8.3.34@o2ib6) Mar 15 12:37:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 13:52:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 3b3604ef-8f30-623b-fd9d-9575d5cb005b (at 10.9.115.9@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ee000, cur 1552683162 expire 1552683012 last 1552682935 Mar 15 13:52:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 14:17:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 405a922c-acd7-226c-f53f-840190be85ca (at 10.9.115.9@o2ib4) Mar 15 14:17:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 15:47:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client bf33b1ba-c0e0-a6ea-46bb-c4f6973ac82b (at 10.8.7.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762492c00, cur 1552690022 expire 1552689872 last 1552689795 Mar 15 15:47:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 16:18:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cec68a9e-fc42-8b83-b21d-285fcd29817f (at 10.8.7.16@o2ib6) Mar 15 16:18:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:22:46 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ed1c4187-1ac6-70a0-6573-26fae52b1735 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582aa2c000, cur 1552695766 expire 1552695616 last 1552695539 Mar 15 17:22:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:22:46 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ed1c4187-1ac6-70a0-6573-26fae52b1735 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98582aa29800, cur 1552695766 expire 1552695616 last 1552695539 Mar 15 17:22:46 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 15 17:22:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 15 17:22:59 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:31:23 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1c455fa3-8795-a399-fc77-f8b740fccf28 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8ac00, cur 1552696283 expire 1552696133 last 1552696056 Mar 15 17:31:35 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 1c455fa3-8795-a399-fc77-f8b740fccf28 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986783c8b000, cur 1552696295 expire 1552696145 last 1552696068 Mar 15 17:31:44 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 1c455fa3-8795-a399-fc77-f8b740fccf28 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c9069c400, cur 1552696304 expire 1552696154 last 1552696077 Mar 15 17:31:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 15 17:31:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 15 17:31:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:33:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d4e21693-5b53-3ca8-0a99-9e1c8548548b (at 10.8.27.7@o2ib6) in 159 seconds. I think it's dead, and I am evicting it. exp ffff98575c531c00, cur 1552696380 expire 1552696230 last 1552696221 Mar 15 17:33:00 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 15 17:33:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1d2da3ac-e307-036e-f8a3-99f8b3ab4ed7 (at 10.8.15.5@o2ib6) Mar 15 17:33:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:34:08 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d4e21693-5b53-3ca8-0a99-9e1c8548548b (at 10.8.27.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c532400, cur 1552696448 expire 1552696298 last 1552696221 Mar 15 17:34:08 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Mar 15 17:34:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) Mar 15 17:34:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:34:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7cee7bf7-9aa1-cc50-5aed-b23b669bf632 (at 10.8.15.2@o2ib6) Mar 15 17:34:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:36:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3e279d24-e2fb-196e-d7ac-e1a73db143bd (at 10.9.112.16@o2ib4) Mar 15 17:36:28 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 15 17:51:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7cee7bf7-9aa1-cc50-5aed-b23b669bf632 (at 10.8.15.2@o2ib6) Mar 15 17:51:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 17:51:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 1c6268a8-ed2d-de9d-a4f1-6f168218e3b8 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984a83a62000, cur 1552697511 expire 1552697361 last 1552697284 Mar 15 17:51:51 fir-io1-s1 kernel: Lustre: Skipped 128 previous similar messages Mar 15 17:56:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3e2bfa45-013a-e48d-7bcc-c486bbeaa49b (at 10.9.108.9@o2ib4) Mar 15 17:56:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 18:02:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 43958e5c-bc4e-55cf-7e1b-8957d5f5266d (at 10.8.18.12@o2ib6) Mar 15 18:02:07 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 15 18:02:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 284b8394-44b9-ab09-772e-2bb7e58f7201 (at 10.8.18.16@o2ib6) Mar 15 18:02:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 18:03:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5ba2142c-879f-ac28-9d4a-a3788afebea0 (at 10.8.7.35@o2ib6) Mar 15 18:03:00 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 15 18:03:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 37327d93-dd03-80bf-ad2c-df17a42702a9 (at 10.8.18.26@o2ib6) Mar 15 18:03:34 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 15 18:04:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0d14142b-4545-89c2-ae1d-9a606fc5f0ac (at 10.8.4.36@o2ib6) Mar 15 18:04:45 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 15 19:04:18 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection to fir-MDT0000 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 15 19:04:18 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 15 19:04:18 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 15 19:04:18 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Mar 15 19:05:34 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 11 seconds Mar 15 19:05:34 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 8 previous similar messages Mar 15 19:05:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 15 19:05:36 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 7 previous similar messages Mar 15 19:05:39 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 15 19:05:59 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 3 seconds Mar 15 19:05:59 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages Mar 15 19:06:24 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 14 seconds Mar 15 19:06:24 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 11 previous similar messages Mar 15 19:06:50 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 1 seconds Mar 15 19:06:50 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 11 previous similar messages Mar 15 19:06:54 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 15 19:06:54 fir-io1-s1 kernel: Lustre: Skipped 24 previous similar messages Mar 15 19:07:20 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 15 19:07:20 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (6): c: 0, oc: 0, rc: 8 Mar 15 19:07:39 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST000a: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 15 19:07:39 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 15 19:07:39 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST000a: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 15 19:07:39 fir-io1-s1 kernel: Lustre: Skipped 21 previous similar messages Mar 15 19:07:45 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds Mar 15 19:07:45 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.52@o2ib7 (5): c: 0, oc: 0, rc: 8 Mar 15 19:08:29 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST0008: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 15 19:08:29 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 15 19:08:29 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0008: Connection restored to 10.0.10.52@o2ib7 (at 10.0.10.52@o2ib7) Mar 15 19:08:29 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2478595 to 0x8c0000402:2478753 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2479835 to 0x6c0000400:2479969 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2479080 to 0xc40000402:2479265 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2479781 to 0x580000400:2480161 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2479583 to 0x5c0000400:2479713 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2479382 to 0xc80000402:2479425 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1721527 to 0x0:1721665 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1722166 to 0x0:1722497 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1721485 to 0x0:1721793 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1722115 to 0x0:1722369 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1721683 to 0x0:1721985 Mar 15 19:08:48 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1721488 to 0x0:1721633 Mar 15 19:08:49 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2695854 to 0x6c0000401:2695873 Mar 15 19:08:49 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2695694 to 0xc40000400:2695713 Mar 15 19:08:49 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2695684 to 0xc80000400:2695713 Mar 15 19:08:49 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2695673 to 0x8c0000400:2695745 Mar 15 19:08:49 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2696329 to 0x5c0000401:2696417 Mar 15 19:08:49 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2696221 to 0x580000401:2696257 Mar 15 19:08:51 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:3364444 to 0x6c0000402:3364865 Mar 15 19:08:51 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:3365546 to 0x5c0000402:3366753 Mar 15 19:08:51 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:3364183 to 0xc40000401:3368161 Mar 15 19:08:51 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:3365543 to 0x580000402:3369153 Mar 15 19:08:51 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:3363912 to 0xc80000401:3364449 Mar 15 19:08:51 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:3363736 to 0x8c0000401:3364097 Mar 15 19:08:54 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0001-lwp-OST0000: This client was evicted by fir-MDT0001; in progress operations using this service will fail. Mar 15 19:08:54 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 15 19:19:36 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client fef6ee51-6f62-1747-47f6-9e7fe72cc4fb (at 10.8.12.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b65c800, cur 1552702776 expire 1552702626 last 1552702549 Mar 15 19:19:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 19:31:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ad6a13d8-84a3-9bfa-30eb-1bebcbd74501 (at 10.8.4.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678677d000, cur 1552703502 expire 1552703352 last 1552703275 Mar 15 19:31:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 15 20:04:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.4.20@o2ib6) Mar 15 20:04:51 fir-io1-s1 kernel: Lustre: Skipped 13 previous similar messages Mar 15 20:11:37 fir-io1-s1 kernel: md: md0: data-check done. Mar 15 21:04:17 fir-io1-s1 kernel: Lustre: 96906:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552709050/real 1552709050] req@ffff984bbdf57200 x1625484389731792/t0(0) o106->fir-OST0004@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552709057 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 15 21:04:17 fir-io1-s1 kernel: Lustre: 96891:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552709050/real 1552709050] req@ffff984dca766c00 x1625484389731840/t0(0) o106->fir-OST000a@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552709057 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 15 21:04:17 fir-io1-s1 kernel: Lustre: 96891:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 15 21:04:24 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552709057/real 1552709057] req@ffff98688e595100 x1625484389731808/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552709064 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 15 21:04:24 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 15 21:04:31 fir-io1-s1 kernel: Lustre: 96891:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552709064/real 1552709064] req@ffff984dca766c00 x1625484389731840/t0(0) o106->fir-OST000a@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552709071 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 15 21:04:31 fir-io1-s1 kernel: Lustre: 96891:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 15 21:04:45 fir-io1-s1 kernel: Lustre: 96915:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552709078/real 1552709078] req@ffff987497166300 x1625484389731824/t0(0) o106->fir-OST0008@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552709085 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 15 21:04:45 fir-io1-s1 kernel: Lustre: 96915:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 15 21:05:06 fir-io1-s1 kernel: Lustre: 96891:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552709099/real 1552709099] req@ffff984dca766c00 x1625484389731840/t0(0) o106->fir-OST000a@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552709106 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 15 21:05:06 fir-io1-s1 kernel: Lustre: 96891:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 12 previous similar messages Mar 15 21:05:47 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 40e1e4d3-8bd4-8965-d2c1-6077287113c0 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480332d000, cur 1552709147 expire 1552708997 last 1552708920 Mar 15 21:05:47 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 15 21:05:48 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552709141/real 1552709141] req@ffff98688e595100 x1625484389731808/t0(0) o106->fir-OST0006@10.8.9.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552709148 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 15 21:05:48 fir-io1-s1 kernel: Lustre: 96491:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 15 21:07:03 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 5ff90012-14af-982f-ec90-7b770b5f5eed (at 10.8.9.8@o2ib6) in 218 seconds. I think it's dead, and I am evicting it. exp ffff98677d41cc00, cur 1552709223 expire 1552709073 last 1552709005 Mar 15 21:07:03 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 15 21:34:54 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6e6960c6-10df-82bd-f4dc-e07be17ca1cd (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f357800, cur 1552710894 expire 1552710744 last 1552710667 Mar 15 21:34:54 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 15 21:35:00 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 6e6960c6-10df-82bd-f4dc-e07be17ca1cd (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983829017400, cur 1552710900 expire 1552710750 last 1552710673 Mar 15 21:39:05 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f67264fb-5450-499c-940a-29f1f1085f28 (at 10.8.3.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678677ac00, cur 1552711145 expire 1552710995 last 1552710918 Mar 15 21:39:05 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 15 22:09:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ced38ea1-bc1c-667b-da28-176debc6acc3 (at 10.8.3.36@o2ib6) Mar 15 22:09:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 02:30:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 70e6ef5d-3469-d21c-3a60-75b9400d5fbf (at 10.8.3.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868a8400, cur 1552728648 expire 1552728498 last 1552728421 Mar 16 02:30:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:01:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c694e053-04d0-ee79-c9a4-0ace9e2f2c9a (at 10.8.3.29@o2ib6) Mar 16 03:01:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:12:34 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 41b6a5b3-4a18-d2d8-bd49-5a77a5b63fbd (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e50400, cur 1552731154 expire 1552731004 last 1552730927 Mar 16 03:12:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:12:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 41b6a5b3-4a18-d2d8-bd49-5a77a5b63fbd (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865f0ece800, cur 1552731158 expire 1552731008 last 1552730931 Mar 16 03:12:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 41b6a5b3-4a18-d2d8-bd49-5a77a5b63fbd (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e54000, cur 1552731160 expire 1552731010 last 1552730933 Mar 16 03:12:40 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 16 03:12:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 16 03:12:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 03:38:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 71bb8fd2-33ab-b36a-ebcb-8446b062dc3f (at 10.8.3.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d648800, cur 1552732735 expire 1552732585 last 1552732508 Mar 16 03:38:55 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 16 03:59:39 fir-io1-s1 kernel: Lustre: 94237:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552733972/real 1552733972] req@ffff9838474c6c00 x1625484720518144/t0(0) o106->fir-OST0002@10.8.15.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552733979 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 03:59:39 fir-io1-s1 kernel: Lustre: 94237:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 23 previous similar messages Mar 16 03:59:53 fir-io1-s1 kernel: Lustre: 96917:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552733986/real 1552733986] req@ffff983813d22a00 x1625484720518160/t0(0) o106->fir-OST0000@10.8.15.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552733993 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 03:59:53 fir-io1-s1 kernel: Lustre: 96917:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 16 04:00:14 fir-io1-s1 kernel: Lustre: 49833:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552734007/real 1552734007] req@ffff984067fd1800 x1625484720518192/t0(0) o106->fir-OST0006@10.8.15.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552734014 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 04:00:14 fir-io1-s1 kernel: Lustre: 49833:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Mar 16 04:00:21 fir-io1-s1 kernel: LustreError: 96889:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.15.7@o2ib6) returned error from glimpse AST (req@ffff98381887cb00 x1625484720518176 status -107 rc -107), evict it ns: filter-fir-OST0004_UUID lock: ffff9863c5534c80/0x49e1862da1e0d49d lrc: 3/0,0 mode: PW/PW res: [0x18b373:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000020000 nid: 10.8.15.7@o2ib6 remote: 0xa8173d6dc3c9bade expref: 6 pid: 96929 timeout: 0 lvb_type: 0 Mar 16 04:00:21 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.8.15.7@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 16 04:00:21 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 16 04:00:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552734021s: evicting client at 10.8.15.7@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff985b8c8d7740/0x49e1862da1e0d40a lrc: 3/0,0 mode: PW/PW res: [0x18b3b7:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000020000 nid: 10.8.15.7@o2ib6 remote: 0xa8173d6dc3c9baa6 expref: 7 pid: 96406 timeout: 0 lvb_type: 0 Mar 16 04:00:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Mar 16 04:00:21 fir-io1-s1 kernel: LustreError: 96889:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 3 previous similar messages Mar 16 04:01:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dcd406ad-ffdd-a7c9-489f-309957a1236e (at 10.8.15.7@o2ib6) Mar 16 04:01:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 04:02:04 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c5297149-76a5-f9a1-90af-2ca4df82605c (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985c3d661400, cur 1552734124 expire 1552733974 last 1552733897 Mar 16 04:02:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 04:11:58 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 60c991c9-bbb6-b45f-4869-dff10ed664d7 (at 10.8.3.25@o2ib6) Mar 16 04:11:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 13:40:43 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client facdaf2a-2506-3ef9-621e-e25dfed11b5f (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800f57000, cur 1552768843 expire 1552768693 last 1552768616 Mar 16 13:40:43 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 16 13:45:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 16 13:45:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:20:05 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 8da97ea5-266b-5127-7d49-8a7312a1f2c6 (at 10.8.4.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984810a2ec00, cur 1552771205 expire 1552771055 last 1552770978 Mar 16 14:20:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:21:35 fir-io1-s1 kernel: Lustre: 110616:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552771288/real 1552771288] req@ffff98381887e600 x1625485266163856/t0(0) o106->fir-OST0002@10.8.7.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552771295 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 14:21:35 fir-io1-s1 kernel: Lustre: 94362:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552771288/real 1552771288] req@ffff9838474c3c00 x1625485266163824/t0(0) o106->fir-OST000a@10.8.7.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552771295 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 16 14:21:35 fir-io1-s1 kernel: Lustre: 94362:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 16 14:21:42 fir-io1-s1 kernel: Lustre: 96240:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552771295/real 1552771295] req@ffff984067fd0f00 x1625485266163904/t0(0) o106->fir-OST0000@10.8.7.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552771302 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 14:21:42 fir-io1-s1 kernel: Lustre: 94362:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552771295/real 1552771295] req@ffff9838474c3c00 x1625485266163824/t0(0) o106->fir-OST000a@10.8.7.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552771302 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 14:21:42 fir-io1-s1 kernel: Lustre: 94362:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 14:21:42 fir-io1-s1 kernel: Lustre: 96240:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 16 14:22:03 fir-io1-s1 kernel: Lustre: 94314:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552771316/real 1552771316] req@ffff983813d22100 x1625485266163936/t0(0) o106->fir-OST0004@10.8.7.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552771323 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 14:22:03 fir-io1-s1 kernel: Lustre: 94314:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 16 14:22:45 fir-io1-s1 kernel: Lustre: 94362:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552771358/real 1552771358] req@ffff9838474c3c00 x1625485266163824/t0(0) o106->fir-OST000a@10.8.7.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552771365 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 14:22:45 fir-io1-s1 kernel: Lustre: 94362:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Mar 16 14:24:02 fir-io1-s1 kernel: Lustre: 94314:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552771435/real 1552771435] req@ffff983813d22100 x1625485266163936/t0(0) o106->fir-OST0004@10.8.7.6@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552771442 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 16 14:24:02 fir-io1-s1 kernel: Lustre: 94314:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 43 previous similar messages Mar 16 14:24:42 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e683a177-915d-b284-42f8-05616b32580f (at 10.8.29.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480315d800, cur 1552771482 expire 1552771332 last 1552771255 Mar 16 14:24:42 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 16 14:46:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b737ae64-7c9e-050b-3dcf-22ab292fbc3e (at 10.9.106.4@o2ib4) Mar 16 14:46:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:47:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) Mar 16 14:47:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:47:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dc527ec4-db62-14c0-459e-fcb26c78037c (at 10.9.107.8@o2ib4) Mar 16 14:47:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:47:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 54df73a3-4915-b589-f8b2-dd262402c8c5 (at 10.9.107.65@o2ib4) Mar 16 14:47:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:47:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 78f38211-beb8-aca3-b985-9281f7d5f62c (at 10.8.29.8@o2ib6) Mar 16 14:47:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:48:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8c99250d-76e9-284a-5406-5556d3865e14 (at 10.8.29.5@o2ib6) Mar 16 14:48:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:48:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e265f84a-d19d-6fce-343c-d86c6eba2d5b (at 10.8.29.3@o2ib6) Mar 16 14:48:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:49:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 400e2c70-3670-eb05-66c0-e754ea5cd280 (at 10.8.29.7@o2ib6) Mar 16 14:49:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 16 14:53:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6d0afe5c-6178-6107-0dc9-7aeabb7be071 (at 10.8.4.9@o2ib6) Mar 16 14:53:31 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 16 14:55:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a9d94d98-e333-fd13-62ca-f5aa87682ca7 (at 10.8.2.17@o2ib6) Mar 16 14:55:50 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 16 15:01:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5c820789-8e24-ad89-a0df-b1759dd671b0 (at 10.9.101.56@o2ib4) Mar 16 15:01:20 fir-io1-s1 kernel: Lustre: Skipped 125 previous similar messages Mar 16 18:12:47 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 45f16ab7-b5e2-a9af-1f35-2918dfa46803 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b3a400, cur 1552785167 expire 1552785017 last 1552784940 Mar 16 18:12:47 fir-io1-s1 kernel: Lustre: Skipped 209 previous similar messages Mar 16 18:46:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7751f34c-206a-edf3-4c97-12b3ac7dcb0e (at 10.8.23.14@o2ib6) Mar 16 18:46:30 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 17 01:00:01 fir-io1-s1 kernel: md: data-check of RAID array md8 Mar 17 01:00:07 fir-io1-s1 kernel: md: data-check of RAID array md2 Mar 17 01:00:13 fir-io1-s1 kernel: md: data-check of RAID array md10 Mar 17 01:00:19 fir-io1-s1 kernel: md: data-check of RAID array md0 Mar 17 03:07:18 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client cf3c5667-8587-41c3-c24b-3c442118af2c (at 10.8.18.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848372d7c00, cur 1552817238 expire 1552817088 last 1552817011 Mar 17 03:07:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 03:07:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6b3fefc8-73fb-ce51-5a1b-5e43dc04a3e8 (at 10.8.18.11@o2ib6) Mar 17 03:07:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 07:08:43 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5a991cae-9882-4a45-5592-20b2d30543de (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98380be3c400, cur 1552831723 expire 1552831573 last 1552831496 Mar 17 07:08:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 08:00:37 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 065586d9-b32b-050a-b982-669e19f627f2 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984804f51c00, cur 1552834837 expire 1552834687 last 1552834610 Mar 17 08:00:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 08:02:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 17 08:02:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 08:09:32 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f2b37e2e-8f06-6693-09ba-1ee534da848f (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865b5462800, cur 1552835372 expire 1552835222 last 1552835145 Mar 17 08:09:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 08:09:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 17 08:09:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 08:50:05 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 50d1fd4a-92aa-00f9-b318-f11b53a61722 (at 10.8.4.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2c800, cur 1552837805 expire 1552837655 last 1552837578 Mar 17 08:50:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 09:01:02 fir-io1-s1 kernel: Lustre: fir-MDT0002-lwp-OST0000: Connection to fir-MDT0002 (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 17 09:01:02 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: 91456:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552838462/real 1552838462] req@ffff985fe3c3f800 x1625486304375088/t0(0) o400->fir-MDT0003-lwp-OST0000@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1552838469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: 91459:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552838462/real 1552838462] req@ffff985fe3c3dd00 x1625486304375328/t0(0) o400->fir-MDT0001-lwp-OST0008@10.0.10.52@o2ib7:12/10 lens 224/224 e 0 to 1 dl 1552838469 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: 91459:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0008: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: fir-MDT0003-lwp-OST0002: Connection to fir-MDT0003 (at 10.0.10.52@o2ib7) was lost; in progress operations using this service will wait for recovery to complete Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 17 09:01:09 fir-io1-s1 kernel: Lustre: 91456:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Mar 17 09:01:35 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 27 seconds Mar 17 09:01:35 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 8 previous similar messages Mar 17 09:01:40 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 0 seconds Mar 17 09:01:40 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 8 previous similar messages Mar 17 09:02:00 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.52@o2ib7: 20 seconds Mar 17 09:02:00 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 2 previous similar messages Mar 17 09:02:25 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Timed out tx for 10.0.10.51@o2ib7: 19 seconds Mar 17 09:02:25 fir-io1-s1 kernel: LNet: 91376:0:(o2iblnd_cb.c:3370:kiblnd_check_conns()) Skipped 11 previous similar messages Mar 17 09:02:30 fir-io1-s1 kernel: Lustre: 91455:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1552838544/real 1552838550] req@ffff985fe3c3fb00 x1625486304376640/t0(0) o400->MGC10.0.10.51@o2ib7@10.0.10.51@o2ib7:26/25 lens 224/224 e 0 to 1 dl 1552838551 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Mar 17 09:02:30 fir-io1-s1 kernel: LustreError: 166-1: MGC10.0.10.51@o2ib7: Connection to MGS (at 10.0.10.51@o2ib7) was lost; in progress operations using this service will fail Mar 17 09:02:47 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 17 09:02:48 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0002-mdtlov_UUID (at 10.0.10.51@o2ib7) Mar 17 09:02:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 09:02:49 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to fir-MDT0003-mdtlov_UUID (at 10.0.10.52@o2ib7) Mar 17 09:02:49 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 17 09:02:55 fir-io1-s1 kernel: LustreError: 167-0: fir-MDT0002-lwp-OST0000: This client was evicted by fir-MDT0002; in progress operations using this service will fail. Mar 17 09:02:55 fir-io1-s1 kernel: LustreError: Skipped 7 previous similar messages Mar 17 09:02:55 fir-io1-s1 kernel: Lustre: fir-MDT0000-lwp-OST0000: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 17 09:02:55 fir-io1-s1 kernel: Lustre: Skipped 27 previous similar messages Mar 17 09:03:20 fir-io1-s1 kernel: Lustre: Evicted from MGS (at 10.0.10.51@o2ib7) after server handle changed from 0x974d7e52601ddf to 0xe9a38a32063642f5 Mar 17 09:03:20 fir-io1-s1 kernel: Lustre: MGC10.0.10.51@o2ib7: Connection restored to 10.0.10.51@o2ib7 (at 10.0.10.51@o2ib7) Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x0:1843586 to 0x0:1843713 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x0:1843263 to 0x0:1843361 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x0:1843220 to 0x0:1843361 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x0:1843378 to 0x0:1843521 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0x0:1844056 to 0x0:1844225 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0x0:1843935 to 0x0:1844065 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000402:2501932 to 0xc40000402:2502113 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000402:2502082 to 0xc80000402:2502209 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000400:2502394 to 0x5c0000400:2502721 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000400:2502764 to 0x580000400:2502977 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000402:2501330 to 0x8c0000402:2501409 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000400:2502566 to 0x6c0000400:2502721 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000401:4474352 to 0x8c0000401:4474625 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000402:4477018 to 0x5c0000402:4477537 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000402:4475160 to 0x6c0000402:4475905 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000401:4474647 to 0xc80000401:4475233 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000402:4479575 to 0x580000402:4481857 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000401:4478464 to 0xc40000401:4481505 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0002: deleting orphan objects from 0x5c0000401:2738375 to 0x5c0000401:2738401 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0000: deleting orphan objects from 0x6c0000401:2737831 to 0x6c0000401:2737857 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0006: deleting orphan objects from 0xc40000400:2737631 to 0xc40000400:2737697 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST000a: deleting orphan objects from 0x580000401:2738233 to 0x580000401:2738305 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0008: deleting orphan objects from 0xc80000400:2737701 to 0xc80000400:2737729 Mar 17 09:03:40 fir-io1-s1 kernel: Lustre: fir-OST0004: deleting orphan objects from 0x8c0000400:2737708 to 0x8c0000400:2737729 Mar 17 09:22:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0a270a28-384b-10a4-3edd-bee22242e3ea (at 10.8.4.35@o2ib6) Mar 17 09:22:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 11:24:12 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2efa0721-d988-fc25-739d-9be810de2f6e (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98383acaf000, cur 1552847052 expire 1552846902 last 1552846825 Mar 17 11:24:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 11:24:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 17 11:24:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 11:31:24 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 54063cc6-6673-be2f-4b71-d24313c833b5 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98646d55f800, cur 1552847484 expire 1552847334 last 1552847257 Mar 17 11:31:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 11:31:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 17 11:31:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 14:40:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client f2c621f2-5d26-565b-bccc-7696037a054f (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985768858400, cur 1552858840 expire 1552858690 last 1552858613 Mar 17 14:40:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 14:41:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 17 14:41:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 16:25:48 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 464aa845-dbed-a8e7-d827-38d5dc3cee22 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984f5e9d2000, cur 1552865148 expire 1552864998 last 1552864921 Mar 17 16:25:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 19:23:20 fir-io1-s1 kernel: md: md4: data-check done. Mar 17 22:13:27 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0c8040e3-4813-12de-26fa-4eb04d12693b (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987259f5b400, cur 1552886007 expire 1552885857 last 1552885780 Mar 17 22:13:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 22:13:30 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0c8040e3-4813-12de-26fa-4eb04d12693b (at 10.9.112.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a92d9fc00, cur 1552886010 expire 1552885860 last 1552885783 Mar 17 22:13:30 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 17 22:21:31 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f53a043d-d024-dbbb-a021-95a52c6fc449 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984961f30000, cur 1552886491 expire 1552886341 last 1552886264 Mar 17 22:21:31 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 17 22:23:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 17 22:23:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 22:51:28 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client ef5944d0-ee26-32a0-7c1d-76b00a08bd53 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a0dc00, cur 1552888288 expire 1552888138 last 1552888061 Mar 17 22:51:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 17 22:52:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 17 22:52:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 08:42:26 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3b756410-0021-ec90-3727-b3e57184eba2 (at 10.8.1.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985815681800, cur 1552923746 expire 1552923596 last 1552923519 Mar 18 08:42:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 08:44:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 18 08:44:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 09:10:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fa3988d8-312e-baa0-298b-1666a8960425 (at 10.8.14.2@o2ib6) Mar 18 09:10:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 09:10:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3dbbb061-d93c-c7e8-88c8-e262ff513397 (at 10.8.14.6@o2ib6) Mar 18 09:10:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 09:11:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7751f34c-206a-edf3-4c97-12b3ac7dcb0e (at 10.8.23.14@o2ib6) Mar 18 09:11:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 09:12:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2286c65a-f3fa-726e-64b2-7afa3a454d38 (at 10.8.3.30@o2ib6) Mar 18 09:12:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 09:14:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 874d84ab-2918-f27e-a1fe-cdc3435eb5ad (at 10.8.2.18@o2ib6) Mar 18 09:14:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 09:14:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.8.9@o2ib6) Mar 18 09:14:28 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 18 09:14:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1c8970d4-d156-fd9f-2a97-4594ebb1da47 (at 10.8.12.6@o2ib6) Mar 18 09:14:53 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 18 09:15:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7796eaa1-24c1-f1a6-996a-1af3c662e968 (at 10.8.7.2@o2ib6) Mar 18 09:15:33 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Mar 18 09:16:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7d0cca0a-850a-1b33-0257-9a89827b9447 (at 10.8.26.11@o2ib6) Mar 18 09:16:41 fir-io1-s1 kernel: Lustre: Skipped 40 previous similar messages Mar 18 09:18:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e9d1b5f8-7ec9-998b-fe00-a6102cb74525 (at 10.9.102.2@o2ib4) Mar 18 09:18:51 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 18 11:58:09 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 205ae51c-a7d2-066e-af67-5c5bb1437452 (at 10.8.4.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98680675d400, cur 1552935489 expire 1552935339 last 1552935262 Mar 18 11:58:09 fir-io1-s1 kernel: Lustre: Skipped 203 previous similar messages Mar 18 12:20:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ca1af5b2-4b74-b03d-4a2b-13a823b2dc8f (at 10.8.15.10@o2ib6) Mar 18 12:20:44 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 18 12:21:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Mar 18 12:21:55 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 18 12:23:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to bd6b0907-bbf0-754e-ba62-411999a5fe50 (at 10.8.15.1@o2ib6) Mar 18 12:23:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 12:26:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7631cb3-40a3-08be-ae74-cf548ae0665c (at 10.8.14.8@o2ib6) Mar 18 12:26:03 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 18 12:30:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 22ee14eb-5a96-ad04-6e5f-188b7aec897d (at 10.8.12.33@o2ib6) Mar 18 12:30:33 fir-io1-s1 kernel: Lustre: Skipped 69 previous similar messages Mar 18 12:48:51 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 51ef8ccf-9de5-e575-ae0c-62dfa31a3aa7 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848831e6c00, cur 1552938531 expire 1552938381 last 1552938304 Mar 18 12:48:51 fir-io1-s1 kernel: Lustre: Skipped 77 previous similar messages Mar 18 12:48:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 51ef8ccf-9de5-e575-ae0c-62dfa31a3aa7 (at 10.8.15.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983801dccc00, cur 1552938534 expire 1552938384 last 1552938307 Mar 18 12:48:54 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 18 13:18:49 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b0dd25ee-158d-59f0-3d75-463b4d516c25 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984961f34c00, cur 1552940329 expire 1552940179 last 1552940102 Mar 18 13:19:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b0dd25ee-158d-59f0-3d75-463b4d516c25 (at 10.8.15.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984961f30000, cur 1552940346 expire 1552940196 last 1552940119 Mar 18 13:19:06 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 18 13:44:56 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 315dec7f-c968-195f-6605-28239e3811fb (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983858617800, cur 1552941896 expire 1552941746 last 1552941669 Mar 18 13:44:56 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 18 14:30:35 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6fce637b-f2db-9a4b-9957-4127f76f17d9 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825fd800, cur 1552944635 expire 1552944485 last 1552944408 Mar 18 14:30:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 14:30:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 6fce637b-f2db-9a4b-9957-4127f76f17d9 (at 10.8.15.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983858610c00, cur 1552944656 expire 1552944506 last 1552944429 Mar 18 14:30:56 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 18 15:37:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0cda1ca9-b849-4b59-7ce7-48abe2de3c2e (at 10.8.15.6@o2ib6) Mar 18 15:37:48 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Mar 18 15:38:41 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c5c9865c-ce96-1257-4be0-c4e1ce9d9f39 (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b85e000, cur 1552948721 expire 1552948571 last 1552948494 Mar 18 15:54:31 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 6584584d-5829-2821-07a0-a17c4d26330f (at 10.8.15.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98622f65d800, cur 1552949671 expire 1552949521 last 1552949444 Mar 18 15:54:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 16:10:56 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552950649/real 1552950649] req@ffff98381887f500 x1625487927056848/t0(0) o106->fir-OST0006@10.9.106.49@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552950656 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 18 16:11:03 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552950656/real 1552950656] req@ffff98381887f500 x1625487927056848/t0(0) o106->fir-OST0006@10.9.106.49@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552950663 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 16:11:10 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552950663/real 1552950663] req@ffff98381887f500 x1625487927056848/t0(0) o106->fir-OST0006@10.9.106.49@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552950670 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 16:11:17 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552950670/real 1552950670] req@ffff98381887f500 x1625487927056848/t0(0) o106->fir-OST0006@10.9.106.49@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552950677 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 16:11:24 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552950677/real 1552950677] req@ffff98381887f500 x1625487927056848/t0(0) o106->fir-OST0006@10.9.106.49@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552950684 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 16:11:38 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552950691/real 1552950691] req@ffff98381887f500 x1625487927056848/t0(0) o106->fir-OST0006@10.9.106.49@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552950698 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 16:11:38 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message Mar 18 16:11:59 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552950712/real 1552950712] req@ffff98381887f500 x1625487927056848/t0(0) o106->fir-OST0006@10.9.106.49@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1552950719 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 16:11:59 fir-io1-s1 kernel: Lustre: 49825:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 18 16:12:02 fir-io1-s1 kernel: LustreError: 49825:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.106.49@o2ib4) returned error from glimpse AST (req@ffff98381887f500 x1625487927056848 status -107 rc -107), evict it ns: filter-fir-OST0006_UUID lock: ffff985509f65100/0x49e1862e040f00fd lrc: 3/0,0 mode: PW/PW res: [0xc40000402:0x26517c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x40000000020000 nid: 10.9.106.49@o2ib4 remote: 0x3a68919bf4c90c7a expref: 6 pid: 96619 timeout: 0 lvb_type: 0 Mar 18 16:12:02 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0006: A client on nid 10.9.106.49@o2ib4 was evicted due to a lock glimpse callback time out: rc -107 Mar 18 16:12:02 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 18 16:12:02 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552950722s: evicting client at 10.9.106.49@o2ib4 ns: filter-fir-OST0006_UUID lock: ffff985509f65100/0x49e1862e040f00fd lrc: 3/0,0 mode: PW/PW res: [0xc40000402:0x26517c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->135167) flags: 0x40000000020000 nid: 10.9.106.49@o2ib4 remote: 0x3a68919bf4c90c7a expref: 7 pid: 96619 timeout: 0 lvb_type: 0 Mar 18 16:12:02 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Mar 18 16:12:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f3023e70-967c-cdca-d170-324afefea199 (at 10.9.106.49@o2ib4) Mar 18 16:12:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 16:12:04 fir-io1-s1 kernel: LustreError: 96903:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.106.49@o2ib4) returned error from glimpse AST (req@ffff9838474c3c00 x1625487928589024 status -107 rc -107), evict it ns: filter-fir-OST0004_UUID lock: ffff983c182af500/0x49e1862dfd666bef lrc: 3/0,0 mode: PW/PW res: [0x8c0000402:0x264d51:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000000000 nid: 10.9.106.49@o2ib4 remote: 0x3a68919bf4c8fb7f expref: 6 pid: 111284 timeout: 0 lvb_type: 0 Mar 18 16:12:04 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0008: A client on nid 10.9.106.49@o2ib4 was evicted due to a lock glimpse callback time out: rc -107 Mar 18 16:12:04 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1552950724s: evicting client at 10.9.106.49@o2ib4 ns: filter-fir-OST0008_UUID lock: ffff983c182a8d80/0x49e1862dfd666bfd lrc: 3/0,0 mode: PW/PW res: [0xc80000402:0x26506c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x40000000000000 nid: 10.9.106.49@o2ib4 remote: 0x3a68919bf4c8fbef expref: 6 pid: 111284 timeout: 0 lvb_type: 0 Mar 18 16:12:04 fir-io1-s1 kernel: LustreError: 96903:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Mar 18 16:12:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client b3a317af-4b80-b14d-dd83-74a6242899eb (at 10.8.26.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848346a5400, cur 1552950738 expire 1552950588 last 1552950511 Mar 18 16:12:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 16:27:54 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 201fc987-298b-0a86-11a6-146ce25c0c5b (at 10.8.9.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852b0f4b400, cur 1552951674 expire 1552951524 last 1552951447 Mar 18 16:27:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 18 16:43:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 11bdb068-3a68-b486-d427-87b2a02899d5 (at 10.8.2.24@o2ib6) Mar 18 16:43:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 16:44:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 87721f3b-4f03-c138-ffa3-cffa8a052df0 (at 10.8.26.5@o2ib6) Mar 18 16:44:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 17:47:38 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 631d966a-bcae-9487-bf11-b260d0f5b869 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984996393c00, cur 1552956458 expire 1552956308 last 1552956231 Mar 18 17:47:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 17:47:52 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 631d966a-bcae-9487-bf11-b260d0f5b869 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98380734e000, cur 1552956472 expire 1552956322 last 1552956245 Mar 18 17:47:58 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 631d966a-bcae-9487-bf11-b260d0f5b869 (at 10.8.23.14@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984996394c00, cur 1552956478 expire 1552956328 last 1552956251 Mar 18 17:47:58 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 18 17:48:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7751f34c-206a-edf3-4c97-12b3ac7dcb0e (at 10.8.23.14@o2ib6) Mar 18 17:48:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 18:51:46 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 6899a6ff-b1c1-d6d5-9839-98e0b5531d18 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984830fb6400, cur 1552960306 expire 1552960156 last 1552960079 Mar 18 18:51:46 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 18 18:51:56 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 6899a6ff-b1c1-d6d5-9839-98e0b5531d18 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853b080e800, cur 1552960316 expire 1552960166 last 1552960089 Mar 18 18:52:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 6899a6ff-b1c1-d6d5-9839-98e0b5531d18 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853b080b000, cur 1552960326 expire 1552960176 last 1552960099 Mar 18 18:52:16 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 6899a6ff-b1c1-d6d5-9839-98e0b5531d18 (at 10.8.15.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9838229de000, cur 1552960336 expire 1552960186 last 1552960109 Mar 18 18:52:16 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 18 22:44:35 fir-io1-s1 kernel: Lustre: 110574:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552974268/real 1552974268] req@ffff984067fd2100 x1625488913983952/t0(0) o106->fir-OST0006@10.8.11.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552974275 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 18 22:44:35 fir-io1-s1 kernel: Lustre: 110574:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 18 22:44:42 fir-io1-s1 kernel: Lustre: 110037:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552974275/real 1552974275] req@ffff98383bdd9500 x1625488913983968/t0(0) o106->fir-OST0008@10.8.11.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552974282 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 22:44:42 fir-io1-s1 kernel: Lustre: 96506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552974275/real 1552974275] req@ffff98384021ad00 x1625488913983936/t0(0) o106->fir-OST0004@10.8.11.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552974282 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 22:44:42 fir-io1-s1 kernel: Lustre: 110037:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 18 22:45:03 fir-io1-s1 kernel: Lustre: 96506:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552974296/real 1552974296] req@ffff98384021ad00 x1625488913983936/t0(0) o106->fir-OST0004@10.8.11.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552974303 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 22:45:03 fir-io1-s1 kernel: Lustre: 96506:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 18 22:45:45 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552974338/real 1552974338] req@ffff98384970ad00 x1625488913983920/t0(0) o106->fir-OST0000@10.8.11.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552974345 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 22:45:45 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 18 22:47:02 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552974415/real 1552974415] req@ffff98384970ad00 x1625488913983920/t0(0) o106->fir-OST0000@10.8.11.9@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552974422 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 18 22:47:02 fir-io1-s1 kernel: Lustre: 96335:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Mar 18 22:47:06 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 506c2262-4cca-3894-72db-d5ccf2174a66 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780867400, cur 1552974426 expire 1552974276 last 1552974199 Mar 18 22:47:06 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 18 22:48:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 18 22:48:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 22:56:18 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2f0937f7-8221-72c5-644a-212dfc93799d (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98383acad000, cur 1552974978 expire 1552974828 last 1552974751 Mar 18 22:56:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 22:57:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 18 22:57:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 23:39:21 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 91ae8bca-d651-0daa-5d63-b74e222862a5 (at 10.8.28.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984833dac400, cur 1552977561 expire 1552977411 last 1552977334 Mar 18 23:39:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 18 23:39:30 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 91ae8bca-d651-0daa-5d63-b74e222862a5 (at 10.8.28.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785dc9400, cur 1552977570 expire 1552977420 last 1552977343 Mar 18 23:39:30 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 19 00:12:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0405d36b-1dfe-417d-33da-88f65ca0bd9f (at 10.8.28.4@o2ib6) Mar 19 00:12:33 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 01:45:35 fir-io1-s1 kernel: Lustre: 96765:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552985128/real 1552985128] req@ffff98382460ad00 x1625489399006464/t0(0) o106->fir-OST000a@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552985135 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 19 01:45:35 fir-io1-s1 kernel: Lustre: 110621:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552985128/real 1552985128] req@ffff983810506f00 x1625489399006576/t0(0) o106->fir-OST0004@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552985135 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 19 01:45:56 fir-io1-s1 kernel: Lustre: 110621:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552985149/real 1552985149] req@ffff983810506f00 x1625489399006576/t0(0) o106->fir-OST0004@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552985156 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 01:45:56 fir-io1-s1 kernel: Lustre: 110621:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Mar 19 01:46:34 fir-io1-s1 kernel: Lustre: 110573:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552985183/real 1552985183] req@ffff985b71b63600 x1625489399006560/t0(0) o106->fir-OST0000@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552985194 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 01:46:34 fir-io1-s1 kernel: Lustre: 110573:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Mar 19 01:47:51 fir-io1-s1 kernel: Lustre: 110574:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1552985260/real 1552985260] req@ffff983840219200 x1625489399006528/t0(0) o106->fir-OST0002@10.8.14.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1552985271 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 01:47:51 fir-io1-s1 kernel: Lustre: 110574:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 34 previous similar messages Mar 19 01:48:30 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4af51c32-5433-8946-8fc2-4f1ad9c274b4 (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853b0809800, cur 1552985310 expire 1552985160 last 1552985083 Mar 19 01:48:30 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 19 01:48:49 fir-io1-s1 kernel: LNet: Service thread pid 110573 was inactive for 200.47s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 19 01:48:49 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 19 01:48:49 fir-io1-s1 kernel: Pid: 110573, comm: ll_ost00_094 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 19 01:48:49 fir-io1-s1 kernel: Call Trace: Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 19 01:48:49 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 19 01:48:49 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1552985329.110573 Mar 19 01:48:50 fir-io1-s1 kernel: LNet: Service thread pid 96765 was inactive for 201.86s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 19 01:48:50 fir-io1-s1 kernel: Pid: 96765, comm: ll_ost00_040 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 19 01:48:50 fir-io1-s1 kernel: Call Trace: Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 19 01:48:50 fir-io1-s1 kernel: Pid: 110574, comm: ll_ost00_095 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 19 01:48:50 fir-io1-s1 kernel: Call Trace: Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 19 01:48:50 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 19 01:48:55 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 4af51c32-5433-8946-8fc2-4f1ad9c274b4 (at 10.8.14.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677bc9c800, cur 1552985335 expire 1552985185 last 1552985108 Mar 19 01:48:55 fir-io1-s1 kernel: LNet: Service thread pid 110574 completed after 206.28s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 19 01:48:55 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Mar 19 02:39:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client a7e9d99b-3529-3851-2fc1-3e109f24d099 (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987094be1400, cur 1552988376 expire 1552988226 last 1552988149 Mar 19 02:39:36 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 19 02:41:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 19 02:41:46 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 06:34:32 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d251a63a-cf5f-7b49-f078-0f10ea97d229 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f67ec00, cur 1553002472 expire 1553002322 last 1553002245 Mar 19 06:34:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 06:37:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 19 06:37:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 06:42:48 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3923848c-a52c-adc9-c644-e9ae293e136d (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983806021c00, cur 1553002968 expire 1553002818 last 1553002741 Mar 19 06:42:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 06:58:11 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 016bfca4-4707-5e56-8faf-ae2a629ca193 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983829016400, cur 1553003891 expire 1553003741 last 1553003664 Mar 19 06:58:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 07:03:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 07:03:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 07:34:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 19 07:34:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 07:39:38 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b19f31e9-0c9f-7e7b-0ceb-f7adedb279ca (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678230ec00, cur 1553006378 expire 1553006228 last 1553006151 Mar 19 07:39:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 07:40:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 07:40:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:12:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f8edd23b-4bf8-0fec-9fdf-f7199ad12e29 (at 10.8.14.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f230000, cur 1553011974 expire 1553011824 last 1553011747 Mar 19 09:12:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:23:47 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client c6090610-4cd1-706d-5e96-56b48e513e07 (at 10.8.11.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f9da9e400, cur 1553012627 expire 1553012477 last 1553012400 Mar 19 09:23:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:25:03 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 3f756571-fb33-26bd-8c78-0a6a3d59beda (at 10.8.26.12@o2ib6) in 205 seconds. I think it's dead, and I am evicting it. exp ffff986bb8ae7c00, cur 1553012703 expire 1553012553 last 1553012498 Mar 19 09:25:03 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 19 09:25:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cafb4a30-f828-c84f-d0a5-247d32765a3e (at 10.8.1.27@o2ib6) Mar 19 09:25:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:25:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 43db6f02-39b1-2d91-8a5a-c0c4a16a2f08 (at 10.8.26.12@o2ib6) Mar 19 09:25:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:26:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 795ed85c-3d14-47f4-c094-a6509583ab56 (at 10.8.11.24@o2ib6) Mar 19 09:26:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:26:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cfaf9282-7f7c-20ef-2a1a-f242e378dd7c (at 10.8.16.8@o2ib6) Mar 19 09:26:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:26:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 664f5f86-7e37-c3dd-9009-3eec77c4bd45 (at 10.8.11.1@o2ib6) Mar 19 09:26:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:26:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1dc9ce85-fab5-042d-3bf3-bedbf1df78c7 (at 10.8.10.36@o2ib6) Mar 19 09:26:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:27:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 553f7c09-7b2f-bfaa-2ec4-3819c2429915 (at 10.8.13.10@o2ib6) Mar 19 09:27:44 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 19 09:45:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 434948ee-bfe5-5d8b-0ab2-6420dff4bd4f (at 10.8.10.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583b90a400, cur 1553013909 expire 1553013759 last 1553013682 Mar 19 09:45:09 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 19 09:47:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 19 09:47:18 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 19 09:53:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fe59dcfb-3bbe-5505-5f06-837da604cf7f (at 10.9.101.55@o2ib4) Mar 19 09:53:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:54:37 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d8dfe83-25e5-247f-c731-de21f1e85c71 (at 10.8.12.10@o2ib6) Mar 19 09:54:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:55:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 090fafdf-b851-44a0-92d1-bfda03f3741e (at 10.8.8.29@o2ib6) Mar 19 09:55:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 09:57:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7f613542-9147-a023-00d7-5cbc7f1f90e0 (at 10.8.11.27@o2ib6) Mar 19 09:57:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 10:11:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 0e6edbf0-d4e4-2bfd-8d10-ebcc53c0105c (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9862a032bc00, cur 1553015473 expire 1553015323 last 1553015246 Mar 19 10:11:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 10:11:20 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 0e6edbf0-d4e4-2bfd-8d10-ebcc53c0105c (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d58400, cur 1553015480 expire 1553015330 last 1553015253 Mar 19 10:11:20 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 19 10:11:25 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0e6edbf0-d4e4-2bfd-8d10-ebcc53c0105c (at 10.8.9.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984802d59400, cur 1553015485 expire 1553015335 last 1553015258 Mar 19 10:33:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 722fc89e-4fb6-0519-04d0-92f8091a9aa0 (at 10.8.14.5@o2ib6) Mar 19 10:33:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 10:37:01 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a557acba-5c99-b832-b8c9-7f1df36fe8bc (at 10.9.101.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868d49aac00, cur 1553017021 expire 1553016871 last 1553016794 Mar 19 10:37:01 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 19 10:50:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 19 10:50:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 10:51:28 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client a706839b-701f-5b4f-f0e7-4b38d85f93fa (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801541800, cur 1553017888 expire 1553017738 last 1553017661 Mar 19 10:51:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 10:51:28 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a706839b-701f-5b4f-f0e7-4b38d85f93fa (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801545000, cur 1553017888 expire 1553017738 last 1553017661 Mar 19 10:51:28 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 19 11:03:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fe59dcfb-3bbe-5505-5f06-837da604cf7f (at 10.9.101.55@o2ib4) Mar 19 11:03:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 11:05:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.29@o2ib4) Mar 19 11:05:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 11:05:10 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c6309390-d5ff-3002-57f3-e7314f018ae2 (at 10.9.101.29@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c6f400, cur 1553018710 expire 1553018560 last 1553018483 Mar 19 11:05:10 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 19 11:33:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 19 11:33:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 11:34:01 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 0017ebfd-95eb-fbc6-6ddb-e28993d1a90c (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e79c800, cur 1553020441 expire 1553020291 last 1553020214 Mar 19 11:34:01 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 19 11:36:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 19 11:36:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 11:37:28 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c7826685-0cd6-a002-cd95-0ea1ac8fe136 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98546cff6800, cur 1553020648 expire 1553020498 last 1553020421 Mar 19 11:37:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 11:57:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 3c73dbf8-3b9c-e12d-8e88-82999a4eea2b (at 10.8.15.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575de9e800, cur 1553021864 expire 1553021714 last 1553021637 Mar 19 11:57:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 11:57:54 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3c73dbf8-3b9c-e12d-8e88-82999a4eea2b (at 10.8.15.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678694f000, cur 1553021874 expire 1553021724 last 1553021647 Mar 19 12:05:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 19 12:05:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 12:06:18 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 02cd65a8-402f-266d-98fb-e5756f8cce59 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677e799800, cur 1553022378 expire 1553022228 last 1553022151 Mar 19 12:06:18 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 19 12:10:19 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 97584adb-1840-2125-be9f-ed6ea8df5749 (at 10.8.1.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762c76c00, cur 1553022619 expire 1553022469 last 1553022392 Mar 19 12:10:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 12:16:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f0b653e6-2546-c659-97c8-5d3c41619c38 (at 10.8.15.9@o2ib6) Mar 19 12:16:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 12:47:04 fir-io1-s1 kernel: Lustre: 96518:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553024817/real 1553024817] req@ffff98382c51c800 x1625491126691808/t0(0) o106->fir-OST0008@10.8.14.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553024824 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 19 12:47:04 fir-io1-s1 kernel: Lustre: 94237:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553024817/real 1553024817] req@ffff98381fe7e600 x1625491126691744/t0(0) o106->fir-OST0004@10.8.14.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553024824 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 19 12:47:04 fir-io1-s1 kernel: Lustre: 94237:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 24 previous similar messages Mar 19 12:47:25 fir-io1-s1 kernel: Lustre: 94237:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553024838/real 1553024838] req@ffff98381fe7e600 x1625491126691744/t0(0) o106->fir-OST0004@10.8.14.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553024845 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 12:47:25 fir-io1-s1 kernel: Lustre: 94237:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Mar 19 12:48:07 fir-io1-s1 kernel: Lustre: 110573:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553024880/real 1553024880] req@ffff9838474c0900 x1625491126691760/t0(0) o106->fir-OST0006@10.8.14.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553024887 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 12:48:07 fir-io1-s1 kernel: Lustre: 110620:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553024880/real 1553024880] req@ffff983a22d58900 x1625491126691728/t0(0) o106->fir-OST0000@10.8.14.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553024887 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 12:48:07 fir-io1-s1 kernel: Lustre: 110620:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Mar 19 12:48:07 fir-io1-s1 kernel: Lustre: 110573:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 19 12:48:21 fir-io1-s1 kernel: LustreError: 110620:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.14.1@o2ib6) returned error from glimpse AST (req@ffff983a22d58900 x1625491126691728 status -107 rc -107), evict it ns: filter-fir-OST0000_UUID lock: ffff98382a903600/0x49e1862e45740467 lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x274026:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x40000000000000 nid: 10.8.14.1@o2ib6 remote: 0x9877bf60f0845ca9 expref: 5 pid: 110620 timeout: 0 lvb_type: 0 Mar 19 12:48:21 fir-io1-s1 kernel: LustreError: 110620:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages Mar 19 12:48:21 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0004: A client on nid 10.8.14.1@o2ib6 was evicted due to a lock glimpse callback time out: rc -107 Mar 19 12:48:21 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Mar 19 12:48:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1553024901s: evicting client at 10.8.14.1@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff98382a905340/0x49e1862e45740483 lrc: 3/0,0 mode: PW/PW res: [0x8c0000402:0x273b08:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4294967296->34359738367) flags: 0x40000000000000 nid: 10.8.14.1@o2ib6 remote: 0x9877bf60f0845ce1 expref: 6 pid: 110620 timeout: 0 lvb_type: 0 Mar 19 12:48:21 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 4 previous similar messages Mar 19 12:49:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5c6d73c4-6d7c-e986-8930-74829de205ff (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d2c400, cur 1553024998 expire 1553024848 last 1553024771 Mar 19 12:49:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 13:17:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2c62a935-22f5-687e-61b8-7ce44c516d90 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868d2800, cur 1553026662 expire 1553026512 last 1553026435 Mar 19 13:17:42 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 19 13:19:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 13:19:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 13:31:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9b01daf5-c264-7fbf-6861-9205db47e51b (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b9d000, cur 1553027476 expire 1553027326 last 1553027249 Mar 19 13:31:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 19 13:31:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 19 13:31:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 13:50:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 19 13:50:45 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 19 13:51:38 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client f50067e8-8b33-9dca-d9d1-f4010211e6dd (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801bfcc00, cur 1553028698 expire 1553028548 last 1553028471 Mar 19 13:51:38 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 13:51:40 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f50067e8-8b33-9dca-d9d1-f4010211e6dd (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984996392400, cur 1553028700 expire 1553028550 last 1553028473 Mar 19 13:51:40 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 19 13:51:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f50067e8-8b33-9dca-d9d1-f4010211e6dd (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984801bf9800, cur 1553028701 expire 1553028551 last 1553028474 Mar 19 13:57:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 828dd8b3-0a36-2519-d144-487922890e9b (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98384f060000, cur 1553029075 expire 1553028925 last 1553028848 Mar 19 13:58:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 19 13:58:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:08:00 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 57f53103-f839-81d5-c131-897db97762e8 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b13800, cur 1553029680 expire 1553029530 last 1553029453 Mar 19 14:08:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:12:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 14:12:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:14:39 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client ab905b54-126d-719c-68ac-791f0a17153b (at 10.9.108.52@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f9da9a800, cur 1553030079 expire 1553029929 last 1553029852 Mar 19 14:14:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:30:10 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 386266ad-b29a-a641-0564-58c80858c967 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983bb7f1dc00, cur 1553031010 expire 1553030860 last 1553030783 Mar 19 14:30:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 19 14:32:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 14:32:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:39:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ab905b54-126d-719c-68ac-791f0a17153b (at 10.9.108.52@o2ib4) Mar 19 14:39:08 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 19 14:42:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 194ae90b-dda5-aeec-e623-aa1c27f6c383 (at 10.8.17.21@o2ib6) Mar 19 14:42:46 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 19 14:47:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e4e1992b-645c-7fa0-95fb-f97f854d138b (at 10.8.26.23@o2ib6) Mar 19 14:47:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:48:22 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ff43ce9a-6722-4dfb-851f-4ce9905ab062 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283dc400, cur 1553032102 expire 1553031952 last 1553031875 Mar 19 14:48:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:52:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 14:52:14 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:53:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 19 14:53:07 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:56:32 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 67fabff9-2aff-6efd-4e69-7d069cb29b1e (at 10.8.26.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853b080d000, cur 1553032592 expire 1553032442 last 1553032365 Mar 19 14:56:32 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 14:57:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9cc869db-86b5-83f4-3d1d-91c30da3920c (at 10.8.14.1@o2ib6) Mar 19 14:57:15 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:01:48 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e1b13546-795a-0c7f-c0f3-58293da03e68 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98683a53d800, cur 1553032908 expire 1553032758 last 1553032681 Mar 19 15:01:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:01:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e1b13546-795a-0c7f-c0f3-58293da03e68 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318eb400, cur 1553032918 expire 1553032768 last 1553032691 Mar 19 15:01:58 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 19 15:02:43 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 19 15:02:43 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:20:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 27e8ee0c-14af-405a-1862-22bcb182600a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98383aca8800, cur 1553034043 expire 1553033893 last 1553033816 Mar 19 15:21:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 19 15:21:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:26:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2d279b6b-ae49-37ac-0a12-0938de9dc4ca (at 10.8.1.29@o2ib6) Mar 19 15:26:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:29:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e4e1992b-645c-7fa0-95fb-f97f854d138b (at 10.8.26.23@o2ib6) Mar 19 15:29:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:39:29 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 26e0c53a-4e65-960e-65a8-e6757fba5388 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852e4704800, cur 1553035169 expire 1553035019 last 1553034942 Mar 19 15:39:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:47:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 752b566d-56d4-609e-539e-fa0972b935ec (at 10.8.20.15@o2ib6) Mar 19 15:47:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:48:45 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a9ea0fe4-9f31-0606-5dea-c6a4e81b7ee7 (at 10.8.20.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864523b8000, cur 1553035725 expire 1553035575 last 1553035498 Mar 19 15:48:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:50:03 fir-io1-s1 kernel: Lustre: 74748:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553035796/real 1553035796] req@ffff9864802ab900 x1625491613828208/t0(0) o106->fir-OST0006@10.8.11.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553035803 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 19 15:50:03 fir-io1-s1 kernel: Lustre: 74748:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Mar 19 15:50:17 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553035810/real 1553035810] req@ffff98381f25ce00 x1625491613828240/t0(0) o106->fir-OST000a@10.8.11.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553035817 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 15:50:17 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Mar 19 15:50:38 fir-io1-s1 kernel: Lustre: 96499:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553035831/real 1553035831] req@ffff9838474c1800 x1625491613828224/t0(0) o106->fir-OST0008@10.8.11.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553035838 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 15:50:38 fir-io1-s1 kernel: Lustre: 96499:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 19 15:51:20 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553035873/real 1553035873] req@ffff985d739c1800 x1625491613828256/t0(0) o106->fir-OST0002@10.8.11.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553035880 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 15:51:20 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 19 15:52:37 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553035950/real 1553035950] req@ffff98381f25ce00 x1625491613828240/t0(0) o106->fir-OST000a@10.8.11.10@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553035957 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 19 15:52:37 fir-io1-s1 kernel: Lustre: 96947:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 44 previous similar messages Mar 19 15:53:16 fir-io1-s1 kernel: LNet: Service thread pid 74748 was inactive for 200.53s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 19 15:53:16 fir-io1-s1 kernel: LNet: Skipped 1 previous similar message Mar 19 15:53:16 fir-io1-s1 kernel: Pid: 74748, comm: ll_ost02_082 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 19 15:53:16 fir-io1-s1 kernel: Call Trace: Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 19 15:53:16 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 19 15:53:16 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1553035996.74748 Mar 19 15:53:17 fir-io1-s1 kernel: LNet: Service thread pid 96499 was inactive for 201.59s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 19 15:53:17 fir-io1-s1 kernel: Pid: 96499, comm: ll_ost00_026 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 19 15:53:17 fir-io1-s1 kernel: Call Trace: Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 19 15:53:17 fir-io1-s1 kernel: Pid: 96947, comm: ll_ost02_064 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 19 15:53:17 fir-io1-s1 kernel: Call Trace: Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 19 15:53:17 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 19 15:53:18 fir-io1-s1 kernel: Pid: 96253, comm: ll_ost02_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 19 15:53:18 fir-io1-s1 kernel: Call Trace: Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 19 15:53:18 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 19 15:53:19 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d5530428-6ade-c9de-ffe0-dea132e98805 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98380be38400, cur 1553035999 expire 1553035849 last 1553035772 Mar 19 15:53:19 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 15:53:19 fir-io1-s1 kernel: LNet: Service thread pid 96947 completed after 202.82s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 19 15:53:19 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 19 17:12:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 19 17:12:44 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 19 18:23:52 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client b7b00dce-254c-3fee-52a4-03caca689aa9 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985756de5c00, cur 1553045032 expire 1553044882 last 1553044805 Mar 19 18:23:52 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 18:28:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 18:28:23 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 19 21:08:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 19 21:08:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 21:11:36 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 98bc7d1b-b251-d2f4-d279-8f037b579329 (at 10.8.23.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9872e8fc5c00, cur 1553055096 expire 1553054946 last 1553054869 Mar 19 21:11:36 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 19 21:40:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 964dd845-5b8d-e1ee-3136-b2e16703b1a9 (at 10.8.23.21@o2ib6) Mar 19 21:40:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 21:42:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client ee572418-61fd-bb0f-2f27-b402c449535f (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d78400, cur 1553056929 expire 1553056779 last 1553056702 Mar 19 21:42:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 19 21:44:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 19 21:44:27 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 00:53:05 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8a895c8b-0904-d7a9-2e7a-626cb7240359 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853b080dc00, cur 1553068385 expire 1553068235 last 1553068158 Mar 20 00:53:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 00:53:09 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8a895c8b-0904-d7a9-2e7a-626cb7240359 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9853b080f400, cur 1553068389 expire 1553068239 last 1553068162 Mar 20 00:53:09 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 20 00:55:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 20 00:55:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 01:35:50 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client a7de0fce-472e-35aa-c25d-8ab8c8ad252b (at 10.8.11.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834684400, cur 1553070950 expire 1553070800 last 1553070723 Mar 20 01:38:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c4f458ee-079e-1f6b-715d-4cc60d32c4b8 (at 10.8.11.4@o2ib6) Mar 20 01:38:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 01:48:04 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d411c561-adfa-7f78-e534-2dbceed521e6 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483a2edc00, cur 1553071684 expire 1553071534 last 1553071457 Mar 20 01:48:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 01:50:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 20 01:50:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 03:04:25 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 20 03:04:25 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 20 06:13:03 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 9e716467-9dbf-5768-e1d9-bc12bbba7e1b (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985835776c00, cur 1553087583 expire 1553087433 last 1553087356 Mar 20 06:13:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 07:52:24 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 58fe5ad1-8676-9b8c-0acf-d3a6f6cbb3c3 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583aaba000, cur 1553093544 expire 1553093394 last 1553093317 Mar 20 07:52:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 07:52:43 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 58fe5ad1-8676-9b8c-0acf-d3a6f6cbb3c3 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9863e663a000, cur 1553093563 expire 1553093413 last 1553093336 Mar 20 07:52:43 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 20 07:52:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 20 07:52:54 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 20 08:46:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ce789f13-bc41-259b-1230-a0fd4905bf23 (at 10.8.11.10@o2ib6) Mar 20 08:46:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 08:51:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a18f5541-141e-0b60-6f2b-a0d2a0024968 (at 10.9.104.48@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985077a2a000, cur 1553097068 expire 1553096918 last 1553096841 Mar 20 08:52:24 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 8c9bebd2-342e-0eed-0fe5-980eceed6b2a (at 10.8.10.28@o2ib6) in 225 seconds. I think it's dead, and I am evicting it. exp ffff985acab6bc00, cur 1553097144 expire 1553096994 last 1553096919 Mar 20 08:52:24 fir-io1-s1 kernel: Lustre: Skipped 89 previous similar messages Mar 20 08:52:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8c9bebd2-342e-0eed-0fe5-980eceed6b2a (at 10.8.10.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986784b3e800, cur 1553097146 expire 1553096996 last 1553096919 Mar 20 08:52:26 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Mar 20 08:56:37 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client dca8880b-a0a6-6e7b-c8b2-9c4e4dd52c70 (at 10.9.113.14@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867804f1400, cur 1553097397 expire 1553097247 last 1553097170 Mar 20 08:56:37 fir-io1-s1 kernel: Lustre: Skipped 7 previous similar messages Mar 20 09:17:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 25268aa7-8aba-977f-1700-754dcdcbd041 (at 10.9.107.14@o2ib4) Mar 20 09:17:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 09:17:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72da9a6e-2827-1c9d-1aa6-7b398153fee1 (at 10.9.106.71@o2ib4) Mar 20 09:17:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 09:17:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2c2d2f6b-3086-1f70-68ed-98263873eaff (at 10.9.107.25@o2ib4) Mar 20 09:17:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 09:18:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 032aae30-e439-a130-1f18-efe924baca21 (at 10.9.106.70@o2ib4) Mar 20 09:18:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 09:18:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9041b005-ca79-5425-d710-65376539b634 (at 10.9.107.59@o2ib4) Mar 20 09:18:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 09:18:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b5d0fd65-06a7-fd14-4925-ef95dfe63868 (at 10.9.107.28@o2ib4) Mar 20 09:18:24 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 20 09:18:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fbb7e0da-3603-8dfb-de71-fd8cea5618ef (at 10.9.106.69@o2ib4) Mar 20 09:18:48 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Mar 20 09:19:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9fdeb7e4-dbe7-36a2-5704-e3cbd89d3a9c (at 10.9.104.25@o2ib4) Mar 20 09:19:22 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 09:20:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 57bd9565-249c-08fa-b75e-115d9c0f2fee (at 10.9.104.26@o2ib4) Mar 20 09:20:33 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 20 09:22:41 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 795ed85c-3d14-47f4-c094-a6509583ab56 (at 10.8.11.24@o2ib6) Mar 20 09:22:41 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 20 09:27:19 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9cc869db-86b5-83f4-3d1d-91c30da3920c (at 10.8.14.1@o2ib6) Mar 20 09:27:19 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Mar 20 09:52:08 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 2d5487f8-fc66-6ff4-f642-a858d8c06d39 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481a6c5800, cur 1553100728 expire 1553100578 last 1553100501 Mar 20 09:52:08 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Mar 20 09:52:23 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 2d5487f8-fc66-6ff4-f642-a858d8c06d39 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483bf74c00, cur 1553100743 expire 1553100593 last 1553100516 Mar 20 09:52:23 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 20 10:06:37 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client c949f5b2-7a2e-f96d-353d-05d9c3a77be2 (at 10.8.7.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758c23c00, cur 1553101597 expire 1553101447 last 1553101370 Mar 20 10:27:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 72786b44-506b-3a4f-18fb-59ce5db7cb7f (at 10.9.107.52@o2ib4) Mar 20 10:27:07 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 20 10:30:16 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client a5c8a1f3-adab-eac4-941d-2952eaff6418 (at 10.8.30.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d229d6c00, cur 1553103016 expire 1553102866 last 1553102789 Mar 20 10:30:16 fir-io1-s1 kernel: Lustre: Skipped 383 previous similar messages Mar 20 10:30:16 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 2d2fd5f3-208c-0377-6a4c-e1c4421cdbef (at 10.8.30.11@o2ib6) in 214 seconds. I think it's dead, and I am evicting it. exp ffff984e2e1b4c00, cur 1553103016 expire 1553102866 last 1553102802 Mar 20 10:30:16 fir-io1-s1 kernel: Lustre: Skipped 114 previous similar messages Mar 20 10:30:18 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to f87e49de-2ac6-e466-4f82-af6dcb4e090b (at 10.9.112.13@o2ib4) Mar 20 10:30:18 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Mar 20 10:32:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 586a6e8e-70fe-af28-3ec3-56d6983d8923 (at 10.8.9.6@o2ib6) Mar 20 10:32:49 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 20 10:37:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0b22cd50-4b3f-cc55-9158-e0958bde4beb (at 10.8.18.27@o2ib6) Mar 20 10:37:08 fir-io1-s1 kernel: Lustre: Skipped 221 previous similar messages Mar 20 10:50:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 34c048ca-f7f5-abef-2125-67a4e23c4ce9 (at 10.8.19.3@o2ib6) Mar 20 10:50:47 fir-io1-s1 kernel: Lustre: Skipped 129 previous similar messages Mar 20 11:01:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 513e4107-cd39-8515-1387-2ee9a5768f3f (at 10.8.12.29@o2ib6) Mar 20 11:01:07 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Mar 20 11:40:08 fir-io1-s1 kernel: Lustre: 96945:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553107201/real 1553107201] req@ffff986a239fd100 x1625493774659520/t0(0) o106->fir-OST0006@10.8.12.18@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553107208 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 20 11:40:08 fir-io1-s1 kernel: Lustre: 96945:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 22 previous similar messages Mar 20 11:40:29 fir-io1-s1 kernel: Lustre: 111281:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553107222/real 1553107222] req@ffff98771d4f1800 x1625493774659536/t0(0) o106->fir-OST0008@10.8.12.18@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553107229 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 11:40:29 fir-io1-s1 kernel: Lustre: 111281:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Mar 20 11:41:11 fir-io1-s1 kernel: Lustre: 96379:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553107264/real 1553107264] req@ffff9874c69ae300 x1625493774659568/t0(0) o106->fir-OST0002@10.8.12.18@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553107271 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 11:41:11 fir-io1-s1 kernel: Lustre: 2629:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553107264/real 1553107264] req@ffff986a09b52400 x1625493774659552/t0(0) o106->fir-OST000a@10.8.12.18@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553107271 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 11:41:11 fir-io1-s1 kernel: Lustre: 2629:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages Mar 20 11:41:11 fir-io1-s1 kernel: Lustre: 96379:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 20 11:42:41 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client e80fe1ea-9195-6988-8429-f40395c91cf7 (at 10.8.12.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f9da9e000, cur 1553107361 expire 1553107211 last 1553107134 Mar 20 11:42:41 fir-io1-s1 kernel: Lustre: Skipped 40 previous similar messages Mar 20 11:45:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 68c817a5-11d6-cdbd-24ae-c48f1f4fb878 (at 10.8.10.9@o2ib6) Mar 20 11:45:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 12:04:44 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 12:04:44 fir-io1-s1 kernel: Lustre: fir-OST0008: Connection restored to 3cc732cd-cd8a-e01a-1ca8-de6a3b7d52cc (at 10.8.25.8@o2ib6) Mar 20 12:05:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 12:05:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 3cc732cd-cd8a-e01a-1ca8-de6a3b7d52cc (at 10.8.25.8@o2ib6) Mar 20 12:09:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 92c5a2a2-e228-d051-8b1a-a4a6b8577967 (at 10.9.102.41@o2ib4) Mar 20 12:09:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 12:10:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8d1694af-d32c-3288-00b1-fe38a72175fb (at 10.9.104.37@o2ib4) Mar 20 12:10:16 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 12:11:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 26c5771f-6bff-b79c-941d-a328fa48123c (at 10.9.102.6@o2ib4) Mar 20 12:11:35 fir-io1-s1 kernel: Lustre: Skipped 59 previous similar messages Mar 20 12:14:10 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d3b1cb87-f594-fafe-f3ca-be9e7cfe9d17 (at 10.8.22.35@o2ib6) Mar 20 12:14:10 fir-io1-s1 kernel: Lustre: Skipped 185 previous similar messages Mar 20 12:24:55 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client b43de966-3bd1-cb7d-7f33-8c2ea23259e2 (at 10.8.20.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7e8c00, cur 1553109895 expire 1553109745 last 1553109668 Mar 20 12:24:55 fir-io1-s1 kernel: Lustre: Skipped 503 previous similar messages Mar 20 12:33:56 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 811d2121-781c-a741-e2fa-8f2029fcc64a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576184b400, cur 1553110436 expire 1553110286 last 1553110209 Mar 20 12:33:56 fir-io1-s1 kernel: Lustre: Skipped 767 previous similar messages Mar 20 12:34:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 20 12:34:09 fir-io1-s1 kernel: Lustre: Skipped 239 previous similar messages Mar 20 12:39:17 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 3187b83d-8988-cdea-c743-36c915b58e40 (at 10.8.6.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834174400, cur 1553110757 expire 1553110607 last 1553110530 Mar 20 12:39:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 12:40:33 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client ab5d64b3-5ec1-d28c-7331-64a8d459ad2b (at 10.8.26.14@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff9867868ff400, cur 1553110833 expire 1553110683 last 1553110611 Mar 20 12:40:33 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Mar 20 12:46:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1e0dbe81-97a8-a9d0-3976-d5a8c6b1ba02 (at 10.9.108.16@o2ib4) Mar 20 12:46:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 12:51:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 084ae2e7-32eb-f718-134b-ac7a3c2328d4 (at 10.9.101.9@o2ib4) Mar 20 12:51:35 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 12:52:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.29@o2ib4) Mar 20 12:52:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 12:53:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 958dbe37-44f4-f3c8-99a0-45b8f975616f (at 10.9.101.60@o2ib4) Mar 20 12:53:03 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 12:54:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b41950de-0614-6c1a-0d53-d43c60fe0f33 (at 10.9.102.1@o2ib4) Mar 20 12:54:18 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 12:58:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 420eb17f-e3a0-1fd9-bcf2-389dfdfba340 (at 10.8.20.5@o2ib6) Mar 20 12:58:18 fir-io1-s1 kernel: Lustre: Skipped 77 previous similar messages Mar 20 13:00:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1e5f7f4c-78fa-5eb7-a0ea-e8f04fabf57f (at 10.8.30.32@o2ib6) Mar 20 13:00:57 fir-io1-s1 kernel: Lustre: Skipped 125 previous similar messages Mar 20 13:06:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8e59dd76-3b27-3c03-ab99-aa40f6b47fdd (at 10.8.13.12@o2ib6) Mar 20 13:06:02 fir-io1-s1 kernel: Lustre: Skipped 323 previous similar messages Mar 20 13:16:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0edeac5b-ee1e-024f-de97-9e0fc3efb1af (at 10.8.6.2@o2ib6) Mar 20 13:16:18 fir-io1-s1 kernel: Lustre: Skipped 346 previous similar messages Mar 20 13:36:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ba196fcc-f372-6c13-d1d6-766c67a1554e (at 10.8.12.31@o2ib6) Mar 20 13:36:53 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 14:08:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to aed0066d-b988-82aa-d3d8-7a27a4fc5a96 (at 10.8.13.20@o2ib6) Mar 20 14:08:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 14:18:23 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 14:18:23 fir-io1-s1 kernel: Lustre: fir-OST000a: Connection restored to 3cc732cd-cd8a-e01a-1ca8-de6a3b7d52cc (at 10.8.25.8@o2ib6) Mar 20 14:18:23 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 14:18:58 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 207c059a-5ed2-57da-6a02-2e571cd2325f (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9838570c2800, cur 1553116738 expire 1553116588 last 1553116511 Mar 20 14:18:58 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 20 14:19:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 14:19:00 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 3cc732cd-cd8a-e01a-1ca8-de6a3b7d52cc (at 10.8.25.8@o2ib6) Mar 20 14:19:00 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 14:19:25 fir-io1-s1 kernel: Lustre: fir-OST000a: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 14:19:33 fir-io1-s1 kernel: Lustre: fir-OST0002: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 14:20:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 14:20:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3cc732cd-cd8a-e01a-1ca8-de6a3b7d52cc (at 10.8.25.8@o2ib6) Mar 20 14:20:07 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 20 14:20:32 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.25.8@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 14:20:56 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 14:20:56 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 14:22:04 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 62ba2d9a-846e-e540-b91f-466c303dcd38 (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985c3d664000, cur 1553116924 expire 1553116774 last 1553116697 Mar 20 14:22:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 14:22:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Client 3a67cbce-ebcb-cb9d-0966-475b6050e93b (at 10.8.25.8@o2ib6) reconnecting Mar 20 14:22:39 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0003_UUID: not available for connect from 10.8.25.8@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 14:22:39 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 3cc732cd-cd8a-e01a-1ca8-de6a3b7d52cc (at 10.8.25.8@o2ib6) Mar 20 14:22:39 fir-io1-s1 kernel: Lustre: Skipped 8 previous similar messages Mar 20 14:30:47 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 98e83ed5-6d59-446c-8f7b-05df06bf758a (at 10.9.101.36@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f58fd4000, cur 1553117447 expire 1553117297 last 1553117220 Mar 20 14:30:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 14:36:32 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 856d5a69-336a-a4d3-87e4-dd9fb79c5879 (at 10.8.18.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a09c00, cur 1553117792 expire 1553117642 last 1553117565 Mar 20 14:36:32 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 14:37:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ee125ca1-89a8-138e-2ca7-035aecd4a796 (at 10.8.20.6@o2ib6) in 213 seconds. I think it's dead, and I am evicting it. exp ffff9854e2654000, cur 1553117868 expire 1553117718 last 1553117655 Mar 20 14:37:48 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 20 14:48:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9d71a68d-4d5a-cecc-964a-d0b52e79f962 (at 10.8.4.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848001fd400, cur 1553118484 expire 1553118334 last 1553118257 Mar 20 14:48:04 fir-io1-s1 kernel: Lustre: Skipped 647 previous similar messages Mar 20 14:51:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 20 14:51:24 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 20 14:52:22 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f0060400-9cd0-2c3e-adbb-0229a7af009a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98678677d000, cur 1553118742 expire 1553118592 last 1553118515 Mar 20 14:52:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 14:59:01 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to ecc5b628-5efd-fbfd-7392-a1abe17de407 (at 10.9.101.36@o2ib4) Mar 20 14:59:01 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 20 14:59:42 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to d3b133e8-4ec8-ebe3-7fc5-79aa16e59c0b (at 10.9.106.35@o2ib4) Mar 20 14:59:42 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Mar 20 15:02:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0e1cc7ee-ac14-2533-62de-8aa817b3cbc6 (at 10.8.4.26@o2ib6) Mar 20 15:02:38 fir-io1-s1 kernel: Lustre: Skipped 22 previous similar messages Mar 20 15:05:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3c41d89d-9b92-2d59-5f67-c6b03989a988 (at 10.9.104.64@o2ib4) Mar 20 15:05:14 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 15:06:53 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5ec0de6b-ae9e-c1b9-fb48-2940b5b4166f (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9847ae51e000, cur 1553119613 expire 1553119463 last 1553119386 Mar 20 15:06:53 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 15:10:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9d602dc8-8051-8982-51eb-1bb4250b93cd (at 10.8.22.18@o2ib6) Mar 20 15:10:20 fir-io1-s1 kernel: Lustre: Skipped 203 previous similar messages Mar 20 15:22:47 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5bb66e6a-08d8-6220-642a-dafa120f0139 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e53800, cur 1553120567 expire 1553120417 last 1553120340 Mar 20 15:22:47 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 15:22:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 5bb66e6a-08d8-6220-642a-dafa120f0139 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c3e800, cur 1553120571 expire 1553120421 last 1553120344 Mar 20 15:22:53 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 5bb66e6a-08d8-6220-642a-dafa120f0139 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839c3b400, cur 1553120573 expire 1553120423 last 1553120346 Mar 20 15:22:53 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 15:23:07 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 20 15:23:07 fir-io1-s1 kernel: Lustre: Skipped 407 previous similar messages Mar 20 15:45:30 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d36389f3-850d-8021-2fce-3e8c14fa2e90 (at 10.8.23.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98724a6dc800, cur 1553121930 expire 1553121780 last 1553121703 Mar 20 15:45:30 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 20 15:52:21 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 54fa294d-51a2-8f9e-e24d-863f96dcbd1b (at 10.9.101.56@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867a04a5800, cur 1553122341 expire 1553122191 last 1553122114 Mar 20 15:52:21 fir-io1-s1 kernel: Lustre: Skipped 611 previous similar messages Mar 20 15:55:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 476e933b-2664-4b57-53cf-d95b660fb2b3 (at 10.9.101.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785c1f000, cur 1553122556 expire 1553122406 last 1553122329 Mar 20 15:55:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 16:07:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 62f38112-c51d-b467-8668-293d1a60bffb (at 10.9.103.40@o2ib4) Mar 20 16:07:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 16:12:47 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15a46d11-bb13-9263-b09b-73d01716030b (at 10.9.104.1@o2ib4) Mar 20 16:12:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 16:15:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8c552fe1-a520-0d81-25ee-2ab3174795af (at 10.9.105.13@o2ib4) Mar 20 16:15:18 fir-io1-s1 kernel: Lustre: Skipped 118 previous similar messages Mar 20 16:20:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a7ee5e6e-4193-d860-551d-c6fefce9827c (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985769984000, cur 1553124008 expire 1553123858 last 1553123781 Mar 20 16:20:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 16:20:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) Mar 20 16:20:24 fir-io1-s1 kernel: Lustre: Skipped 221 previous similar messages Mar 20 16:34:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2c9768a0-0ebf-5ca5-977c-76a19199ac74 (at 10.9.101.5@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98679fbfac00, cur 1553124882 expire 1553124732 last 1553124655 Mar 20 16:34:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 16:38:13 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client b0c348d2-5728-bcff-54ca-e6ded1621be9 (at 10.8.25.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483513bc00, cur 1553125093 expire 1553124943 last 1553124866 Mar 20 16:38:13 fir-io1-s1 kernel: Lustre: Skipped 329 previous similar messages Mar 20 16:46:16 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8ca0d763-ddf9-424b-6f83-6714563736cc (at 10.8.17.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984c9069ec00, cur 1553125576 expire 1553125426 last 1553125349 Mar 20 16:46:16 fir-io1-s1 kernel: Lustre: Skipped 203 previous similar messages Mar 20 17:01:30 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15a46d11-bb13-9263-b09b-73d01716030b (at 10.9.104.1@o2ib4) Mar 20 17:01:30 fir-io1-s1 kernel: Lustre: Skipped 281 previous similar messages Mar 20 17:02:46 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d5a74680-e7af-ebfb-7dfd-72e2645d277b (at 10.9.101.51@o2ib4) Mar 20 17:02:46 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 20 17:05:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8ce54e8a-98e9-eb75-31d3-9152b3013e49 (at 10.8.17.13@o2ib6) Mar 20 17:05:21 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Mar 20 17:10:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1ca40c95-5615-186b-162f-92f0324c3c09 (at 10.8.26.21@o2ib6) Mar 20 17:10:23 fir-io1-s1 kernel: Lustre: Skipped 323 previous similar messages Mar 20 17:11:15 fir-io1-s1 kernel: Lustre: 94240:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553127068/real 1553127068] req@ffff984cb2b69b00 x1625496838153168/t0(0) o106->fir-OST0008@10.8.26.24@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553127075 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 20 17:11:15 fir-io1-s1 kernel: Lustre: 94240:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 51 previous similar messages Mar 20 17:11:36 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553127089/real 1553127089] req@ffff987497162d00 x1625496838153248/t0(0) o106->fir-OST0000@10.8.26.24@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553127096 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 17:11:36 fir-io1-s1 kernel: Lustre: 96375:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 20 17:12:18 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553127131/real 1553127131] req@ffff9852cea5c500 x1625496838153184/t0(0) o106->fir-OST000a@10.8.26.24@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553127138 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 17:12:18 fir-io1-s1 kernel: Lustre: 96357:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Mar 20 17:13:35 fir-io1-s1 kernel: Lustre: 96346:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553127208/real 1553127208] req@ffff984e1f216c00 x1625496838153216/t0(0) o106->fir-OST0002@10.8.26.24@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553127215 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 17:13:35 fir-io1-s1 kernel: Lustre: 96346:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41 previous similar messages Mar 20 17:14:28 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client a73c92c5-a880-e64e-0d9b-93c1decd2638 (at 10.9.104.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9862a43f9000, cur 1553127268 expire 1553127118 last 1553127041 Mar 20 17:14:28 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Mar 20 17:15:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 7c8cd262-1540-9440-72b3-b1d4951bc141 (at 10.8.7.35@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff986785c47000, cur 1553127344 expire 1553127194 last 1553127118 Mar 20 17:15:44 fir-io1-s1 kernel: Lustre: Skipped 137 previous similar messages Mar 20 17:19:29 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 52868190-18c7-1725-83e6-feb668ec0fb9 (at 10.9.102.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865f0ecd800, cur 1553127569 expire 1553127419 last 1553127342 Mar 20 17:19:29 fir-io1-s1 kernel: Lustre: Skipped 269 previous similar messages Mar 20 17:28:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2f6fe8b6-836c-dd12-6a69-66ec0db776e6 (at 10.9.106.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575c533c00, cur 1553128082 expire 1553127932 last 1553127855 Mar 20 17:28:02 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 17:33:31 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c1058372-4fdb-dd48-7766-74d1ae77251a (at 10.9.101.28@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a92d9b000, cur 1553128411 expire 1553128261 last 1553128184 Mar 20 17:33:31 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 17:34:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4e48592f-b97d-5c93-9da4-86c872d7a486 (at 10.9.107.43@o2ib4) Mar 20 17:34:27 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Mar 20 17:35:45 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 15977d36-cdb8-43c9-109d-47180b552ba3 (at 10.9.106.36@o2ib4) Mar 20 17:35:45 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 17:38:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6e436ee1-56e5-e1f3-3459-58b43a359102 (at 10.9.108.17@o2ib4) Mar 20 17:38:29 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 20 17:43:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cb67b941-504b-3226-9e75-e94440d73a8e (at 10.9.104.3@o2ib4) Mar 20 17:43:31 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 20 17:47:21 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client f74dffee-1211-08cc-579f-e3fbe93c1b8c (at 10.8.9.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9864ac6e8400, cur 1553129241 expire 1553129091 last 1553129014 Mar 20 17:47:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 17:54:58 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 8fa4f5c3-ede0-631d-8d92-2a217b137a17 (at 10.9.104.65@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983801b55400, cur 1553129698 expire 1553129548 last 1553129471 Mar 20 17:54:58 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 18:01:27 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 22505c78-b9c2-e28a-88c4-7dadc4be41e9 (at 10.9.101.28@o2ib4) Mar 20 18:01:27 fir-io1-s1 kernel: Lustre: Skipped 352 previous similar messages Mar 20 18:15:47 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 66d5ed4b-5e65-7676-63fe-3f1978a4b814 (at 10.9.103.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dffa800, cur 1553130947 expire 1553130797 last 1553130720 Mar 20 18:15:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 18:15:50 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 66d5ed4b-5e65-7676-63fe-3f1978a4b814 (at 10.9.103.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dff8400, cur 1553130950 expire 1553130800 last 1553130723 Mar 20 18:15:50 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 18:15:55 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 66d5ed4b-5e65-7676-63fe-3f1978a4b814 (at 10.9.103.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480332a800, cur 1553130955 expire 1553130805 last 1553130728 Mar 20 18:16:00 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 66d5ed4b-5e65-7676-63fe-3f1978a4b814 (at 10.9.103.18@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98514dff8c00, cur 1553130960 expire 1553130810 last 1553130733 Mar 20 18:23:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e0d0d81e-4164-f236-7d8a-b466fb3eea50 (at 10.9.104.65@o2ib4) Mar 20 18:23:25 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 20 18:31:32 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e305a56e-2848-5110-d816-9f9e12e50119 (at 10.8.30.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985765736800, cur 1553131892 expire 1553131742 last 1553131665 Mar 20 18:31:32 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 18:32:48 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 26ffca9a-41a5-3ce8-902c-ae89522244f7 (at 10.9.102.16@o2ib4) in 162 seconds. I think it's dead, and I am evicting it. exp ffff986785d2fc00, cur 1553131968 expire 1553131818 last 1553131806 Mar 20 18:32:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 18:34:04 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client da7252d8-69e3-ba7b-3700-5d7659da07db (at 10.8.6.33@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff983812d72000, cur 1553132044 expire 1553131894 last 1553131818 Mar 20 18:34:04 fir-io1-s1 kernel: Lustre: Skipped 317 previous similar messages Mar 20 18:34:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 18:34:24 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 18:36:50 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 9032fdbf-5d7e-cc23-e194-ce64548cb3db (at 10.8.21.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857654f3000, cur 1553132210 expire 1553132060 last 1553131983 Mar 20 18:36:50 fir-io1-s1 kernel: Lustre: Skipped 443 previous similar messages Mar 20 18:38:06 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 1cfcc461-3b25-0b43-e3e5-7576cd1b4068 (at 10.9.102.54@o2ib4) in 222 seconds. I think it's dead, and I am evicting it. exp ffff986c4c7c9000, cur 1553132286 expire 1553132136 last 1553132064 Mar 20 18:38:06 fir-io1-s1 kernel: Lustre: Skipped 701 previous similar messages Mar 20 18:39:22 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 64524dae-4747-fc27-2842-d3399e815a00 (at 10.9.103.22@o2ib4) in 213 seconds. I think it's dead, and I am evicting it. exp ffff9847fae79c00, cur 1553132362 expire 1553132212 last 1553132149 Mar 20 18:39:22 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 18:40:38 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client c4fd2462-b91d-1c4f-eb1c-dfe6a873e91e (at 10.8.8.28@o2ib6) in 185 seconds. I think it's dead, and I am evicting it. exp ffff985762607400, cur 1553132438 expire 1553132288 last 1553132253 Mar 20 18:40:38 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 18:41:54 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 2f305ed3-b495-abbd-27d1-4d81849ef1d7 (at 10.9.104.65@o2ib4) in 166 seconds. I think it's dead, and I am evicting it. exp ffff98690126a000, cur 1553132514 expire 1553132364 last 1553132348 Mar 20 18:41:54 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 18:43:10 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 89e5fef3-4324-e047-528f-89b54771f78e (at 10.8.24.6@o2ib6) in 222 seconds. I think it's dead, and I am evicting it. exp ffff9848008f6000, cur 1553132590 expire 1553132440 last 1553132368 Mar 20 18:43:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 18:44:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 39b74f02-0c4c-fd51-e621-4bd6eb7173c0 (at 10.9.103.18@o2ib4) Mar 20 18:44:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 18:45:49 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 15fef58b-faf5-9755-47e9-038710ff38d5 (at 10.8.14.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851d495d000, cur 1553132749 expire 1553132599 last 1553132522 Mar 20 18:45:49 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 18:52:28 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 7d393a8e-e1f8-fcf7-03ee-821528b24da2 (at 10.9.104.62@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986ba3b46000, cur 1553133148 expire 1553132998 last 1553132921 Mar 20 18:52:28 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 20 18:57:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 861509dc-4523-55e8-f09a-16b6fca3f713 (at 10.9.107.51@o2ib4) Mar 20 18:57:32 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 20 19:05:55 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ba4bdd9b-8782-3afa-5abf-8f6e916672eb (at 10.9.107.67@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868ae400, cur 1553133955 expire 1553133805 last 1553133728 Mar 20 19:05:55 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 20 19:07:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 6999d791-63e6-9f77-9076-8032dae4068f (at 10.9.102.66@o2ib4) Mar 20 19:07:33 fir-io1-s1 kernel: Lustre: Skipped 587 previous similar messages Mar 20 19:16:34 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 018e5a85-e389-6baf-5044-f19cd70cb6dd (at 10.8.23.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852e4703000, cur 1553134594 expire 1553134444 last 1553134367 Mar 20 19:16:34 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 19:17:38 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 606475d1-7679-816f-4619-03e8971f8853 (at 10.8.8.32@o2ib6) Mar 20 19:17:38 fir-io1-s1 kernel: Lustre: Skipped 537 previous similar messages Mar 20 19:18:22 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:18:22 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 20 19:18:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:19:27 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:19:27 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 19:19:59 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:19:59 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 20 19:20:32 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:21:53 fir-io1-s1 kernel: Lustre: fir-OST0004: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:21:53 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 20 19:24:36 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:24:36 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 20 19:28:39 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 19:28:39 fir-io1-s1 kernel: Lustre: Skipped 459 previous similar messages Mar 20 19:30:40 fir-io1-s1 kernel: Lustre: fir-OST0006: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:30:40 fir-io1-s1 kernel: Lustre: Skipped 15 previous similar messages Mar 20 19:34:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f67fecb3-3bbf-12b1-aba1-62e99c3a174a (at 10.9.101.2@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986f6a65e400, cur 1553135693 expire 1553135543 last 1553135466 Mar 20 19:34:53 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 20 19:39:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 19:39:09 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Mar 20 19:39:51 fir-io1-s1 kernel: Lustre: fir-OST0004: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:39:51 fir-io1-s1 kernel: Lustre: Skipped 20 previous similar messages Mar 20 19:41:00 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0007_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:41:00 fir-io1-s1 kernel: LustreError: Skipped 4 previous similar messages Mar 20 19:42:15 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:46:10 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:46:10 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 20 19:46:11 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client f029c833-89d6-9b59-62cf-9daabfc0eac1 (at 10.8.20.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b764400, cur 1553136371 expire 1553136221 last 1553136144 Mar 20 19:46:11 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 19:47:00 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:47:00 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 20 19:48:14 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0007_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:48:14 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 20 19:49:15 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 26c5771f-6bff-b79c-941d-a328fa48123c (at 10.9.102.6@o2ib4) Mar 20 19:49:15 fir-io1-s1 kernel: Lustre: Skipped 45 previous similar messages Mar 20 19:49:25 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0007_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:49:25 fir-io1-s1 kernel: LustreError: Skipped 1 previous similar message Mar 20 19:50:03 fir-io1-s1 kernel: Lustre: fir-OST0006: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 19:50:03 fir-io1-s1 kernel: Lustre: Skipped 27 previous similar messages Mar 20 19:52:15 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:52:15 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 20 19:53:06 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 19:53:06 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 20 19:58:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 46dc7cdd-116d-c0c4-a9dc-76cd584455af (at 10.9.103.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9871058d4000, cur 1553137131 expire 1553136981 last 1553136904 Mar 20 19:58:51 fir-io1-s1 kernel: Lustre: Skipped 1283 previous similar messages Mar 20 19:59:48 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 19:59:48 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 20 20:00:32 fir-io1-s1 kernel: Lustre: fir-OST0004: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:00:32 fir-io1-s1 kernel: Lustre: Skipped 33 previous similar messages Mar 20 20:08:19 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0007_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:08:19 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 20 20:09:53 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 20:09:53 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 20 20:11:37 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:11:37 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 20 20:12:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:12:02 fir-io1-s1 kernel: Lustre: Skipped 37 previous similar messages Mar 20 20:12:26 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:12:26 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 20 20:17:02 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 7bb34ed3-94f6-1734-4915-37dc5bb7fb17 (at 10.8.20.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868ff000, cur 1553138222 expire 1553138072 last 1553137995 Mar 20 20:17:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 20:19:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 76e809db-e90a-7c07-20d4-e3130ed3be85 (at 10.9.104.30@o2ib4) Mar 20 20:19:59 fir-io1-s1 kernel: Lustre: Skipped 615 previous similar messages Mar 20 20:22:30 fir-io1-s1 kernel: Lustre: fir-OST0004: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:22:30 fir-io1-s1 kernel: Lustre: Skipped 19 previous similar messages Mar 20 20:24:47 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:24:47 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 20 20:25:38 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:25:38 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 20 20:30:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a43e43e4-04ca-0e28-cb02-64f3d87aa8c5 (at 10.8.24.10@o2ib6) Mar 20 20:30:00 fir-io1-s1 kernel: Lustre: Skipped 567 previous similar messages Mar 20 20:31:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 0cf05ced-8cb0-ae3f-8651-23e02cecf98b (at 10.8.17.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984811215c00, cur 1553139086 expire 1553138936 last 1553138859 Mar 20 20:31:26 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 20:34:04 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:34:04 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 20 20:34:29 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:34:29 fir-io1-s1 kernel: Lustre: fir-OST0006: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:34:29 fir-io1-s1 kernel: Lustre: fir-OST000a: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:34:29 fir-io1-s1 kernel: Lustre: Skipped 31 previous similar messages Mar 20 20:34:29 fir-io1-s1 kernel: Lustre: Skipped 31 previous similar messages Mar 20 20:34:29 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 20 20:34:54 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0005_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:34:54 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 20 20:40:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b30edc85-7a93-af90-f7e4-3b5a24d9d571 (at 10.8.23.15@o2ib6) Mar 20 20:40:34 fir-io1-s1 kernel: Lustre: Skipped 189 previous similar messages Mar 20 20:41:02 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:41:02 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 20 20:43:18 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:43:18 fir-io1-s1 kernel: LustreError: Skipped 9 previous similar messages Mar 20 20:44:48 fir-io1-s1 kernel: Lustre: fir-OST0006: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:44:48 fir-io1-s1 kernel: Lustre: Skipped 31 previous similar messages Mar 20 20:46:04 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0007_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:46:04 fir-io1-s1 kernel: LustreError: Skipped 7 previous similar messages Mar 20 20:50:35 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 20:50:35 fir-io1-s1 kernel: Lustre: Skipped 62 previous similar messages Mar 20 20:52:16 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 20:52:16 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 20 20:52:21 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client d8bce502-6501-f158-4236-16c870cc9a87 (at 10.8.25.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800fda800, cur 1553140341 expire 1553140191 last 1553140114 Mar 20 20:52:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 20:53:37 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 2452f36c-2e1c-a070-489d-7b1a32e90bc3 (at 10.8.11.30@o2ib6) in 152 seconds. I think it's dead, and I am evicting it. exp ffff987068889000, cur 1553140417 expire 1553140267 last 1553140265 Mar 20 20:53:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 20:54:57 fir-io1-s1 kernel: Lustre: fir-OST0004: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 20:54:57 fir-io1-s1 kernel: Lustre: Skipped 31 previous similar messages Mar 20 20:57:48 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 75f0b783-9da7-86f4-3ab0-04cc14c1b585 (at 10.8.12.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984804f55400, cur 1553140668 expire 1553140518 last 1553140441 Mar 20 20:57:48 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 21:00:47 fir-io1-s1 kernel: Lustre: fir-OST0006: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 21:00:47 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 20 21:03:27 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 21:03:27 fir-io1-s1 kernel: LustreError: Skipped 15 previous similar messages Mar 20 21:05:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 21:05:14 fir-io1-s1 kernel: Lustre: Skipped 39 previous similar messages Mar 20 21:05:17 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 971e0808-7ac1-c6bb-96ea-79297b6f29be (at 10.8.11.22@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857618e7800, cur 1553141117 expire 1553140967 last 1553140890 Mar 20 21:05:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 21:12:48 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 21:12:48 fir-io1-s1 kernel: Lustre: Skipped 30 previous similar messages Mar 20 21:17:20 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client eb28cb4e-51ef-c06d-f3ae-32f97f51023b (at 10.8.23.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811fa800, cur 1553141840 expire 1553141690 last 1553141613 Mar 20 21:17:20 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 21:21:17 fir-io1-s1 kernel: Lustre: fir-OST0008: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 21:21:17 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 20 21:23:03 fir-io1-s1 kernel: Lustre: fir-OST0002: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 21:23:03 fir-io1-s1 kernel: Lustre: Skipped 36 previous similar messages Mar 20 21:25:34 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 21:25:34 fir-io1-s1 kernel: LustreError: Skipped 26 previous similar messages Mar 20 21:28:54 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 21:28:54 fir-io1-s1 kernel: LustreError: Skipped 5 previous similar messages Mar 20 21:29:35 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 8a1ea84a-46e6-aa4e-06a4-29413ad1db8e (at 10.8.11.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868ff000, cur 1553142575 expire 1553142425 last 1553142348 Mar 20 21:29:35 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 21:31:49 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 21:31:49 fir-io1-s1 kernel: LustreError: Skipped 9 previous similar messages Mar 20 21:32:14 fir-io1-s1 kernel: Lustre: fir-OST0000: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 21:32:14 fir-io1-s1 kernel: Lustre: Skipped 34 previous similar messages Mar 20 21:35:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 21:35:18 fir-io1-s1 kernel: Lustre: Skipped 51 previous similar messages Mar 20 21:41:53 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 21:41:53 fir-io1-s1 kernel: LustreError: Skipped 11 previous similar messages Mar 20 21:43:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 21:43:36 fir-io1-s1 kernel: Lustre: Skipped 10 previous similar messages Mar 20 21:44:13 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4a4a3f59-01c7-507f-93d1-ac7d8d922b93 (at 10.8.11.35@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009ff800, cur 1553143453 expire 1553143303 last 1553143226 Mar 20 21:44:13 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 20 21:45:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 21:45:54 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 21:53:39 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 21:53:39 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 21:58:32 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 37cc079a-c37c-cf86-ab23-66b4c360eeae (at 10.8.22.4@o2ib6) Mar 20 21:58:32 fir-io1-s1 kernel: Lustre: Skipped 54 previous similar messages Mar 20 21:58:47 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e6bbf2ae-f22b-03ba-227e-75512924cd65 (at 10.9.103.35@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d7de57000, cur 1553144327 expire 1553144177 last 1553144100 Mar 20 21:58:47 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 22:00:49 fir-io1-s1 kernel: Lustre: 110621:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144442/real 1553144442] req@ffff9875d6b76600 x1625501533223904/t0(0) o106->fir-OST000a@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144449 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 20 22:00:49 fir-io1-s1 kernel: Lustre: 49830:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144442/real 1553144442] req@ffff983dddffe900 x1625501533223888/t0(0) o106->fir-OST0008@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144449 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 20 22:00:49 fir-io1-s1 kernel: Lustre: 110035:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144442/real 1553144442] req@ffff9875d6b73900 x1625501533223872/t0(0) o106->fir-OST0006@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144449 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 20 22:00:49 fir-io1-s1 kernel: Lustre: 49830:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 28 previous similar messages Mar 20 22:00:49 fir-io1-s1 kernel: Lustre: 110035:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 28 previous similar messages Mar 20 22:00:53 fir-io1-s1 kernel: Lustre: 110636:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144446/real 1553144446] req@ffff986327651800 x1625501534219808/t0(0) o106->fir-OST0008@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144453 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 20 22:00:53 fir-io1-s1 kernel: Lustre: 110636:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 20 22:01:00 fir-io1-s1 kernel: Lustre: 49832:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144453/real 1553144453] req@ffff983dddffa100 x1625501535671888/t0(0) o106->fir-OST0006@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144460 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 20 22:01:00 fir-io1-s1 kernel: Lustre: 49832:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Mar 20 22:01:10 fir-io1-s1 kernel: Lustre: 49830:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144463/real 1553144463] req@ffff983dddffe900 x1625501533223888/t0(0) o106->fir-OST0008@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144470 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 22:01:10 fir-io1-s1 kernel: Lustre: 49830:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 33 previous similar messages Mar 20 22:01:28 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144481/real 1553144481] req@ffff985f17985100 x1625501534219792/t0(0) o106->fir-OST0006@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144488 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 22:01:28 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 67 previous similar messages Mar 20 22:01:32 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 22:01:32 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 20 22:02:06 fir-io1-s1 kernel: Lustre: 96507:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144519/real 1553144519] req@ffff986a239ff500 x1625501534748000/t0(0) o106->fir-OST0006@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144526 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 22:02:06 fir-io1-s1 kernel: Lustre: 96507:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 181 previous similar messages Mar 20 22:03:21 fir-io1-s1 kernel: Lustre: 49822:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553144594/real 1553144594] req@ffff98380541f200 x1625501534360464/t0(0) o106->fir-OST0008@10.8.4.26@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553144601 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 20 22:03:21 fir-io1-s1 kernel: Lustre: 49822:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 414 previous similar messages Mar 20 22:08:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7589c81e-2d34-67a0-8460-0dfbb24f93e8 (at 10.8.18.6@o2ib6) Mar 20 22:08:33 fir-io1-s1 kernel: Lustre: Skipped 126 previous similar messages Mar 20 22:09:35 fir-io1-s1 kernel: Lustre: fir-OST0004: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 22:09:35 fir-io1-s1 kernel: Lustre: Skipped 13 previous similar messages Mar 20 22:15:52 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client e12bee3e-89e8-03cd-3abb-63cdc0c915d6 (at 10.9.106.7@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9869b5592000, cur 1553145352 expire 1553145202 last 1553145125 Mar 20 22:15:52 fir-io1-s1 kernel: Lustre: Skipped 107 previous similar messages Mar 20 22:19:03 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8a515cc0-b236-b863-3928-4f88b7b0bba9 (at 10.8.21.6@o2ib6) Mar 20 22:19:03 fir-io1-s1 kernel: Lustre: Skipped 841 previous similar messages Mar 20 22:19:28 fir-io1-s1 kernel: LustreError: 137-5: fir-OST0005_UUID: not available for connect from 10.8.21.6@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. Mar 20 22:19:28 fir-io1-s1 kernel: LustreError: Skipped 2 previous similar messages Mar 20 22:20:05 fir-io1-s1 kernel: Lustre: fir-OST0002: Client a311e923-fdd0-7b46-b973-73d57b609bed (at 10.8.21.6@o2ib6) reconnecting Mar 20 22:20:05 fir-io1-s1 kernel: Lustre: Skipped 32 previous similar messages Mar 20 22:30:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5aed28dc-3466-bb16-a128-8f738b650102 (at 10.8.27.21@o2ib6) Mar 20 22:30:02 fir-io1-s1 kernel: Lustre: Skipped 40 previous similar messages Mar 20 22:31:00 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 25f0beb6-3c52-1d8e-a3ed-cb69a6f6eb23 (at 10.8.24.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576184ec00, cur 1553146260 expire 1553146110 last 1553146033 Mar 20 22:31:00 fir-io1-s1 kernel: Lustre: Skipped 251 previous similar messages Mar 20 22:40:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 97c8e837-ba4b-d84f-94e1-60778fc028be (at 10.8.8.37@o2ib6) Mar 20 22:40:11 fir-io1-s1 kernel: Lustre: Skipped 688 previous similar messages Mar 20 22:45:49 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 18eef1f0-9182-60c7-3399-b4a6b05b8cf3 (at 10.8.13.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867825fcc00, cur 1553147149 expire 1553146999 last 1553146922 Mar 20 22:45:49 fir-io1-s1 kernel: Lustre: Skipped 473 previous similar messages Mar 20 23:00:55 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2ae35386-7dbf-e9e4-77f5-0f6e19b82987 (at 10.8.30.25@o2ib6) Mar 20 23:00:55 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 23:01:10 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 199061ed-6dad-57cb-6c9d-5e64db5ef6e4 (at 10.8.20.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785e1fc00, cur 1553148070 expire 1553147920 last 1553147843 Mar 20 23:01:10 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 20 23:02:52 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f2609dba-8f86-a71a-181c-0f3c1a31c50b (at 10.9.104.70@o2ib4) Mar 20 23:02:52 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 20 23:05:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to dbfdab11-8ec7-41b0-1485-dd07c4a504bd (at 10.9.104.47@o2ib4) Mar 20 23:05:23 fir-io1-s1 kernel: Lustre: Skipped 83 previous similar messages Mar 20 23:13:22 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 9e42425c-3999-3161-1632-bbd8f8c557c4 (at 10.8.14.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583bbae800, cur 1553148802 expire 1553148652 last 1553148575 Mar 20 23:13:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 23:14:17 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.13.28@o2ib6) Mar 20 23:14:17 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 20 23:29:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4abc7b2d-5f6a-da5f-d550-baa3bf9eb296 (at 10.8.20.8@o2ib6) Mar 20 23:29:13 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 20 23:30:07 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client ed53be0e-9561-14a1-7cd8-58b52e16307e (at 10.8.26.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984f5e9d4800, cur 1553149807 expire 1553149657 last 1553149580 Mar 20 23:30:07 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 23:41:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7751f34c-206a-edf3-4c97-12b3ac7dcb0e (at 10.8.23.14@o2ib6) Mar 20 23:41:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 20 23:47:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client a534713f-8e81-6ffa-dfd7-154b5d486b6c (at 10.9.104.70@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480315f400, cur 1553150846 expire 1553150696 last 1553150619 Mar 20 23:47:26 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 20 23:54:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8f273097-d34e-d604-e427-2da4f99ca32a (at 10.9.106.26@o2ib4) Mar 20 23:54:44 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 20 23:58:07 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client c03bbd7c-e2c3-746c-2e4a-f57d35040b05 (at 10.9.106.10@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9863d7957c00, cur 1553151487 expire 1553151337 last 1553151260 Mar 20 23:58:07 fir-io1-s1 kernel: Lustre: Skipped 98 previous similar messages Mar 21 00:14:21 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client 9e1bf5a9-e176-047c-f31a-cc2226ce6141 (at 10.8.22.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986c39bc2800, cur 1553152461 expire 1553152311 last 1553152234 Mar 21 00:14:21 fir-io1-s1 kernel: Lustre: Skipped 20 previous similar messages Mar 21 00:19:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e3d7d750-2a26-e81f-1160-2f3ee9d7f849 (at 10.9.106.10@o2ib4) Mar 21 00:19:31 fir-io1-s1 kernel: Lustre: Skipped 113 previous similar messages Mar 21 00:24:26 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 23f9a8de-c6a0-76ea-9378-57e47966c02c (at 10.8.21.23@o2ib6) Mar 21 00:24:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 00:25:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 01444670-f696-61f8-66b1-612465f02e7c (at 10.8.17.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848947a5000, cur 1553153144 expire 1553152994 last 1553152917 Mar 21 00:25:44 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 21 00:27:59 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a82d0417-5354-7090-8c74-27f558bf90cb (at 10.9.103.27@o2ib4) Mar 21 00:27:59 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 21 00:34:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ab5bcaa-8d5e-27d9-5913-f9d8f76ca855 (at 10.8.11.17@o2ib6) Mar 21 00:34:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 00:54:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4cdb4c1f-c631-73f5-0cc6-576f1959d1eb (at 10.8.17.8@o2ib6) Mar 21 00:54:04 fir-io1-s1 kernel: Lustre: Skipped 23 previous similar messages Mar 21 00:58:13 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client dd0fb95a-abf4-abdc-4c4d-6b9f333cc5be (at 10.8.26.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986c10229c00, cur 1553155093 expire 1553154943 last 1553154866 Mar 21 00:58:13 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 21 01:26:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 43db6f02-39b1-2d91-8a5a-c0c4a16a2f08 (at 10.8.26.12@o2ib6) Mar 21 01:26:31 fir-io1-s1 kernel: Lustre: Skipped 14 previous similar messages Mar 21 01:28:36 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client db748cb7-b21a-0e70-4beb-5b2e23eaafa7 (at 10.9.103.23@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98727caa5c00, cur 1553156916 expire 1553156766 last 1553156689 Mar 21 01:28:36 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 01:31:35 fir-io1-s1 kernel: LustreError: 73277:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2760704(3809280) req@ffff984833e2dc50 x1628590553288064/t0(0) o4->e9614b82-773c-e706-5a4a-cdf089441a3d@10.9.102.35@o2ib4:548/0 lens 600/472 e 3 to 0 dl 1553157103 ref 1 fl Interpret:/0/0 rc 0/0 Mar 21 01:31:35 fir-io1-s1 kernel: Lustre: fir-OST0004: Bulk IO write error with e9614b82-773c-e706-5a4a-cdf089441a3d (at 10.9.102.35@o2ib4), client will retry: rc = -110 Mar 21 01:31:35 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 21 01:31:35 fir-io1-s1 kernel: LustreError: 73277:0:(sec.c:2362:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages Mar 21 01:31:44 fir-io1-s1 kernel: Lustre: fir-OST0004: Client e9614b82-773c-e706-5a4a-cdf089441a3d (at 10.9.102.35@o2ib4) reconnecting Mar 21 01:31:44 fir-io1-s1 kernel: Lustre: Skipped 18 previous similar messages Mar 21 01:31:44 fir-io1-s1 kernel: Lustre: fir-OST0004: Connection restored to 9bd541c8-5e18-2470-8262-fd1a455e43c1 (at 10.9.102.35@o2ib4) Mar 21 01:31:44 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 21 01:49:37 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client e017e038-5601-9bb0-415b-7e187b32b720 (at 10.8.18.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f5400, cur 1553158177 expire 1553158027 last 1553157950 Mar 21 01:49:37 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 01:56:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2e6063a2-ca9b-ae5d-abce-e3daf4d673e4 (at 10.9.103.23@o2ib4) Mar 21 01:56:09 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 21 02:01:54 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client d4da1cd5-c954-5248-1daa-725453bd3004 (at 10.9.103.25@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9873165abc00, cur 1553158914 expire 1553158764 last 1553158687 Mar 21 02:01:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:13:06 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 41f50ccb-339b-d2d0-b563-663f958c318b (at 10.8.31.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786925000, cur 1553159586 expire 1553159436 last 1553159359 Mar 21 02:13:06 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:17:11 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 148ae75a-e083-e480-3765-d63daa0c5525 (at 10.9.108.55@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985762747000, cur 1553159831 expire 1553159681 last 1553159604 Mar 21 02:17:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:18:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 531debd8-6dde-8101-c0c5-b86120a894b1 (at 10.8.18.20@o2ib6) Mar 21 02:18:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:22:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 39f91ffc-f969-2d2d-6e9d-ce4a2e82af12 (at 10.9.103.25@o2ib4) Mar 21 02:22:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:24:18 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client d6a0e0dd-e328-9d2e-1120-aecdb18da00a (at 10.9.104.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867814d8800, cur 1553160258 expire 1553160108 last 1553160031 Mar 21 02:24:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:26:53 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c820b3c4-0169-94a4-482e-1b5c7213329f (at 10.9.114.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677dc5b800, cur 1553160413 expire 1553160263 last 1553160186 Mar 21 02:26:53 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 02:28:09 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 3d1eb8fd-032d-6924-97a8-a976e84a0afd (at 10.8.26.27@o2ib6) in 221 seconds. I think it's dead, and I am evicting it. exp ffff98583bb58800, cur 1553160489 expire 1553160339 last 1553160268 Mar 21 02:28:09 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 02:29:25 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d4a39c37-bddc-65c8-8277-ae48b63287f9 (at 10.8.26.17@o2ib6) in 207 seconds. I think it's dead, and I am evicting it. exp ffff984a83a64c00, cur 1553160565 expire 1553160415 last 1553160358 Mar 21 02:29:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:30:41 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 0a950a0b-46a0-6038-f802-3eabcd4b5f39 (at 10.9.104.11@o2ib4) in 156 seconds. I think it's dead, and I am evicting it. exp ffff9847fe7c4800, cur 1553160641 expire 1553160491 last 1553160485 Mar 21 02:30:41 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:31:57 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 7263c1ce-0e70-2ff4-422a-ec88f2558e2d (at 10.8.26.15@o2ib6) in 184 seconds. I think it's dead, and I am evicting it. exp ffff986784e69000, cur 1553160717 expire 1553160567 last 1553160533 Mar 21 02:31:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:36:29 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 5351f4fd-8176-f034-3392-548be352e5b0 (at 10.8.27.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984e7da28800, cur 1553160989 expire 1553160839 last 1553160762 Mar 21 02:36:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:41:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 8ba1b18f-72db-d059-bab7-d3e649c57271 (at 10.9.108.55@o2ib4) Mar 21 02:41:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:41:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to edc5d3f2-dae7-69fe-e7fb-9c4cf59a4b4c (at 10.8.31.8@o2ib6) Mar 21 02:41:11 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:44:45 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 52e7d481-4b33-918f-098b-03613e3d4b0f (at 10.9.102.13@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984818e9cc00, cur 1553161485 expire 1553161335 last 1553161258 Mar 21 02:44:45 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:46:34 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to ab905b54-126d-719c-68ac-791f0a17153b (at 10.9.108.52@o2ib4) Mar 21 02:46:34 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:49:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fb273c41-c272-402d-98b5-3e5f91dba50e (at 10.9.114.15@o2ib4) Mar 21 02:49:21 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:50:01 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 920ceaaa-cffa-e736-eb6e-7719488af49d (at 10.9.104.54@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786884800, cur 1553161801 expire 1553161651 last 1553161574 Mar 21 02:50:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:50:13 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b7eb786c-4da9-9431-a597-6f5f4ba4c9ed (at 10.9.114.12@o2ib4) Mar 21 02:50:13 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 02:52:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3c603ef7-b311-5012-38a9-d1ff9ba9b526 (at 10.9.104.13@o2ib4) Mar 21 02:52:21 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 21 02:56:51 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2d8e1d81-8c01-f081-9434-1558c3a99426 (at 10.8.26.27@o2ib6) Mar 21 02:56:51 fir-io1-s1 kernel: Lustre: Skipped 6 previous similar messages Mar 21 02:57:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 824f0587-167c-fe30-e5f5-4a8a8b3eb359 (at 10.8.26.17@o2ib6) Mar 21 02:57:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 03:00:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to eb4e1d20-3a1f-d68d-546e-5c1cf1ecb74b (at 10.9.104.11@o2ib4) Mar 21 03:00:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 03:04:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 7122cf14-0523-fe12-768f-cd0ed99220da (at 10.8.27.11@o2ib6) Mar 21 03:04:21 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 03:14:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4600638c-d686-27e8-2646-fd49b60d2ae1 (at 10.9.102.13@o2ib4) Mar 21 03:14:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 03:27:02 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to c3afa774-0e39-3459-0af2-ddf264214c5b (at 10.9.104.54@o2ib4) Mar 21 03:27:02 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 03:44:17 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client c2a2ee23-08dd-29c9-da2a-ec047e61fec1 (at 10.9.104.59@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758838000, cur 1553165057 expire 1553164907 last 1553164830 Mar 21 03:44:17 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 03:55:40 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 92b61c05-3a95-81e4-be4c-d266657c7214 (at 10.9.104.60@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984818e9ec00, cur 1553165740 expire 1553165590 last 1553165513 Mar 21 03:55:40 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 04:04:42 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 73c371ae-4e8a-4c83-10e9-0bb8bc3c3687 (at 10.8.24.29@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98384f065c00, cur 1553166282 expire 1553166132 last 1553166055 Mar 21 04:04:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 04:21:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 56f157dc-9f05-a349-697a-ac16ba31313e (at 10.9.104.59@o2ib4) Mar 21 04:21:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 04:32:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3a285969-e764-0f86-6def-3c9abf088372 (at 10.8.24.29@o2ib6) Mar 21 04:32:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 04:33:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.104.60@o2ib4) Mar 21 04:33:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 04:39:39 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 4a2f628b-1f30-149a-2690-bf94c1ce3493 (at 10.8.20.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f2400, cur 1553168379 expire 1553168229 last 1553168152 Mar 21 04:39:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:06:54 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 32abea4a-7005-2965-cd85-5d6ade53f88d (at 10.9.104.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677b616c00, cur 1553170014 expire 1553169864 last 1553169787 Mar 21 05:06:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:07:50 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f6f2dc0e-bc9f-2120-e971-29d8049b1247 (at 10.8.20.12@o2ib6) Mar 21 05:07:50 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:16:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client fd4445fa-c245-6ae0-d0c8-841c2e52a80d (at 10.9.104.61@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838a9000, cur 1553170611 expire 1553170461 last 1553170384 Mar 21 05:16:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:21:28 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 11f418ca-bf11-a6b4-8d8a-9b285f03259c (at 10.8.25.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867868ac800, cur 1553170888 expire 1553170738 last 1553170661 Mar 21 05:21:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:22:44 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4daa05a1-3fa7-1bdc-64a4-00c4c6f8e29b (at 10.8.22.22@o2ib6) in 188 seconds. I think it's dead, and I am evicting it. exp ffff98677d507000, cur 1553170964 expire 1553170814 last 1553170776 Mar 21 05:22:44 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:24:00 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4edbb0a6-4cf3-90e1-8ff5-dc388586ec7b (at 10.8.22.18@o2ib6) in 208 seconds. I think it's dead, and I am evicting it. exp ffff9864523ba400, cur 1553171040 expire 1553170890 last 1553170832 Mar 21 05:24:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:26:51 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client c43cb276-6467-e460-1d7e-ee5e11399161 (at 10.9.104.65@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857606b5400, cur 1553171211 expire 1553171061 last 1553170984 Mar 21 05:26:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:43:29 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 00cd381d-5246-e5cd-af5e-792229d3fea2 (at 10.9.104.63@o2ib4) Mar 21 05:43:29 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:49:28 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 96dc3f28-35a7-d1a0-d554-ac4259066293 (at 10.8.25.15@o2ib6) Mar 21 05:49:28 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:51:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 9d602dc8-8051-8982-51eb-1bb4250b93cd (at 10.8.22.18@o2ib6) Mar 21 05:51:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:51:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 16fe1f06-91d4-6364-b5d9-1d6caad6f915 (at 10.8.22.22@o2ib6) Mar 21 05:51:18 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:53:23 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client f218ebe2-3065-61c8-f640-d41bc7d4ee94 (at 10.9.106.63@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d36a800, cur 1553172803 expire 1553172653 last 1553172576 Mar 21 05:53:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 05:54:16 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 74adb3c7-1310-d82a-c0b0-2c64f425de3b (at 10.9.104.61@o2ib4) Mar 21 05:54:16 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:04:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e0d0d81e-4164-f236-7d8a-b466fb3eea50 (at 10.9.104.65@o2ib4) Mar 21 06:04:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:07:04 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2d6dda5f-e20d-88c9-f714-74dd0fa20536 (at 10.8.24.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c671800, cur 1553173624 expire 1553173474 last 1553173397 Mar 21 06:07:04 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:08:20 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 056e33f2-7bec-ce74-8ca2-743799013531 (at 10.8.12.3@o2ib6) in 224 seconds. I think it's dead, and I am evicting it. exp ffff9847fd263c00, cur 1553173700 expire 1553173550 last 1553173476 Mar 21 06:08:20 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 06:09:36 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 55854ce0-15d0-3581-cfe3-79fd0081b026 (at 10.8.23.13@o2ib6) in 214 seconds. I think it's dead, and I am evicting it. exp ffff986f06878800, cur 1553173776 expire 1553173626 last 1553173562 Mar 21 06:09:36 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 06:14:25 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 63b89084-a198-377b-0bdb-5ed4c4e0cd41 (at 10.9.106.63@o2ib4) Mar 21 06:14:25 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:36:08 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0aaa3f75-926a-3a00-e300-1693464069e6 (at 10.8.24.36@o2ib6) Mar 21 06:36:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:36:56 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 70e66a5b-5f5d-f3d1-390c-84f46bd302af (at 10.8.25.32@o2ib6) Mar 21 06:36:56 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:37:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.8.23.13@o2ib6) Mar 21 06:37:54 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:37:57 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1c866079-2b52-1a82-44ba-82659851888e (at 10.8.23.30@o2ib6) Mar 21 06:37:57 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:39:22 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 576a6a50-b74b-7be9-c975-c468ccc99865 (at 10.8.12.3@o2ib6) Mar 21 06:39:22 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:39:42 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to cd77514a-c92d-dc94-95c2-44bf91c41d35 (at 10.8.11.32@o2ib6) Mar 21 06:39:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 06:53:11 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client e53cd0cd-f24d-7c08-b3f9-5cdb350e9bd3 (at 10.9.101.44@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d0c00, cur 1553176391 expire 1553176241 last 1553176164 Mar 21 06:53:11 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 07:08:23 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client d504b511-2ad2-592a-26d7-5aca7d3a1784 (at 10.9.103.31@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9873ff216400, cur 1553177303 expire 1553177153 last 1553177076 Mar 21 07:08:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 07:21:00 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to (at 10.9.101.44@o2ib4) Mar 21 07:21:00 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 07:26:26 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 919e44d9-c353-9398-ff6e-e686647448f9 (at 10.8.31.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef11400, cur 1553178386 expire 1553178236 last 1553178159 Mar 21 07:26:26 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 07:30:05 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to a3c57bef-a739-0ea9-6582-283914517ba2 (at 10.9.103.31@o2ib4) Mar 21 07:30:05 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 07:31:12 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 3ee1d828-ecde-1b2b-d364-76f45ea23c4f (at 10.8.23.11@o2ib6) Mar 21 07:31:12 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 07:32:13 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client bfb10931-62c8-4698-42a5-cff142f62395 (at 10.8.23.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985f0e4c1800, cur 1553178733 expire 1553178583 last 1553178506 Mar 21 07:32:13 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 21 07:35:18 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2e6063a2-ca9b-ae5d-abce-e3daf4d673e4 (at 10.9.103.23@o2ib4) Mar 21 07:35:18 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 07:35:49 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 4600638c-d686-27e8-2646-fd49b60d2ae1 (at 10.9.102.13@o2ib4) Mar 21 07:35:49 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 07:37:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 75dcc589-f93a-4a55-56e0-316e7a2edbec (at 10.8.23.18@o2ib6) Mar 21 07:37:24 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 21 07:41:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 14aae05e-d3ff-54ad-8b93-c5dd42954ce5 (at 10.8.23.31@o2ib6) Mar 21 07:41:01 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 21 08:00:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 62a94e57-5eb9-1e28-4a37-9d6b953b2a83 (at 10.8.23.26@o2ib6) Mar 21 08:00:11 fir-io1-s1 kernel: Lustre: Skipped 149 previous similar messages Mar 21 08:01:11 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553180464/real 1553180464] req@ffff98381f25f800 x1625507837600208/t0(0) o104->fir-OST0004@10.9.107.12@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1553180471 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 21 08:01:11 fir-io1-s1 kernel: Lustre: 110701:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 240 previous similar messages Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 36981:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553180485/real 1553180485] req@ffff98384ea71e00 x1625507837600112/t0(0) o104->fir-OST0002@10.9.107.12@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1553180492 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553180485/real 1553180485] req@ffff98381f25c800 x1625507837600096/t0(0) o104->fir-OST0008@10.9.107.12@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1553180492 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553180485/real 1553180485] req@ffff98381cdf8000 x1625507837600192/t0(0) o104->fir-OST0006@10.9.107.12@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1553180492 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553180485/real 1553180485] req@ffff986a239fc500 x1625507837600160/t0(0) o104->fir-OST000a@10.9.107.12@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1553180492 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 96933:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 21 08:01:32 fir-io1-s1 kernel: Lustre: 36981:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Mar 21 08:02:14 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553180527/real 1553180527] req@ffff986a239fc500 x1625507837600160/t0(0) o104->fir-OST000a@10.9.107.12@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1553180534 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 08:02:14 fir-io1-s1 kernel: Lustre: 96561:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 30 previous similar messages Mar 21 08:03:31 fir-io1-s1 kernel: Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553180604/real 1553180604] req@ffff98381cdf8000 x1625507837600192/t0(0) o104->fir-OST0006@10.9.107.12@o2ib4:15/16 lens 296/224 e 0 to 1 dl 1553180611 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 08:03:31 fir-io1-s1 kernel: Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 64 previous similar messages Mar 21 08:03:38 fir-io1-s1 kernel: LustreError: 96933:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.107.12@o2ib4) failed to reply to blocking AST (req@ffff98381f25c800 x1625507837600096 status 0 rc -110), evict it ns: filter-fir-OST0008_UUID lock: ffff984951749440/0x49e1862f7c9a7537 lrc: 4/0,0 mode: PR/PR res: [0x1e3aec:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400010020 nid: 10.9.107.12@o2ib4 remote: 0xd936939a385e0cb2 expref: 5 pid: 96405 timeout: 3528512 lvb_type: 1 Mar 21 08:03:38 fir-io1-s1 kernel: LustreError: 74749:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.9.107.12@o2ib4) failed to reply to blocking AST (req@ffff98381f25ec00 x1625507837600256 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff983e0474f980/0x49e1862f7c9a7f40 lrc: 4/0,0 mode: PR/PR res: [0x1e38ce:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400010020 nid: 10.9.107.12@o2ib4 remote: 0xd936939a385e0da7 expref: 5 pid: 49823 timeout: 3528512 lvb_type: 1 Mar 21 08:03:38 fir-io1-s1 kernel: LustreError: 138-a: fir-OST0000: A client on nid 10.9.107.12@o2ib4 was evicted due to a lock blocking callback time out: rc -110 Mar 21 08:03:38 fir-io1-s1 kernel: LustreError: Skipped 3 previous similar messages Mar 21 08:03:38 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.9.107.12@o2ib4 ns: filter-fir-OST0000_UUID lock: ffff983e0474f980/0x49e1862f7c9a7f40 lrc: 3/0,0 mode: PR/PR res: [0x1e38ce:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 17179869184->18446744073709551615) flags: 0x60000400010020 nid: 10.9.107.12@o2ib4 remote: 0xd936939a385e0da7 expref: 6 pid: 49823 timeout: 0 lvb_type: 1 Mar 21 08:03:38 fir-io1-s1 kernel: LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 3 previous similar messages Mar 21 08:03:38 fir-io1-s1 kernel: LustreError: 96933:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 4 previous similar messages Mar 21 08:04:00 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 10c87d86-61ec-1f6e-9cd4-c0064d4bc875 (at 10.8.8.10@o2ib6) in 189 seconds. I think it's dead, and I am evicting it. exp ffff9868147b7000, cur 1553180640 expire 1553180490 last 1553180451 Mar 21 08:04:00 fir-io1-s1 kernel: Lustre: Skipped 179 previous similar messages Mar 21 08:04:01 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 10c87d86-61ec-1f6e-9cd4-c0064d4bc875 (at 10.8.8.10@o2ib6) in 190 seconds. I think it's dead, and I am evicting it. exp ffff986785cd9800, cur 1553180641 expire 1553180491 last 1553180451 Mar 21 08:04:01 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 21 08:06:09 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 722fc89e-4fb6-0519-04d0-92f8091a9aa0 (at 10.8.14.5@o2ib6) Mar 21 08:06:09 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 08:08:44 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 37fbf52b-acd3-2039-1295-2f5becb5bcd2 (at 10.9.101.44@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848807f6c00, cur 1553180924 expire 1553180774 last 1553180697 Mar 21 08:08:44 fir-io1-s1 kernel: Lustre: Skipped 2 previous similar messages Mar 21 08:15:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 22ee14eb-5a96-ad04-6e5f-188b7aec897d (at 10.8.12.33@o2ib6) Mar 21 08:15:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 08:16:30 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b53c44e1-5903-3921-0e46-dadcbed5aa59 (at 10.8.9.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677f002400, cur 1553181390 expire 1553181240 last 1553181163 Mar 21 08:16:30 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 08:16:54 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 2a988819-ab2a-bc15-49a4-e9fcfd75b5c8 (at 10.8.6.26@o2ib6) Mar 21 08:16:54 fir-io1-s1 kernel: Lustre: Skipped 53 previous similar messages Mar 21 08:19:21 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 49825401-fbcc-183b-03d9-d3fc1f8e75ef (at 10.8.6.7@o2ib6) Mar 21 08:19:21 fir-io1-s1 kernel: Lustre: Skipped 28 previous similar messages Mar 21 08:24:53 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b2364f4b-9129-81e8-7e2f-15aa4210b663 (at 10.9.107.12@o2ib4) Mar 21 08:24:53 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 21 08:30:51 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 2a91d180-a701-77df-0cf5-1e7f94e37f24 (at 10.9.103.15@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834456800, cur 1553182251 expire 1553182101 last 1553182024 Mar 21 08:30:51 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 08:34:04 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 5fda523c-9891-cc11-7a0e-50a252e6fb83 (at 10.9.103.8@o2ib4) Mar 21 08:34:04 fir-io1-s1 kernel: Lustre: Skipped 4 previous similar messages Mar 21 08:40:08 fir-io1-s1 kernel: Lustre: fir-OST000a: haven't heard from client 87f2008c-e312-2bfc-1941-e2a854a2dd3d (at 10.8.11.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983821036c00, cur 1553182808 expire 1553182658 last 1553182581 Mar 21 08:40:08 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 08:59:44 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 879d94a8-a845-ea21-f6e8-a2d093701c88 (at 10.9.103.15@o2ib4) Mar 21 08:59:44 fir-io1-s1 kernel: Lustre: Skipped 30 previous similar messages Mar 21 09:07:04 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client b22b8bcc-4d2f-9121-bcc7-f2775c7e8547 (at 10.8.24.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986816f3dc00, cur 1553184424 expire 1553184274 last 1553184197 Mar 21 09:07:04 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 09:07:23 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 722fc89e-4fb6-0519-04d0-92f8091a9aa0 (at 10.8.14.5@o2ib6) Mar 21 09:07:23 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 09:11:39 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0f8c1926-cb87-db8f-9eb9-d0323b54c0f6 (at 10.8.11.20@o2ib6) Mar 21 09:11:39 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 09:20:09 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client d9b63d00-c5f5-abd8-cb92-9c6858b85bc1 (at 10.9.108.46@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98654faab400, cur 1553185209 expire 1553185059 last 1553184982 Mar 21 09:20:09 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 21 09:30:26 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client b34c5fb9-955f-0dcc-e5da-a55da03747b2 (at 10.8.24.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758838800, cur 1553185826 expire 1553185676 last 1553185599 Mar 21 09:30:26 fir-io1-s1 kernel: Lustre: Skipped 47 previous similar messages Mar 21 09:31:42 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 5d1d168d-d026-ae89-cd06-3f011f74000e (at 10.9.105.47@o2ib4) in 216 seconds. I think it's dead, and I am evicting it. exp ffff9849c8262400, cur 1553185902 expire 1553185752 last 1553185686 Mar 21 09:31:42 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 09:34:20 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to e5be9ff2-873f-0542-6c5f-13af50413057 (at 10.8.30.1@o2ib6) Mar 21 09:34:20 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 09:35:01 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 0dbe37ca-2471-ddda-9bbd-b589c5cc0a2b (at 10.8.22.11@o2ib6) Mar 21 09:35:01 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 09:36:49 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 5bf05006-db64-6bbe-163a-3eab990b5d25 (at 10.8.22.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867838ad800, cur 1553186209 expire 1553186059 last 1553185982 Mar 21 09:36:49 fir-io1-s1 kernel: Lustre: Skipped 5 previous similar messages Mar 21 09:37:11 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to b4e607c7-6408-22c3-e114-9f5caf76c169 (at 10.8.11.9@o2ib6) Mar 21 09:37:11 fir-io1-s1 kernel: Lustre: Skipped 17 previous similar messages Mar 21 09:38:05 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 9fd9e922-b643-dd3e-5653-6f30e7f1d8fa (at 10.8.30.3@o2ib6) in 224 seconds. I think it's dead, and I am evicting it. exp ffff986b4f0c9800, cur 1553186285 expire 1553186135 last 1553186061 Mar 21 09:38:05 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 09:41:08 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client ce21d408-9816-6f7e-cb89-8666c0be38b5 (at 10.8.30.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985573c67800, cur 1553186468 expire 1553186318 last 1553186241 Mar 21 09:41:08 fir-io1-s1 kernel: Lustre: Skipped 35 previous similar messages Mar 21 09:41:16 fir-io1-s1 kernel: Lustre: 96366:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553186469/real 1553186469] req@ffff98380c4f1200 x1625508846329632/t0(0) o106->fir-OST0006@10.8.21.27@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553186476 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 21 09:41:16 fir-io1-s1 kernel: Lustre: 96366:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Mar 21 09:41:30 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds Mar 21 09:41:30 fir-io1-s1 kernel: LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.204@o2ib7 (0): c: 0, oc: 0, rc: 6 Mar 21 09:41:30 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff985178673600 Mar 21 09:41:30 fir-io1-s1 kernel: LustreError: 91376:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff985005f54800 Mar 21 09:41:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Client 16c53570-4c33-b669-c4e4-d9850974da88 (at 10.8.21.11@o2ib6) reconnecting Mar 21 09:41:36 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 21 09:41:36 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to f6844397-c58e-1716-47c7-98dc229eec16 (at 10.8.21.11@o2ib6) Mar 21 09:41:37 fir-io1-s1 kernel: Lustre: 94211:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553186490/real 1553186490] req@ffff98381887f200 x1625508848474800/t0(0) o105->fir-OST000a@10.8.6.11@o2ib6:15/16 lens 360/224 e 0 to 1 dl 1553186497 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 21 09:41:37 fir-io1-s1 kernel: Lustre: 94211:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Mar 21 09:42:13 fir-io1-s1 kernel: Lustre: fir-OST0008: Client 16c53570-4c33-b669-c4e4-d9850974da88 (at 10.8.21.11@o2ib6) reconnecting Mar 21 09:42:13 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 21 09:42:15 fir-io1-s1 kernel: Lustre: 96942:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553186528/real 1553186528] req@ffff98732dbadd00 x1625508854623632/t0(0) o106->fir-OST0004@10.8.26.31@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553186535 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 21 09:42:15 fir-io1-s1 kernel: Lustre: 96942:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 456 previous similar messages Mar 21 09:42:24 fir-io1-s1 kernel: Lustre: fir-OST0004: haven't heard from client 8d9bbf51-39c9-86fa-3d86-d21a00647c4f (at 10.8.25.14@o2ib6) in 205 seconds. I think it's dead, and I am evicting it. exp ffff984832bdf800, cur 1553186544 expire 1553186394 last 1553186339 Mar 21 09:42:24 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 21 09:42:25 fir-io1-s1 kernel: LustreError: 96665:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ req@ffff98546cff6c50 x1628592064544944/t0(0) o3->f9709d24-98d8-0cac-f2ff-a99b8ba62700@10.8.24.16@o2ib6:565/0 lens 488/440 e 3 to 0 dl 1553186565 ref 1 fl Interpret:/0/0 rc 0/0 Mar 21 09:42:25 fir-io1-s1 kernel: Lustre: fir-OST0002: Bulk IO read error with f9709d24-98d8-0cac-f2ff-a99b8ba62700 (at 10.8.24.16@o2ib6), client will retry: rc -110 Mar 21 09:42:25 fir-io1-s1 kernel: LustreError: 96665:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 1 previous similar message Mar 21 09:42:51 fir-io1-s1 kernel: Lustre: fir-OST0002: Client f9709d24-98d8-0cac-f2ff-a99b8ba62700 (at 10.8.24.16@o2ib6) reconnecting Mar 21 09:42:51 fir-io1-s1 kernel: Lustre: Skipped 1 previous similar message Mar 21 09:43:30 fir-io1-s1 kernel: Lustre: 96935:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553186603/real 1553186603] req@ffff98384ea73000 x1625508853114464/t0(0) o106->fir-OST000a@10.8.30.17@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1553186610 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 09:43:30 fir-io1-s1 kernel: Lustre: 96935:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 902 previous similar messages Mar 21 09:43:40 fir-io1-s1 kernel: Lustre: fir-OST0008: haven't heard from client 4b312c57-6d94-bdf1-2def-67f662cac7ad (at 10.8.26.30@o2ib6) in 224 seconds. I think it's dead, and I am evicting it. exp ffff9847fe8f2c00, cur 1553186620 expire 1553186470 last 1553186396 Mar 21 09:43:40 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Mar 21 09:44:17 fir-io1-s1 kernel: Lustre: fir-OST000a: Client 82e6e9c0-80cc-a194-c64b-34e20270950c (at 10.8.21.18@o2ib6) reconnecting Mar 21 09:44:17 fir-io1-s1 kernel: Lustre: Skipped 3 previous similar messages Mar 21 09:44:43 fir-io1-s1 kernel: LNet: Service thread pid 96368 was inactive for 200.41s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 21 09:44:43 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 21 09:44:43 fir-io1-s1 kernel: Pid: 96368, comm: ll_ost01_045 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 09:44:44 fir-io1-s1 kernel: Call Trace: Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 09:44:44 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 09:44:44 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1553186684.96368 Mar 21 09:44:45 fir-io1-s1 kernel: LNet: Service thread pid 110701 was inactive for 201.75s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 21 09:44:45 fir-io1-s1 kernel: Pid: 110701, comm: ll_ost02_110 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 09:44:45 fir-io1-s1 kernel: Call Trace: Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 09:44:45 fir-io1-s1 kernel: Pid: 96619, comm: ll_ost01_070 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 09:44:45 fir-io1-s1 kernel: Call Trace: Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 09:44:45 fir-io1-s1 kernel: Pid: 49822, comm: ll_ost00_072 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 09:44:45 fir-io1-s1 kernel: Call Trace: Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 09:44:45 fir-io1-s1 kernel: Pid: 74743, comm: ll_ost02_078 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 09:44:45 fir-io1-s1 kernel: Call Trace: Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 09:44:45 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 09:44:45 fir-io1-s1 kernel: LNet: Service thread pid 94240 was inactive for 202.17s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one. Mar 21 09:44:45 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 21 09:44:56 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 63070c4c-8616-f633-4373-b2f7ed21e3a6 (at 10.8.26.31@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff9862a43ff400, cur 1553186696 expire 1553186546 last 1553186470 Mar 21 09:44:56 fir-io1-s1 kernel: Lustre: Skipped 71 previous similar messages Mar 21 09:44:56 fir-io1-s1 kernel: LNet: Service thread pid 96619 completed after 212.46s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 21 09:44:56 fir-io1-s1 kernel: LNet: Skipped 17 previous similar messages Mar 21 09:46:12 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 03b583e3-d1e7-c994-abb5-e4c7b812b37b (at 10.8.22.17@o2ib6) in 221 seconds. I think it's dead, and I am evicting it. exp ffff984801243400, cur 1553186772 expire 1553186622 last 1553186551 Mar 21 09:46:12 fir-io1-s1 kernel: Lustre: Skipped 95 previous similar messages Mar 21 09:48:01 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553186874/real 1553186874] req@ffff98384ea70000 x1625508911818224/t0(0) o106->fir-OST0002@10.9.102.27@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1553186881 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 21 09:48:01 fir-io1-s1 kernel: Lustre: 74736:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 454 previous similar messages Mar 21 09:48:24 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to fb0e351b-5ab8-b43d-813c-60db20cd78c1 (at 10.9.101.45@o2ib4) Mar 21 09:48:24 fir-io1-s1 kernel: Lustre: Skipped 50 previous similar messages Mar 21 09:48:44 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 6a6765e6-4c08-b88e-fc58-44594de42f64 (at 10.8.25.25@o2ib6) in 206 seconds. I think it's dead, and I am evicting it. exp ffff986786a32000, cur 1553186924 expire 1553186774 last 1553186718 Mar 21 09:48:44 fir-io1-s1 kernel: Lustre: Skipped 155 previous similar messages Mar 21 09:53:29 fir-io1-s1 kernel: Lustre: fir-OST0002: haven't heard from client 30924fcc-e650-8173-7e23-7a8d6a7bd74b (at 10.8.27.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583baf7000, cur 1553187209 expire 1553187059 last 1553186982 Mar 21 09:53:29 fir-io1-s1 kernel: Lustre: Skipped 123 previous similar messages Mar 21 10:00:31 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 1bf63035-2382-2247-57ec-f4958613068d (at 10.8.24.11@o2ib6) Mar 21 10:00:31 fir-io1-s1 kernel: Lustre: Skipped 11 previous similar messages Mar 21 10:03:34 fir-io1-s1 kernel: Lustre: fir-OST0000: haven't heard from client fe3112f1-9114-6bb7-2c17-43d97d26c2f7 (at 10.8.11.9@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984f5e9d6400, cur 1553187814 expire 1553187664 last 1553187587 Mar 21 10:03:34 fir-io1-s1 kernel: Lustre: Skipped 115 previous similar messages Mar 21 10:10:33 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to d19ea69a-0a10-0e0b-8ba2-f4e7c0778c3d (at 10.8.30.3@o2ib6) Mar 21 10:10:33 fir-io1-s1 kernel: Lustre: Skipped 29 previous similar messages Mar 21 10:15:31 fir-io1-s1 kernel: Lustre: 96903:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553188524/real 1553188524] req@ffff98381887cb00 x1625508947852160/t0(0) o106->fir-OST0008@10.9.104.52@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1553188531 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Mar 21 10:15:31 fir-io1-s1 kernel: Lustre: 96903:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1910 previous similar messages Mar 21 10:15:49 fir-io1-s1 kernel: Lustre: fir-OST0006: haven't heard from client 81e5a25a-1468-52d2-db69-99b369231b19 (at 10.9.115.4@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867818ac800, cur 1553188549 expire 1553188399 last 1553188322 Mar 21 10:15:49 fir-io1-s1 kernel: Lustre: Skipped 41 previous similar messages Mar 21 10:16:13 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553188566/real 1553188566] req@ffff98382567e300 x1625508947852144/t0(0) o106->fir-OST0006@10.9.104.52@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1553188573 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 10:16:13 fir-io1-s1 kernel: Lustre: 96254:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 16 previous similar messages Mar 21 10:17:30 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1553188643/real 1553188643] req@ffff98381cdff500 x1625508947852176/t0(0) o106->fir-OST000a@10.9.104.52@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1553188650 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Mar 21 10:17:30 fir-io1-s1 kernel: Lustre: 96253:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Mar 21 10:18:44 fir-io1-s1 kernel: LNet: Service thread pid 96254 was inactive for 200.03s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Mar 21 10:18:44 fir-io1-s1 kernel: LNet: Skipped 3 previous similar messages Mar 21 10:18:44 fir-io1-s1 kernel: Pid: 96254, comm: ll_ost01_014 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 10:18:44 fir-io1-s1 kernel: Call Trace: Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 10:18:44 fir-io1-s1 kernel: LustreError: dumping log to /tmp/lustre-log.1553188724.96254 Mar 21 10:18:44 fir-io1-s1 kernel: Pid: 96253, comm: ll_ost02_012 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 10:18:44 fir-io1-s1 kernel: Call Trace: Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 10:18:44 fir-io1-s1 kernel: Pid: 96903, comm: ll_ost00_057 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 Mar 21 10:18:44 fir-io1-s1 kernel: Call Trace: Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffff850c1c31>] kthread+0xd1/0xe0 Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 Mar 21 10:18:44 fir-io1-s1 kernel: [<ffffffffffffffff>] 0xffffffffffffffff Mar 21 10:18:56 fir-io1-s1 kernel: LNet: Service thread pid 96903 completed after 211.74s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Mar 21 10:18:56 fir-io1-s1 kernel: LNet: Skipped 2 previous similar messages Mar 21 10:20:40 fir-io1-s1 kernel: Lustre: fir-OST0000: Connection restored to 44c34e5e-d358-e5f1-f032-e5118620e81b (at 10.8.24.9@o2ib6) Mar 21 10:20:40 fir-io1-s1 kernel: Lustre: Skipped 329 previous similar messages